iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
The specific experiment named in this headline cannot be verified from the available account. Its application, baseline, decision boundary, implementation, test set, measurements, and regressions were not recoverable, so it would be misleading to claim that particular changes caused particular breakages.
There is still a useful answer for engineers evaluating the same design move. Related Claude Code evaluations show how to test whether instructions are actually invoked, how a trigger change can create a measurable regression, and why small, stochastic experiments need careful controls. The sections below separate those documented observations from a practical method for running and reporting a genuine “logic moved out of the LLM” comparison.
What can be established about the titled experiment
Two searches for the exact title and a close title variant did not locate a verifiable primary account. No attributable description establishes which decisions were originally made by a language model, which decisions were moved into deterministic code, or what failed afterward. The following details therefore remain unresolved:
Recommended Free Tools
- The application and model version.
- The original and revised architectures.
- The exact boundary between model judgment and program logic.
- The baseline task set and pass criteria.
- The number of runs, prompts, and context variations.
- Any changes in latency, token use, cost, correctness, or failure visibility.
That distinction matters. A plausible engineering story is not evidence that this particular experiment happened. The figures that can be discussed come from separate Claude Code reports summarized in NewSumm’s October 6, 2026 roundup; they should not be attributed to the titled account.
What “moving decision logic” means in an agent
In a model-centered design, the prompt and surrounding context ask the LLM to decide what to do: whether a condition is met, which tool to call, whether an output is safe, or whether a workflow should stop. Moving decision logic out of that layer means implementing at least part of the decision in code with explicit inputs, branches, schemas, and tests.
The change is not a simple switch from “AI” to “no AI.” Most practical systems use a boundary:
- Deterministic code handles rules that must be repeatable, such as type checks, permission checks, numeric limits, required fields, idempotency keys, and allowed state transitions.
- The model handles interpretation, classification, drafting, extraction from ambiguous language, and other tasks where multiple answers may be acceptable.
- Orchestration code decides when to ask the model, when to reject its result, when to retry, and when to escalate.
That boundary can improve auditability, but it can also expose assumptions that the model previously papered over. A rule may reject inputs that were previously accepted; a trigger may no longer fire; or a fallback path may become visible because the model is no longer allowed to improvise. Those are hypotheses to test, not established outcomes of the missing account.
Adjacent evidence: invocation and trigger reliability
A linked-reference test
One Claude Code skill experiment examined whether the model would open a linked reference.md file. In the small test, the skill’s SKILL.md linked to the reference in 16 runs, and the model read it in all 16. When the link was absent, it read the reference in 0 of 4 runs. The reported result is therefore 16 of 16 linked-reference reads and 0 of 4 unlinked-reference reads.
Those counts measure one narrow behavior: opening a reference file under one skill setup. They do not establish a general trigger probability for Claude Code, other agents, or longer sessions. The test used one task and one small skill listing. A larger context, competing instructions, a different model version, or a multi-step workflow could produce a different result.
Rank #2
A trigger-description sabotage test
A separate report deliberately changed a skill’s trigger description and ran an evaluation suite repeatedly. Its overall suite score fell from 1.00 to 0.36. A trigger-only tripwire fell from 1.00 to 0.00. The author used repeated runs to distinguish a consistently broken trigger from ordinary model variability.
This is a configuration-drift experiment, not a measurement from the titled account. It illustrates why a single failed run is weak evidence: model behavior is stochastic. A credible regression claim needs a fixed test set, repeated trials, and a metric that isolates the behavior being investigated.
A scan of public skills
Another report scanned 215 public Claude Code skills and classified 69% as unlikely or borderline to trigger reliably. The figure came from the SkillCrossroads scan summarized in the same October 6, 2026 roundup. It is a rubric-based result for that sample and those checks, not a population estimate for every skill, agent, or LLM application. The underlying report was not independently opened for this account, so the percentage should not be generalized beyond its stated scope.
What a before-and-after comparison must measure
“What broke” is too broad to be a metric. A useful comparison names the decision boundary and records observable outcomes on the same task set.
Decision ownership
For every branch, document whether the model, deterministic code, or a human owns the decision. For example, code might decide that an amount exceeds a limit, while the model extracts the amount from an invoice. Without this map, a change in output cannot be attributed to moving logic.
Rank #3
Correctness and action validity
Score both the final answer and the action trace. A response can look correct while calling the wrong tool, omitting a required field, or taking an unsafe action. Record invalid calls, missing calls, duplicate calls, and actions that violate the allowed state machine.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchesRobustness
Run wording variants, reordered context, missing optional fields, noisy documents, and conflicting instructions. Deterministic branches should remain stable when irrelevant wording changes. Model-owned steps should be evaluated for the degree of variation that the application can tolerate.
Failure visibility
Measure whether failures are rejected explicitly, silently converted into a fallback, or returned as plausible but wrong output. Moving a branch into code may increase visible errors while reducing silent errors; both effects belong in the report.
Operational measures
Record latency, model calls, input and output tokens, retries, and infrastructure cost only if they were actually measured. Do not infer savings from the architecture alone. A code check can reduce a model call, but added validation, retries, or a second model pass can erase that benefit.
How to run a defensible experiment
- Freeze the baseline. Version the prompt, model, tool definitions, instruction files, application code, and test data. Save the exact configuration before moving any branch.
- Write the decision contract. For each decision, specify inputs, allowed outputs, rejection conditions, side effects, and who owns the branch. Include examples of valid and invalid cases.
- Build a fixed task set. Include ordinary cases, boundary values, malformed inputs, adversarial wording, and cases that should be refused. Keep this set unchanged between versions.
- Separate trigger tests from outcome tests. A trigger test asks whether the instruction, skill, or tool was invoked. An outcome test asks whether the final action was correct. Combining them can hide a trigger failure behind a lucky answer.
- Repeat each case. Use enough repetitions to estimate variability and report the run count. A lone success or failure cannot show whether a stochastic behavior changed.
- Capture traces. Store prompts, retrieved references, tool calls, arguments, validation errors, retries, and final results. Trace data lets you identify the first divergent step instead of judging only the final text.
- Run the revised system under identical conditions. Change only the decision ownership you are testing. Keep model version, temperature or equivalent sampling settings, context, and tool availability fixed where possible.
- Classify every regression. Label failures as missed trigger, wrong branch, invalid action, schema error, refusal, timeout, or evaluator disagreement. A single aggregate score cannot explain what broke.
Failure modes that commonly appear at the boundary
Rules moved too early
If code rejects an input before the model can normalize it, harmless variations may fail. For example, a parser that accepts only one date format may discard a request that the model previously interpreted correctly. The remedy is to define a normalization step and test all supported formats before applying the strict rule.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Rules left implicit
The opposite problem occurs when the model still owns a safety or authorization decision that the application assumes is deterministic. A prompt instruction is not an access-control mechanism. Permission checks, monetary limits, and irreversible actions need enforceable programmatic gates.
Trigger drift
Instruction wording, skill descriptions, file links, and tool names can change independently of application code. A trigger-only tripwire catches the case where a capability is never invoked, even though final-answer grading might occasionally pass by chance.
Fallback loops
A deterministic validator may reject a model result and send it back for repair. If the repair prompt does not explain the exact violation, the system can repeat the same invalid output, increasing latency and cost. Record retry counts and impose a bounded attempt policy.
Evaluator blind spots
Exact-match grading can mark a semantically correct answer wrong, while loose text grading can miss an invalid side effect. Use structured assertions for tool calls and state transitions, plus human review for ambiguous language tasks.
How to interpret the available numbers
The adjacent figures demonstrate measurement techniques, not universal reliability rates. The linked-reference result has 16 positive observations and 4 negative observations in separate conditions. Such a small, single-skill setup cannot support a claim about all references or all sessions. The trigger sabotage result reports a large score change in one suite after an intentional configuration change; it does not predict the size of regressions elsewhere. The 69% scan result describes a sample judged with a particular rubric, not the entire ecosystem.
Keep the denominator, condition, and attribution attached whenever a number appears. “16 of 16” means linked references were read in that reported setup; it does not mean a model will always read every linked file. “0.36” means the configuration-drift author’s suite score after the described change; it does not quantify the titled experiment.
A practical architecture for safer ownership
Start by making deterministic code the final authority for irreversible effects. Let the model propose structured intent, then validate that intent against schemas, permissions, business rules, and current state. If validation fails, return a specific error that the model can repair or escalate to a human. Keep interpretation separate from execution so that a fluent answer cannot bypass a gate.
For triggers, make activation observable. Log which instruction or skill was selected, which references were opened, and which tools were considered. Add a small tripwire test that fails when a required capability is not invoked. Run it alongside end-to-end tests so a passing final response cannot conceal a missing trigger.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Finally, version the decision contract and its tests with the code. A changed description, link, schema, or model can alter behavior even when application logic is untouched. Treat those files as production dependencies, not informal documentation.
What a trustworthy account of this experiment would contain
A complete first-person report could resolve the open questions by publishing:
- The original prompt-and-tool workflow and the revised workflow.
- A diagram or table showing each decision’s owner before and after.
- The model, version, sampling settings, and application environment.
- The fixed task set, number of repeated runs, and evaluator definitions.
- Separate scores for trigger invocation, action validity, final correctness, and failure visibility.
- Representative traces for both regressions and improvements.
- Measured latency, token, and cost changes, with their test conditions.
Until those details are available, the responsible conclusion is limited: the titled account’s breakages are unverified. The neighboring Claude Code measurements show that invocation can be tested directly, trigger wording can produce a measurable regression, and small samples require careful qualification. They provide a method for evaluating a logic boundary, not a substitute for the missing experiment.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
