iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Mayank Kaul’s experiment found that removing a 1,350-line CLAUDE.md changed whether an agent followed two rules—but both rules conflicted with conventions visible in the repository. The result is a useful case study in how coding agents weigh project instructions against code and history, not proof that shorter instruction files are generally better.
What Kaul tested
In an August 16, 2026 post, Kaul describes testing 20 mechanically checkable rules against one ticket-sized coding task. He ran 30 trials across six configurations and used a deterministic checker to inspect the resulting file tree, rather than relying only on agents’ descriptions of what they had done. With the full 1,350-line CLAUDE.md present, 399 of 400 rule checks passed.
![Claude AI for Beginners Bible: [5 in 1] The Ultimate Guide to Automate Your Work, Save Hours Every Week, and Use AI for Real-World Results](https://m.media-amazon.com/images/I/415+fSJacsL._SL160_.jpg)
Across the configurations, two rules accounted for the reported difference when the file was removed. That result is narrow: it describes what happened in this repository, with this task and setup. It does not establish that long instruction files are unnecessary or that the same pattern will hold in other projects.
Free tools Windows power users keep installed
One-click scans. No signup required.
Which two rules changed—and why
Where to put a changelog entry
The file told the agent to add a bullet under ## Unreleased. But the repository’s changelog kept “Nothing yet.” under that heading and placed actual entries in released sections. Its header also described creating a new version section when cutting a release. The agent could therefore infer a different workflow from the file’s visible examples and guidance.
#1 Best Overall
Which version number to bump
The file directed the agent to increase the patch digit. Repository history, however, showed a minor-version increase for a change that added public API surface. That history supplied evidence that the project’s versioning practice could differ from the instruction’s blanket patch-bump rule.
Kaul calls the resulting behavior violations, while acknowledging that the agents followed defensible conventions documented in the repository. The distinction matters: a checker can determine whether an output matches a stated rule, but it cannot by itself decide whether that rule is current or whether the repository’s established practice is preferable.
What the configuration results say
For this particular task, repository, model, and test setup, Kaul reports these results for two configurations:
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problems| Instruction setup | Rule-check score | Reported cost per run |
|---|---|---|
| Nineteen lines plus a Skill | 100/100 | $0.696 |
| Full 1,350-line file | 99/100 | $1.094 |
He also reports that the full-file setup cost more in all 25 pairwise comparisons. These are the author’s results for this experiment, not general price or performance benchmarks. The one-point score difference and cost figures should not be treated as predictions for other tasks, repositories, or models.
Why removing the file did not remove its influence
The initial attempt had confounds: the deleted file remained in Git history and could potentially be recovered, while memory mechanisms and a test-harness comment also pointed toward the rules file. Kaul says he rebuilt the base repository so the file was absent from every scanned object-store blob.
Even after that change, he describes the file-absent configurations as cases where the model had been told that a contract existed but could not access it—not cases where it had never heard of a contract. That qualification limits what the comparison can show about a genuinely undocumented project. Meanwhile, the changelog and version history remained visible, giving the agent evidence about the repository’s conventions.
How to use the finding in your own repository
Kaul’s practical suggestion is to inspect each instruction and ask whether a careful reader could infer it from the repository itself. A rule that conflicts with code, examples, or history deserves investigation—not automatic deletion. The instruction may be stale, or the repository may be consistently demonstrating a practice the team should change.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →- Check the rule against current artifacts. Compare it with the relevant files, examples, and recent history, not just with what the team remembers intending.
- Decide which source should govern. If the instruction is still correct, clarify why the repository appears to contradict it or update the repository. If the convention has changed, revise the instruction.
- Try a real task more than once. Observe whether the rule changes the resulting files, rather than judging only from an agent’s explanation.
- Verify the output mechanically where possible. A deterministic check can reveal whether a stated requirement was met, while human review still decides whether the requirement and project convention are sound.
Kaul summarizes his interpretation this way: “The rules worth keeping are the ones your codebase contradicts, which is uncomfortable, because those are also the ones most likely to be wrong.” That is a pointed observation about this experiment, not a universal rule for deciding which instructions belong in a project file.
Best Value
What this experiment cannot establish
The test covered one task, one repository, one model, and five runs per arm, with no statistical testing of compliance outcomes. The repository also contained several examples of its conventions, making those conventions unusually easy to infer. A greenfield repository was not tested. The findings therefore do not show that a compact file plus a Skill will generally match or outperform a long file, nor that repository evidence will always outweigh explicit instructions.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
