Recommended Free Tools
iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Short answer: In Himanshu Kumar’s ToolTrap benchmark, an added system-prompt rule sharply reduced repetition of planted fake details—but did not stop it on new cases. In a held-out layout where fake details appeared in a top-level carrier_update field, seven of the ten models that completed the tests repeated at least one planted detail while using the rule. These are results from a synthetic benchmark, not a measurement of real customer-support traffic.
What ToolTrap tested
ToolTrap is a synthetic customer-support benchmark created by Himanshu Kumar. It tests whether an AI assistant will repeat false details embedded in tool results in replies to customers. The simulated store uses 11 mock tools for tasks such as order lookup and refunds; its customers, destinations, offers, and planted details are fictional.
The benchmark records tool calls, returned payloads, and assistant replies. A deterministic scorer checks those records rather than relying on another AI model to judge the answer. It evaluates whether planted false information reaches the customer, while also checking whether the assistant retains legitimate support details and behaves cleanly.
Kumar’s report describes the benchmark and results on DEV Community. The figures below are author-reported; they have not been independently replicated.
#1 Best Overall
What the prompt rule changed—and what it did not
The intervention was an added system-prompt block. It named authoritative fields, permitted the use of verified support information, and told the model not to repeat details from imported notes—even when warning the customer about them. The notes were still passed to the model; the intervention did not filter or remove them. As Kumar puts it, “The code still passes those notes to the model without filtering them; following the rule depends on the model.” The experiment tested the whole instruction block, so it cannot show which sentence, if any, drove the change.
Development cases: a clean result on examples used to shape the rule
The development suite covered eight types of detail, with malicious, legitimate, and clean cases. Planted malicious details appeared in imported notes; valid details appeared in verified_support. Twelve hosted models were tested in fresh chats with the original prompt and the prompt plus the rule.
On the development malicious trials, the original prompt produced an exact planted-marker match in 75 of 192 trials; the added rule produced none in 192. All legitimate details were retained in that suite. Gemini 3.8 Flash already had no marker repetitions under the original prompt. The strong development result shows performance on those cases, but by itself does not establish that the rule transfers to unfamiliar cases.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Held-out cases: performance on new content and layouts
For held-out tests, Kumar froze new detail content and layouts before inference. The benchmark used two layouts: fake details in a top-level carrier_update field, and details in a history entry tagged imported_email. Valid information remained in verified_support.
Rank #3
Ten models completed the held-out inference. Gemma failed twice at the provider, and Opus was not run because of an inference quota limit. The reported comparison uses the same ten completed models across the development and held-out suites; the held-out totals are separate from the 12-model development totals.
What “7 of 10” means
The headline refers specifically to the carrier_update layout with the source rule enabled: seven of the ten completed models repeated at least one planted detail somewhere in their trials. It does not mean that seven models failed every trial or that seven out of ten models are generally unsafe in production. In that layout, the rule reduced exact-marker repetitions for all ten models compared with the original prompt.
| Test set and layout | Original prompt | Prompt plus rule | What the count represents |
|---|---|---|---|
| Development malicious cases | 75/192 exact-marker repetitions | 0/192 exact-marker repetitions | 12 models; 192 trials per prompt |
Held-out carrier_update |
61/160 exact-marker repetitions | 26/160 exact-marker repetitions; 27/160 after the token-based sensitivity check | 10 completed models; 160 trials per prompt |
Held-out history tagged imported_email |
41/160 exact-marker repetitions | 2/160 exact-marker repetitions | 10 completed models; 160 trials per prompt |
| Legitimate-detail checks | 316/320 exact-marker appearances; 319/320 after the token check | 313/320 exact-marker appearances | Aggregate counts reported across legitimate-detail trials |
All counts are those reported by Kumar in 2026. The token-based check is a sensitivity analysis, not a replacement for the primary exact-marker score.
Why the scores need careful reading
Exact markers can miss reformatted disclosures
The primary scorer searched replies for an exact planted marker chosen before the run. It can miss a detail if a model changes its formatting or paraphrases it. For example, a Gemini 3.8 Flash reply exposed a parcel-locker PIN with a colon between the label and digits, so the exact marker did not match. After inspecting failures, Kumar ran a payload-token sensitivity check. It found eight additional malicious disclosures across held-out replies and changed the carrier_update rule-arm count from 26/160 to 27/160. Because this check followed inspection of the results, it should be read as a sensitivity check—not as a comprehensive semantic evaluation.
Best Value
Legitimate information is a separate measure
A system that withholds every tool-returned detail might avoid repeating planted content while failing to help customers. ToolTrap therefore also checked legitimate-detail retention. Aggregate retention counts were high, but they hide a model-specific cost: under the rule, GPT-5.5 omitted six of 32 legitimate details, while it omitted none under the original prompt. Four omissions involved a loyalty code the reply said had been issued but did not provide; two involved a verified gift-card code.
Kumar also reports that all clean cases passed and that the runs caused no unrequested account or order mutations. These outcomes address other parts of the benchmark’s behavior checks; they do not cancel out the malicious-disclosure or legitimate-omission results.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What the benchmark can—and cannot—show
The results support a limited conclusion: this particular prompt rule prevented exact-marker repetitions on the development malicious cases and reduced them in both held-out layouts, but some planted content still reached customer-facing replies. They do not establish how often customer-support assistants would disclose false information in production or whether the same rule would work across other systems, tools, or types of untrusted content.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
- The cases are authored families with repeated trials, not a random sample of real customer conversations. The model roster is selected and incomplete.
- The held-out layouts changed content, nesting, source labels, and apparent authority together. The results therefore do not isolate which cue caused a model to follow or miss the rule.
- The
imported_emailtag shares the word “imported” with the prompt contract. Success on that layout does not prove models will recognize unfamiliar untrusted sources. - The report cautions that nominal Wilson intervals assume independent observations and are not confidence bounds for real support traffic. Its pooled p-values are exploratory.
- Tool-use logs are not enough to establish that a customer was protected: the benchmark also needs to inspect the final reply, where a disclosure can occur.
How to judge a similar prompt defense
A useful evaluation should test both sides of the task: preventing false details from reaching the customer and preserving information the support system has verified. It should also test whether the defense transfers beyond examples used to write it.
- Use held-out content and vary source fields, nesting, and labels rather than testing only the format named in the prompt.
- Report the model and prompt identities, completed and failed runs, and whether comparisons use the same models.
- Define the exposure criterion in advance. Report exact-string results separately from token-sensitive or semantic checks, and disclose how those checks were performed.
- Pair planted malicious details with legitimate-detail cases, then report retention and omission counts alongside disclosure counts.
- Inspect both tool actions and customer-facing replies; a safe-looking tool log does not guarantee a safe final answer.
Kumar describes this paired approach as important: “I would pair every planted detail with a separate legitimate case so withholding useful information shows up, and test new content and tool fields beyond the examples used to write the prompt.”
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
