iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
LLMs can catch some defects in multi-agent prompts, but one small test suggests they are not dependable as a standalone release gate. In Mike Demo’s simulated football-team benchmark, the tested models missed known defects, rejected a clean prompt as needing fixes, and did not always find the same issue on repeat runs. Treat an LLM as a fallible reviewer whose results need to be checked—not as proof a prompt is safe to ship.
What the test asked the models to do
Mike Demo tested whether LLMs could audit JSON prompt files that define the roles and coordination rules for agents in the first Agentic Football Cup, a simulated football league. The players were LLM agents controlled by prompts; the article is not about physical robots.
Each model received a retired team-prompt file and the same stress-test audit prompt used by a human. It had to return defect IDs, GREEN/YELLOW/RED flags, direct quotations supporting its findings, and one final decision: SHIP AS-IS, FIX-THEN-SHIP, or DO NOT SHIP. It was told to report issues rather than rewrite the prompt. Read Demo’s account.
What prompt builds were in the benchmark?
The benchmark used three versions, each serving a different purpose:
#1 Best Overall
- OFFICIALLY LICENSED NFL TRANSFORMERS BOT: This Transformers figure, KC-59, represents the Kansas City Chiefs and features team colors, a mini helmet and a football accessory.
- CONVERTS FROM ROBOT TO FOOTBALL HELMET: Change the Transformers action figure from robot toy mode to helmet mode in 23 steps. Figure is 5.5 inches (14 cm) tall in robot mode
- FOOTBALL ACCESSORIES: Figure comes with a mini helmet that the figure can wear in robot mode and a mini football accessory that the robot figure can hold
- ARTICULATED FOR PLAY AND DISPLAY: Transformers figures feature articulation for display-worthy poses or action-packed play
- GIFT NFL x TRANSFORMERS COLLECTIBLES: This NFL x Transformers figure makes a great gift for fans of the Kansas City Chiefs, Transformers toys, and boys and girls 8 and up who love robot toys
- v2.5A.1: A faulty build with three known defects: a defensive-corner collision, an intercept lane-handoff loophole, and a conflict in the anti-spam rule’s scope.
- v2.5A.2: A clean control with no defects. A RED finding counted as a false positive, while the expected decision was SHIP AS-IS.
- v2.5A.5: A harder build with eight possible findings of different severities.
Scoring separated known-defect recall, false positives on the clean control, and whether the model’s final ship decision matched the benchmark’s expected answer. That separation matters: an auditor can avoid inventing serious defects and still miss real ones—or refuse to approve a clean prompt.
What the three working models found
Demo reports nine runs in total: Google Gemini 2.5 Flash, Google Gemini 2.5 Pro, and OpenAI GPT-5.4 mini each assessed all three builds. Results differed by build and model:
Rank #2
- OFFICIALLY LICENSED NFL TRANSFORMERS BOT: This Transformers figure, Starblitz, represents the Dallas Cowboys and features team colors, a mini helmet and football accessory.
- CONVERTS FROM ROBOT TO FOOTBALL HELMET: Change the Transformers action figure from robot toy mode to helmet mode in 23 steps. Figure is 5.5 inches (14 cm) tall in robot mode
- FOOTBALL ACCESSORIES: Figure comes with a mini helmet that the figure can wear in robot mode and a mini football accessory that the robot figure can hold
- ARTICULATED FOR PLAY AND DISPLAY: Transformers figures feature articulation for display-worthy poses or action-packed play
- GIFT NFL x TRANSFORMERS COLLECTIBLES: This NFL x Transformers figure makes a great gift for fans of the Dallas Cowboys, Transformers toys, and boys and girls 8 and up who love robot toys
| Prompt build | Reported result |
|---|---|
| v2.5A.2, clean control | All three recommended FIX-THEN-SHIP instead of SHIP AS-IS. None made a RED finding, but all missed the required decision. |
| v2.5A.1, three known defects | GPT-5.4 mini identified two defects; both Gemini models identified none in the reported sweep. |
| v2.5A.5, eight possible findings | Gemini 2.5 Pro identified four findings, Gemini 2.5 Flash identified three, and GPT-5.4 mini identified none. |
These are Demo’s results from this benchmark, not independent evaluations or a general ranking of the models. The article’s line, “If you used one of these as your audit gate, nothing would ever get shipped,” refers specifically to all three models recommending FIX-THEN-SHIP on the clean control.
Free tools Windows power users keep installed
One-click scans. No signup required.
Why one audit run is not enough
A smoke test caught a defensive-corner collision that did not appear in a later resweep of the same model and build. That change is a practical warning: a single pass may not reveal everything an auditor will find—or miss—on another run.
Rank #3
- OFFICIALLY LICENSED NFL TRANSFORMERS BOT: This Transformers figure, Goldrush, represents the San Francisco 49ers and features team colors and themed accessories
- CONVERTS FROM ROBOT TO FOOTBALL HELMET: Change the Transformers action figure from robot toy mode to helmet mode in 23 steps. Figure is 5.5 inches (14 cm) tall in robot mode
- FOOTBALL ACCESSORIES: Figure comes with a mini helmet that the figure can wear in robot mode and a mini football accessory that the robot figure can hold
- ARTICULATED FOR PLAY AND DISPLAY: Transformers figures feature articulation for display-worthy poses or action-packed play
- COLLECTIBLE NFL x TRANSFORMERS GIFTS: Looking for unique football fan gifts? This Goldrush figure makes a great gift for fans of the San Francisco 49ers or anyone who collects Transformers toys
Repeat runs help show whether a finding is stable. They do not, by themselves, prove that a prompt is defect-free. For an audit workflow, compare each run with known ground truth and record whether it finds the issue, quotes relevant evidence, avoids false alarms, and reaches the right release decision.
How to evaluate an LLM prompt auditor
- Prepare a clean control and defect-bearing examples. Include prompts with known issues as well as at least one prompt whose expected result is approval. A clean example reveals both invented defects and excessive caution.
- Use one rubric across auditors. Keep the audit instructions, prompt builds, expected defect IDs, and decision criteria the same so differences are interpretable.
- Require evidence for each finding. Ask the model to identify the rule or defect and quote the exact prompt text it considers relevant. Check that the quotation actually supports the claim.
- Score distinct outcomes separately. Track known-defect recall, false positives on clean prompts, final-decision accuracy, and consistency across repeated runs. Do not collapse these into one pass/fail score.
- Keep a human or other verification step. Use model findings to direct attention, then validate them against the prompt and the intended coordination rules before changing a release decision.
Demo mentions latency and cost as possible comparison dimensions but does not report per-model measurements for either. The article reports about $0.52 in total spend, including failed attempts; that is not a per-model or per-run price.
Rank #4
- OFFICIALLY LICENSED NFL TRANSFORMERS BOT: This Transformers figure, Tundra Prime, represents the Green Bay Packers and features team colors, a mini helmet and a football accessory.
- CONVERTS FROM ROBOT TO FOOTBALL HELMET: Change the Transformers action figure from robot toy mode to helmet mode in 23 steps. Figure is 5.5 inches (14 cm) tall in robot mode
- FOOTBALL ACCESSORIES: Figure comes with a mini helmet that the figure can wear in robot mode and a mini football accessory that the robot figure can hold
- ARTICULATED FOR PLAY AND DISPLAY: Transformers figures feature articulation for display-worthy poses or action-packed play
- GIFT NFL x TRANSFORMERS COLLECTIBLES: This NFL x Transformers figure makes a great gift for fans of the Green Bay Packers, Transformers toys, and boys and girls 8 and up who love robot toys
What the results do—and do not—show
The scope is narrow: three models, three prompt builds, and nine reported runs in one author-created simulated-football benchmark. The figures are author-reported and do not establish how LLMs perform across other multi-agent systems, prompt formats, audit rubrics, or deployment conditions.
Recommended Free Tools
Demo also reports that attempts with Claude Sonnet 4.5 repeatedly failed with HTTP 429 capacity errors, while a DeepSeek R1 0528 attempt could not be parsed by the benchmark SDK. Those are observations about the author’s platform attempts, not evidence about the models’ general availability or audit capability.
Quick Recap
Best Value
- OFFICIALLY LICENSED NFL TRANSFORMERS BOT: This Transformers figure, Steelsmash, represents the Pittsburgh Steelers and features team colors, a mini helmet and a football accessory.
- CONVERTS FROM ROBOT TO FOOTBALL HELMET: Change the Transformers action figure from robot toy mode to helmet mode in 23 steps. Figure is 5.5 inches (14 cm) tall in robot mode
- FOOTBALL ACCESSORIES: Figure comes with a mini helmet that the figure can wear in robot mode and a mini football accessory that the robot figure can hold
- ARTICULATED FOR PLAY AND DISPLAY: Transformers figures feature articulation for display-worthy poses or action-packed play
- GIFT NFL x TRANSFORMERS COLLECTIBLES: This NFL x Transformers figure makes a great gift for fans of the Pittsburgh Steelers, Transformers toys, and boys and girls 8 and up who love robot toys
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
