Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
When an LLM is asked to write a job ad without employer-specific facts, it often fills the gaps with concrete claims. In Karlis Gutans’s October 5, 2026 benchmark post, four models placed between 35% and 50% of their extracted claims in the “concrete” category. That is a measure of how readily they commit to specifics—not whether those specifics are true, useful, or good hiring copy. The test uses one brief and a generated-ad judge that has not been validated on generated ads, so the figures are a preliminary snapshot, not a dependable model ranking.
What the benchmark asked models to do
Gutans’s Kaggle benchmark gave models a brief for a junior backend engineer job ad. It specified the seniority and topics to address—services, programming languages and data stores, code review, team size, location, pay, and the first six months—but provided no employer identity or company-specific facts. A model could make a specific claim, use a placeholder, or stay at the level of general direction.
That setup matters: without employer facts, a concrete claim about the job is unsupported by the prompt. The benchmark therefore measures how often a model supplies specifics anyway, not how accurately it describes a real vacancy.
How the score works
A fixed judge model, separate from the contestants, first extracts claims as exact substrings from each generated ad. It then labels each claim as concrete, general direction, or an empty slogan. Code rejects claims that do not appear verbatim in the ad, downgrades a concrete label unless the claim quotes the particular detail, flags template placeholders, and excludes bare skill nouns and “English” as particulars.
#1 Best Overall
The reported score is the share of extracted claims labeled concrete: concrete claims divided by all claims, on a scale from 0 to 1. A higher score is not a better result. In a fact-free brief, it means the ad contains a larger share of specific assertions the prompt did not support.
What the reported runs show
The figures below are the author’s reported results for the post’s current leaderboard, not independently reproduced measurements. Each model received one brief and produced one ad, so treat the decimal scores as descriptions of those runs rather than stable estimates of model behavior.
| Model | Concrete-claim score | Reported run cost | Reported runtime | Result |
|---|---|---|---|---|
| Claude Haiku 4.5 | 0.50 | $0.037 | 29 seconds | Scored |
| Gemini 3.7 Flash | 0.48 | $0.062 | 44 seconds | Scored |
| Gemini 2.5 Flash | 0.37 | $0.074 | 60 seconds | Scored |
| GPT-5.4 mini | 0.35 | $0.069 | 45 seconds | Scored |
| Grok 4.5 | not stated (DEV Community post, October 5, 2026) | not stated (DEV Community post, October 5, 2026) | not stated (DEV Community post, October 5, 2026) | Error; excluded because the benchmark does not publish a partial score |
For the four scored runs, the author says cost and runtime include generating the ad and making both judge calls; most of the tokens were used by the judge. The table does not show that a lower score means a more accurate or more useful ad, nor that any one model consistently behaves that way across different prompts.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →What one ad looked like under the judge
In the Gemini 3.7 Flash example, the generated ad was 2,528 characters long. The judge extracted 29 claims: 14 concrete, 13 general-direction claims, and two empty slogans; it found no placeholders. The post says none of the requested topics was left blank. That is a useful illustration of how the labels were applied, but it is only one ad—not a separate validation of the judge or a broad estimate for Gemini.
Why this is not a model ranking
One prompt leaves substantial uncertainty
The benchmark runs one brief, yielding roughly 30 claims per model. For Gemini 3.7 Flash’s 14 concrete claims out of 29, the post reports an approximate 95% interval of 0.31–0.66. The author says every model falls within that interval. The apparent grouping of the two higher and two lower scores is therefore a direction to investigate, not evidence that the models form distinct performance tiers.
The judge has not been checked on generated ads
The current generated-ad judge has not been hand-checked on the kind of output this benchmark scores. The post cites 86.7% agreement with hand labels for an earlier version of its rules on real job postings. That earlier result used a different judge model and older prompts, so it does not establish how reliably the present setup classifies generated ads.
Rank #4
The score does not measure hiring outcomes
The benchmark does not establish whether these ads attract qualified applicants, improve hiring quality, or help employers fill roles. Those questions require outcomes data from employers or job boards, which the leaderboard does not provide.
What a more informative next test would require
Gutans proposes running all 20 briefs multiple times, hand-checking a sample of generated claims, and reporting results by topic—especially pay. Repeated briefs would help show whether a model’s score persists across different prompts and runs; manual checks would test the judge on generated text; topic-level results could reveal whether unsupported specificity is concentrated in sensitive details. Measuring applicant quality would require a different study with real recruiting outcomes, not just generated ads.
Best Value
How this fits earlier job-ad research
A related but distinct 2022 peer-reviewed paper by Borchers and co-authors studied bias and realism in zero-shot GPT-3 job ads. Its abstract reports that diversity-encouraging prompt engineering did not significantly improve bias or realism, while fine-tuning—especially on unbiased real ads—could improve realism and reduce bias. That work addresses different questions and does not replicate Gutans’s specificity benchmark.
Sources: Karlis Gutans, “Give an LLM no facts and ask for a job ad. What does it commit to?”, DEV Community, October 5, 2026; Conrad Borchers et al., “Looking for a Handsome Carpenter! Debiasing GPT-3 Job Advertisements,” Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing, Association for Computational Linguistics, July 2022. The DEV Community source link is to the publication because the post’s specific page URL is not established here.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
