Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Meta faced a credible criticism over how it presented Llama 4 Maverick’s LM Arena result: the highly ranked entry was an experimental, conversationally customized model, while the public Maverick checkpoint later ranked much lower. LM Arena said Meta should have disclosed the customization more clearly. That supports a benchmark-transparency criticism, but it does not prove that Meta trained Llama 4 on benchmark test answers or committed fraud.
What triggered the accusation?
When Meta announced Llama 4 Scout and Maverick on April 5, 2025, it highlighted results from several evaluations, including LM Arena. Meta’s materials identified the Arena entry as an “experimental chat version” and described it as optimized for conversationality. The model name used for that entry was Llama-4-Maverick-03-26-Experimental.
The concern was not simply that Meta submitted a model for evaluation. It was that a customized experimental model earned the prominent result, while readers could reasonably take the headline comparison to describe the publicly released Maverick model. LM Arena later said: “Meta should have made it clearer that ‘Llama-4-Maverick-03-26-Experimental’ was a customized model to optimize for human preference.”
That statement from LM Arena is the strongest basis for calling the episode a disclosure failure. It supports saying Meta was criticized for benchmark presentation or faced a risk of misleading comparisons. It does not establish that Meta secretly altered test answers or that the experiment itself violated a stated rule.
#1 Best Overall
- NO WIRES, MORE FUN — Break free from cords. Game, play, exercise and explore immersive worlds — untethered and without limits.
- 2X GRAPHICAL PROCESSING POWER — Enjoy lightning-fast load times and next-gen graphics for smooth gaming powered by the SnapdragonTM XR2 Gen 2 processor.
- EXPERIENCE VIRTUAL REALITY — Take gaming to a new level and blend virtual objects with your physical space to experience two worlds at once.
- 2+ HOURS OF BATTERY LIFE — Charge less, play longer and stay in the action with an improved battery that keeps up.
- 33% MORE MEMORY — Elevate your play with 8GB of RAM. Upgraded memory delivers a next-level experience fueled by sharper graphics and more responsive performance.
Was the high-scoring Arena model the same as the public release?
Not clearly. The Arena submission was labeled experimental and described as customized for conversationality. The publicly released checkpoint was Llama-4-Maverick-17B-128E-Instruct. Those names and descriptions matter: an Arena score belongs to the precise model submitted under the evaluation conditions, not automatically to every checkpoint in the same model family.
LM Arena said it added the public Hugging Face Maverick version to its leaderboard and released more than 2,000 head-to-head battle results for public review. In a reported 2025 leaderboard snapshot, the unmodified public Maverick ranked around 32nd, below GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro. The experimental entry was reported with an Elo score of 1417. These are results associated with different model identities and should not be read as a controlled before-and-after test of one unchanged model.
Rank #2
- NO WIRES, MORE FUN — Break free from cords. Game, play, exercise and explore immersive worlds — untethered and without limits.
- 2X GRAPHICAL PROCESSING POWER — Enjoy lightning-fast load times and next-gen graphics for smooth gaming powered by the SnapdragonTM XR2 Gen 2 processor.
- EXPERIENCE VIRTUAL REALITY — Take gaming to a new level and blend virtual objects with your physical space to experience two worlds at once.
- 2+ HOURS OF BATTERY LIFE — Charge less, play longer and stay in the action with an improved battery that keeps up.
- 33% MORE MEMORY — Elevate your play with 8GB of RAM. Upgraded memory delivers a next-level experience fueled by sharper graphics and more responsive performance.
| Comparison point | Experimental Arena entry | Public Maverick checkpoint |
|---|---|---|
| Model identity | Llama-4-Maverick-03-26-Experimental; Meta described it as an experimental chat version optimized for conversationality. | Llama-4-Maverick-17B-128E-Instruct, the public checkpoint discussed in the reported follow-up snapshot. |
| Reported result | Elo 1417, as reported for the experimental entry in 2025. | Around rank 32 in TechCrunch’s reported 2025 snapshot. |
| Evaluation | LM Arena human-preference battles. | LM Arena leaderboard result for the public model; the cited reporting does not provide a matched retest protocol against the experimental entry. |
| Disclosure issue | LM Arena said the customization should have been made clearer. | The public checkpoint was added for comparison; its reported rank does not by itself explain the performance difference. |
The table summarizes reported results, not a reproducible head-to-head experiment. Exact submission settings, prompt protocol, and evaluation dates would be needed to attribute the gap to a particular customization or other cause. A leaderboard rank can also change as new models and votes are added, so “around 32nd” describes that reported snapshot, not a current position.
Free tools Windows power users keep installed
One-click scans. No signup required.
Did Meta manipulate benchmarks or train on test sets?
“Manipulation” can mean different things. If it means presenting a tailored, experimental model result without making the distinction sufficiently prominent, LM Arena’s criticism supports that concern. If it means training on hidden benchmark answers, the available evidence does not establish it.
Rank #3
- Transform your reality and do everything you love in totally new ways. Welcome to Meta Quest 3S.
- Explore thousands of unreal experiences with mixed reality, where you can blend digital objects into the room around you or dial up the immersion in VR.
- Have more fun with friends in Quest. Whether you’re stepping into an immersive game with people from around the world, watching a live concert together in Meta Horizon or inviting everyone over to cast your play onto the TV.
- Multi-tasking has never been this easy. Pull up multiple screens at once to browse the web, watch YouTube and direct message with friends — all while keeping your room in view.
Meta vice president of generative AI Ahmad Al-Dahle rejected the separate test-set allegation, saying it was “simply not true” that Meta trained Scout or Maverick on benchmark test sets. That is Meta’s denial; the cited material does not independently prove the denial or substantiate the allegation. It is therefore inaccurate to state as fact that Llama 4 was trained on hidden test answers.
Nor does the record here establish legal fraud or intentional deception. A careful description is that Meta was criticized for insufficient disclosure about a customized Arena submission, while the test-set-training allegation remains unproven in the cited evidence.
Rank #4
- CARDBOARD MONKENAUT — Get our best Gorilla Tag bundle yet with this Amazon exclusive deal. Purchase Meta Quest 3 to get exclusive items, including the Gorilla Space Program Suit and Helmet, plus 2,000 SHINY ROCKS.
- NEARLY 30% LEAP IN RESOLUTION — Experience every thrill in breathtaking detail with sharp graphics and stunning 4K+ Infinite Display.
- NO WIRES, MORE FUN — Break free from cords. Game, play and explore in immersive worlds — untethered and without limits.
- 2X GRAPHICAL PROCESSING POWER — Enjoy lightning-fast load times and next-gen graphics for smooth gaming powered by the Snapdragon XR2 Gen 2 processor.
- EXPERIENCE VIRTUAL REALITY — Blend virtual objects with your physical space and experience two worlds at once in your VR headset.
Why can Arena and benchmark scores tell different stories?
LM Arena evaluates model responses through human-preference battles: users compare answers and their preferences contribute to a leaderboard rating. Fixed task benchmarks instead score performance on defined tasks under a particular test setup. A model can do well on conversational preference without ranking similarly on coding, reasoning, or other task-specific evaluations.
Meta’s Llama 4 materials also listed standard benchmark results. For example, its model card reported LiveCodeBench pass@1 scores of 32.8 for Scout and 43.4 for Maverick within Meta’s stated evaluation window. Those figures measure a coding benchmark outcome; they are not Arena Elo scores and cannot validate or invalidate the experimental model’s human-preference rating.
Best Value
- CARDBOARD MONKENAUT — Get our best Gorilla Tag bundle yet with this Amazon exclusive deal. Purchase Meta Quest 3S to get exclusive items, including the Gorilla Space Program Suit and Helmet, plus 2,000 SHINY ROCKS.
- NO WIRES, MORE FUN — Break free from cords. Game, play and explore immersive worlds — untethered and without limits.
- 2X GRAPHICAL PROCESSING POWER — Enjoy lightning-fast load times and next-gen graphics for smooth gaming powered by the Snapdragon XR2 Gen 2 processor.
- EXPERIENCE VIRTUAL REALITY — Take gaming to a new level and blend virtual objects with your physical space to experience two worlds at once in your VR headset.
- 2+ HOURS OF BATTERY LIFE — Charge less, play longer and stay in the action with an improved battery that keeps up. *Based on the graphic performance of the Qualcomm Snapdragon XR2 Gen 2 platform vs the Meta Quest 2 platform.
When comparing AI model claims, keep these details together:
- Exact model name and checkpoint: distinguish an experimental or tuned submission from the public release.
- Evaluation type: human preference, coding tasks, and other fixed benchmarks answer different questions.
- Disclosure: check whether tuning, prompting, or other customization is described.
- Date and leaderboard snapshot: rankings can shift as votes and competitors change.
- Reproducibility: determine whether another party can test the same checkpoint under a documented protocol.
Can you trust LM Arena scores?
Use an Arena score as evidence about preference performance in that Arena setting, not as a universal measure of model quality. Its value depends on knowing which model answered, whether it was customized, how comparisons were gathered, and when the leaderboard snapshot was taken.
LM Arena’s release of more than 2,000 battle results for public review offered material for scrutiny, and its public-model addition enabled a separate comparison. Those steps improve transparency, but they do not make results for experimental and public checkpoints interchangeable or turn an Arena rating into a fixed benchmark score.
For a practical decision, compare the exact model version you can access on tasks that resemble your own use. Treat a leaderboard position as one data point, and look for disclosure of the checkpoint and evaluation conditions before drawing conclusions about the released product.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
