iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
acc and acc_norm are accuracy results based on different ways of scoring candidate answers. In likelihood-based multiple-choice evaluations, acc typically chooses the answer with the highest summed token log-probability; acc_norm adjusts candidate scores for answer length before choosing. The adjustment can reduce a penalty against longer answers, but it can also overcorrect. Neither metric is automatically length-neutral: the benchmark’s scoring implementation and the way it formats choices matter.
What the two metrics measure
Both metrics report the fraction of benchmark questions answered correctly. They differ in how the evaluation chooses a candidate answer for each question—not in how the final fraction is calculated.
accis accuracy using the task’s ordinary candidate-scoring rule. In likelihood-based multiple-choice tasks, this commonly means comparing the summed log-probabilities of the candidate completions.acc_normis accuracy after the candidate scores have been normalized for answer length before candidates are compared.
The precise implementation is task-dependent. In real evaluations, length usually means the number of tokens, not characters. The Hugging Face Open LLM Leaderboard organization representative summarized the convention as normalization “on the length of the answer,” while noting that the actual result depends on how the options are passed to the model (Hugging Face discussion, February 2024).
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Why answer length changes likelihood scores
A sequence’s log-probability is the sum of the log-probabilities of its tokens. Since token log-probabilities are generally negative, adding more tokens tends to make a longer completion’s summed score lower. That can disadvantage a correct answer expressed as a longer sequence when it is compared with a shorter candidate.
#1 Best Overall
Length normalization adjusts the candidate score to account for sequence length—often by dividing by token count—before selecting an answer. This can counter the summed-score penalty, but it is a heuristic, not a guarantee that length no longer affects the result. Depending on the candidates and evaluation conditions, normalization can overcorrect and favor longer answers. The balance depends in part on the completion-length distribution in the benchmark.
What is being normalized—and why formatting matters
The adjustment applies to candidate scores, usually with token length as the denominator or basis. It does not mean that the final accuracy percentage is divided by answer length. Nor should acc_norm be assumed to mean a universal formula: the exact normalization and scoring procedure depend on the task implementation.
Candidate serialization also matters. An evaluation may score only an option letter, such as “B,” or score the full answer text. Those choices produce sequences of different lengths and therefore can change both the likelihoods and the effect of normalization. The Hugging Face representative explicitly noted that the score can vary depending on whether the options are passed as a full sequence or just the choice letter (discussion).
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Clear out junk files and repair common Windows errors3Fix the driver behind crashes, sound loss and screen glitchesHow to interpret and compare reported scores
Do not compare two numbers labeled acc or acc_norm until you know that they were produced under comparable conditions. Check the model’s technical report and the benchmark or harness documentation for the score definition and setup.
- Task and split: Record the benchmark, task configuration, and evaluation split.
- Prompt and examples: Check the prompt format and number of few-shot examples, since these define the context in which candidates are scored.
- Candidate formatting: Establish whether the model is evaluated on option letters or full answer strings.
- Scoring rule: Confirm whether scores use summed likelihood, length normalization, or another procedure, and how the normalization is implemented.
- Model and evaluation software: Record the model revision and evaluation harness or version so the result can be reproduced.
Historical documentation illustrates why these details matter. The archived Hugging Face Open LLM Leaderboard v1 used the EleutherAI Language Model Evaluation Harness, selected metrics by benchmark, and noted slight result variation with batch size because of padding. The leaderboard was archived in June 2024, so this is documentation of its historical setup, not a description of a current leaderboard (Open LLM Leaderboard v1 documentation).
What newer work says about alternatives
Koen Oostermeijer’s paper “Accuracy and Normalized Accuracy under Length Bias: Analysis, Guidelines, and a Bayesian Alternative,” published in the Proceedings of the 43rd International Conference on Machine Learning in 2026 (PMLR volume 306, pages 94698–94713), analyzes when ordinary and length-normalized accuracy are appropriate. It reports that length normalization frequently overcorrects and that the bias depends on completion lengths and evaluation conditions (PMLR paper).
The paper also introduces Bayesian accuracy, which uses an explicit prior over answer length to compute posterior candidate probabilities. PMLR describes it as a drop-in replacement for likelihood-based multiple-choice evaluation that requires no additional forward passes. The authors report lower empirical length bias than standard and length-normalized accuracy across the benchmarks and few-shot settings they studied. That result is specific to the study; it does not establish that the method eliminates bias in every benchmark or application.
Choosing a metric for an evaluation
For a result you need to interpret, start with the benchmark’s documented metric and the model report’s definition rather than choosing a score by name alone. If selecting a method for an evaluation, compare how each rule handles the benchmark’s candidate lengths and serialization, then assess its behavior on the target benchmark and few-shot setup. Reproducibility and compute cost matter too: the 2026 paper reports that Bayesian accuracy requires no additional forward passes in its stated setup, but that should not be generalized beyond that setup.
Quick Recap
Best Value
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
