iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
A numeric-answer grader can mark correct arithmetic wrong if it extracts the first number in a response rather than the final answer. In Erik Hill’s account of his own grading harness, replies that repeated a number from the prompt put that number ahead of the correct result, and a regular expression compared the wrong value.
How a correct answer became a grading failure
Hill describes a grader that uses a regular expression to find a number in a model’s output, then compares that match with the expected answer. Because it uses the first match, the result can depend on how the response is phrased—not just on whether the arithmetic is right.
For “Compute 3 + 4 * 5,” the expected result is 23, but a response that starts by restating the expression begins with 3. For “What is 0.3 minus 0.1,” the expected result is 0.2, while a response that repeats the question begins with 0.3. In each case, the grader can reject a response containing the right calculation because it has selected an earlier number.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
What Hill reported in his test suite
In his 2026 account, Hill says seven of the suite’s eight tasks began with a number different from the expected answer, making them vulnerable to this first-match behavior. He reports that the float-subtraction task failed across nine models, and that one model failed in 60 of 60 runs even though its responses included the correct result.
Those are results from Hill’s own harness and test setup, not an independent benchmark or a field-wide failure rate. He says the other vulnerable tasks passed because the tested models answered tersely; that does not show the grader was robust to responses that restate the prompt.
Did changing the instruction help?
Hill tested three wordings for the same float-subtraction task across three models, for nine reported calls. The as-shipped instruction, “Give the number to one decimal place,” led the grader to extract 0.3 and fail all three calls. With “Give only the number” and “Number only,” it extracted 0.2 and passed all three calls for each wording.
Rank #2
This small experiment suggests that wording affected whether responses restated the problem in those calls. It covers one task and three models, however, and Hill says it does not establish how broadly the effect generalizes.
Free tools Windows power users keep installed
One-click scans. No signup required.
Why the grader fix is not obvious
Choosing the last number instead of the first could help when a response shows its work and ends with the answer. But it could also accept a response that shows calculations and finishes with a wrong value. Requiring a bare-number response could avoid ambiguity in extraction, but it also makes formatting part of the task. Hill’s account identifies the tradeoff rather than establishing one universally reliable fix.
Rank #3
- Carefully designed questions: Ensuring a solid understanding of concepts
- Engaging activities: Offering a mix of enjoyable exercises
- Problem-solving techniques: Providing strategies for tackling challenges
- Vibrant, full-color visuals: Enhancing learning with captivating illustrations
The practical diagnostic is to inspect both the extraction rule and the full response: if a model repeats a number from the prompt, check whether the grader compares that first number rather than the answer. As Hill puts it, “If you grade numeric answers with a regex: check which end of the string you are reading, and check it against an output that restates the question.” Read Hill’s account on DEV Community.
Quick Recap
Rank #4
- Full of different activities to help your child develop their skills
- Contains one sixty-four page workbook
- Available in a variety of different age groups
- Available in different themed activity books
- Made in USA
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
