Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
A model’s token price tells you what one input or output unit costs; it does not tell you what your team pays to get an accepted code change. To compare coding agents or routing workflows, divide the total attributable cost of a fixed evaluation by the number of patches that pass acceptance checks you defined in advance. Count unsuccessful attempts and retries in the cost, and report the total spend and passing-patch count alongside the ratio.
What does “cost per passing patch” measure?
Cost per passing patch is an evaluation metric: the total cost attributable to all attempts in a defined run, divided by the number of patches that satisfy a predeclared acceptance rule.
Cost per passing patch = total attributable evaluation cost ÷ passing patches
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Repair Windows errors before they cause bigger problems3Scan for outdated or missing drivers - takes under a minuteIt is an editorial calculation, not a formally standardized industry metric. Its value is that it puts cost beside an explicit measure of accepted work. A low per-token price can still produce a high cost per passing patch if attempts fail, require retries, or use substantial tool and agent time.
#1 Best Overall
For example, suppose a run incurs $120 in model and agent charges and produces 10 passing patches. Its cost per passing patch is $12. If the same spend produces 20 passing patches, the figure is $6. The ratio is only meaningful when the task set, acceptance rule, cost boundary, and workflow are clear.
What should count as a passing patch?
Decide what “pass” means before comparing systems. Specify the tasks, repository versions, required tests, and any review criteria. Otherwise, two reported costs may use different definitions of success and cannot be fairly compared.
Benchmark acceptance
For a benchmark, name the dataset and task version and follow its evaluation procedure. SWE-bench uses real GitHub issues: an agent receives a repository and issue description and produces a patch. In OpenAI’s description of the benchmark, FAIL_TO_PASS tests check the issue behavior and PASS_TO_PASS tests check for regressions. OpenAI’s “Introducing SWE-bench Verified,” updated February 24, 2025, states: “Both sets of tests are required to pass for the edit to fully resolve the original GitHub issue.” A benchmark resolution is therefore a result under that benchmark’s tests—not a guarantee that a patch is correct in every use or accepted by your team.
Internal acceptance
For an internal evaluation, use the checks your team actually requires, such as a specified test suite and a consistent review rule. If you want to measure human-reviewed acceptance, define what reviewers judge and apply the same process to every workflow. Report automated-test results separately from review outcomes rather than blending them into an unexplained single pass count.
Which costs belong in the numerator?
Set a cost boundary and use it consistently. A practical comparison usually includes every billed charge that is part of the workflow being tested, including failed attempts, retries, tool calls, and agent or runtime charges. Do not include only the final successful attempt: doing so hides the cost of getting there.
- Model charges: billed input and output usage, including usage from failed attempts and retries.
- Agent and tool charges: billed runtime, hosted-agent, or tool use that belongs to the evaluated workflow.
- Other infrastructure: include it if it is part of the comparison’s chosen cost boundary; otherwise disclose that it is excluded.
- Engineering review: decide whether reviewer time is included. If it is, state the method used to cost it; if not, say the reported figure excludes review.
Keep engineering labor and infrastructure visible as separate categories when possible. A model-and-agent bill is not the same as total organizational cost, and silently mixing those categories makes the ratio hard to interpret. If prices or labor rates vary, record the rates and evaluation date so the reported amount has a clear basis.
How can you run a fair comparison?
Use the same tasks and evaluation conditions for each candidate workflow. A changed retry budget or test procedure can affect both cost and pass count, so treat it as a different experiment rather than a like-for-like comparison.
- Choose and freeze the task set. Record the repositories, task identifiers, and versions. Use the same set for each workflow, and state any tasks excluded before evaluation.
- Write the acceptance rule. Specify the tests and any review criteria that determine a pass. Set the rule before seeing results.
- Hold the workflow steady. Keep tools, environment, retry budget, timeout, and stopping rule consistent. If a workflow is a model-routing sequence, define its routing rules and handoffs in advance.
- Track every attempt. Log task, attempt number, model or workflow, billed model and agent costs, tool use, elapsed time if relevant, outcome, and test result. Preserve patches and run logs.
- Classify failures honestly. Distinguish a patch that fails the acceptance checks from an evaluation infrastructure error. Record both, and state whether infrastructure errors are excluded from the pass denominator or trigger a rerun under a predeclared rule.
- Calculate and publish the raw values. Add the attributable costs for the entire evaluation, count passing patches under the declared rule, and divide. Report total cost and passing count next to the ratio.
For a reproducible result, retain per-task outcomes, test output, run logs, and applied diffs. The official SWE-bench evaluation guide distinguishes resolved, unresolved, empty-patch, and error outcomes, and notes that results are cached by run ID and instance ID. When changing predictions for an already evaluated instance, use a new run ID so the saved result does not obscure the new run.
How should you report more than the ratio?
A single ratio can hide a small or uneven sample. Put the underlying counts and operational measures next to it so readers can see what drove the result.
Rank #4
| Measure | What to report | Why it matters |
|---|---|---|
| Total attributable cost | Run total, with the included cost categories and evaluation period | Shows the numerator and the scope of the calculation. |
| Passing patches | Count and share of evaluated tasks that met the declared rule | Makes the denominator and success rate visible. |
| Cost per passing patch | Total attributable cost divided by passing patches | Connects spend to accepted results. |
| Retries and tool use | Counts or billed usage under the same logging rules | Helps explain why workflows with different token prices have different total costs. |
| Latency | Elapsed time under a stated measurement method, if turnaround matters | Shows a trade-off that cost alone cannot capture. |
| Patch quality | Review findings or other post-test quality checks, when available | Automated test success does not establish broader maintainability or correctness. |
If no patch passes, the ratio is undefined—not zero. Report the total cost, zero passing patches, and the failure breakdown instead. For small task sets, show per-task results as well as aggregate figures; a handful of outcomes can shift the ratio sharply.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What can a passing result—and a benchmark—actually tell you?
A passing patch is evidence that the patch met the tests and rules you chose. It is not proof of universal correctness. OpenAI’s SWE-bench Verified analysis discusses misleading outcomes caused by underspecified issues, tests that are overly specific or unrelated, and environment setup failures.
Recommended Free Tools
In OpenAI’s 2024 analysis of 1,699 randomly selected samples annotated during construction of SWE-bench Verified, 38.3% were flagged for underspecified problem statements, 61.1% for tests that might unfairly reject valid solutions, and 68.3% were filtered for one or more quality concerns. These percentages describe that annotated analysis set, not the final 500-sample Verified subset. OpenAI also says 93 software developers with Python experience were involved in screening samples for the subset. Those construction details help explain why a benchmark’s task quality and evaluation setup matter; they do not establish how your own repository tasks will behave.
Best Value
For SWE-bench specifically, distinguish an evaluation error from a patch that was evaluated and failed. The project’s guide gives examples of errors such as a patch not applying, a container failing to start, or a run timing out. State how such errors were handled instead of silently counting them as failures, excluding them, or treating them as passes.
What does the published cost example show?
In a 2026 personal report, Tyler Folkman said a Sol-only workflow produced four passing patches for $2.74, while a Sol-to-Kimi handoff produced three for $11.12. Dividing the reported spend by the reported pass count gives about $0.69 and $3.71 per passing patch, respectively. Those calculations describe only the figures in that report; the anecdote is not an independent benchmark, and the available account does not establish a general ranking or a comparable result across teams. It illustrates why token prices alone are not enough to judge a workflow.
How do you use the metric to choose a workflow?
First compare results under the same task set, cost boundary, and acceptance rule. Then inspect the per-task outcomes and the cost ledger: a favorable aggregate ratio may depend on a few easy tasks, while a higher ratio may accompany faster completion or stronger review results. If those factors matter to your team, show them alongside cost rather than assuming one number captures the whole decision.
No universal winner follows from cost-per-patch arithmetic alone. Without a verified comparison using the same current tasks, harness, workflow conditions, and price basis, the defensible choice is to measure the systems you can actually run under your own acceptance criteria.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
