Recommended Free Tools
iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
The reliable way to measure an AI coding task is to instrument every API call, preserve each provider’s raw usage payload, normalize the token categories, apply a dated price registry, and reconcile the result with the provider’s financial records. Count the complete task—not just the text visible in the final answer. Include input context, cached input, output, exposed reasoning or thought tokens, tool calls, retries, sub-agent calls, and context-reuse calls. Then report both the task total and its distribution across many runs.
A model’s advertised price per million tokens is only one input to the calculation. OpenAI notes that models can tokenize the same text differently and can generate different amounts of output or reasoning, so a lower unit price does not necessarily produce a lower total cost. Prices, model aliases, usage fields, and billing rules change; every price record should therefore carry an effective date and its source.
What you need to measure
A coding task is a business or engineering unit such as “fix the authentication bug in repository X,” not a single HTTP request. One task can trigger several model turns, retrieval calls, shell or editor tools, retries after timeouts, and calls made by sub-agents. Measuring only the last response understates both tokens and cost.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →For each request and response, record these dimensions:
#1 Best Overall
- Identity: provider, exact model identifier, endpoint, task or session ID, request ID when available, and your application version.
- Time: a UTC timestamp for request start and completion. Use UTC consistently; OpenAI states that dashboard data is displayed in UTC.
- Execution: status, latency, retry number, timeout information, and whether the call was made by a primary agent, tool, or sub-agent.
- Usage: the raw provider usage object plus normalized input, output, cached-input, reasoning or thought, total-token, and any provider-specific billable-unit fields.
- Cost context: price-registry version, currency, geography or service tier, and effective date used for the estimate.
Keep raw and normalized data side by side. Raw fields let you reprocess historical records when a provider adds a field or changes its schema; normalized columns let you compare providers without discarding provider-specific details.
A provider-neutral usage ledger
Store one ledger row per provider request, including requests that are later grouped into one coding task. A practical schema is:
| Column | Purpose |
|---|---|
event_id |
Unique identifier for this request/response pair. |
task_id and session_id |
Link calls to the coding task and conversational run. |
provider, model, endpoint |
Identify the service and exact model variant. |
started_at_utc, completed_at_utc |
Support time-window analysis and reconciliation with dashboards. |
status, retry_index, timeout |
Distinguish successful, failed, retried, and abandoned calls. |
input_tokens |
Normalized non-cached input tokens. |
cached_input_tokens |
Input tokens billed under a cache category, when reported. |
output_tokens |
Visible or generated output tokens. |
reasoning_tokens or thought_tokens |
Hidden deliberation counted by a provider, when exposed or inferable from its usage object. |
total_tokens |
Provider-reported total, retained even when it does not equal your sum. |
billable_units_json |
Audio, image, tool, character, or other provider-specific units. |
raw_usage_json |
Untouched usage payload for audit and future parsing. |
price_version, currency, estimated_cost |
Make each estimate reproducible. |
Do not overwrite a missing value with zero. Use a null value and a field such as not_reported when a provider does not expose a category. A reported zero means the provider measured zero; an absent field means you do not know.
Capture usage from each API
OpenAI endpoints
OpenAI uses different field names by endpoint. Chat Completions responses use prompt_tokens, completion_tokens, and total_tokens. Responses API results use input_tokens, output_tokens, and total_tokens. Map both forms into your normalized columns while preserving the original names.
For streamed Chat Completions, request a final usage chunk by enabling stream_options.include_usage. Your collector must continue reading until that terminal usage event; summing only intermediate text chunks will not produce the authoritative total.
OpenAI’s dashboard supports UTC display, project filters, exports, and tokens-per-minute intervals. Use those views to spot spikes and missing logging, but do not treat dashboard activity as the invoice. OpenAI distinguishes dashboard usage from billing data and directs finance reconciliation to its Costs endpoint. Its Usage API provides granular activity; the Costs endpoint is the appropriate comparison for financial records.
Built-in tool tokens are billed at the selected model’s token rates. Record tool invocations and their associated request/response events so a task ledger explains why a model call used more tokens than the visible answer suggests.
Anthropic requests
Anthropic responses expose usage fields that should be captured exactly as returned, then mapped to your common schema. Keep any cache-read, cache-write, or other provider-specific categories in billable_units_json if they do not fit your standard columns.
Anthropic documents a Usage and Cost API intended for reconciliation. Many organizations use prepaid usage credits, while others have monthly invoicing arrangements. The documented charging behavior states that successful API calls and completed tasks are charged while failed requests generally are not. “Generally” matters: retain status and provider response data, and verify your account’s treatment of edge cases rather than applying a universal assumption.
Gemini responses
Gemini returns actual request usage in the response’s usage_metadata. Persist that object, including every token category it contains. Google states that when billing is enabled, pricing can include the full thought tokens generated to produce a summary even when those thoughts are not shown in the output. Therefore, visible response length is not a substitute for the billed usage metadata.
Normalize tokens without losing meaning
Normalization is a translation layer, not a license to erase differences. Define a canonical set of columns and a mapping table for each provider and endpoint. At minimum, use:
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
- Input: tokens supplied to the model and not classified as cached.
- Cached input: tokens charged under a cache-hit or cache-read category.
- Output: generated answer or code tokens.
- Reasoning/thought: hidden deliberation tokens when the provider reports them or specifies that they are billable.
- Total: the provider’s own total, not a recomputed replacement.
Also capture tool-call arguments, tool results, and serialized conversation history if they become model input. A repository snapshot, compiler log, test output, or repeated system instruction can dominate input usage even when the final patch is small.
When categories overlap, do not add them blindly. Some providers include reasoning in output or total fields; others report it separately. Document the provider-specific relationship in your mapping version and calculate a “normalized billable total” only after confirming the provider’s definition.
Build a versioned price registry
Prices must be data, not constants buried in application code. Create one record per provider, model, token category, geography or service tier, currency, and effective date. Keep the source name and URL in the registry when available, and retain superseded records rather than editing them in place.
For a category priced per million tokens, calculate:
category_cost = (category_tokens / 1,000,000) × price_per_1M_tokens
Rank #3
Sum category costs across a call, then sum calls across the task:
task_cost = Σ(input_cost + cached_input_cost + output_cost + reasoning_cost + provider_specific_costs)
Apply the price that was effective for the call’s billing date and account tier, not today’s price. Keep currency conversion separate from token arithmetic; store the exchange-rate date and method if you convert currencies for reporting. If a provider uses a non-token unit, calculate it with that provider’s documented rate and keep it separate from token categories.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Fix the driver behind crashes, sound loss and screen glitches3Repair Windows errors before they cause bigger problemsList prices and observed task costs answer different questions. List prices help you model a planned call; observed costs include actual tokenization, retries, cache behavior, tool use, and provider-side reasoning.
Attribute every call to a coding task
Generate a stable task_id when a user or workflow starts a coding objective. Propagate it through agent, tool, and retry layers. Include:
- Initial planning and clarification turns.
- Repository or documentation retrieval calls.
- Compiler, test, shell, browser, or code-indexing tool results sent back to the model.
- Retries after rate limits, timeouts, network failures, or client errors.
- Sub-agent calls and context-reuse calls.
- Final review, patch generation, and summarization turns.
Store a parent-child relationship for sub-agents so you can report both the full task and each component. A retry that was billed is part of the task even if its response was discarded. A failed request that was not charged should remain visible with zero or unknown cost and its status, rather than disappearing.
Measure tasks with distributions, not one average
For a meaningful period, aggregate all calls by task_id. Report total cost, input/output/reasoning/cache subtotals, call count, retry count, latency, and quality outcome. Publish at least the median and 90th-percentile (p90) task cost. The median describes a typical task; p90 shows what a budget must withstand when context or tool use expands.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Segment results by task type, repository size, model, agent configuration, and success criterion. A single blended average can hide that small bug fixes are cheap while dependency upgrades or long-running test loops are expensive.
A 2026 arXiv study, “How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks,” found that agentic tasks can consume substantially more tokens than code chat or code-reasoning tasks, that input tokens often drive total cost, and that repeated runs of the same task varied by up to 30×. The study also found that higher token use did not reliably translate to higher accuracy. Treat those as study findings, not a universal benchmark: measure your own workload and quality criteria.
Why your estimate differs from the provider bill
You counted characters or visible words
Tokenizers split code, punctuation, whitespace, identifiers, and non-ASCII text differently. The same repository excerpt can therefore produce different token counts on different models. Visible answer length omits system instructions, conversation history, tool results, and hidden reasoning.
You ignored cached input
A provider may report cached input separately and charge it at a different rate. If your local estimate treats every input token as a full-price token, it will diverge in either direction depending on the cache rate and hit pattern.
You omitted retries and tools
Rate-limit retries, timeout retries, test-output loops, and tool-result messages can add more input than the final code change. Task aggregation must include them, even when a user sees one answer.
You used the wrong endpoint or model price
Aliases can move to a new model, and Chat Completions and Responses use different usage field names. Resolve the exact model identifier and endpoint from each event, then select the price record whose effective date and service tier match.
You compared dashboard activity with financial records
OpenAI explicitly distinguishes dashboard usage from billing data. Use its Usage API for granular activity and the Costs endpoint for finance reconciliation. For Anthropic, use the Usage and Cost API; for Gemini, reconcile your stored usage_metadata against the account’s billing export.
A repeatable measurement procedure
- Create the task context. Generate
task_id, record the UTC start time, repository or project identifier, workflow version, and intended success criterion. - Wrap every provider call. Log provider, exact model, endpoint, request ID, timestamps, status, retry index, tool-call metadata, and the untouched request/response usage objects.
- Capture streaming completion. For OpenAI Chat Completions, enable
stream_options.include_usageand persist the terminal usage chunk. - Normalize after receipt. Map endpoint-specific fields into input, cached input, output, reasoning/thought, total, and provider-specific units. Preserve nulls for unreported categories.
- Resolve pricing. Look up the registry record by provider, model, category, tier, currency, and effective date. Calculate category costs with the per-million formula.
- Join the task graph. Attach retries, tools, sub-agents, and context-reuse calls to the originating
task_id. - Run reconciliation checks. Compare daily or monthly aggregates with provider usage and cost records. Investigate gaps, duplicate events, timezone mismatches, and calls missing a price version.
- Report distributions and quality. Publish median and p90 cost, token-category shares, latency, retry rate, and task success or review score. Keep list-price simulations in a separate report.
How to compare providers for coding work
Use the same repository slices, prompts, tool policy, stopping criteria, and success tests when comparing models. Then evaluate these axes independently:
Free tools Windows power users keep installed
One-click scans. No signup required.
| Comparison axis | Question to answer |
|---|---|
| Input price | What is charged for uncached prompt and context tokens? |
| Output price | What is charged for generated code, explanations, and tool calls? |
| Cached-input price | How are repeated system prompts or repository context priced? |
| Reasoning/thought billing | Are hidden tokens reported and charged, and where do they appear in totals? |
| Tokenizer behavior | How many tokens does each model use for the same code and logs? |
| Context and tools | What context-window limits and tool serialization patterns affect input volume? |
| Retry and timeout rules | Are failed, partial, or retried requests charged? |
| Reporting APIs | Can you reconcile event-level records with provider financial data? |
| Task outcome | What cost, latency, and quality distribution results from the complete workflow? |
Do not rank a provider by price per million tokens alone. OpenAI’s guidance is explicit: different tokenization and output or reasoning volumes can make a nominally cheaper model more expensive for the same work.
Operational checks and failure recovery
Missing usage fields
Mark the event as incomplete, retain the raw response, and prevent it from entering a “complete” cost report until a parser update or provider reconciliation fills the gap. Never infer usage from character count when an authoritative field is expected.
Duplicate or out-of-order events
Deduplicate on provider request ID plus endpoint, while retaining a separate client-attempt ID for retries. Order by provider timestamp when available and fall back to your UTC receipt time.
Timezone mismatch
Store UTC in every event and convert only in presentation. OpenAI’s dashboard is UTC; a local-midnight report can otherwise assign calls to the wrong day.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallPrice changed mid-period
Join each event to its effective-dated price record. Recalculate historical reports with the original record and label any restatement that uses a corrected provider price.
Unexpected cost spike
Drill from task total to token category, then to call and tool. Look first for a repository snapshot repeated on every turn, runaway test logs, a retry loop, a model alias change, or newly billed thought tokens. Compare the affected task’s p90 and call graph with a normal run.
Minimum dashboard for engineering and finance
An internal dashboard should expose:
- Daily and monthly spend by provider, model, project, and task type.
- Median and p90 cost per completed coding task.
- Input, cached-input, output, reasoning/thought, and other billable-unit shares.
- Tokens and cost per successful patch, test pass, or accepted change.
- Retry, timeout, and failure rates, with charged versus uncharged outcomes.
- Data freshness, missing-usage count, unmatched-price count, and reconciliation variance.
Set alerts on both money and behavior: a sudden rise in input tokens can precede a bill increase even when output remains stable, while a rise in retries can inflate cost without improving quality.
Practical interpretation
If you need one number for a completed task, use the reconciled sum of all billable calls linked to its task ID. If you need a budget for future work, use a distribution—at least median and p90 from comparable tasks—and add an explicit allowance for retries and unusually large contexts. If you need to choose between providers, run the same instrumented workflow and compare cost, latency, and accepted-result rate together.
The defensible conclusion is operational rather than a universal dollar average: no single published average represents every coding task. Accurate measurement comes from a durable event ledger, effective-dated prices, provider reconciliation, and task-level analysis that treats token usage as variable rather than predictable from visible code length.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
