Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstalliTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
There is no universal “cost per AI query.” A defensible estimate starts with your actual workload—requests, input and output tokens, caching, peaks, retries and model mix—then applies the billing meters and infrastructure required to meet your quality, latency, availability and geographic targets. Report a low, base and high range, show every assumption, and compare managed APIs, provisioned capacity and self-hosted inference on the same service level.
Token charges are only one part of the bill. Peak concurrency can force you to reserve capacity that sits idle between bursts; self-hosting adds accelerators, servers, networking, storage, security, power, cooling and operations. The workflow below turns those variables into a monthly cost per useful request or token that you can validate against production measurements.
What actually drives the cost of generative AI at scale?
Four variables dominate most estimates:
- Token volume and model mix: request counts, prompt tokens, generated tokens, cached-token share and the models used by each feature determine metered usage.
- Workload shape: peak requests per second, concurrency, burst duration, maximum output settings and retries determine how much capacity must be available at once.
- Service envelope: quality, first-token latency, end-to-end latency, availability, data-residency and region requirements constrain the model and deployment choices.
- Operating model: managed usage, committed or provisioned throughput, and self-hosting place costs in different buckets. A raw accelerator hourly rate is not a normalized comparison.
Keep these dimensions separate in your spreadsheet. A low daily token count can still require expensive provisioned capacity if traffic arrives in short bursts, while a large asynchronous workload may run cheaply when it can use spare capacity and tolerate queueing.
Free tools Windows power users keep installed
One-click scans. No signup required.
How do I calculate LLM inference cost?
Use a repeatable sequence. Do not begin with a provider’s headline price; begin with the traffic and service level your application must deliver.
#1 Best Overall
- The H100 NVL graphics card is designed to scale the support of large language models, such as GPT3-175B, in mainstream PCIe-based server systems, providing up to 12X the throughput performance of HGX A100 systems when configured with 8 units.
- Equipped with advanced features, including 94GB of high-speed HBM3 memory, NVLink connectivity for enhanced inter-GPU communication, and an impressive memory bandwidth of 3938 GB/sec, the H100 NVL is built for high-performance AI inference tasks.
- The card showcases a robust performance spectrum across various compute types: 68 TFLOPS for FP64, 134 TFLOPS for both FP64 Tensor Core and FP32, escalating up to 7916 TFLOPS/TOPS for FP8 and INT8 Tensor Core operations, all benefiting from sparsity optimizations.
- It enables standard mainstream servers to deliver high-performance capabilities for generative AI inference, simplifying the deployment process for partners and solution providers with fast time to market and ease of scalability.
- The H100 NVL's power efficiency is optimized with a configurable maximum power consumption ranging between 2x 350-400W, supporting extensive computational tasks without excessive power usage.
1. Define the workload by feature and model
Break the application into call types such as chat turns, document extraction, summarization, classification and tool-calling. For each type, record:
- Requests per day and per month.
- Input tokens per request, output tokens per request and their distributions, not only averages.
- The share of input tokens that the provider treats as cached, plus the cached-token meter and rate.
- Peak requests per second, maximum concurrency and the shape and duration of bursts.
- Reasoning, retrieval, tool-use or other hidden-token overhead that increases billed usage.
- Expected retries, timeouts and error rates.
- The model selected for that feature and the percentage of traffic routed to each model.
Use measured traces whenever they exist. Otherwise create low, base and high scenarios and label every assumption. A single average prompt multiplied by monthly requests hides the long-tail outputs and peaks that often determine capacity.
2. State the service envelope
Write down the result the system must provide, not just the number of tokens it should process:
- Minimum quality or evaluation score.
- First-token and complete-response latency objectives.
- Maximum concurrent requests and acceptable queue time.
- Availability target and behavior during overload.
- Allowed regions and data-residency restrictions.
- Whether asynchronous or batch responses are acceptable.
These constraints can change the least-cost architecture. A model that is inexpensive per token may fail your latency target at peak load, while a larger model may reduce retries or post-processing enough to lower cost per useful result.
3. Calculate metered usage
For each model and meter, calculate billable input, cached input and output separately. A general monthly usage equation is:
usage cost = (billable input tokens × input rate)
+ (cached input tokens × cached-input rate)
+ (output tokens × output rate)Apply the provider’s current rates, model names, regions and billing units. Then add paid tools, retrieval, storage or other services only when your architecture uses them. Keep commitments, minimum-capacity charges, overage and hosting charges as separate lines so a discount is not confused with a fixed obligation.
4. Build low, base and high cases
Vary the assumptions that are least certain: request growth, output length, cache hit share, peak-to-average ratio, retries and model routing. Present a range rather than a single false-precision number. The range should show which assumption causes the largest change; that is where measurement or product design work has the highest financial value.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchesHow much does AI cost per query?
“Per query” is meaningful only after you define the query. Two applications can send the same number of requests while using radically different token counts, models, latency targets and retry policies.
Compute a useful-request metric after calculating the monthly bill:
cost per useful request = total monthly cost ÷ successful requests
cost per useful output token = total monthly cost ÷ accepted output tokensCount a request as useful according to your product’s acceptance rule—for example, a response that passes validation or completes the user’s task. Include failed attempts and retries in total cost, but do not count rejected outputs as useful work. For multi-model systems, calculate the metric by feature first, then report a weighted application total.
Keep quality and latency constant when comparing models. A cheaper token rate that produces more retries, longer prompts, or human review can have a higher cost per useful result than a higher-priced model.
How do managed API charges work?
Managed services usually expose usage meters, commitments or hosted deployments. The exact rates and meter names change, so verify them in the provider documentation for the intended region and model at the time you prepare the estimate.
Usage-based deployment
Multiply each model’s billable input, cached input and output tokens by its current rates. Add tool calls, retrieval, storage and other services only where they are part of the request path. Model traffic by feature so a high-volume, low-complexity workload is not accidentally priced as if every request used your most capable model.
Rank #2
- Graphics Card Interface: Pci E
Committed or hosted deployment
Separate recurring commitments from variable overage. A commitment can lower the marginal meter rate but still costs money when demand is below the contracted level. Hosted fine-tuned deployments can also charge while they are deployed, even during low usage periods; Microsoft Learn states: “Fine-tuned deployments incur hosting charges while deployed, even during low usage periods.” See Microsoft Foundry’s cost-management documentation for the applicable billing concepts.
Items to verify before using a price
- Whether input, output and cached tokens have separate meters.
- Whether token rounding, minimums or batch rules apply.
- Region-specific rates and model availability.
- Commitment duration, minimum capacity and overage treatment.
- Charges for fine-tuned model hosting when traffic is low.
- Whether tools, retrieval and storage are billed outside the model meter.
How do provisioned throughput and capacity costs work?
Provisioned capacity is sized for the traffic you must serve at the required latency, not for your average daily token count. Peak concurrency, burst length, maximum output settings and headroom determine how many capacity units you need and how much of that capacity is idle between bursts.
Use utilization as a constraint, not a target of 100%
Microsoft’s provisioned-deployment guidance estimates utilization from prompt tokens (after subtracting cached tokens) and the configured max_tokens, then corrects the estimate against actual token counts. Treat 100% utilization as an overload condition: Microsoft documents that it can trigger immediate HTTP 429 responses. Size headroom for normal variation, retries and bursts, and monitor actual utilization after launch. Details are in Microsoft’s production guide for provisioned throughput deployments.
Translate traffic into the provider’s capacity unit
Capacity units are model-specific. Google documents provisioned throughput in GSUs with model-specific burndown rates; a GSU allocation that works for one model or token mix cannot be assumed to work for another. Follow the measurement procedure in Google Cloud’s provisioned-throughput documentation, using representative prompts and output lengths.
Include idle and commitment cost
For each scenario, calculate paid capacity hours, expected utilization, unused capacity and the commitment period. A system that needs ten units for a five-minute burst may pay for those units continuously unless it can scale or queue work. Conversely, an asynchronous workload may fit a smaller allocation if its service envelope permits delayed responses.
Is it cheaper to host an LLM or use an API?
Neither option is always cheaper. Compare them at the same model quality, traffic distribution, latency, availability, region and data-residency requirement, then include idle capacity and operational labor.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →| Option | Primary billing basis | What must be included | Operational profile |
|---|---|---|---|
| Managed token API | Input, cached-input and output usage, plus any selected tools or storage | Variable token charges, retries, tool and retrieval charges, and any minimums or overage | Provider operates serving hardware; limits, model availability and regional terms apply |
| Committed or provisioned throughput | Reserved capacity or provider-specific units, with possible overage | Commitment or hosting charges, required headroom, utilization and idle capacity | More predictable throughput, but capacity can be paid for during quiet periods |
| Self-hosted inference | Accelerator and server acquisition or rental plus operating expenses | Serving software, networking, storage, security, power, cooling, facility overhead, operations and idle time | Maximum control over deployment and data location; you own capacity planning and reliability |
Use “cost per useful request” or “cost per useful token” as the comparison unit. Do not compare a cloud GPU hourly quote with an API token rate without converting both to the throughput and utilization your workload can actually sustain.
How do I estimate the total cost of self-hosted inference?
Build a total-cost-of-ownership model with one line for every resource needed to meet the service envelope.
Hardware and serving
- Accelerators and servers, purchased or rented.
- Serving software, model licenses and engineering time to integrate them.
- Redundant capacity for failures, maintenance and upgrades.
- Storage for model weights, caches, logs and input data.
Network, security and facility
- Low-latency networking between clients, inference workers and data stores.
- Ingress and egress charges where applicable.
- Identity, access control, encryption, monitoring and security operations.
- Power, cooling, rack or facility overhead and the energy consumed while accelerators are active or idle.
Measure throughput with your real prompts
Benchmark the exact model and serving stack with representative prompt lengths, output limits, concurrency, batching and latency objectives. Record usable output tokens per second at the utilization you can sustain without violating your error or latency target. Convert the resulting monthly infrastructure and operations cost into cost per useful token or request. A benchmark at a different batch size or output length is an input to the model, not proof of your production economics.
AWS’s cost guidance emphasizes selecting an appropriately sized model—“Pick the right type and size of model for your use case. (You may not need the largest model.)”—and accounting for data preparation and storage, training or fine-tuning, data residency, low-latency networking, security and purpose-built chips or GPUs. Read the AWS infrastructure cost guide for the full category list.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
What do published cost and energy benchmarks actually tell you?
Published figures are useful anchors only when your configuration matches the tested model, hardware, software and workload. They are not guaranteed cloud rates or universal per-query costs.
| Published figure | Scenario and source | How to use it |
|---|---|---|
| Approximately $0.09 per million tokens | NVIDIA’s H100 page, citing SemiAnalysis InferenceX benchmarks from April 2026; GPT-OSS-120B on H100 with vLLM at 66 tokens per second per user | Scenario-specific benchmark estimate; do not treat it as a hardware quote, provider list price or cross-provider comparison. See NVIDIA’s H100 benchmark page. |
| 0.34 Wh per query median | Microsoft Research’s September 2025 study of frontier-scale models above 200 billion parameters on an H100 node under its specified workload, utilization and PUE assumptions | Energy per query, not a dollar amount. See the study and its methodology. |
| 4.32 Wh per query median | The same Microsoft Research study’s test-time-scaling scenario, using 15 times more tokens per typical query | Shows how additional test-time computation changes energy; it is not a universal inference value or monetary estimate. |
To convert an energy figure into a local operating cost, you would need your electricity price, facility efficiency, utilization and hardware configuration. The published watt-hour values alone cannot produce a reliable dollar estimate.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.How should I compare managed, provisioned and self-hosted estimates?
Normalize every option to the same traffic and service envelope before drawing a conclusion.
Rank #3
- Discrete graphics card memory 40 GB
- Memory bandwidth (max) 1555 GB/s
- Graphics processor family NVIDIA
- Graphics processor A100
- Fix the workload distribution. Use identical request counts, token distributions, cache share, output limits, peaks and retry assumptions.
- Fix the quality target. Select models or configurations that meet the same evaluation threshold; otherwise a lower price may simply reflect lower capability.
- Fix latency and availability. Include the capacity and redundancy needed for the same first-token, end-to-end and uptime objectives.
- Price all meters. Include input, cached input, output, tools, storage, commitments, overage, hosting and self-hosted facility costs as applicable.
- Apply utilization honestly. Use sustainable throughput and account for idle capacity, burst headroom, maintenance and failures.
- Report useful-unit economics. Show monthly total, cost per useful request and cost per useful output token, with low, base and high cases.
Keep list prices, measured benchmarks and modeled estimates labeled as such. A provider’s published rate is not evidence that your prompt mix will achieve the same throughput, and a lab benchmark is not a contract price.
How do I validate and update the estimate?
After deployment, reconcile the model with production data. Compare predicted input, cached and output tokens with billing meters; compare predicted concurrency and utilization with observed peaks; and track retries, 429 responses, latency and accepted outputs. Replace assumptions with measured distributions as soon as enough traffic is available.
Recalculate when traffic mix, model routing, prompt templates, output limits, cache behavior, region, provider rates or commitment terms change. Vendor pricing and capacity documentation are volatile, so verify rates, meter names, model availability and regional conditions each time you make a material purchasing or architecture decision.
What common estimation mistakes inflate or hide cost?
Using daily averages for capacity
Average tokens conceal bursts. Size provisioned or self-hosted capacity against peak concurrency and latency, then model the idle hours explicitly.
Multiplying a headline token rate by requests
Requests have different input and output lengths, cached-token treatment and model routes. Meter each component separately and include retries and tools.
Recommended Free Tools
Assuming cached tokens are free
Caching can reduce billable usage, but only the provider’s documented cached-token meter determines the charge. Measure the actual cache share instead of assuming every repeated prefix is free.
Ignoring maximum output settings
Provisioned-utilization calculations can use configured max_tokens, while actual output may be lower. Model both the configured limit and observed token counts, and leave headroom for long responses.
Comparing hardware prices with API prices
An accelerator purchase or rental omits servers, networking, storage, security, power, cooling, operations and idle capacity. Convert the complete self-hosted total into useful throughput before comparing it with a managed meter.
Treating energy as a dollar forecast
Watt-hours describe energy, not money. Electricity price, facility efficiency, utilization and hardware configuration are required for a monetary calculation.
Locking in an outdated rate
Model and region availability, meter names, commitments and prices change. Link your estimate to the provider documentation you used and record the date and region alongside every rate.
Frequently Asked Questions
Should cached tokens be included in an LLM cost estimate?
Yes. Track cached-token volume separately from uncached input and apply the provider’s documented cached-input meter. A cache hit changes the bill only when the service defines a separate rate or exclusion, so validate the assumption against billing data.
How often should a production AI cost model be refreshed?
Refresh it whenever traffic mix, prompts, output limits, model routing, cache behavior, region, provider pricing or commitment terms change. Reconcile forecasts with actual token meters and utilization after each material release rather than allowing the original assumptions to persist.
What does a 429 response mean for capacity planning?
A 429 at a provisioned deployment indicates that available capacity or utilization limits were exceeded at that moment. Treat the event as evidence that peak demand, burst headroom, maximum-token settings or retry behavior needs to be represented in the capacity model.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Can an energy-per-query number tell me my monthly electricity bill?
No. Convert watt-hours to money only after supplying your electricity tariff, facility power-usage effectiveness, accelerator utilization and the hardware configuration that matches your workload. A study’s energy scenario cannot substitute for those local operating inputs.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
