iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
TOPS and tokens per second measure different things. TOPS is a chip’s theoretical peak operations throughput under a stated precision and counting method. Tokens per second is the speed a particular language model actually generates text on a complete system, with a specific runtime, prompt, context, power mode and memory configuration. You cannot convert TOPS directly into a token-rate forecast, and an accelerator with twice the TOPS will not necessarily generate twice as many tokens per second.
What TOPS measures
TOPS means trillions of operations per second. It is normally used to describe the peak arithmetic throughput of an AI accelerator such as a neural-processing unit (NPU), GPU tensor engine or other matrix-math block. The figure is a specification for the compute engine, not a measurement of an application completing a useful task.
Qualcomm’s technical explanation gives an illustrative dense-math relationship:
TOPS = 2 × MAC unit count × frequency / 1 trillion
#1 Best Overall
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
A MAC (multiply-accumulate) performs a multiplication and an addition. Counting both as operations produces the factor of two in the equation. The result changes with the number of MAC units, clock frequency, architecture and the numeric format used by those units.
Why the precision label is essential
AI inference TOPS is commonly quoted at INT8 precision, but an accelerator may support several formats, such as INT8, INT16, FP16 or other vendor-specific modes. Lower-bit integer arithmetic can increase throughput and reduce energy use, while a different format may be required for a particular model or operator. A TOPS number without its precision is therefore incomplete for a serious comparison.
Counting conventions also differ. Dense TOPS assumes every arithmetic lane is doing useful work. Sparse TOPS can assume that some weights or activations are zero and can be skipped. A sparse figure must not be compared with a dense figure as if they used identical work assumptions.
Peak capability is not sustained application speed
A peak specification assumes ideal scheduling and a stream of operations that keeps the relevant execution units busy. Real software may spend time moving data, executing unsupported operators elsewhere, waiting for memory, compiling kernels or synchronizing CPU, GPU and NPU work. Thermal limits and a laptop’s power policy can also lower sustained clocks. Qualcomm’s Peter Burns summarized the practical implication in a technical explainer published April 24, 2024: “A high TOPS number alone does not guarantee optimal AI performance.”
What tokens per second measures
A language model emits text in tokens, not necessarily whole words. A token may be a complete word, a word fragment, punctuation or a short sequence of characters, depending on the model’s tokenizer. Tokens per second (tokens/s or tok/s) is the observed output rate for a particular model running a particular workload.
For example, a report might state that a quantized model produced 18 output tokens per second on one laptop. That number describes that model, quantization, prompt, context length, runtime, device configuration and measurement window. It is not a property of the laptop’s NPU in isolation, and it does not predict the rate for another model.
Token counts are not word counts
NVIDIA’s performance documentation uses tokens per second for AI workloads and notes that a token can be a word or a smaller fragment. Consequently, 20 tokens per second does not necessarily mean 20 complete words per second. The ratio varies with language, punctuation, numbers, code and the tokenizer used by the model. Two models answering the same question can produce different token counts for similar-looking text. When reporting a rate, identify the model and tokenizer rather than converting it into an assumed words-per-minute figure.
Separate first-token latency from generation rate
Two timings matter to a person waiting for an answer:
- Time to first token (TTFT): the delay from submitting the request until the first output token appears. It includes prompt processing, model loading or memory transfers and scheduling overhead.
- Time per output token: the interval between subsequent generated tokens. Its inverse is the ongoing output rate, usually reported as tokens per second.
A system can have a short TTFT but a modest streaming rate, or a longer TTFT followed by very fast generation. Publishing only one number hides that difference. A cold start, in which the model must be loaded, can also have a very different TTFT from a warm run with weights and runtime caches already resident.
Rank #2
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
Prefill and decode are different phases
Prompt processing, often called prefill, handles the input context before output begins. It can perform large, relatively parallel matrix operations. Decode then generates output one token at a time while updating the key-value cache. A benchmark that reports prompt-processing speed is answering a different question from one that reports decode tok/s. A long prompt can increase TTFT even if the later decode rate is unchanged, while a long requested answer makes sustained decode performance more visible.
Why TOPS does not convert to tokens per second
LLM decoding is a chain of operations with data dependencies, memory traffic and control work. TOPS describes arithmetic capacity; token generation describes the end-to-end result of running a model. The path between them contains several bottlenecks.
Free tools Windows power users keep installed
One-click scans. No signup required.
Memory bandwidth and model size
During autoregressive decoding, each new token uses the model’s weights and the growing key-value (KV) cache. Qualcomm identifies memory bandwidth as a common limit for token generation. A model with more parameters generally requires more data to be read, while a longer context increases KV-cache traffic and memory consumption. If the arithmetic units are waiting for weights or cache data, additional TOPS cannot be used.
Capacity matters as well as bandwidth. If the model and its KV cache fit comfortably in available memory, the runtime can keep data close to the accelerator. If they do not fit, the runtime may offload layers, move data between processors or fail to load the model. Those transfers can dominate latency. Shared system memory, dedicated graphics memory and their respective bandwidths should therefore be identified in a device comparison.
Precision and quantization
Quantizing a model from a higher-precision format to INT8, INT4 or another supported format can reduce memory use and increase throughput, but it may alter quality and operator compatibility. The accelerator’s advertised TOPS may be measured at INT8 while the chosen model performs some layers at FP16 or on a different processor. Those mixed paths make a simple TOPS-to-tok/s calculation invalid.
“Four-bit” is not a complete description by itself. A report should identify the quantization method, weight format and accumulation format where known. Two files labeled with the same bit width can have different memory layouts or runtime support, and a model can spend enough work in higher precision that the advertised low-bit peak is not representative.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallPrompt length, context and batch size
Long prompts require more prefill work before decoding begins. A large context window can increase KV-cache size and memory traffic. Batch size changes how efficiently matrix operations are filled: throughput for many simultaneous requests may rise even while the latency of one interactive request changes. Always state whether a result covers prompt processing, output generation or both.
Concurrency also changes what “tokens per second” means. A single-user test might report the rate for one stream. A server test may report aggregate tokens across several requests. Those values should not be compared without stating the number of active requests, scheduling policy and whether the figure is per request or system-wide.
Software and system integration
The model runtime, compiler, kernel library, driver and operating-system API determine which operations run on the NPU, GPU or CPU. Unsupported layers can force transfers between processors. Graph partitioning, kernel fusion, caching, thread scheduling and synchronization all affect the measured rate. Two devices with similar hardware specifications can therefore produce different results with the same model.
Rank #3
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
Software versions are part of the result, not a footnote. A backend may gain an optimized kernel in a later release, or a driver update may change which operators are assigned to the NPU. Repeating a test after an update requires recording the new runtime and driver rather than treating the number as a permanent hardware property.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →How to read headline TOPS figures
| Headline figure | What it establishes | What it does not establish |
|---|---|---|
| More than 40 TOPS | Microsoft’s Copilot+ PC category requires an NPU capability above 40 TOPS. | It does not state the tokens per second, model compatibility or local-LLM experience of a qualifying PC. |
| Up to 45 TOPS | Qualcomm’s 2024 Snapdragon X Series claim for peak NPU capability. | It is not a direct measurement of any LLM’s generation speed and is subject to model, precision and software conditions. |
Microsoft’s Copilot+ threshold is a platform eligibility requirement, not a benchmark result. Microsoft also notes that many NPUs support integer math in lower-bit formats such as INT8 for performance and power efficiency. The threshold tells you that the hardware belongs to a capability class; it does not promise that every AI application will use the NPU or that a particular model will run locally.
The wording “more than 40” is also different from a measured application score. It identifies a capability threshold for a product category. A machine just above the threshold and one far above it can still have different memory systems, cooling, software support and model compatibility. Check the current device documentation rather than assuming that category membership supplies those details.
How to compare two AI devices fairly
Use TOPS as one architectural clue, then compare measured workload results under matching conditions. A valid comparison should disclose all of the following:
- Model identity: exact model name, release or model version and tokenizer.
- Quantization and precision: for example, INT4 weights with FP16 accumulation, not merely “4-bit.”
- Prompt and context: prompt text or token count, maximum context setting and whether a warm KV cache was used.
- Batch size and concurrency: one interactive stream is not equivalent to a batch throughput test.
- Runtime and version: the inference engine, accelerator backend, driver and operating-system build.
- Device configuration: system memory, shared versus dedicated memory, cooling profile and connected power supply.
- Power mode and sustained behavior: battery, balanced or performance mode, plus test duration if throttling is possible.
- Measurement method: TTFT, output tok/s, total latency and the number of tokens or seconds included.
- Memory specifications: capacity and bandwidth, especially for models that approach the available memory limit.
- Energy information: energy per request or sustained power when battery life, fan noise or thermals matter.
There is no single universal test protocol established by the sources cited here. Treat results as comparable only when these variables are aligned or explicitly adjusted.
A practical local-LLM report template
For a reproducible result, record:
- The exact model file, tokenizer and quantization.
- Prompt token count, requested output length and context-window setting.
- Whether the reported rate covers prefill, decode or decode only.
- Runtime name and version, backend, driver and operating-system version.
- CPU, GPU and NPU model, system memory and memory mode.
- Power source and performance setting, with temperature or duration if sustained behavior is relevant.
- TTFT, output tokens per second, total elapsed time and the measurement window.
This checklist is reporting guidance rather than an industry standard, but without it another reader cannot reproduce or meaningfully compare the number.
A simple comparison workflow
- Choose the workload first. Select the model, quantization, prompt length and output length that represent your real use. Do not choose a benchmark solely because it flatters one device.
- Confirm support. Verify that the runtime can execute the model’s operators on the intended NPU, GPU or CPU path. A nominally compatible device may still offload unsupported layers.
- Warm and cold test separately. Measure startup and TTFT when the model is unloaded, then measure a warmed run. Label which result is which.
- Measure both phases. Record prompt-processing behavior, TTFT and decode tok/s instead of collapsing them into one average.
- Repeat long enough to expose throttling. A short burst can reflect peak clocks. A longer session shows whether temperature, battery state or power limits reduce the rate.
- Report uncertainty. Include repeated-run variation, rounding and any failed or partially offloaded configurations. A single best-case number is not a complete performance picture.
Worked example: why a two-times-TOPS claim can disappoint
Imagine two NPUs advertised at 20 and 40 TOPS. The 40-TOPS part may have twice the ideal INT8 arithmetic throughput, yet an LLM test could show a smaller advantage or no advantage at all. If both systems are waiting on the same memory bandwidth, the extra arithmetic lanes remain idle. If the runtime supports only part of the model on the NPU, CPU or GPU work can dominate. A longer context, different quantization or a lower power mode can erase the theoretical gap.
The reverse can also happen: a device with a lower TOPS rating may deliver a higher tok/s result because it has faster memory, a better-optimized runtime, more effective operator support or a model that fits its cache and memory layout. The result is not contradictory; the two metrics answer different questions.
When TOPS is useful
- Platform screening: It can identify whether a device meets a stated NPU class, such as the more-than-40-TOPS Copilot+ requirement.
- Architecture context: It gives a rough ceiling for arithmetic throughput when precision and counting assumptions match.
- Workload planning: It helps engineers estimate whether an accelerator has enough compute for a known operator mix, alongside bandwidth and memory capacity.
- Power comparisons: TOPS per watt can be informative when measured with the same precision and methodology, though it still does not equal application performance per watt.
When tokens per second is the better metric
- Interactive chat: TTFT and decode tok/s describe how quickly a user sees and reads an answer.
- Local-model selection: The rate for your chosen model and quantization matters more than a generic accelerator label.
- Application capacity: For a service handling concurrent requests, report per-request latency and aggregate throughput at the target batch size.
- Regression testing: Tok/s and TTFT can reveal whether a runtime, driver or model change improved the actual workload.
Common mistakes and how to avoid them
Assuming a linear conversion
There is no reliable formula that turns a vendor’s TOPS number into LLM tok/s. Publish measured tok/s for the exact workload instead.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Clear out junk files and repair common Windows errors3Fix the driver behind crashes, sound loss and screen glitchesRank #4
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
Mixing precision or sparsity claims
Do not compare INT8 dense TOPS with FP16 or sparse TOPS without explaining the assumptions. The arithmetic count and usable model path are different.
Reporting a peak number as sustained performance
“Up to” indicates a theoretical or best-case ceiling. State whether clocks, thermals and power mode were controlled, and include a sustained test when the device will be used for long sessions.
Ignoring memory limits
A model that does not fit in available memory may swap, offload layers or fail to load. Check model size, quantization, KV-cache growth and memory bandwidth before interpreting TOPS.
Comparing unlike benchmark setups
A short prompt with a warm model, one user and decode-only timing is not equivalent to a cold-start test with a long prompt and prompt processing included. Align the setup or label the differences.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Confusing category eligibility with capability
A laptop can meet a platform requirement and still lack the memory, runtime support or thermal headroom for the model you want. Treat a threshold as a filter for candidates, then verify the complete workload.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Choosing an AI PC for on-device LLMs
Start with the models and experience you need, not the largest TOPS label. Confirm that the runtime supports the device’s NPU backend and your model’s operators. Then verify memory capacity, memory bandwidth, cooling and power behavior. Look for independent measurements that report the same model, quantization, context and timing definitions you care about.
An NPU above 40 TOPS may satisfy the Copilot+ PC hardware category, but that fact alone does not prove that a chosen LLM runs locally, fits in memory or reaches a useful interactive rate. Check current device-specific support and model compatibility before buying, because platform requirements and software support can change.
For mobile or laptop use, consider the trade-off between peak speed and sustained, quiet operation. A performance mode may improve decode tok/s while using more power and producing more fan noise. Battery operation can impose a different power policy from AC operation. If portability matters, compare energy per request and sustained behavior, not only the first few seconds of a benchmark.
Recommended Free Tools
Bottom line: use both metrics, for different decisions
TOPS answers, “How much peak arithmetic throughput does this accelerator advertise under these precision and counting assumptions?” Tokens per second answers, “How quickly did this model generate output in this complete software and hardware setup?” Use TOPS to understand a platform’s theoretical capability and eligibility class. Use TTFT, output tok/s, memory data and a fully described benchmark to judge the LLM experience you will actually get.
Best Value
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
Frequently Asked Questions
Can I estimate LLM tokens per second from an NPU’s TOPS rating?
No reliable direct conversion exists. Token generation also depends on memory bandwidth, model parameters, context length, quantization, batch size, runtime, processor placement and power behavior.
Does a Copilot+ PC guarantee fast local LLM performance?
No. The more-than-40-TOPS NPU requirement defines a Windows hardware category. It does not specify a model, runtime, memory configuration or tokens-per-second result.
What should a benchmark report besides tokens per second?
Report time to first token separately, along with model and quantization, prompt and context length, batch size, runtime and version, device configuration, power mode, memory details and whether prompt processing is included.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Is a token the same as a word?
No. A token can be a whole word, a fragment, punctuation or a short character sequence. The relationship varies by tokenizer, language and content, so tok/s should not be converted directly into words per second.
Why can a lower-TOPS device be faster in an LLM test?
It may have greater memory bandwidth, better operator support, a more optimized runtime, more effective processor placement or a model that fits its memory layout better. TOPS measures peak arithmetic throughput, not complete application speed.
Should prompt processing and output generation be reported separately?
Yes. Prompt or prefill work affects time to first token, while decode tok/s describes ongoing generation. Combining them can hide whether a system starts slowly, streams slowly or behaves differently under long contexts.
Does INT8 TOPS predict performance for an INT4 model?
Not directly. The advertised figure may use a different precision, and the runtime may use FP16 or another format for some layers. State the model’s quantization, accumulation format and actual execution path.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →What does “up to” mean in a TOPS specification?
It generally describes a theoretical or best-case peak under stated precision and counting assumptions. It is not a sustained application benchmark and does not promise a particular LLM generation rate.
The Bottom Line
TOPS is a peak accelerator specification; tokens per second is an observed model result. Read the precision, counting convention, memory system and software path behind any TOPS claim, then judge an LLM device with TTFT and decode tok/s measured on the exact model and conditions you care about.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
