Measure GPU utilization alongside inference throughput and latency—not as a standalone score. Establish a repeatable baseline with representative requests, collect several GPU activity and resource signals during the run, and then test one change at a time. Batching or a different serving scheduler may increase throughput, but a higher utilization reading is useful only if the service still meets its latency and accuracy requirements.
How do I measure GPU utilization for AI inference?
Start by defining what a successful inference run means for your service. Record the model and precision, GPU type and configuration, request mix, input and output lengths, concurrency, target throughput, and latency objective. There is no universal GPU-utilization target that applies to every inference workload; judge measurements against the service outcome you need.
- Run representative traffic. Use a repeatable test that reflects the requests and concurrency you expect in practice. Keep the workload consistent so changes between runs can be compared.
- Capture device conditions. Record the GPU identity and configuration. During the run, NVIDIA’s TensorRT benchmarking guidance demonstrates
nvidia-smi dmon -s pcuto monitor clocks, power, temperature, and utilization. Compare those readings with the serving system’s throughput and latency, rather than treating device telemetry as the whole result. See NVIDIA’s TensorRT performance benchmarking guidance. - Collect complementary activity signals. DCGM profiling exposes graphics/compute engine activity, SM activity, SM occupancy, Tensor Core activity, device-memory activity, and PCIe and NVLink traffic. These values are averaged over a sampling interval, not instantaneous readings. The DCGM feature overview describes the metrics and their interpretation.
- Match the sampling interval to the test. The Triton GenAI-Perf telemetry guide says DCGM Exporter’s 30-second default collection interval is too infrequent for detailed benchmarking. DCGM profiling documentation describes configurable intervals and a 1 Hz default in its feature overview; the fields and configuration available depend on hardware and software version. Avoid drawing conclusions about short benchmark phases from samples that are too far apart. See Triton GenAI-Perf’s GPU telemetry guide.
- Repeat the run under comparable conditions. Track clocks, power, and temperature alongside application results. Floating clocks and throttling can make runs less stable, so a change in throughput should not automatically be attributed to a software adjustment.
What do GPU utilization signals actually tell me?
| Signal | What it helps show | How to interpret it |
|---|---|---|
| General GPU utilization | Whether the device is reported as busy during sampled periods. | Useful as a broad indicator, but it does not identify which resource is active or whether the application is meeting its service objective. |
| SM activity | Activity on the streaming multiprocessors. | Active warps may be waiting on memory requests, so activity is not synonymous with productive computation. NVIDIA says a value of 0.8 or greater is necessary, but not sufficient, for effective GPU use; a value below 0.5 likely indicates ineffective use. This is metric guidance, not a universal service target. |
| SM occupancy | How many warps are resident on the SMs relative to the supported capacity. | It provides context for execution, but occupancy alone is not a score to maximize. |
| Tensor Core activity | Activity on Tensor Core execution resources. | Helps assess whether this kind of compute work is present; interpret it with model precision, throughput, and other signals. |
| Device-memory activity | Activity involving GPU memory. | Consider data movement or memory pressure when it is high, then verify the cause with application timings or a profiler. |
| PCIe and NVLink traffic | Interconnect activity between the GPU and other components or devices. | Can help identify significant data movement, but a counter alone does not prove that transfers are the bottleneck. |
These metrics are complementary. For example, high SM activity with weak throughput does not by itself prove the GPU is doing useful work; low device activity does not by itself explain whether the gap comes from request arrivals, preprocessing, or another host-side delay. Use the application’s own timing and, when needed, a developer profiler to investigate the cause. Continuous DCGM counters can help compare phases or replicas, but do not identify the source line, CUDA kernel, or instruction responsible for a reading. NVIDIA also advises coordinating access to hardware counters: pause DCGM collection while developer profiling tools need the same resources, then resume collection afterward.
Why is my GPU utilization low during inference?
A low reading is a clue to investigate, not a diagnosis. Use the pattern across device signals and application timing to choose what to check next.
#1 Best Overall
- Valued Carpenter Pencil Set: You will get 2 pcs solid carpenter pencils with 26 piece 2.8 mm refills, 1 replaceable sharpener, 1 plastic storage box.The complete carpenter pencils combination allows you to finish your work faster and more easily
- Deep Hole Marker Pencil: The deep-hole construction pencils adopts 45mm elongated tip design, which is more convenient to mark in the small hole or in other tight areas that other carpenter markers cannot reach
- Carpenter Pencils with Sharpener: The sharpener is screwed into the top of the work pencil, which won't get lost either. Built-in pencil sharpener that keep the lead with pointed and smooth to Improves line of sight in fine work
- Stronger Solid Lead: This work pencil is matched with a 2.8 mm thick lead , which is much thicker and stronger during the drawing process of construction work, it will not break or damage easily
- Marks on Various Surfaces: 3 colors solid construction pencil can marks on various surfaces,such as metal, plastic, wood, paper etc. Ideals for woodworkers, contractors, craftsmen, builders, merchants and masons
| Observed pattern | What to investigate |
|---|---|
| Low device activity during periods when requests are expected | Check request arrival patterns, concurrency, batching, preprocessing, and gaps in host-side work. Confirm with serving timings rather than assuming any one of these is the cause. |
| Activity rises and falls with uneven request arrivals | Test whether dynamic or opportunistic batching can combine independently arriving requests. Measure any added waiting time against the latency objective. |
| High memory activity or substantial interconnect traffic | Investigate memory pressure and data movement. Corroborate the suspected bottleneck with a profiler and application timing. |
| SM activity is high but throughput is disappointing | Do not infer productive computation from SM activity alone: active warps can be waiting on memory requests. Compare the other resource signals and profile if the counters do not explain the result. |
| Results vary considerably between otherwise similar runs | Compare clocks, power, temperature, and signs of throttling. Recheck that workload, concurrency, and sampling interval were comparable. |
How can I increase GPU utilization without increasing latency?
There is no single setting that improves every model and traffic pattern. Treat each option below as a benchmark candidate: hold the workload steady, change one factor, and compare throughput, latency—including tail latency when available—GPU activity, memory or KV-cache pressure, and run-to-run stability.
Test batch sizes against the latency budget
Batching can expose more parallel work and improve throughput, but larger batches are not automatically faster. Test candidate batch sizes with your actual model and traffic. For independent incoming requests, compare dynamic or opportunistic batching as well; waiting to form a batch can add latency. NVIDIA also notes that on Ada Lovelace GPUs or later, smaller batches can improve throughput when they help inputs and outputs fit in L2 cache. The useful batch size is therefore hardware- and workload-dependent. See NVIDIA’s TensorRT performance optimization guidance.
Rank #2
- Ergonomically Designed: Work in tight areas with a compact design that gets into tough spots
- Compact and Lightweight: Both tools are designed to fit into difficult to reach spaces. The 1/4" impact driver has a length of 5.55 in. and weighs just 2.8 lbs, while the 1/2" drill/driver measures only 7.5 in. and weighs 3.6 lbs
- Both the DEWALT impact driver and electric drill driver feature integrated LED work lights with a convenient 20-second delay, ensuring enhanced visibility in dimly lit or challenging work areas
- One-Handed Loading - Keep one hand free with a 1/4 in. hex chuck that accepts 1 in. bit tips
- Power drill cordless with 1/2" single sleeve ratcheting chuck provides tight bit gripping strength, making bit changes faster and more secure
Compare TensorRT-LLM scheduler policies
For the Triton TensorRT-LLM backend, compare max_utilization and guaranteed_no_evict under the traffic and KV-cache constraints you actually serve. max_utilization greedily packs requests to pursue throughput, but reaching KV-cache limits can lead to pause/resume overhead. guaranteed_no_evict prioritizes ensuring that a started request is not paused. Check throughput and latency together rather than selecting a policy from its name. Details are in the Triton TensorRT-LLM backend documentation.
Benchmark engine and execution settings
NVIDIA’s TensorRT optimization guidance covers CUDA graphs, multi-streaming, layer fusion, and targeting Tensor Cores. Test relevant settings against a baseline instead of assuming they will help your workload. If a setting changes precision or numerical behavior, verify accuracy alongside throughput and latency.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchesRank #3
- 【Great Compatibility】This Katerk 1/4 inch hex shank bit holder is specifically designed for 1/4 inch hex shank drill bits. It's compatible with most 1/4 fast hex handles, hex sockets, various electric screwdrivers, and handheld screwdrivers. The bit holder makes it a valuable addition for any handyman.
- 【Secure and Safe】Built with a secure backup nut design, each drill bit holder securely locks onto your bits, ensuring they stay firmly in place. Additionally, our bit holder incorporates a high-quality steel ball rolling design that holds up to several kilograms of weight, ensuring your various drill bits don't fall off.
- 【Easy One-Handed Operation】The bit holder for impact driver allows you to change bits single-handedly, simplifying your workflow. Its multi-color design further allows for quick identification of the drill bit you need.
- 【Compact and Convenient】Thanks to its compact size, this 1/4 inch bit holder is easy to carry around. The bit holder allows for easy attachment to various tools, making this a convenient addition to your construction accessories. The Katerk bit holder is cast from high-quality alloy material, promising a long product lifespan. Despite its rugged strength, the bit holder remains lightweight, making it portable.
- 【Cool Christmas Gift For Men Stocking Stuffers】 This screwdriver bit holder, driver bit holder, impact bit holder, can be given as a gift to your loved one, especially for anyone involved in construction or electrical work. It's a must-have for stocking stuffers for men and women, tools gifts for dad, tech gadgets for men, gifts for dad, gifts for him, gifts for husband, gifts for boyfriend, cool gadgets for men, and cool gifts for dad.
How should I decide whether an optimization worked?
Compare each candidate with the baseline using the same request mix, input and output lengths, concurrency, and test procedure. Keep the result that improves the service outcome—not merely the utilization number.
- Did throughput improve, and did latency objectives—including tail latency where measured—remain satisfied?
- Which activity signals changed: SM, Tensor Core, memory, or interconnect activity?
- Did memory or KV-cache pressure change in a way that explains the result?
- Were clocks, power, temperature, and throttling comparable across runs?
- If counters still leave the cause unclear, do profiler traces and application timings support the suspected explanation?
A low utilization reading alone is not evidence that you need another or larger accelerator. First establish what limits the current workload and whether a tested change improves the required throughput and latency. The cited NVIDIA documentation is vendor guidance; it does not establish one best setting or utilization target for every inference stack.
Quick Recap
Best Value
- Milwaukee Ink all Fine Point Marker, Black, 4 Per Pack
- 4 per pack Features Clog Resistant Marker Tip Writes through Dusty, Wet and Oily Surfaces Durable Marker Tip for Writing on Concrete, OSB and Rough Surfaces
- Clog resistant tip writes on dusty, wet and oily surfaces and is optimized for rough surfaces such as OSB, cinderblock and concrete
- Hard hat clip- attaches for easy access
- Quick dry time with reduced smearing and marking
Rank #4
- Long Nib and Deep Hole Marker: Our mechanical carpenter pencil with 45mm nib is designed for easy marking of deep holes or narrow areas. These construction pencils are the great choice for woodworking tools, construction tools, carpenter tools, contractor tools, wood carpentry tools and architect tools
- Extra Refills in 2 Colors for Versatile Marking: The construction mechanical pencil comes with 12 extra 2.8mm refills, including 6 red and 6 black refills. The black refill is suitable for light surfaces, while the red wax is perfect for dark surfaces. Our carpenter mechanical pencil makes sure that you'll have an ample supply for extended use
- Built-in Sharpener: Our construction pencil comes with a built-in sharpener to ensure the mechanical pencil tip is always sharp and ready for use. Never buy an extra pencil sharpener again. A great tool for any woodworker pencil, contractor pencils. The refill can easily be extended or retracted with a simple click of the pencils mechanical, allowing you to work more efficiently and accurately
- Portable Clip Design: Our deep hole construction pencil features a portable clip design, easy to carry and attach to your pocket or tool box, so that you can keep the carpenter pencils mechanical close at hand, making it a convenient tool to have on the go. Great gifts choice for carpenters
- Stronger Pencil Lead: The black refills are made of lead, sturdy and smooth. The red refills are made of wax, clear and light. These marking pencils are much thicker and stronger than normal pencils during the marking process of construction work, suitable for various surfaces, such as glasses, metal, boards, floors, walls, furniture, etc. The written marks can be easily wiped with a wet paper towel when needed
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
