iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
SM120 has less shared memory than SM100, but that difference alone does not explain why vLLM crashes when FP8 KV cache is enabled. NVIDIA documents a lower per-block and per-SM shared-memory ceiling on compute capability 12.0; vLLM’s KV-cache dtype controls cache storage, not a particular kernel’s shared-memory use. The actual cause depends on the GPU, model, backend, software versions, and the first error in the logs.
How large is the SM120 shared-memory gap?
NVIDIA’s CUDA 13.0 Blackwell Tuning Guide reports these limits:
| # | Preview | Product | Price | |
|---|---|---|---|---|
| 1 |
|
ASUS Dual GeForce RTX 5060 Ti 16GB GDDR7 OC Edition Gaming Graphics Card | $777.19 | Buy on Amazon |
| 2 |
|
ASUS TUF Gaming GeForce RTX™ 5080 16GB GDDR7 OC Edition Graphics Card | $1,830.91 | Buy on Amazon |
| Compute capability | Shared memory capacity per SM | Maximum shared memory per thread block | Source |
|---|---|---|---|
| 12.0 (SM120) | 128 KB | 99 KB | NVIDIA, CUDA 13.0 Blackwell Tuning Guide |
| 10.0 (SM100) | 228 KB | 227 KB | NVIDIA, CUDA 13.0 Blackwell Tuning Guide |
These are two related but distinct limits. “Per SM” describes the shared-memory capacity available on a streaming multiprocessor; “per thread block” is the maximum a single block can use. A kernel that requests more shared memory per block than the applicable limit cannot run in that configuration, even if the GPU has unused device memory elsewhere.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
The guide describes shared memory as a fast, limited resource for kernel data. It also notes that data that cannot fit there must use global memory instead. It does not say that vLLM’s KV cache is stored in per-block shared memory, or that enabling FP8 KV cache necessarily makes a kernel exceed the SM120 limit.
#1 Best Overall
- AI Performance: 767 AI TOPS
- OC mode: 2632 MHz (OC mode)/ 2602 MHz (Default mode)
- Powered by the NVIDIA Blackwell architecture and DLSS 4
- Axial-tech fan design features a smaller fan hub that facilitates longer blades and a barrier ring that increases downward air pressure
- A 2.5-slot design maximizes compatibility and cooling efficiency for superior performance in small chassis
What does the FP8 KV-cache setting change?
In vLLM, --kv-cache-dtype sets the data type used to store the KV cache. The serve CLI documentation lists FP8 choices including fp8, fp8_e4m3, and fp8_e5m2. That identifies the requested cache representation; it does not, by itself, identify which attention kernel runs or prove that shared memory is exhausted.
FP8 can also refer to quantized model weights or activations, which is a different part of the execution path. When reporting a failure, specify whether FP8 is enabled for the KV cache, the model’s weights or activations, or more than one of these. A failure involving FP8 model-weight dispatch is not evidence that FP8 KV storage caused it.
Device-memory budgeting is also separate from the shared-memory limits above. vLLM documents --gpu-memory-utilization and --kv-cache-memory-bytes as controls related to GPU memory allocation and cache sizing. A device-memory allocation failure should not be treated as proof of a per-block shared-memory failure.
Which failure does the error actually point to?
“vLLM crashed” is not a diagnosis. The failure stage and first error help distinguish allocation problems from kernel, backend, or stream-capture failures. The reports below have different scopes and should not be generalized into one SM120-wide FP8 compatibility verdict.
Rank #2
- Powered by the NVIDIA Blackwell architecture and DLSS 4. System Requirements: Minimum 850W PSU with 16-pin 12V-2x6 (12VHPWR) connector required. Verify before purchasing.
- Military-grade components deliver rock-solid power and longer lifespan for ultimate durability. Compatibility: 348mm (13.7") length, 3.6 slots, 4.3 lbs. Confirm case clearance and slot spacing. GPU bracket included.
- Protective PCB coating helps protect against short circuits caused by moisture, dust, or debris
- 3.6-slot design with massive fin array optimized for airflow from three Axial-tech fans
- Phase-change GPU thermal pad helps ensure optimal thermal performance and longevity, outlasting traditional thermal paste for graphics cards under heavy loads
| Reported context | What was reported | How to interpret it |
|---|---|---|
| DGX Spark, FP8 workload | NVIDIA’s vLLM Release 26.02 notes, last updated September 29, 2026, describe CUDA stream-capture errors tied to illegal synchronization operations in FlashInfer kernels; the notes say a FlashInfer fix is available. | A stream-capture/backend issue in the documented platform context—not proof of SM120 shared-memory exhaustion on all workstation GPUs. |
| Nemotron Super V3 FP8 attention | The same NVIDIA release notes specify the Triton attention backend for this model’s FP8 operation. | A model-specific backend requirement, not a universal setting for other models. |
| RTX PRO 6000 Blackwell under WSL2, hybrid Mamba KV allocation | The author of a vLLM issue reports a large contiguous Mamba-state allocation failure and says switching to FP8 KV did not help. The author suggests hidden WSL2 memory overhead as a possible cause. | An issue report about allocation, with the WSL2 explanation presented as the submitter’s hypothesis—not a project-confirmed root cause or a shared-memory diagnosis. |
| SM120, FP8 block-scaled DeepSeek model | The author of a separate vLLM issue reports a Cutlass dispatch_scaled_mm failure during profiling. |
A model-weight kernel-dispatch report; it should not be labeled an FP8 KV-cache failure. |
| RTX PRO 4000 SM120, low-batch workload | A community field report dated April 3, 2026 describes tests using vLLM 0.15.1, 0.17.1, and 0.18.1 with CUDA 13.0, and recommends --kv-cache-dtype auto for that workload because of reported FP8 dequantization overhead. |
A single workload-specific performance report, not a general FP8 compatibility rule or evidence that auto is always faster. |
These examples show why the exact traceback matters: allocation failure, a kernel dispatch error, an illegal synchronization operation during graph capture, and a shared-memory resource error are different symptoms. An FP8 setting can be present in several of those scenarios without being the underlying cause.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.How should you diagnose an FP8-related vLLM crash?
- Identify the failure stage. Record whether it occurs during model loading, warmup or profiling, graph capture, or request execution. Preserve the first error and full traceback rather than reporting only the final process exit.
- Record the full software and hardware context. Include the exact GPU SKU and compute capability, vLLM version or image tag, CUDA toolkit/runtime, driver, PyTorch, and FlashInfer versions.
- Disambiguate FP8 use. State the model name and whether FP8 applies to the KV cache, quantized weights or activations, or multiple components. Include the attention backend and relevant launch flags, including cache dtype and memory-utilization or cache-sizing settings.
- Change one variable at a time. If testing a different attention backend or KV-cache dtype, keep the other settings fixed and note whether the same failure changes. This can help distinguish a backend or kernel path from a cache-allocation problem.
- Separate memory-budget tests from kernel tests. For an error that specifically indicates a device or shared/unified-memory OOM, investigate the relevant memory budget; NVIDIA’s release notes advise reducing utilization in shared/unified-memory OOM contexts. For a stream-capture or kernel-dispatch error, investigate the backend, model path, and applicable release guidance instead of treating a cache-size adjustment as a confirmed fix.
Do not infer a universal workaround from the SM120 ceiling. A useful report includes startup logs, the first error, the traceback, all relevant flags, and the exact versions; if you test alternatives, state which single setting changed and what happened.
What can—and cannot—be concluded about SM120 and FP8 KV cache?
The documented architecture difference is real: CUDA 13.0 lists lower shared-memory limits for compute capability 12.0 than for 10.0. That constraint can matter to kernels selected or compiled for SM120. The available documentation and issue reports do not establish that FP8 KV cache itself exceeds the per-block limit, nor do they identify one root cause for every workstation Blackwell crash. A specific diagnosis requires the actual error and the GPU, model, backend, and software-version context.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
