iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
For the Q4_K_M GGUF and 32K context in the published examples, start with --n-cpu-moe 22 on a 12 GB GPU, 13 on a 16 GB GPU, or 0 on a 24 GB GPU. These are configuration-specific starting points, not universal settings: the GGUF quantization, context length, KV-cache type, runtime build, and memory used by other settings can change what fits. Use the value as a starting point, then verify it on your machine.
What --n-cpu-moe changes
In the behavior described by the configuration article, --n-cpu-moe N keeps expert feed-forward tensors for the first N model layers in system RAM and runs those experts on the CPU. The remaining expert tensors are GPU-resident; attention, shared weights, and the KV cache remain on the GPU. This shifts some expert-weight memory away from VRAM while adding CPU work when those experts are used. Check the help output and model-load log for your particular llama.cpp build before relying on those implementation details; builds and forks can differ. The article’s author, Donald Lee, describes the approach as something that can be computed from the GGUF rather than guessed through repeated restarts.
Starting values for the published Q4_K_M examples
The following values and approximate decode rates are reported for particular configurations in the 2026 DEV Community article. They are not a controlled comparison across GPU models: hardware, quantization, runtime, and other settings vary, so the rates should not be used to rank GPU tiers or predict another system’s speed. The article names an RTX 3060 for the 12 GB / 32K case, an RTX 5060 Ti for the 16 GB / 32K case, and an RTX 4090 for the 24 GB / 32K case.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →| GPU memory and named GPU | Context | Starting --n-cpu-moe |
Reported VRAM | Source-reported decode rate |
|---|---|---|---|---|
| 12 GB, RTX 3060 | 32K | 22 | 11.8 GiB | 24–41 tok/s |
| 12 GB | 128K | 26 | 11.8 GiB | 16–27 tok/s |
| 16 GB, RTX 5060 Ti | 32K | 13 | 15.8 GiB | 31–53 tok/s |
| 16 GB | 128K | 17 | 15.9 GiB | 21–35 tok/s |
| 24 GB, RTX 4090 | 32K | 0 | 21.7 GiB | 81–142 tok/s |
These examples use the article’s named unsloth/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf. Its figures are configuration results rather than guarantees. The article also lists differing memory bandwidths for the example GPUs (360 GB/s, 448 GB/s, and the RTX 4090 configuration), among other system-specific factors.
#1 Best Overall
- Axial-tech fans now feature a smaller fan hub that facilitates longer blades and a barrier ring that increases downward air pressure
- 2.5-slot design allows for greater build compatibility while maintaining cooling performance
- 0dB technology lets you enjoy light gaming in relative silence
- Dual BIOS switch lets you toggle between Quiet and Performance BIOS profiles
- Dual ball fan bearings last up to twice as long as sleeve bearing designs
Why context length changes the value
The KV cache grows with the context available to the model and consumes GPU memory alongside GPU-resident weights and runtime buffers. In the article’s stated model configuration, its FP16 KV-cache estimate is 20,480 bytes per token: it counts K and V across 10 full-attention layers with two KV heads, dimension 256, and two bytes per value. That estimate is specific to the described configuration and cache type.
In the same Q4_K_M examples, moving from 32K to 128K raises the suggested value from 22 to 26 on 12 GB and from 13 to 17 on 16 GB. A separate 2026 community guide for a different APEX/abliterated GGUF and Windows-native ik_llama.cpp b5095 build on an RTX 4070 SUPER reports preferred values of 16 at 32K, 17 at 64K, 20 at 128K, and 22 at 258K. That guide corroborates the direction of the context effect, not the exact values for the Q4_K_M file above.
Rank #2
- Powered by the NVIDIA Blackwell architecture and DLSS 4
- Powered by GeForce RTX 5070 Ti
- Integrated with 16GB GDDR7 256bit memory interface
- PCIe 5.0
- WINDFORCE cooling system
How to tune the setting on your system
- Identify the exact model file. Record the GGUF name, quantization, and file size. A value for one quant or model variant is not automatically valid for another.
- Set the workload you actually need. Choose the target context, KV-cache type, batch and micro-batch sizes, and number of parallel request slots. Context is divided among parallel slots, while batch settings and slots affect memory headroom. The llama.cpp tuning tutorial discusses these configuration factors.
- Choose the closest example as a starting point. Use the table only when its model file and context are reasonably close to yours. If your target context is longer, expect to budget more memory for KV cache; do not assume the example’s N will still fit.
- Load and inspect actual memory use. Check the runtime’s load log and GPU memory use with your own workload. Increase N in small steps around the lowest value that loads reliably with headroom. More CPU offload can reduce VRAM pressure, but it also moves expert work to the CPU.
- Measure prompt processing and generation separately. Long prompt prefill and token generation need not respond the same way to a setting change. Keep the prompt length and workload consistent when comparing runs.
- Record the environment with every result. Note llama.cpp version or fork, backend/build, GPU, CPU, system RAM, exact quant, context, KV type, batch sizes, parallel slots, N, and measured prefill and decode rates. A result from Windows-native
ik_llama.cppshould not be presented as a result from a different build.
What other reported results can—and cannot—tell you
A 12 GB guide illustrates workload trade-offs
The separate 2026 community guide reports 64.0 tok/s at 32K with --n-cpu-moe 16, 60.7 tok/s at 64K with 17, 55.3 tok/s at 128K with 20, and 50.9 tok/s at 258K with 22. These are author-reported results for that guide’s APEX/abliterated GGUF, Windows-native ik_llama.cpp b5095 build, RTX 4070 SUPER, i5-14600KF, and 32 GB DDR4—not a benchmark of the Q4_K_M examples in the table.
KV-cache choice can affect prompt processing
In that same guide’s stated profile, a 45K-token input took 397 seconds with q8 KV and 85.5 seconds with q4_0 KV. The author also reports an overall “KL 0.10” for the guide’s setup; that figure should not be treated as a general quality result for other models, prompts, or cache configurations. The comparison is a reason to measure the cache choice under your own quality and performance requirements, not a promise of the same speedup.
Rank #3
- Powered by the NVIDIA Blackwell architecture and DLSS 4
- Powered by GeForce RTX 5060
- Integrated with 8GB GDDR7 128bit memory interface
- PCIe 5.0
- WINDFORCE cooling system
A 24 GB full-context recipe is not a universal fit guarantee
Patrick Gawron reports a working recipe at 262,144-token context using IQ4_XS weights, f16 KV, and all layers on GPU without CPU expert offload. The article does not provide measured exact VRAM use or tokens per second. It establishes that --n-cpu-moe 0 can work in at least one 24 GB setup with that recipe; it does not establish that every 24 GB card, quantization, batch size, or parallel configuration will fit.
How to calculate a starting point from a GGUF
The DEV Community article’s method estimates memory by separating non-expert tensors from per-layer expert tensors, then accounting for GPU-resident weights, KV cache, and runtime overhead. For its named Q4_K_M GGUF, the author reports 22,123,538,944 tensor bytes in total, about 486,539,264 bytes of expert tensors per layer across most of 40 layers, and 2,555,013,632 bytes for the other tensors. These are file-specific measurements, not constants for Qwen3.6-35B-A3B or other quantizations.
Rank #4
- Powered by Radeon RX 9070 XT
- WINDFORCE Cooling System
- Hawk Fan
- Server-grade Thermal Conductive Gel
- RGB Lighting
Its fit calculation subtracts the input embedding it says remains CPU-resident from non-expert weights, adds the experts left on the GPU, the KV cache for the requested context, and roughly 1 GiB for CUDA context and compute buffers. Expert tensor sizes vary slightly by quant and layer, so the article advises summing actual layer tensor sizes rather than multiplying a nominal per-layer figure blindly. Treat this as a practical estimate: the runtime’s actual allocation and load log are the final check.
Recommended Free Tools
When changing the GPU is not the first move
If the model loads with a higher N and meets your latency needs, tuning may be sufficient; an upgrade is not required simply because some experts run on the CPU. Consider a GPU upgrade only if your target context, throughput, or concurrent workload cannot be met with the VRAM and CPU capacity available. These examples support the general point that 12, 16, and 24 GB configurations can have different trade-offs, but they do not establish a current best-value card or a controlled performance-per-dollar comparison.
Quick Recap
Best Value
- Axial-tech fans now feature a smaller fan hub that facilitates longer blades and a barrier ring that increases downward air pressure
- Phase-change GPU thermal pad helps ensure optimal heat transfer, lowering GPU temperatures for enhanced performance and reliability
- 2.5-slot design allows for greater build compatibility while maintaining cooling performance
- Dual-ball fan bearings last up to twice as long as standard conventional sleeve bearings designs
- 0dB technology lets you enjoy light gaming in relative silence
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
