iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Yes, one RTX 4090 can be the GPU in a hybrid llama.cpp setup for a model with more than 100 billion total parameters—but the model does not thereby fit in the card’s VRAM. The full quantized weight set must still be accessible through host memory or another supported storage-backed loading path, and the machine also needs room for the KV cache, runtime buffers, and the operating system. llama.cpp provides controls for placing MoE weights on the CPU and limiting how many layers are stored in VRAM; those controls do not mean the server automatically selects individual experts for the GPU.
What “100B+” means for an MoE model
An MoE model’s total parameter count and its active parameters per token answer different questions. Total parameters describe the model’s complete weight set. Active parameters describe the portion used during a particular forward pass. The active count can help explain computation, but it does not eliminate the need to make the other weights available when they may be selected.
| # | Preview | Product | Price | |
|---|---|---|---|---|
| 1 |
|
VIPERA NVIDIA GeForce RTX 4090 Founders Edition Graphic Card | $4,425.00 | Buy on Amazon |
| 2 |
|
ASUS TUF Gaming GeForce RTX™ 5080 16GB GDDR7 OC Edition Graphics Card | $1,830.91 | Buy on Amazon |
A January 2026 community guide describes DeepSeek V3 as having 671 billion total parameters and 37 billion active during each forward pass. That distinction helps explain why an MoE model may require less computation per token than a dense model with the same total parameter count. It does not mean a 24 GB GPU can hold the model’s complete weights.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →Model-file size is a separate constraint
As one concrete, non-universal example, bartowski’s mutable DeepSeek-V3-0324 GGUF repository listed its Q4_K_M file at 404.43 GB and identified the model as 671B parameters; the listing was accessed October 7, 2026. Other model variants and quantizations can have different file sizes. The listing does not establish an official recommended quantization or a precise RAM requirement.
#1 Best Overall
- 16,384 NVIDIA CUDA Cores
- Supports 4K 120Hz HDR, 8K 60Hz HDR and variable refresh rate as indicated in HDMI 2.1A
- New streaming multiprocessors: up to 2x power and power efficiency
- Fourth generation tensor cores: up to 2x AI power
- Third-generation RT cores: up to 2x ray tracing performance
Do not convert a model-file size directly into a minimum-RAM figure. Runtime allocations, context and KV-cache needs, the operating system, and implementation details affect whether a setup loads. Check the actual file you intend to use and leave additional capacity for runtime needs.
How llama.cpp controls CPU and GPU placement
The llama.cpp server README documents controls for keeping MoE weights in CPU memory and for limiting the maximum number of layers stored in VRAM. They describe placement by weight category or layer count—not a choice of individual experts based on which ones are popular.
| Option | Documented purpose | What it does not mean |
|---|---|---|
--cpu-moe or -cmoe |
Keep all Mixture of Experts weights in CPU memory. | It does not select a subset of individual experts to keep on the GPU. |
--n-cpu-moe N or -ncmoe N |
Keep MoE weights from the first N layers in CPU memory. |
N counts layers, not individual experts. |
--n-cpu-ffn N or -ncffn N |
Keep dense FFN weights from the first N layers in CPU memory. |
Dense FFN weights are not the same category as MoE expert weights. |
--n-gpu-layers N or -ngl N |
Set the maximum number of layers stored in VRAM. The documented accepted values include an exact count, auto, and all. |
It is not a control for choosing individual experts within an MoE layer. |
--fit and related fit options |
Adjust unset arguments to fit available device memory; the README describes a target margin and a minimum context-size setting. | It does not guarantee that a particular model, context, and runtime load will fit. |
These descriptions are version-sensitive. Consult the current llama.cpp server README for the build you use, and use its startup logs to verify actual placement and memory use. In particular, do not infer from the option names that the server automatically finds and retains the most frequently selected experts on the GPU; the documented controls above do not establish that behavior.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Repair Windows errors before they cause bigger problems3Scan for outdated or missing drivers - takes under a minutePlan the machine around the full workload
A 4090 is the GPU component, not a substitute for the rest of the system. Its VRAM must accommodate whatever layers and weights are placed there as well as context-related and compute allocations. The complete model weights must also be accessible from host memory or a supported storage-backed loading path. The CPU and memory subsystem matter because weights placed on the CPU must be served from there during inference.
- Model and quant: Record the exact repository, file, and quantization. Compare the actual file footprint before downloading; a smaller file does not, by itself, establish equivalent model quality.
- Host memory: Budget for the weights and leave room for the operating system and runtime overhead. The available sources do not support a universal RAM minimum.
- VRAM: Leave room for the context/KV cache and compute buffers rather than allocating all available VRAM to layers.
- Context and workload: Decide the intended context length and batch settings before tuning placement. Memory use and throughput can change with those choices.
- Software stack: Record the llama.cpp revision, build backend, and relevant driver/runtime conditions. Option behavior and results can vary by build.
For example, the 404.43 GB Q4_K_M listing above is a warning about scale, not a promise that a machine with exactly that much RAM will run the model. Compare the file against your actual memory and runtime budget before committing to a download or launch.
Choose a placement strategy, then verify it
There is no universal command that fits every MoE architecture and machine. Start with the placement you intend to test, change one constraint at a time, and confirm the result from the server’s startup output.
Rank #2
- Powered by the NVIDIA Blackwell architecture and DLSS 4. System Requirements: Minimum 850W PSU with 16-pin 12V-2x6 (12VHPWR) connector required. Verify before purchasing.
- Military-grade components deliver rock-solid power and longer lifespan for ultimate durability. Compatibility: 348mm (13.7") length, 3.6 slots, 4.3 lbs. Confirm case clearance and slot spacing. GPU bracket included.
- Protective PCB coating helps protect against short circuits caused by moisture, dust, or debris
- 3.6-slot design with massive fin array optimized for airflow from three Axial-tech fans
- Phase-change GPU thermal pad helps ensure optimal thermal performance and longevity, outlasting traditional thermal paste for graphics cards under heavy loads
- Confirm the exact model and file. Note its architecture, quantization, and file size. Make sure the complete weight set is accessible to the machine.
- Check current option syntax. Read the server README for the llama.cpp revision you built. The options above are documented controls, but README descriptions are mutable.
- Decide which MoE weights belong on the CPU. Use
--cpu-moefor all MoE weights or--n-cpu-moe Nto keep MoE weights from the firstNlayers on the CPU. These are layer-based controls, not individual-expert selectors. - Set the VRAM layer ceiling separately. Use
--n-gpu-layerswith the documented value form appropriate to your test. It controls the maximum number of layers stored in VRAM; it is distinct from the CPU MoE options. - Reserve memory for context and runtime. Specify or preserve a context setting that matches your intended use, and do not assume the largest possible GPU layer count is the best fit.
- Launch and inspect startup logs. Verify the reported placement and memory allocation for the actual model and build. If loading fails, reduce the GPU layer ceiling or adjust CPU placement, then retry while keeping context and other settings in view.
- Measure your own workload. Record prompt-processing and generation throughput separately, along with context length, batch/ubatch settings, quant, hardware, and llama.cpp revision.
--fit can help by adjusting unset arguments to available device memory, according to the README. Treat it as a fit aid rather than a guarantee: inspect what the server actually selected and whether the resulting context and placement meet your needs.
Recommended Free Tools
What to expect from CPU offloading
CPU placement makes a larger weight set usable in a hybrid configuration when the system has adequate capacity, but it does not make CPU-resident weights behave like VRAM-resident weights. Performance depends on the model’s architecture and routing, quantization, how much resides on each device, the CPU and memory subsystem, context/KV requirements, and the exact build. Prompt processing and token generation are different workloads, so measure them separately rather than relying on one throughput figure.
The cited sources do not establish standardized head-to-head performance for upstream llama.cpp across 100B+ models on one RTX 4090. Consequently, no single speed estimate or RAM recommendation is justified for all such setups. A result is useful only with its model file, quant, context, hardware, and software build attached.
Experimental expert caching is not the same as upstream offloading
A September 2, 2026 llama.cpp community RFC proposed a persistent expert-slot pool for MoE CPU offload under the name --moe-expert-cache and described a companion branch. Its author reported a test involving an RTX 4090 24 GB, Qwen3.8-Flash-Next, an 8K context, and greedy generation. The discussion is an experimental, branch-specific report—not evidence that the option is available in an upstream release or that the same result applies to another model, quantization, or machine.
The RFC says cache benefits depend on routing skew and working-set size; it also cautions that setting the cache size equal to top-k can be a poor choice. Treat this as a feature to evaluate only on the branch that implements it, and do not use its reported speedup as a general RTX 4090 performance promise.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Quick Recap
Decision checklist
- Can the system access the complete quantized weight set, not merely the active parameters per token?
- Is there enough host-memory capacity for weights plus operating-system and runtime needs?
- Does the VRAM plan leave room for the intended context/KV cache and compute buffers?
- Are the CPU MoE and GPU layer controls understood as separate placement settings?
- Have startup logs confirmed where weights landed on this model and build?
- Are performance measurements tied to the exact quant, context, workload, and software revision?
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
