iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
You can run a generative AI model locally on an NVIDIA GPU with a CUDA-capable card, a compatible driver and runtime, and a model whose memory requirements fit your available VRAM. The lowest-friction route is Ollama; llama.cpp is the better choice when you need direct control over model files, GPU-layer offload, context size, or a local HTTP server.
The reliable sequence is: verify CUDA support, install the platform prerequisites, choose a runtime and quantized model, launch it, then watch the GPU during loading and generation. A successful installation alone does not prove that inference is using the GPU.
What local generative AI actually means
A local model runs on hardware you control rather than sending prompts to a hosted inference service. The model weights, prompt processing and generated tokens are handled by software on your computer. Your NVIDIA GPU accelerates the tensor operations, while system RAM and the CPU may still be used for model loading, storage, orchestration, or layers that do not fit in VRAM.
For this guide, “NVIDIA GPU” means a CUDA-capable device supported by NVIDIA’s CUDA GPU reference. Do not assume that every NVIDIA-branded card can use a given CUDA runtime: check the exact GPU family and compute capability first. CUDA capability, available VRAM, model quantization, context length, and workload concurrency determine what is practical.
#1 Best Overall
- AI Performance: 767 AI TOPS
- OC mode: 2632 MHz (OC mode)/ 2602 MHz (Default mode)
- Powered by the NVIDIA Blackwell architecture and DLSS 4
- Axial-tech fan design features a smaller fan hub that facilitates longer blades and a barrier ring that increases downward air pressure
- A 2.5-slot design maximizes compatibility and cooling efficiency for superior performance in small chassis
1. Check the GPU before installing anything
Identify the exact device
On Windows, open Device Manager and expand Display adapters, or open PowerShell and run:
nvidia-smiOn Linux, run the same command in a terminal. Record the model name, total VRAM, driver version, and whether the command can see the device. If nvidia-smi fails, fix the driver installation before troubleshooting Ollama, llama.cpp, or a model.
Confirm CUDA capability
Compare the exact GPU with NVIDIA’s CUDA GPU reference and its supported-device list. Compute capability is the hardware feature level that CUDA software targets. A card missing from the supported list should not be assumed to work through the CUDA path, even if another NVIDIA card from the same era does.
Use VRAM bands as planning guidance
NVIDIA’s current Run LLMs on Your NVIDIA Device playbook groups hardware into the following planning bands. These are example recommendation ranges, not universal minimums. Actual use changes with quantization, context length, multimodal projectors, batch size, number of simultaneous requests, and CPU offload.
| Available VRAM | Practical planning implication |
|---|---|
| 8–16 GB | Plan for compact or aggressively quantized models and modest context windows; leave headroom for the runtime and graphics system. |
| 16–24 GB | More medium-sized quantized models become practical, with greater room for context or a multimodal component. |
| 24–48 GB | Suitable for larger quantized models and longer contexts, subject to the model architecture and runtime overhead. |
| 48–128 GB | Allows substantially larger models or higher-concurrency local serving than a typical desktop card. |
| 128–256 GB | Appropriate for very large models, larger contexts, or serving several requests when the software supports it. |
| 256–384 GB | Provides headroom for exceptionally large model files and demanding serving configurations. |
| 384–500 GB | Targets the largest local deployments and multi-GPU memory pools. |
| 500 GB+ | Enterprise-scale local capacity; model fit still depends on quantization, context, batching, and implementation. |
Think of VRAM as a budget rather than a yes-or-no specification. A model file that appears smaller than your card’s capacity can still fail because inference needs memory for the key-value cache, temporary tensors, runtime allocations, a vision or multimodal projector, and the operating system’s graphics reservation.
2. Install the CUDA prerequisites
Windows
NVIDIA’s CUDA 13.4 Windows guidance lists four relevant prerequisites: a CUDA-capable NVIDIA GPU, a supported Windows release, a supported Visual Studio installation, and the CUDA Toolkit. Follow the guide’s compatibility table for your exact Windows and Visual Studio versions rather than assuming that the newest release is supported.
Beginning with CUDA 13.1, the NVIDIA driver is installed separately from the CUDA Toolkit. Install a driver that supports the toolkit and your GPU, reboot when requested, and then install the toolkit. Keeping those two components conceptually separate makes failures easier to diagnose: nvidia-smi tests the driver and device, while nvcc --version reports the compiler toolkit available on the system.
Rank #2
- Powered by the NVIDIA Blackwell architecture and DLSS 4
- Powered by GeForce RTX 5060
- Integrated with 8GB GDDR7 128bit memory interface
- PCIe 5.0
- WINDFORCE cooling system
- Install or update the NVIDIA display driver for the GPU.
- Reboot and run
nvidia-smi; confirm that the card and driver are shown without an error. - Install a Visual Studio version listed as supported by the CUDA 13.4 guide if you are compiling CUDA software.
- Install the CUDA Toolkit and open a new terminal so its paths are available.
- Run
nvcc --versionif you need to build a runtime; a prebuilt application may not require the compiler, but it still requires a compatible driver.
Linux
Use NVIDIA’s CUDA 13.4 Linux guide for the supported distribution, host-compiler versions, repository or package method, environment setup, and verification commands. Distribution packages and NVIDIA’s repository packages can differ, so do not mix installation instructions from unrelated CUDA releases.
- Confirm that your distribution and kernel are supported by the CUDA guide.
- Install the NVIDIA driver using the guide’s method, reboot, and verify the device with
nvidia-smi. - Install the supported host compiler and CUDA Toolkit.
- Apply the environment configuration shown for your package method, then start a new shell.
- Run the guide’s verification sample or commands before installing a model runtime.
If you only use a distributed Ollama package or a prebuilt llama.cpp binary, you may not need to compile CUDA code locally. You still need a working NVIDIA driver and a runtime build that includes NVIDIA support.
3. Pick a runtime that matches your goal
| Runtime or stack | Setup effort | Model format or integration | GPU control | Server/API mode | Best fit |
|---|---|---|---|---|---|
| Ollama | Lowest friction; managed installation and model library | Models distributed through its library and supported package formats | Mostly automatic; device selection can be controlled with CUDA_VISIBLE_DEVICES |
Local service and command-line interaction | Beginners who want to pull a model and start chatting quickly |
| llama.cpp | Moderate; choose a CUDA build and manage model files | Primarily GGUF files | Direct GPU-layer offload, context, batching, and memory settings | llama-server HTTP service or llama-cli |
Users who need transparent, fine-grained control |
| PyTorch | Higher; Python environment, framework, and model code | Model-specific checkpoints and Python libraries | Programmatic device placement and custom kernels | Implemented by your application | In-process development, experiments, and custom pipelines |
| TensorRT | Higher; conversion and NVIDIA-specific optimization work | Engine or converted model formats | Highly optimized NVIDIA deployment | Usually integrated into an application or serving stack | Performance-focused NVIDIA deployment |
| SGLang or vLLM | Higher; server configuration and larger-system assumptions | Framework-supported model checkpoints | Advanced scheduling and GPU memory management | Production-oriented API servers | Serving workloads with concurrency and operational controls |
| WebNN/WebGPU | Browser or web-application oriented | Web-compatible model paths | Abstracted through browser APIs | Web application rather than a native local service | Models embedded in web experiences |
NVIDIA describes Ollama as a simple local model service and llama.cpp as a lightweight C/C++ runtime that works across operating systems and hardware. The distinction matters: Ollama hides more implementation detail, while llama.cpp exposes the choices that affect memory use and GPU offload. The same NVIDIA comparison separates both from PyTorch for development, TensorRT for optimized NVIDIA deployment, and SGLang or vLLM for production serving.
4. Select a model and quantization that fit
Parameter count is only the first estimate
Larger parameter counts generally require more memory, but the model’s numerical precision changes the size substantially. Quantization stores weights with fewer bits, reducing memory at the cost of some accuracy or output quality. A quantized model can make a larger parameter count usable on a consumer GPU, while an unquantized version may not fit at all.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallOutdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchChoose quantization together with your VRAM band and expected context. Do not select the largest model that barely fits on paper. Leave capacity for the key-value cache and temporary buffers; otherwise a short prompt may work while a long conversation or second request triggers an out-of-memory error.
Account for context and concurrency
- Context length: A larger maximum context increases cache memory. Start with a moderate value and raise it only after a stable test.
- Batching: Processing several prompts or requests concurrently consumes additional working memory.
- Multimodal inputs: Vision-language models may require a separate projector file in addition to the main model weights.
- CPU offload: Moving some layers to system RAM can make an oversized model load, but it generally reduces throughput and increases latency.
- Runtime overhead: Different builds allocate buffers differently, so the same GGUF file can have slightly different peak requirements.
Use compatible files
For llama.cpp, download a GGUF model that matches the runtime and architecture. If the model is multimodal, obtain the matching projector file from the same model release and use the runtime’s projector option. A main model and projector from different releases may fail to load or produce unusable results.
5. Easiest path: run a model with Ollama
Ollama is the simplest managed route when you want a local service rather than manual control of every model file. Install the Windows or Linux package for your platform, then open a terminal.
Rank #3
- Powered by the NVIDIA Blackwell architecture and DLSS 4
- Powered by GeForce RTX 5070 Ti
- Integrated with 16GB GDDR7 256bit memory interface
- PCIe 5.0
- WINDFORCE cooling system
- Install Ollama and allow its service or background process to start.
- Pull a model from the Ollama library with
ollama pull <model>. Replace<model>with the library identifier you selected for your VRAM and use case. - Start an interactive session with
ollama run <model>. - Send a short prompt first, then test a longer context and the real workload you care about.
- Keep
nvidia-smiopen in another terminal while the model loads and generates text.
Ollama normally chooses an available compatible NVIDIA device. On a machine with several NVIDIA GPUs, set CUDA_VISIBLE_DEVICES before starting Ollama to restrict which devices it can see. In Windows PowerShell, for example:
Free tools Windows power users keep installed
One-click scans. No signup required.
$env:CUDA_VISIBLE_DEVICES='0'
ollama run <model>On Linux, use:
export CUDA_VISIBLE_DEVICES=0
ollama run <model>The number is the device index visible to the process, not necessarily the slot label printed on the chassis. Check the mapping with nvidia-smi before pinning a multi-GPU workload.
6. More control: run the model with llama.cpp
Install a CUDA-enabled build
Use a llama.cpp release or build that explicitly includes CUDA support for your operating system. If you compile it yourself, use the CUDA Toolkit and host compiler versions supported by your platform. A CPU-only binary can open the same model file but will not provide GPU acceleration.
Start an interactive command-line session
Place the compatible GGUF file in a known directory and run a command similar to:
llama-cli -m /path/to/model.gguf -ngl 999 -c 4096In this example, -ngl requests GPU layers and -c sets the context size. Flag names can differ between llama.cpp releases; run llama-cli --help and use the spelling shown by your build. If all layers do not fit, lower the GPU-layer count or use CPU offload instead of forcing an out-of-memory allocation.
Expose a local HTTP server
For applications that need an API, launch the server binary:
llama-server -m /path/to/model.gguf -ngl 999 -c 4096 --host 127.0.0.1 --port 8080On Windows, the executable may be named llama-server.exe. The NVIDIA playbook demonstrates this style of launch with a Hugging Face GGUF model and, for multimodal use, a matching projector URL. Use the exact model and projector locations published with the model release; do not substitute an unrelated projector. Open the endpoint shown by llama-server --help for your version, because server routes and option names evolve.
Rank #4
- Powered by the NVIDIA Blackwell architecture and DLSS 4 OC mode: 2640MHz/Default mode: 2610MHz (Boost Clock)
- Military-grade components deliver rock-solid power and longer lifespan for ultimate durability
- Protective PCB coating helps protect against short circuits caused by moisture, dust, or debris
- 3.125-slot design with massive fin array optimized for airflow from three Axial-tech fans
- Phase-change GPU thermal pad helps ensure optimal thermal performance and longevity, outlasting traditional thermal paste for graphics cards under heavy loads
Tune only after a baseline works
- Begin with a conservative context such as 4,096 tokens and one request.
- Confirm that the model loads on the GPU before increasing context or concurrency.
- Raise the GPU-layer count until VRAM is nearly, but not completely, exhausted.
- If the process fails, lower the layer count, choose a smaller quantization, or shorten the context.
- Record the model file, quantization, context, GPU-layer count, driver, and runtime version whenever you compare performance.
7. Prove that CUDA is being used
Do not treat a successful response as proof of acceleration. A CPU fallback can still generate text, just much more slowly.
Watch the device during loading
Run nvidia-smi repeatedly while the model loads and while it generates. Evidence of GPU use includes the model process appearing in the process list, VRAM allocation increasing by an amount consistent with the model, and GPU utilization rising during prompt processing or generation. Utilization may briefly fall between tokens or while the program waits for input, so check during active work rather than at idle.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Read runtime logs
llama.cpp usually reports how many layers were assigned to the GPU and whether CUDA was initialized. Ollama’s service logs and diagnostic output can indicate which backend and device were selected. A message that mentions only CPU execution, zero GPU layers, or a failed CUDA initialization warrants investigation even if the command completed.
Check after suspend or resume
Ollama documents a Linux-specific failure mode in which suspend/resume can cause the service to miss the NVIDIA GPU and fall back to the CPU. After resuming, check nvidia-smi, restart the Ollama service if necessary, and repeat the monitoring test. Do not assume that a session which used CUDA before sleep is still using it afterward.
8. Troubleshoot the common failure modes
The GPU is not visible
- Run
nvidia-smioutside the AI runtime. If it fails, repair the driver or hardware installation first. - Verify that the exact GPU appears in NVIDIA’s CUDA-supported list.
- On Linux, check that the installed kernel module matches the running kernel and reboot after driver changes.
- On Windows, confirm that the driver was installed separately from the CUDA Toolkit when using CUDA 13.1 or later.
The model will not load or reports out of memory
- Confirm that the runtime is seeing the intended GPU and that no other process is consuming VRAM.
- Use a smaller parameter count or a more aggressive quantization.
- Shorten the context window.
- Reduce concurrent requests or batch size.
- Disable or replace a large multimodal projector if the workload does not need images.
- Enable CPU offload or lower the number of GPU layers.
Changing only the model file may not solve the problem if a long context or multiple active requests is the real cause.
The program runs, but the GPU stays idle
- Check that you installed a CUDA-enabled runtime rather than a CPU-only build.
- Inspect logs for CUDA initialization errors or a zero GPU-layer assignment.
- For Ollama, check
CUDA_VISIBLE_DEVICESand restart the service after changing it. - For llama.cpp, verify the GPU-layer flag supported by your build and confirm that the requested layers fit in VRAM.
- Check again after Linux suspend/resume, when Ollama can fall back to CPU.
Generation is unexpectedly slow
First establish whether the GPU is actually active. If it is, performance still depends on model architecture, quantization, context length, batch size, PCIe transfers caused by offload, thermals, and other processes. There is no universal tokens-per-second figure that applies to every NVIDIA GPU, model, quantization level, context, and runtime, so compare only using a recorded, reproducible configuration.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchesThe multimodal model fails while a text model works
Check that the projector file matches the exact model release and architecture, that the runtime supports multimodal inputs, and that the projector is passed using the option documented by your llama.cpp build. A text-only test cannot validate the image path.
9. A repeatable first-run checklist
- Record the exact NVIDIA GPU and total VRAM.
- Confirm CUDA capability with NVIDIA’s supported-device reference.
- Install a compatible driver; on Windows, treat the driver and CUDA Toolkit as separate installations for CUDA 13.1 and later.
- Verify
nvidia-smibefore installing the model runtime. - Choose Ollama for managed simplicity or llama.cpp for direct control.
- Select a quantized model whose weights, context cache, projector, and runtime overhead fit with headroom.
- Run one short prompt before testing long contexts, images, or multiple requests.
- Watch VRAM, utilization, and runtime logs during loading and generation.
- Record model, quantization, context, runtime, driver, and GPU-layer settings for repeatability.
What the NVIDIA software ecosystem adds
NVIDIA says its developer ecosystem includes more than 400 CUDA-X libraries and a 150-million-device installed base of local NVIDIA GPUs. Those figures describe ecosystem scale, not a guarantee of speed or compatibility for a particular model. The practical benefit is that CUDA support is available across several layers: managed local services, C/C++ runtimes, Python frameworks, optimized deployment engines, and production serving systems.
Choose the simplest layer that satisfies your requirement. A local chat session does not need a production scheduler; a custom research pipeline may outgrow a managed runner; and a high-concurrency service may need a serving framework rather than an interactive command-line tool.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
