Run Your Own AI Model Locally: A Practical Ollama Setup Guide for 2026

iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more

Run your own AI model locally with Ollama by installing the runtime, running ollama run gemma3, and calling the local API at http://localhost:11434. Ollama works on macOS, Windows, and Linux, but only a local model runs entirely on your computer; a model tagged -cloud sends inference to Ollama’s servers.

This guide takes you from hardware checks to a working model, GPU verification, API integration, storage relocation, customization, privacy controls, and troubleshooting. Ollama is a runtime and API, not a model or a locally hosted version of ChatGPT: the open-weight model you select determines its capabilities, license, behavior, and hardware requirements.

#	Preview	Product	Price
1		ASUS Dual GeForce RTX 5060 Ti 16GB GDDR7 OC Edition Gaming Graphics Card	$781.99	Buy on Amazon
2		GIGABYTE GeForce RTX 5070 Ti Gaming OC 16G Graphics Card, 16GB 256-bit GDDR7, PCIe 5.0, WINDFORCE...	$1,162.49	Buy on Amazon
3		ASUS TUF Gaming GeForce RTX™ 5080 16GB GDDR7 OC Edition Graphics Card	$1,830.91	Buy on Amazon
4		GIGABYTE GeForce RTX 5060 WINDFORCE OC 8G Graphics Card, Cooling System, 8GB 128-bit GDDR7, PCIe...	$459.99	Buy on Amazon
5		ASUS TUF Gaming GeForce RTX 5070 12GB GDDR7 OC EditionGaming Graphics Card	$937.39	Buy on Amazon

Key takeaways

• Ollama runs open-weight language models on macOS, Windows, and Linux and normally exposes the local API at http://localhost:11434.
• ollama run <model> downloads a model when necessary and opens an interactive chat; ollama pull downloads without starting a chat.
• As a planning estimate, 7B–8B models commonly target 16 GB of system RAM or about 8 GB of VRAM, while 27B–32B models commonly target 32–64 GB of RAM or 24 GB or more of VRAM.
• A model’s download size is not its total memory requirement because context length, KV cache, runtime overhead, and concurrent requests require additional memory.
• Running Ollama locally does not guarantee local inference: a model with a -cloud tag is processed remotely.
• ollama ps, plus nvidia-smi on NVIDIA systems, provides the most useful first check for GPU utilization.

What is Ollama?

Ollama is a cross-platform runtime, model manager, command-line interface, and local HTTP API for running compatible open-weight models. The official Ollama quickstart describes the basic workflow: install Ollama, run a model, and interact with it through the terminal or API.

Ollama downloads model packages, stores them on disk, loads them into system memory or GPU memory, and manages the inference service. Ollama also supports Modelfiles for changing system instructions and runtime parameters, and it can connect local models to coding tools and other applications.

#1 Best Overall

Sale

ASUS Dual GeForce RTX 5060 Ti 16GB GDDR7 OC Edition Gaming Graphics Card

• AI Performance: 767 AI TOPS
• OC mode: 2632 MHz (OC mode)/ 2602 MHz (Default mode)
• Powered by the NVIDIA Blackwell architecture and DLSS 4
• Axial-tech fan design features a smaller fan hub that facilitates longer blades and a barrier ring that increases downward air pressure
• A 2.5-slot design maximizes compatibility and cooling efficiency for superior performance in small chassis

$781.99

Buy on Amazon

Ollama is not itself an AI model. Commands such as gemma3, deepseek-r1, and gpt-oss:20b identify models or model tags available in the Ollama model library. Model names, tags, quantizations, licenses, and available sizes can change, so check the current model page before deploying a model commercially or building an application around a specific tag.

What does “local” mean in Ollama?

In Ollama, “local” describes more than the address of the command or API. Three different arrangements are possible:

A local URL does not prove that computation is local. The selected model matters. Ollama’s cloud-model documentation explains the distinction between local models and cloud models.

For a fully offline workflow, use a local model, disable cloud features according to the Ollama FAQ, and keep the service bound to localhost. Also check whether a connected application, browser tool, extension, or script sends data to another service independently of Ollama.

Can your computer run an Ollama model?

Ollama can run on CPU-only computers, but “can start” and “is pleasant to use” are different standards. Practical performance depends on model parameters, quantization, context length, available RAM or VRAM, memory bandwidth, thermal limits, and the number of simultaneous requests.

These are planning estimates, not Ollama guarantees. A model file that fits on an SSD or appears smaller than available VRAM can still fail during inference because the runtime needs memory for the context window, KV cache, framework overhead, and concurrent requests.

Model-specific sizes can vary substantially. For example, the DeepSeek-R1 library page lists variants from 1.5B through 671B parameters; the page showed an approximately 5.2 GB 8B model and an approximately 404 GB 671B model during the August 2026 research pass. Treat those figures as page-specific rather than permanent requirements.

What else should you check before installing?

• Storage: Reserve space for model files, temporary downloads, and future models. Model collections can consume tens or hundreds of gigabytes.
• Task: Chat, coding, reasoning, vision, embeddings, and tool calling may favor different models.
• Context: A long context window uses more memory and can reduce speed.
• Backend: NVIDIA, Apple Metal, AMD ROCm, Vulkan, and CPU execution have different requirements.
• License: Read the individual model license before commercial use, redistribution, or fine-tuning.

Which operating systems does Ollama support?

Ollama documents native installation paths for macOS, Windows, and Linux, as well as Docker deployments. Native installation is the simplest starting point because Docker adds volume, networking, and GPU-passthrough decisions.

How do you install Ollama on macOS?

On macOS, use the official downloadable application from the Ollama download page, or use the documented installer command:

curl -fsSL https://ollama.com/install.sh | sh

The downloadable application is the more approachable option if you do not want to pipe a shell script into a shell. After installation, open Terminal and run:

ollama

Ollama should open its interactive terminal menu or prompt you to run a model. Ollama supports Apple GPU acceleration through Apple’s Metal API; a native macOS installation is preferable when you want to use the Apple GPU.

How do you install Ollama on Windows?

Ollama runs as a native Windows application, normally installs without Administrator rights, runs in the background, and makes the ollama command available in Command Prompt, PowerShell, and other terminals, according to the official Windows documentation.

1. Download and run the official Windows installer.
2. Open a new PowerShell window after installation.
3. Verify the command:

ollama -v

Then start a model:

ollama run gemma3

Ollama’s Windows documentation includes NVIDIA and AMD Radeon support, but the exact driver and backend requirements depend on the current Ollama release, GPU, and installation environment. Check the current GPU support documentation and your GPU vendor’s driver page instead of relying on an old minimum-driver number.

Rank #2

GIGABYTE GeForce RTX 5070 Ti Gaming OC 16G Graphics Card, 16GB 256-bit GDDR7, PCIe 5.0, WINDFORCE Cooling System, GV-N507TGAMING OC-16GD Video Card

• Powered by the NVIDIA Blackwell architecture and DLSS 4
• Powered by GeForce RTX 5070 Ti
• Integrated with 16GB GDDR7 256bit memory interface
• PCIe 5.0
• WINDFORCE cooling system

$1,162.49

Buy on Amazon

AMD acceleration may use ROCm/HIP or Vulkan depending on the supported configuration. Some Radeon systems may need Vulkan as a fallback.

How do you install Ollama on Linux?

On a supported Linux system, install Ollama with the official script:

curl -fsSL https://ollama.com/install.sh | sh

Start the server manually if the installer did not start it:

ollama serve

In a second terminal, verify the installation:

ollama -v

For a systemd-managed installation, use:

sudo systemctl start ollama
sudo systemctl status ollama

For AMD GPU acceleration on Linux, Ollama documents an additional ROCm package for AMD64 systems:

curl -fsSL https://ollama.com/download/ollama-linux-amd64-rocm.tar.zst 
  | sudo tar x -C /usr

Match the package to your architecture. ARM64 systems require the ARM64 package rather than the AMD64 package. The Linux installation documentation is the authoritative place to check the current package and service setup.

What is the quickest way to run your first local model?

The quickest way to run your own AI model locally with Ollama is to verify the installation and execute ollama run gemma3. Ollama downloads the model if it is not already stored locally, starts the service, and opens an interactive session.

# macOS or Linux installation
curl -fsSL https://ollama.com/install.sh | sh

# Verify
ollama -v

# Download without opening a chat
ollama pull gemma3

# See downloaded models
ollama list

# Start an interactive session
ollama run gemma3

On Windows, install through the official installer first, then run the verification and model commands in PowerShell.

Inside the chat, enter your prompt. Run the following to see commands supported by the version you installed:

/help

Use the live help output instead of an old screenshot because interactive commands and menus can change between releases.

What do the main Ollama commands do?

Which Ollama model should you choose?

Choose an Ollama model by task, memory, context requirements, modality, backend compatibility, quantization, and license rather than by parameter count alone. The current model library includes general-purpose, reasoning, coding, vision, multilingual, embedding, and tool-capable options.

The model library includes examples such as Gemma 3 and Gemma 4, DeepSeek-R1, Qwen, gpt-oss, Llama, and embedding models. Availability and tags are volatile. No model is universally “best” without a defined task, benchmark, hardware setup, and quality target.

How do you verify GPU acceleration?

Start a model, then inspect the loaded process in another terminal:

ollama run gemma3

ollama ps

Inspect the processor or memory-allocation information shown by your installed version. On NVIDIA systems, also run:

Rank #3

ASUS TUF Gaming GeForce RTX™ 5080 16GB GDDR7 OC Edition Graphics Card

• Powered by the NVIDIA Blackwell architecture and DLSS 4. System Requirements: Minimum 850W PSU with 16-pin 12V-2x6 (12VHPWR) connector required. Verify before purchasing.
• Military-grade components deliver rock-solid power and longer lifespan for ultimate durability. Compatibility: 348mm (13.7") length, 3.6 slots, 4.3 lbs. Confirm case clearance and slot spacing. GPU bracket included.
• Protective PCB coating helps protect against short circuits caused by moisture, dust, or debris
• 3.6-slot design with massive fin array optimized for airflow from three Axial-tech fans
• Phase-change GPU thermal pad helps ensure optimal thermal performance and longevity, outlasting traditional thermal paste for graphics cards under heavy loads

$1,830.91

Buy on Amazon

nvidia-smi

Interpret the result carefully:

• Fully loaded on a GPU: The model fits in GPU memory and should generally outperform CPU-only inference, although architecture, memory bandwidth, context length, and thermals still matter.
• Partially offloaded: Some layers remain in system RAM, so generation may be slower than full GPU execution.
• CPU or split execution: The model may not fit in VRAM, the driver may be unsupported, or Ollama may be running in an environment without GPU passthrough.

Ollama documents NVIDIA acceleration, Apple Metal, AMD ROCm, and experimental Vulkan support on Windows and Linux. The current GPU support page lists NVIDIA compute capability 5.0 or newer and driver version 531 or newer, while platform-specific documentation can impose different requirements. Verify the exact GPU rather than treating one number as universal.

Ollama’s FAQ says the scheduler generally prefers a GPU when a model fits entirely on one GPU; when the model does not fit, Ollama can spread the model across available GPUs.

What if Vulkan selects the wrong GPU?

Ollama documents GGML_VK_VISIBLE_DEVICES for Vulkan GPU selection. To disable Vulkan visibility, the documented value is:

GGML_VK_VISIBLE_DEVICES=-1

Use the current GPU documentation for the platform-specific syntax and device-selection value.

How do you use Ollama’s local API?

Ollama’s local API normally listens at http://localhost:11434 and local access normally requires no API key. The official quickstart documents the chat endpoint and the JSON request format.

Chat with curl

curl http://localhost:11434/api/chat 
  -d '{
    "model": "gemma3",
    "messages": [
      {
        "role": "user",
        "content": "Explain how local language models work in three paragraphs."
      }
    ],
    "stream": false
  }'

Generate text with curl

curl http://localhost:11434/api/generate 
  -d '{
    "model": "gemma3",
    "prompt": "Why is the sky blue?",
    "stream": false
  }'

PowerShell users can send a request with:

Invoke-WebRequest `
  -Method POST `
  -Body '{"model":"gemma3","prompt":"Why is the sky blue?","stream":false}' `
  -Uri http://localhost:11434/api/generate

Call Ollama from Python

pip install ollama

from ollama import chat

response = chat(
    model="gemma3",
    messages=[
        {
            "role": "user",
            "content": "Give me five practical uses for a local AI model."
        }
    ],
)

print(response.message.content)

Call Ollama from JavaScript

npm install ollama

import ollama from "ollama";

const response = await ollama.chat({
  model: "gemma3",
  messages: [
    {
      role: "user",
      content: "Give me five practical uses for a local AI model."
    }
  ]
});

console.log(response.message.content);

Streaming is commonly enabled by default. Set "stream": false when your application needs one complete JSON response rather than a sequence of streamed chunks. Applications should handle a stopped server, missing model, timeout, malformed request, and out-of-memory response.

The local API is different from Ollama’s hosted API. Direct requests to the hosted endpoint require authentication, as described in the Ollama API authentication documentation. Do not expose port 11434 directly to the public internet without authentication, network restrictions, and a clear security design.

How do context length and concurrency affect memory?

Ollama’s FAQ documents a default context window of 4,096 tokens and the OLLAMA_CONTEXT_LENGTH setting for changing it. A longer context allows more input history but increases memory use.

macOS or Linux:

export OLLAMA_CONTEXT_LENGTH=8192

Windows PowerShell:

$env:OLLAMA_CONTEXT_LENGTH = "8192"

Ollama also documents settings for parallelism and loaded models:

Required RAM scales with parallelism multiplied by context length. If a model works at 4,096 tokens but fails at 32,768 tokens, the problem is usually memory capacity rather than proof that the model is incompatible.

Where does Ollama store models, and how do you move them?

Ollama stores model files separately from the application. The documented default locations are:

Windows documentation separately notes that the Ollama application itself needs at least 4 GB; model storage is additional. Check the FAQ and storage documentation before moving a large collection.

Set OLLAMA_MODELS to relocate models. On a Linux systemd installation, edit the service:

Rank #4

Sale

GIGABYTE GeForce RTX 5060 WINDFORCE OC 8G Graphics Card, Cooling System, 8GB 128-bit GDDR7, PCIe 5.0, Manufactured by NVIDIA, DisplayPort & HDMI - Video Output Interface, GV-N5060WF2OC-8GD Video Card

• Powered by the NVIDIA Blackwell architecture and DLSS 4
• Powered by GeForce RTX 5060
• Integrated with 8GB GDDR7 128bit memory interface
• PCIe 5.0
• WINDFORCE cooling system

$459.99

Buy on Amazon

sudo systemctl edit ollama

Add an environment override:

[Service]
Environment="OLLAMA_MODELS=/mnt/ai-models"

Restart the service after changing the configuration. A destination must have enough free space, and the Linux ollama service user must be able to read and write the directory. Moving the setting does not automatically migrate existing files; copy the model directory if you want to preserve downloaded models.

An external drive can save internal disk space but may introduce slower loading, disconnection, and permission problems. An NVMe SSD is generally a more sensible location than a slow removable drive for models used frequently.

How do you create a custom assistant with a Modelfile?

A Modelfile changes a model’s runtime behavior, parameters, and system instructions; it does not retrain or fine-tune the underlying model. Create a file named Modelfile:

FROM gemma3

PARAMETER temperature 0.2
PARAMETER num_ctx 8192

SYSTEM """
You are a concise technical assistant.
State uncertainty clearly and do not invent commands.
"""

Build and run the customized model:

ollama create local-tech-assistant -f Modelfile
ollama run local-tech-assistant

Modelfile syntax can change between releases. If the command rejects a parameter, check the current Ollama documentation and the installed version’s help output rather than copying an old example unchanged.

How do you troubleshoot common Ollama problems?

Why does the terminal say “ollama: command not found”?

The usual causes are an incomplete installation, a terminal opened before PATH changed, or a Linux binary installed outside the current path. First open a new terminal and run:

ollama -v

On Linux, locate the command with:

which ollama

On Windows PowerShell, use:

Get-Command ollama

If the command remains unavailable, rerun the official installer.

Why does a model download fail?

Check internet connectivity, free disk space, firewall or proxy rules, and whether the model tag still exists in the current library. Retry the download:

ollama pull gemma3

If a specific tag fails, open the current model page instead of guessing a replacement tag. A partial or corrupted download may also require removing and downloading the model again.

Why is the Ollama server unavailable?

Start the server manually:

ollama serve

For a Linux service installation, check and restart the service:

sudo systemctl status ollama
sudo systemctl restart ollama

Then test the local API:

curl http://localhost:11434/api/tags

How do you fix an out-of-memory crash?

Use a smaller model or smaller quantized variant first. Then reduce OLLAMA_CONTEXT_LENGTH, close GPU-heavy applications, set OLLAMA_NUM_PARALLEL=1, and avoid loading multiple models. CPU fallback or a cloud model can be alternatives when the desired model cannot fit.

Increasing swap is not a universal fix. Swap may prevent an immediate crash but can make interactive generation unusably slow.

Why is Ollama not using the GPU?

Run ollama ps while a model is active, then check the NVIDIA driver with nvidia-smi if applicable. Also verify Apple Metal support, AMD ROCm compatibility, Vulkan configuration, model fit in VRAM, and whether the model is running inside an environment without GPU passthrough.

Docker GPU support is documented for Linux and Windows with WSL2 using the NVIDIA Container Toolkit. Docker Desktop on macOS does not provide GPU passthrough for Ollama, so a Dockerized macOS deployment may fall back to CPU. Use native macOS installation when Apple GPU acceleration is important.

Best Value

ASUS TUF Gaming GeForce RTX 5070 12GB GDDR7 OC EditionGaming Graphics Card

• Powered by the NVIDIA Blackwell architecture and DLSS 4 OC mode: 2640MHz/Default mode: 2610MHz (Boost Clock)
• Military-grade components deliver rock-solid power and longer lifespan for ultimate durability
• Protective PCB coating helps protect against short circuits caused by moisture, dust, or debris
• 3.125-slot design with massive fin array optimized for airflow from three Axial-tech fans
• Phase-change GPU thermal pad helps ensure optimal thermal performance and longevity, outlasting traditional thermal paste for graphics cards under heavy loads

$937.39

Buy on Amazon

Why is local generation slow?

Slow responses can result from CPU-only inference, partial GPU offload, an oversized model, excessive context length, thermal throttling, slow external storage, multiple requests, or an architecture that does not use the hardware efficiently. Compare ollama ps with operating-system monitoring tools, then test a smaller model or shorter context to isolate the cause. Fixed tokens-per-second claims are not portable between hardware and model configurations.

How private is a local Ollama setup?

A fully local setup can keep prompts and responses on the computer when the selected model is local, connected applications do not transmit the data elsewhere, and the API remains inaccessible to untrusted networks. Local inference does not automatically control logs, browser tools, plugins, operating-system telemetry, or the behavior of applications that use Ollama.

Ollama’s pricing page says cloud prompt and response data is not logged or used for training for its cloud service, but cloud inference still sends data away from the local device for remote processing. “Not used for training” is not the same as “never leaves your computer.” For highly sensitive work, use local-only mode and a network-isolated workflow where appropriate.

Ollama permits local browser origins by default and supports OLLAMA_ORIGINS for additional browser origins. Adding origins can help local browser tools but also increases the need to understand which applications can call the service. Keep the default localhost setup unless remote access is an explicit requirement.

Local privacy checklist

• Use a model without a -cloud tag.
• Disable cloud features for an offline-only workflow.
• Keep port 11434 on localhost unless you have designed authentication and network controls.
• Audit applications, extensions, scripts, and integrations that can call the local API.
• Read the selected model’s license before commercial deployment.
• Do not put confidential material into a cloud-tagged model merely because the command itself runs locally.

Should you use Ollama locally or Ollama Cloud?

Use local Ollama when privacy, offline access, model-location control, or avoiding per-token local inference charges matters more than maximum model size and throughput. Use Ollama Cloud when your computer cannot fit the desired model or you want hosted compute without managing a large local GPU.

Ollama’s official pricing page showed Free at $0 and Ollama Pro at $20 per month or $200 per year billed annually during the August 16, 2026 research pass. The page described Pro as offering larger cloud models, three cloud models at once, and 50 times the cloud usage of Free. The same page showed Max at $100 per month with new sign-ups temporarily paused, Team at an introductory $25 per seat per month with a five-seat minimum and marked “Coming soon,” and Enterprise with custom pricing. Recheck the page before purchase because plan availability and prices can change.

Ollama Pro is relevant when the Ollama workflow is attractive but local hardware is insufficient; Ollama Pro is not necessary for running downloaded local models.

How does Ollama compare with other local AI tools?

Ollama is usually the most convenient choice of these options when a developer wants a local service and API without managing every low-level inference setting. llama.cpp is a better fit when precise backend and runtime control matters more than convenience. LM Studio is a sensible alternative for users who strongly prefer a GUI.

What should you do after the first successful run?

1. Run ollama ps while the model is active and confirm whether execution is CPU, GPU, or split.
2. Run a small API request against http://localhost:11434/api/chat or /api/generate.
3. Check model storage and move OLLAMA_MODELS if the system drive is too small.
4. Set a conservative context length before experimenting with long documents.
5. Read the selected model’s license and capability notes.
6. Keep cloud features disabled if the goal is offline, on-device processing.
7. Only then consider a Modelfile, remote access, Docker, or multiple concurrent requests.

Frequently Asked Questions

Can Ollama run without a GPU?

Yes, Ollama can run models using the CPU, but CPU-only inference may be too slow for comfortable interactive use, especially with larger models. A smaller model and shorter context usually provide a better experience on CPU-only hardware.

Does using Ollama mean my prompts stay private?

Prompts can remain on your computer when you use a local model, keep the API on localhost, and prevent connected applications from sending data elsewhere. A model tagged -cloud is processed remotely, so a local Ollama command does not by itself guarantee on-device privacy.

How much RAM does Ollama need for an 8B model?

An 8B model commonly targets about 16 GB of system RAM or roughly 8 GB of VRAM as a planning estimate, but the actual requirement varies with quantization, context length, runtime overhead, and concurrent requests. A download size smaller than available memory does not guarantee that inference will fit.

What is the Ollama API address?

The default Ollama local API address is http://localhost:11434. Local access normally does not require authentication, but the hosted Ollama API requires an account or API key, and exposing the local port to other networks without security controls is risky.

The Bottom Line

Ollama is a practical way to run open-weight AI models on a personal computer, expose them through a local API, and keep a correctly configured workload on-device. Start with a model that fits comfortably rather than barely fitting, verify execution with ollama ps, keep the API on localhost, and treat every -cloud tag as remote inference. Choose Ollama Cloud only when larger models or hosted performance outweigh the privacy and recurring-cost advantages of local execution.

Quick Recap

SaleBestseller No. 1

ASUS Dual GeForce RTX 5060 Ti 16GB GDDR7 OC Edition Gaming Graphics Card

AI Performance: 767 AI TOPS; OC mode: 2632 MHz (OC mode)/ 2602 MHz (Default mode); Powered by the NVIDIA Blackwell architecture and DLSS 4

$781.99

Bestseller No. 2

GIGABYTE GeForce RTX 5070 Ti Gaming OC 16G Graphics Card, 16GB 256-bit GDDR7, PCIe 5.0, WINDFORCE Cooling System, GV-N507TGAMING OC-16GD Video Card

Powered by the NVIDIA Blackwell architecture and DLSS 4; Powered by GeForce RTX 5070 Ti; Integrated with 16GB GDDR7 256bit memory interface

$1,162.49

Bestseller No. 3

ASUS TUF Gaming GeForce RTX™ 5080 16GB GDDR7 OC Edition Graphics Card

3.6-slot design with massive fin array optimized for airflow from three Axial-tech fans; Auto-Extreme precision automated manufacturing helps ensure higher reliability

$1,830.91

SaleBestseller No. 4

GIGABYTE GeForce RTX 5060 WINDFORCE OC 8G Graphics Card, Cooling System, 8GB 128-bit GDDR7, PCIe 5.0, Manufactured by NVIDIA, DisplayPort & HDMI - Video Output Interface, GV-N5060WF2OC-8GD Video Card

Powered by the NVIDIA Blackwell architecture and DLSS 4; Powered by GeForce RTX 5060; Integrated with 8GB GDDR7 128bit memory interface

$459.99

Bestseller No. 5

ASUS TUF Gaming GeForce RTX 5070 12GB GDDR7 OC EditionGaming Graphics Card

3.125-slot design with massive fin array optimized for airflow from three Axial-tech fans; Auto-Extreme precision automated manufacturing helps ensure higher reliability

$937.39

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.