iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
You can build a useful AI agent without per-request cloud charges by running the model on your own computer, exposing a narrow FastAPI service, and adding ChromaDB for retrieval. Ollama supplies local chat, completion, embedding, and model-management APIs; FastAPI provides typed HTTP boundaries and production controls; Chroma stores searchable embeddings and metadata. “Zero-cost” means no cloud API bill, not no cost at all: you still provide electricity, storage, memory, and engineering time. Start with a single-user prototype, then add authentication, limits, observability, backups, pinned model versions, and a protected network path before other people depend on it.
What does a local FastAPI, Ollama, and ChromaDB agent look like?
The practical architecture has four boundaries. A client sends a typed request to FastAPI. FastAPI validates it, applies policy, and calls an agent service. The agent service retrieves relevant records from ChromaDB and sends a carefully constructed prompt to Ollama. Ollama runs the selected model on the local machine and returns a response or a streamed sequence of tokens.
Client
|
v
FastAPI (auth, validation, limits, observability)
|
+-- Agent loop (retrieval, tools, prompt policy)
| |
| +-- ChromaDB (embeddings, metadata, search)
| |
| +-- Ollama API (chat/completion/embeddings)
|
+-- Logs, metrics, health and readiness checksKeep the service boundary independent of the model provider. A provider module should be the only code that knows Ollama’s HTTP paths, streaming format, model tag, and keep-alive behavior. That separation lets you change models without rewriting authentication, retrieval, or request handling.
Can I run an AI agent without cloud API costs?
Yes. Ollama serves local models over HTTP, and local requests do not require an API key. The official local base URL is http://localhost:11434/api; its OpenAI-compatible base is http://localhost:11434/v1. You avoid a recurring per-token provider bill, but hardware, power, disk space, maintenance, and your own development time remain real costs.
#1 Best Overall
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
| Decision axis | Local Ollama stack | Hosted model API |
|---|---|---|
| Privacy and network dependence | Prompts and retrieved documents can stay on your network; internet access is optional after models are downloaded. | Requests normally leave your environment and depend on a provider connection and policy. |
| Recurring spend | No per-request cloud charge; you own the computer and operating costs. | Usage is usually metered, while hardware operations are outsourced. |
| Latency | Can be quick on suitable CPU/GPU hardware and avoids a network round trip; weak hardware may be slow. | Consistent provider infrastructure can help, but network latency and queueing apply. |
| Memory and acceleration | You must supply enough RAM/VRAM and configure GPU support. | Provider supplies inference hardware. |
| Context length and tools | Varies by model; smaller local models often have smaller context windows, and tool calling must be checked for the exact model. | Capabilities and limits are defined by the provider’s model version. |
| Retrieval control | You control Chroma collections, chunking, metadata filters, and retention. | Retrieval may be a provider feature or a separate service with its own data controls. |
| Operations | You patch the host, monitor processes, protect the endpoint, and back up data. | The provider operates the inference layer, but you still own application security and data governance. |
How do I build a local AI agent with FastAPI and Ollama?
Build the smallest useful contract first: one request model, one response model, one bounded agent operation, and explicit failure responses. A prototype can run in one process, but keep the interfaces ready for a later production deployment.
1. Install the runtime components
Install Ollama for your operating system, start its local server, and pull a model using its exact model:tag name. In a Python environment, install FastAPI, an ASGI server, an HTTP client, Pydantic, and the Chroma client appropriate to your deployment. Keep dependency versions in a lock file so a rebuild does not silently change behavior.
2. Define configuration outside the code
Use environment variables or a secret manager rather than hard-coded addresses and credentials. A useful baseline is:
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Repair Windows errors before they cause bigger problems3Scan for outdated or missing drivers - takes under a minuteOLLAMA_BASE_URL=http://localhost:11434/api
OLLAMA_MODEL=your-model:tag
OLLAMA_STREAM=false
OLLAMA_KEEP_ALIVE=5m
OLLAMA_TIMEOUT_SECONDS=60
CHROMA_PATH=./data/chroma
MAX_INPUT_CHARS=12000
MAX_TOOL_CALLS=4Local-only deployments should not contain cloud API keys. If the service later supports a remote provider, use a separate configuration profile and make provider selection explicit.
3. Create typed request and response models
Accept only the fields your contract needs. For example, a request can contain message, an optional conversation_id, and a retrieval filter. Enforce maximum lengths, reject unknown or malformed filter fields, and return a stable response containing the answer, model identifier, retrieval references, and a request ID. Do not echo secrets or raw internal prompts.
Rank #2
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
4. Wrap Ollama in a provider module
The provider should set the model tag, timeout, stream mode, and keep-alive policy on every call. Ollama documents chat and completion generation, embeddings, model pull/create/list/delete operations, running-model inspection, and version reporting. When your API needs one complete JSON response, send stream:false rather than making every caller assemble streamed fragments.
Record the model name and the Ollama version in structured logs. For reproducibility, also record the model digest when the model-management API exposes it. Never let a client choose an arbitrary local URL or model path; map an approved logical model name to a server-side configuration.
Free tools Windows power users keep installed
One-click scans. No signup required.
5. Add health and readiness endpoints
A liveness endpoint should answer whether the FastAPI process is running. A readiness endpoint should verify that the configured Chroma collection is available and that Ollama responds within a short timeout. Do not make liveness call a large generation request. During startup or model loading, report “not ready” so a reverse proxy does not send traffic to an unusable instance.
How do I add ChromaDB retrieval to Ollama?
Use Chroma for a separate ingestion path and a query path. Ingestion turns source documents into stable, searchable records; querying retrieves only the passages needed for the current request and supplies them as untrusted context to the model.
Ingest documents deterministically
- Normalize: convert files or records to consistent text, preserve titles and headings, and remove accidental boilerplate.
- Chunk: split by semantic boundaries where possible. Make chunk size and overlap configuration values so you can tune them without rewriting code.
- Assign stable IDs: derive an ID from a source identifier and a content or version hash. Re-running ingestion should update the same record instead of creating duplicates.
- Store metadata: retain source URI or document ID, title, section, tenant, language, access classification, and updated timestamp. Metadata enables filters and audit trails.
- Embed and upsert: choose one embedding model for a collection, document that choice, and upsert embeddings with the text and metadata.
Chroma supports dense, sparse, hybrid, full-text, and regular-expression search, along with metadata filters and multimodal retrieval. It can run locally or be self-hosted. Select the simplest search mode that meets your quality requirement, then test it with representative queries rather than assuming more retrieval is always better.
Rank #3
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
Query before generation when evidence matters
At request time, embed the user query with the same embedding setup, apply authorization and metadata filters, and retrieve a bounded number of passages. Include source identifiers in the internal context so the answer can cite or expose provenance. If no passage clears your relevance threshold, tell the model to say that the collection does not contain enough evidence instead of filling the gap from its general training.
Protect retrieved text from prompt injection
Retrieved documents are data, not instructions. Delimit them clearly and tell the model that commands inside a document cannot override system policy. Strip or quarantine active content, keep tool permissions outside the retrieved text, and require a separate policy decision before any tool call can change files, send messages, or access a network.
How should the agent loop handle tools and failures?
Keep the loop bounded and observable. A safe sequence is: validate the user request, retrieve context if required, ask the model for either a final answer or a structured tool request, validate that request against an allowlist, execute it with a deadline, append the result as tool output, and repeat only up to the configured call limit.
- Separate roles: system instructions, user content, retrieved passages, and tool output should be distinct fields or clearly delimited sections.
- Validate arguments: use typed schemas for every tool and reject extra fields, unsafe paths, oversized payloads, and unapproved destinations.
- Set deadlines: apply an overall request deadline shorter than the reverse-proxy timeout, plus individual Ollama, Chroma, and tool timeouts.
- Retry selectively: retry transient connection failures with exponential backoff and jitter. Do not retry validation errors, authorization failures, or a tool that may already have performed a side effect.
- Support cancellation: propagate client disconnects to the Ollama request where the HTTP client permits it, and stop abandoned generations.
- Return structured errors: distinguish invalid input, unauthorized access, dependency timeout, model loading, and internal failures without exposing stack traces.
What production controls are still required?
FastAPI’s development server is convenient for iteration, but deployment means making the application available with performance and stability. “Deploying a FastAPI application is relatively easy,” according to the framework documentation; the difficult work is choosing and operating a reliable boundary around it.
Authentication and authorization
Require authentication for every non-health route. Use short-lived tokens or a mutually authenticated private network, and authorize access to collections by tenant or document classification. Never treat an obscure URL or the local Ollama port as authentication.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Rank #4
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
Rate, size, and concurrency limits
Limit request body bytes, message characters, retrieved-token budget, concurrent generations, and tool calls. A queue with a visible maximum is safer than allowing unlimited requests to exhaust RAM or VRAM. Return a clear overload response and include a retry hint when appropriate.
Logging and metrics
Emit JSON logs with request ID, authenticated principal, route, latency, status, model tag, model digest when available, retrieval count, tool count, and token or character limits. Redact prompts, retrieved content, and secrets by default; provide an explicit, access-controlled debug mode if sampled payloads are essential. Track request latency, timeout counts, model-load time, queue depth, retrieval misses, and error rates.
Backups and model updates
Back up Chroma’s persistent data and ingestion configuration on a schedule that matches how much data you can recreate. Test restoration, not just backup creation. Promote model updates through a staging collection and representative evaluation queries; pin the production model tag and record its digest so a later pull cannot silently alter answers.
Network exposure
Keep Ollama on its protected local interface whenever possible. If another machine must reach it, place an authenticated reverse proxy or private network in front of it, restrict source addresses, encrypt the connection, and avoid exposing the unauthenticated local port directly to the internet. FastAPI should be the policy-enforcing public boundary, not Ollama itself.
What hardware do I need for a 7B local model?
Docker’s 2026 local-model guidance gives a planning range of 8–16 GB of RAM for a typical 7B model. That is not a guarantee: quantization, context length, concurrent requests, the embedding model, operating-system overhead, and other loaded models can raise the requirement. GPU acceleration can dramatically improve speed, but the usable model size is constrained by available VRAM and how the runtime offloads layers.
Best Value
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
| Workload | Planning considerations | What to measure before launch |
|---|---|---|
| Single-user prototype | CPU inference may be acceptable for short prompts; leave headroom for the operating system and Chroma. | Time to first token, total response time, and peak RAM during model load. |
| Several interactive users | Concurrency increases memory pressure and queueing; a GPU may be important for responsive output. | Peak RAM/VRAM with the intended context and simultaneous requests. |
| Long retrieval context | Context windows are often smaller on local models, and long prompts consume memory and reduce throughput. | Answer quality and latency at the maximum retrieved passage count. |
| Tool-using agent | Tool-calling support must be verified for the exact model and tag. | Structured-call validity, refusal behavior, and recovery after tool errors. |
Start with a small representative corpus and one realistic request at a time. Measure on the hardware you will operate; a model’s parameter count alone cannot predict your service’s latency or memory ceiling.
How do I move from a prototype to a stable service?
- Freeze the contract: document request limits, response fields, error codes, authentication, and supported model tags.
- Separate processes: run FastAPI, Ollama, and persistent Chroma storage with explicit startup, shutdown, and restart behavior.
- Load-test safely: use synthetic or approved data, start below the expected concurrency, and watch memory, queue depth, and timeout rates.
- Stage changes: test code, embedding-model, chunking, and generation-model changes against a fixed evaluation set before promotion.
- Deploy behind a proxy: terminate TLS, enforce request limits, authenticate callers, and set proxy timeouts longer than the application deadline.
- Operate it: alert on readiness failures, disk growth, repeated model-load errors, backup failures, and rising retrieval misses.
- Plan recovery: document how to restore Chroma, repull a pinned model, rotate credentials, and disable a dangerous tool without taking down read-only answers.
What commonly breaks, and how do I fix it?
- “Connection refused” from FastAPI: verify Ollama is running, confirm the configured base URL includes the correct
/apipath, and check that a firewall or container network is not blocking the local address. - Responses stop midway: use a request deadline that exceeds normal generation time, inspect server logs for model eviction, and choose
stream:falsewhen the client cannot handle partial streams. - RAM or VRAM exhaustion: reduce concurrency and retrieved context, unload unused models, use a smaller or more heavily quantized model, or add memory/GPU capacity.
- Irrelevant retrieval: inspect chunks and metadata, ensure query and document embeddings use the same model, tune chunk size and overlap, and test metadata filters independently.
- Duplicate or stale documents: replace random IDs with deterministic IDs based on source and version, then run a reconciliation job that removes records no longer present in the source.
- Unsafe tool execution: remove tools from the default policy, tighten schemas and allowlists, and require explicit authorization for every side-effecting operation.
- Works locally but fails after deployment: compare environment variables, filesystem permissions, model availability, proxy timeouts, and container-to-host networking; readiness checks should reveal which dependency is unavailable.
Frequently Asked Questions
Does Ollama require an API key on a private network?
Local Ollama requests do not require an API key. You still need authentication at your FastAPI or reverse-proxy boundary when anyone besides the trusted host can submit requests.
Can ChromaDB and Ollama run on different machines?
Yes, provided the network path is private or protected and both services have explicit authentication and authorization controls. Keep the Ollama endpoint off the public internet, and account for network latency when setting retrieval and generation deadlines.
Recommended Free Tools
How can I make local-model answers reproducible?
Pin the model tag and record its digest, Ollama version, system prompt, sampling settings, embedding model, chunking parameters, and retrieved document IDs. Store these values with evaluation results so a changed model or corpus can be identified.
What should I do if the model’s context window is too small?
Retrieve fewer, higher-scoring chunks, reduce chunk size, summarize older conversation turns, and keep tool outputs concise. If the task still needs more context, select a model with a larger documented window and retest memory usage on your hardware.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
