A local LLM typically uses more memory as a conversation grows because it keeps a key-value (KV) cache of attention data for earlier tokens. That cache lets the model generate the next token without recomputing all the earlier attention data. In ordinary full-attention models, the cache grows roughly in proportion to the number of retained tokens.
“RAM” can mean system RAM, GPU VRAM, or unified memory. Which one rises depends on where your runtime places the model weights, KV cache, and temporary work buffers. The cache is one part of the total—not the whole memory reading.
What is the KV cache, and why does it grow during a chat?
When a transformer generates text one token at a time, its attention layers produce key (K) and value (V) vectors. The model retains those vectors for earlier token positions in a KV cache. On the next generation step, it can reuse them instead of calculating the earlier key/value pairs again. This is a speed-memory tradeoff: retaining attention state takes memory, but avoids repeated work. Hugging Face explains how cache tensors advance along their sequence-length dimension as tokens are processed.
Each additional retained token adds another slice of K and V data across the model’s cache-bearing attention layers. A longer prompt therefore uses cache space before the model starts replying; generated tokens add to it as well, as long as they remain in the active context. For standard full-attention layers, this is approximately linear growth—not a fixed cost per conversation. Different models can have substantially different per-token costs.
#1 Best Overall
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
How to estimate KV cache memory per token
A conventional first estimate for cache capacity is:
KV cache bytes ≈ B × T × 2 × L × Hkv × D × S
Rank #2
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
- B: number of concurrent sequences or batch slots.
- T: retained tokens per sequence.
- 2: one set of values for keys and one for values.
- L: attention layers that retain cache.
- Hkv: key/value heads per layer.
- D: head dimension.
- S: bytes per cached value. FP16 or BF16 commonly uses two bytes per value.
For an estimate per token, set T to 1. For a rough estimate at a chosen context, use the intended retained-token count. Use KV heads—not necessarily the model’s total query heads. Grouped-query and multi-query attention use fewer KV heads than query heads, which can reduce cache size. Hugging Face’s cache documentation describes the relevant tensor dimensions and cache behavior.
This formula estimates the main tensor storage; it is not a guaranteed reading from a particular runtime. Quantized caches may require metadata, and layouts, hybrid attention designs, and implementation details can change the actual allocation. Cache precision options and their behavior vary by runtime and model. llama.cpp’s server documentation lists separate K- and V-cache data type options, including floating-point and quantized types.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Rank #3
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
Why the memory meter shows more than the cache
A growing memory reading can combine several allocations. A llama.cpp maintainer’s allocation breakdown distinguishes model weights, KV buffer, output buffer, and compute buffers; it is a useful way to interpret the categories, not a promise that every backend reports them identically. The discussion describes those separate allocations.
- Model weights: The model parameters loaded or memory-mapped for inference. Their footprint is driven mainly by the model and its weight representation, rather than by how many tokens you have chatted.
- KV cache: Attention state for retained tokens. Context length, architecture, cache type, and concurrent sequences affect its size.
- Compute buffers: Temporary workspace for inference. In llama.cpp, batch-related settings and Flash Attention can affect compute allocation.
- Output and runtime buffers: Additional structures and backend-specific allocations.
These allocations may land in system RAM, VRAM, or both, depending on runtime configuration and offloading. A single “RAM” number cannot tell you which category grew unless your operating system or inference runtime exposes more detailed allocation information.
Rank #4
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
Why context size does not always equal memory already used
A configured maximum context is a capacity limit, not necessarily a report of current cache occupancy. Some implementations allocate cache as tokens arrive; others reserve capacity in advance. The result can be gradual growth in one setup and a larger allocation at model load or context initialization in another. Check the behavior of your specific runtime and model rather than assuming a universal allocation strategy. Hugging Face documents cache growth and sliding-window behavior.
Attention architecture matters, too. Full-attention layers generally retain state for the active context. In a sliding-window layer, older positions can fall out once the configured window is full, so that layer’s cache need not grow with the entire conversation indefinitely. Hybrid models may combine layers with different behavior, making a single per-token estimate less exact. The Transformers documentation describes sliding-window cache behavior.
Best Value
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
What affects cache size and where the pressure lands?
- Retained context: More prompt and response tokens generally mean more cache in full-attention layers.
- Model architecture: Cache-bearing layer count, KV-head count, head dimension, and attention pattern determine how much state each token adds.
- Cache precision: A smaller representation can reduce bytes per value, but speed and quality effects depend on the model and implementation. Do not assume a particular precision is lossless or faster.
- Concurrency: Multiple active sequences need context state. A runtime may use a unified pool or allocate capacity per slot; batch settings can also affect compute buffers. llama.cpp documents unified KV-cache options. Its CLI documentation covers context and cache controls.
- Offloading: Moving model or cache state between GPU and host memory shifts pressure between VRAM and system RAM, and can affect performance. Exact behavior depends on runtime configuration.
These controls do not all solve the same problem. Reducing context or concurrent sequences targets cache demand; changing cache precision targets bytes per cached value; offloading changes which memory pool bears some of the load. Transformers documents cache strategies and their differing behavior.
How to diagnose a rising memory reading
- Identify the memory pool. Check whether the meter reports system RAM, GPU VRAM, or unified memory. Note that a runtime can use more than one pool.
- Compare stages. Record the reading after model load, after prompt ingestion, and during generation. A rise after the prompt or as tokens are generated is consistent with added context state, though workspace and backend allocations can contribute too.
- Inspect runtime allocation details. If available, use the runtime’s logs or diagnostics to distinguish weights, KV cache, compute buffers, and other allocations. Category names and reporting differ by runtime.
- Estimate the cache for your configuration. Find the cache-bearing layer count, KV-head count, head dimension, retained-token target, cache element type, and number of simultaneous sequences. Apply the formula above, then leave headroom for weights, compute buffers, the operating system, and implementation overhead.
- Change one relevant setting at a time. If memory is constrained, try a shorter context or fewer simultaneous sequences, then check supported cache precision, sliding-window behavior, or cache offload. Measure the actual result: these changes affect different parts of the memory and performance tradeoff.
Do not use a single “memory needed for X tokens” figure without naming the model, runtime, cache type, concurrency, and allocation strategy. Without those details, it can be misleading to compare a context capacity with a memory reading.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
