The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Stop putting the entire transcript into every prompt. Assemble a bounded context for each request: keep the current instructions and necessary recent turns, add a compact summary of older session history, and retrieve only the durable facts relevant to the task. Preserve important decisions and constraints explicitly, and measure whether the agent still recalls them—not just how many tokens the system saves.
Why full-history prompts become a problem
If every invocation resends every prior turn, the prompt grows as the conversation ages. That uses more input tokens and can eventually crowd the request toward the model’s context limit. The AWS Well-Architected Agentic AI Lens identifies this accumulation as a cost and context-management concern, and recommends selective retrieval, compression, pruning, lifecycle controls, and per-session cost visibility.
There is no universal cost per session: the bill depends on the model and its pricing, whether and how caching applies, output length, tool use, and call frequency. The architectural issue is broader than price: a prompt has finite room, and every included instruction, memory, document excerpt, tool result, recent turn, and generated response competes for it.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Repair Windows errors before they cause bigger problems3Fix the driver behind crashes, sound loss and screen glitchesThink of context as a budget assembled for each request
A context window is a bounded working set, not a durable memory store. Microsoft’s multi-agent reference architecture describes short-term context as a shared budget and recommends reserving room for the model’s output. Treat each prompt as an intentional selection of material, not as a growing transcript dump.
#1 Best Overall
A practical assembly order is to include required instructions and task input, then a bounded amount of stable profile information, relevant retrieved memory, recent turns needed to understand the exchange, and a compact summary of older session history. Set limits for each injected section and leave output headroom. The order can vary with the application, but no section should be able to expand without a limit and silently displace the current task.
- Label each block: identify whether material is an instruction, current-session state, a retrieved memory, or source-document content. Preserve provenance and access scope with retrieved items.
- Cap profile and retrieval content: a large profile or an overbroad search result can take space away from recent conversation and the answer.
- Keep source content in view: retrieved user or document text is data, not trusted instruction. Do not grant it authority merely because it appears in the prompt.
- Reserve response space: account for the likely output as well as the input when setting the prompt budget.
Choose the right place for each kind of memory
“Memory” describes several different jobs. Separating them helps avoid both a bloated prompt and an agent that forgets what matters.
| Information type | What it is for | Where it belongs |
|---|---|---|
| Active-session state | Maintaining continuity through the current task | A bounded recent-turn window plus a rolling summary for older turns |
| Durable semantic memory | Reusing compact facts or preferences across sessions | A memory store, retrieved selectively when relevant |
| Episodic memory | Recalling a prior interaction when its history matters | Retained interaction records with retrieval and lifecycle controls |
| Procedural or organizational knowledge | Answering from maintained policies, documentation, or other authoritative material | The appropriate knowledge base, document system, or tool—not a copied personal-memory entry |
Microsoft’s reference architecture distinguishes memory from authoritative knowledge bases. Keep organizational documents in permission-controlled sources and retrieve them at query time. That preserves the source of truth and allows access checks and freshness to remain tied to the underlying system, rather than relying on a stale copy in conversational memory.
Free tools Windows power users keep installed
One-click scans. No signup required.
Compare the main ways to control context growth
| Approach | Useful when | Main trade-off |
|---|---|---|
| Full conversation history | Sessions are short, or the task genuinely depends on every detail | Repeatedly sends old material, increasing token use and context pressure as history grows (AWS Well-Architected Agentic AI Lens). |
| Sliding window | The latest turns usually contain what a simple conversation needs | Earlier decisions and constraints fall out of the prompt, even when the user expects the agent to retain them (Microsoft short-term memory guidance). |
| Rolling summary plus recent turns | Long sessions need continuity while prompt size stays bounded | A summary may omit details; preserve high-value state explicitly and retain source history for drill-down (Microsoft short-term memory guidance). |
| On-demand retrieval from durable memory | Tasks need selected facts from a large history or across sessions | Relevance, latency, and irrelevant results must be controlled and measured (AWS Well-Architected Agentic AI Lens). |
| Long-context model, optionally with caching | Requests repeatedly use a large fixed input or need a large document in view | A longer window does not ensure reliable multi-fact retrieval or eliminate request costs; caching behavior and pricing vary by provider and model (Google AI for Developers, “Long context | Gemini API”). |
The AWS Well-Architected Agentic AI Lens captures the retrieval principle: “Retrieval is selective rather than bulk, so the tokens spent per reasoning step reflect the context that actually matters to the current task.” That is a design goal, not a guarantee: retrieval can still add noise or miss a fact.
Build a summary and retrieval lifecycle
- Track the context budget. Estimate the prompt’s token use, including instructions, recent turns, retrieved material, and expected output. Trigger compaction before the model’s hard limit. Microsoft gives 70–80% of the context window as an example trigger range; it is a starting point, not a universal optimum.
- Compact incrementally. Summarize older turns before pressure becomes urgent instead of waiting until the prompt is nearly full. Maintain a rolling summary as the session proceeds.
- Carry forward decision-critical state. Keep decisions, constraints, identifiers, and open items explicit, rather than trusting a broad narrative summary to retain them. Keep recent turns verbatim when their exact wording matters.
- Retrieve narrowly. Search durable memory for material relevant to the current task, then bound the number and size of results. Monitor relevance, chunk sizing, top-k, and retrieval latency; do not inject results merely because they were returned.
- Keep provenance and permissions. Store where a memory came from and its access scope. Retrieve organizational facts from permission-controlled sources when needed, rather than copying their contents into a user’s durable memory.
- Define lifecycle rules. Decide what is transient session state, what is archived, what may be promoted into durable memory, and how each category is deleted. Treat raw transcript retention and retaining selected facts for future recall as separate choices.
When a summary is uncertain about a detail that could change the answer, retrieve the relevant source history or authoritative document rather than silently filling the gap. This keeps compression from turning missing context into invented certainty.
Evaluate useful recall, not compression alone
Measure the architecture against representative tasks. A smaller memory is not an improvement if it loses the constraint, decision, or prior fact that makes the answer correct. Compare alternatives using:
- Recall quality: can the agent recover the facts, decisions, and constraints needed for the target task?
- Tokens per successful task: include retries and extra calls, not only the first prompt.
- Latency and retrieval overhead: track how long memory search and context assembly add.
- Retrieval noise: assess whether irrelevant or conflicting memories distract the model.
- Freshness and access control: check whether retrieved information is current and available to the requesting user.
- Storage, indexing, deletion, and operational complexity: include the costs and lifecycle work outside the model prompt.
AWS recommends observing costs at agent and session level alongside performance. Track tokens, latency, retrieval frequency, and task quality together so a lower token count cannot conceal degraded recall.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Published compression results are evidence that specialized methods can help, not a forecast for a new agent. A March 2026 preprint, “Structured Distillation for Personalized Agent Memory,” reports an average representation reduction from 371 to 38 tokens per exchange—about 11×—on 4,182 conversations and 14,340 exchanges across six software engineering projects. Its best pure-distilled configuration reported 0.717 MRR against 0.745 for the best verbatim baseline, or 96% of that baseline; outcomes varied with retrieval method. Those results are specific to the paper’s corpus and setup.
Microsoft Research’s PlugMem page reports evaluations across long multi-turn conversation questions, cross-article fact finding, and decisions while browsing. The authors report that PlugMem outperformed generic retrieval and task-specific designs while using less memory context. Treat that as the authors’ benchmark result, not an independent replication or a universal ranking of memory systems.
A July 23, 2026 arXiv preprint, “Agentic Context Management: Solving Agent Memory and Cost by Treating Them as Lifecycle and Architecture Problems,” argues that naive accumulation can make token cost grow quadratically over a conversation and that crude summarization can trade cost for accuracy. This is the paper’s analysis, not a law that applies to every conversation or billing setup. Its useful engineering implication is to validate compaction for fidelity instead of assuming that shorter is better.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.When a longer context window is—and is not—the answer
A longer window can make it practical to keep more material in view, especially for large documents or repeated work over fixed inputs. It does not turn the window into durable memory, guarantee perfect retrieval across multiple facts, or automatically lower cost. Google’s Gemini API guidance discusses caching as an optimization and describes provider- and model-specific cost illustrations; its approximate four-times comparison for Gemini Flash applies to the described caching scenario, not as a general multiplier.
Recommended Free Tools
Choose a longer context when the task benefits from seeing a large body of material together and the model’s measured quality, latency, and cost fit the workload. Choose selective retrieval and durable memory when only a small fraction of a large history is usually relevant. Many systems will combine both: bounded recent context, compact session state, and targeted retrieval.
A practical decision rule
Start with the failure cases your users actually encounter: which earlier decisions, constraints, or facts must survive, and which material is rarely needed again? Build the smallest context assembly that reliably answers those cases, then compare it with alternatives on recall, tokens per successful task, latency, retrieval noise, freshness, access control, lifecycle behavior, and implementation complexity. There is no evidence-backed universal architecture or token-saving percentage; the right balance is the one that remains useful on your workload without paying to resend irrelevant history.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
