Context window
In one sentence
The context window is how much text a model can consider at once, counting both what it has been given and what it produces, and everything outside the window does not exist as far as the model is concerned.
Not to be confused with Retrieval-augmented generation, or Conversation memory.
Definition
The context window is how much text a model can hold in mind at one time, counting both what you gave it and what it is writing back.
Anything outside the window does not exist as far as the model is concerned.
The window is measured in tokens, which are subword units rather than words. It holds everything the model can attend to in a single inference: the system prompt, the conversation history, the retrieved passages, the tool definitions, the tool results, and the response being generated.
What competes for the space in a voice agent
- The system prompt and behavioral instructions.
- Persona, tone and formatting rules.
- Tool and function definitions, which can be surprisingly large.
- Retrieved passages from RAG.
- Conversation history, growing with every turn.
- The response itself.
- These compete, and as a conversation lengthens something has to be sacrificed. What gets sacrificed should be a design decision rather than a truncation accident.
Why a large context window does not remove the need for RAG
- Cost scales with tokens processed. Sending a large corpus with every request is expensive per turn, and voice conversations have many turns.
- Latency scales with tokens processed too. Against a sub-second time to first audio target, a large prompt is disqualifying whether or not it fits.
- Attention degrades over long contexts. Models attend unevenly, and material in the middle of a very long context receives less effective attention than material at either end. Filling the window is not the same as the model using it.
- Precision suffers. Fifty relevant passages produce better answers than five thousand mostly irrelevant ones.
Management strategies as conversations grow
- Sliding window. Keep the most recent N turns and discard the older ones. Simple, and it loses early context including whatever the visitor stated at the outset.
- Summarization. Compress older turns into a running summary. Preserves the gist, loses the detail, and the summarization step costs time of its own.
- Structured state extraction. Pull the durable facts, such as the visitor's name, intent and constraints, into explicit fields held outside the conversational history. More engineering, and it holds up considerably better.
- The third approach is generally correct for voice, because voice conversations are shorter than text conversations but more likely to contain a constraint stated early that matters at the end.
Common misconception
That larger context windows have made retrieval and context management obsolete. The constraint moved. It used to be what fits, and now it is what is affordable, fast and effectively attended to. In voice, where latency is the binding constraint, context economy matters more than it does in text, not less.
Why it matters commercially
Context management determines whether a conversation holds together. The failure is specific and recognizable: the visitor states something in their first sentence, and the agent has forgotten it by the fifth turn. Users experience that as the system not listening, which is the same complaint they make about poor turn-taking and an entirely different cause.
In voice specifically
Voice conversations run shorter than text conversations, which takes pressure off the window, and they are more likely to carry a constraint stated in the first sentence that decides the answer to the fifth question. That combination is why holding durable facts outside the conversation history matters more in voice than raw window size does.
Where AsqVox fits
Website voice conversations are typically short, which reduces context pressure, but a visitor frequently states their situation once at the beginning and expects it to hold. Structured state extraction is the relevant pattern for that shape of conversation.
Visual
What has to fit, and what gets pushed out
Bigger windows changed what fits. Not what is fast or affordable.
Statistics
Every figure carries its source and year. Vendor numbers are labelled as vendor numbers, and where no reliable figure exists this page says so rather than borrowing one.
Context window sizes change rapidly across model releases, and any specific figure published in a durable reference would be stale within months. Current provider documentation is the correct source.
-no reliable figureThis is a genuine editorial decision rather than a missing number, and it is worth stating on the page. A reference that quotes a window size is wrong within a release cycle and gets cited anyway.
Cost and latency both scale with tokens processed, which is why context economy remains an engineering concern regardless of how large the available window is.
Scales with tokensindustry rangeEngineering characteristic of token-priced inference, 2026 - The reason the window-size question is the wrong question. What fits stopped being the constraint; what is affordable and fast did not.
Attention degradation over long contexts is a well-documented characteristic of transformer models, commonly described as reduced effective attention to mid-context material.
Weakest in the middleindustry rangeWell-documented transformer characteristic, 2026 - The direction of the effect is established. The magnitude is model-specific, so cite the direction and do not attach a percentage to it.
Sub-second time to first audio is the working target, and prompt size sits inside it.
Under 1,000 msindustry rangeIndustry working threshold, 2026 - A working convention rather than a published standard. It is what turns a large prompt into a product decision instead of a configuration one.
Independent measurements put model time to first token between around 0.78 seconds for xAI Grok Voice and around 2.98 seconds for Gemini 3.1 Flash Live. Prompt size is one of the variables inside those figures.
0.78s to 2.98sindependentArtificial Analysis, via Softcery, April 2026, 2026 - These are time to first token, so a complete pipeline sits above them. Cite them as the model contribution to the budget, and remember that what you put in the prompt moves the figure.
Above roughly 1,500 milliseconds in total, conversations are widely reported to feel broken.
1,500 msindustry rangeIndustry working threshold, 2026 - A shared working line rather than a measured cliff. Filling a large window is one of the easiest ways to cross it without noticing.
Examples
In practice
A support agent handles long conversations poorly, losing account context after roughly a dozen turns. Rather than expanding the window, the team extracts the durable facts, being account type, region, product and stated problem, into a structured state object injected into every prompt at fixed cost. Conversation history is then summarized aggressively. Both latency and coherence improve, because the important information is now guaranteed present rather than competing for space.
The everyday version
The context window is how much the system can keep in mind at any one moment. Mention in your opening sentence that you are a returning customer, and unless it was built to keep hold of that fact, it may well have lost it by the time you reach your actual question.
Usage
Who says it
- AI engineers, constantly. It is a primary design constraint.
- Product teams, when they are scoping how long and how complex a conversation is allowed to get.
- Rarely a buyer term, though does it remember what I said earlier is the buyer version of the same question.
Where it turns up
- Next to model selection, maximum conversation length, memory behavior and session limits.
- In cost modeling, since tokens processed per turn is a direct cost driver.
Common misuse
- Assuming a large context window removes the need for retrieval. It changes the economics, not the requirement.
- Equating window capacity with effective use. Attention is uneven.
- Quoting specific window sizes in durable documentation. They change too fast.
Questions people ask
Does a bigger context window replace RAG?
No. It changes the economics, not the requirement. Cost and latency both scale with tokens processed, so sending a whole corpus with every turn is expensive and slow, and against a sub-second response target a large prompt is disqualifying whether or not it fits. Precision suffers too: fifty relevant passages produce better answers than five thousand mostly irrelevant ones.
Why does my voice agent forget what I said at the start?
Because conversation history competes with the system prompt, persona rules, tool definitions and retrieved passages for the same space, and as the conversation grows something gets pushed out. The usual fix is not a bigger window but structured state extraction: pull the durable facts, such as the visitor's name, intent and constraints, into explicit fields held outside the history so they are guaranteed present in every prompt.
How big is a context window?
Sizes change fast enough across model releases that any figure published in a durable reference is stale within months, so check current provider documentation rather than trusting a quoted number. The more useful question is not how much fits but how much is affordable, fast and actually attended to, because models attend unevenly across a long context.
What is attention degradation in a long context?
Models do not attend evenly to everything in the window. Material in the middle of a very long context receives less effective attention than material at the beginning or the end, which means filling the window is not the same as the model using it. The direction of the effect is well documented for transformer models. The magnitude is model-specific, so treat it as a design caution rather than a number.
Last reviewed 30 July 2026. Written and reviewed by Dhruv Dholakia, founder of AsqVox.