Conversation memory
In one sentence
Session state is what a voice agent remembers within a single conversation and conversation memory is what it remembers across conversations, and the two are different problems carrying different risks.
Not to be confused with Context window.
Definition
Session state is what an agent holds on to inside one conversation. Conversation memory usually means what carries over from one conversation to the next.
They get talked about as though they were one thing. They are two problems, and their risks are not comparable.
Telling the two apart is worth the trouble, because they get run together constantly and their privacy stakes are not the same.
Session state, within one conversation
- Conversation history, being the turns so far.
- Extracted entities, meaning the lasting facts stated over the conversation.
- Task progress, meaning how far the visitor has got through a multi-step flow.
- Page and behavioral context on a website: which page they are on, how far they scrolled, how long they stayed and where they came from.
- Users take session state for granted. State a constraint in your opening line and you expect it to stick, and when it does not the agent reads as not listening.
Cross-session memory, across conversations
- Lasting facts about someone who came back: what they asked before, what they said they preferred, what they bought.
- Useful, and a good deal more sensitive. It needs identity, and identity means either signing in or being tracked, each with its own fallout.
- The uncanny effect is genuine. Greet a returning visitor with something they do not recall telling you and it feels like being watched, not served, and that line arrives sooner than product teams expect.
- Where the law is consent-first, profiling an identifiable person over time is a lawful-basis question before it is a product one.
Three ways to hold session state
- Raw history. Hold every turn. Simple, eats context, and frays as the conversation runs on.
- Summarized history. Squeeze the older turns down. Keeps the gist, spends a model call, drops detail.
- Structured extraction. Lift the lasting facts into named fields kept apart from the conversation history and fed in at a fixed cost. More to build, far steadier, and the right default for voice.
Failure modes worth designing against
- Losing a stated constraint. The visitor opened by saying they were in India and gets quoted UK pricing by the end.
- Holding on to something already corrected. The visitor said Tuesday, switched to Wednesday, and the agent books Tuesday.
- Dragging stale state between sessions. A preference set six months back applied to a fresh situation.
- Spilling state between tenants or visitors, which is a security incident, not a bug.
Common misconception
That more memory is plainly better. Memory is a feature that comes with a privacy bill and a correction problem. Something that keeps everything and forgets nothing is harder to put right and more unsettling to meet than something with a scope you can define and explain.
Why it matters commercially
A session-state slip is the most familiar coherence failure in a voice agent, and users name it with the same phrase they use for bad turn-taking: it was not listening. Cross-session memory is a real differentiator and a real regulatory liability, and it belongs built on purpose rather than left to pile up.
In voice specifically
A text interface leaves the earlier turns on the screen, so a visitor can see what they already said and correct the record by scrolling up. Speech leaves nothing behind. Whatever the agent dropped is simply gone, and the visitor finds out about it when the answer comes back wrong. That is why holding durable facts in structured fields, rather than trusting them to survive inside the conversation history, is the right default for voice.
Where AsqVox fits
Website voice conversations tend to be brief, so session state is under less strain than in a drawn-out support call, though a visitor will still state their situation once and expect it to stick. A website agent can see page and behavioral context that a phone agent cannot, which is both an edge and a data-handling duty.
Visual
Two kinds of remembering, two kinds of risk
Remembering within a conversation is service. Remembering across them is a decision.
Statistics
Every figure carries its source and year. Vendor numbers are labelled as vendor numbers, and where no reliable figure exists this page says so rather than borrowing one.
India's DPDP Act 2023 had its Rules notified in November 2025, with phased full compliance by 13 May 2027. It is consent-primary, meaning persistent processing of personal data generally requires consent, and it carries fixed penalties up to Rs 250 crore rather than turnover-linked ones.
13 May 2027industry rangeIndia DPDP Act 2023, Rules notified November 2025, 2025 - Notably it does not create a separate category for biometric or voice data. Consent-primary is the part that decides cross-session memory: persistence is the thing that needs a basis, not the recording itself.
EU AI Act Article 50 transparency obligations become enforceable on 2 August 2026.
2 August 2026industry rangeEU AI Act, Article 50, 2026 - A disclosure obligation rather than a memory one. It governs what a person is entitled to know about who they are speaking to, which sits alongside the consent question rather than answering it.
Under GDPR, persistent profiling of identifiable individuals requires a lawful basis and triggers data subject rights including access and erasure.
Lawful basis requiredindustry rangeGDPR, 2026 - This is what makes cross-session memory a compliance design question and not only a product one. Erasure is the clause that catches architectures out, because deleting a visitor has to reach conversational stores, embeddings and logs.
Context window pressure grows with conversation length, and both cost and time to first token scale with the tokens processed. Against a sub-second time to first audio target, unbounded history is a performance problem before it is a memory problem.
Scales with tokensindustry rangeEngineering characteristic of token-priced inference, 2026 - The reason structured extraction wins on voice twice over. It is more reliable, and it is cheaper and faster than carrying every turn forward.
No published research establishes the point at which cross-session personalization becomes uncomfortable to users, despite that being the central design question.
-no reliable figurePractitioner judgement and testing are the only guides available. Anyone quoting a comfort threshold is quoting an opinion, and the threshold is lower than product teams assume.
No benchmark exists for session state retention accuracy in voice agents.
-no reliable figureWhich means the most recognizable failure in the category, forgetting what the visitor said at the start, has no shared measure. Instrument it yourself against your own transcripts.
Examples
In practice
A visitor opens by saying they are asking about the Indian market, then puts four follow-up questions. By the fifth turn the regional constraint has dropped out of the conversation history and the agent quotes UK pricing. The team pulls region, product and intent into a structured state object fed into every prompt at a fixed cost. The constraint can no longer go missing, because it is no longer fighting conversational history for room.
The everyday version
Session state is recalling what your customer said two minutes ago. Conversation memory is recalling them from last month. The first is simply listening properly, and customers count on it. The second is a weightier call, because some people welcome it and some find it creepy, and there are rules around it.
Usage
Who says it
- AI and product engineers, though the words slide around. Memory, state, context and history get thrown about loosely and swapped for each other, so ask which is meant.
- Privacy and compliance teams, specifically about cross-session persistence.
- Buyers, as "does it remember me", which asks about the second kind and deserves a careful answer.
Where it turns up
- Alongside data retention, personalization, authentication, data subject access and erasure, and consent management.
- Erasure is the clause that trips architectures up. Removing a visitor's data has to clear it from conversational stores, embeddings and logs.
Common misuse
- Running session state and cross-session memory together. Their privacy profiles differ, and so does what users expect.
- Treating memory as a clean feature win. It brings a correction problem and a consent obligation.
- Keeping transcripts forever by default instead of by decision.
Questions people ask
What is the difference between session state and conversation memory?
Session state is what the agent holds inside a single conversation: the turns so far, the facts stated, where the visitor has got to in a task, and on a website the page and behavioral context. Conversation memory usually means what persists between conversations, such as previous questions, stated preferences or prior purchases. Users expect the first, and a failure there reads as not listening. The second requires identity, which requires authentication or tracking, and that makes it a consent question rather than only a product one.
Why does a voice agent forget what I told it earlier?
Usually because the constraint was left in the conversation history, where it competes with the system prompt, the tool definitions and the retrieved passages for the same limited space, and it loses as the conversation grows. The fix that holds is structured extraction: pull the durable facts such as region, product and intent into explicit fields kept outside the history and injected at fixed cost, so they cannot be squeezed out by turn eight.
Is it legal for an AI agent to remember visitors between visits?
It depends on the regime and on whether the visitor is identifiable. Under GDPR, persistent profiling of identifiable individuals requires a lawful basis and triggers data subject rights including access and erasure. India's DPDP Act 2023 is consent-primary, with Rules notified in November 2025 and phased full compliance by 13 May 2027. Cross-session memory is therefore a compliance design question and not only a product one.
Should a voice agent remember returning visitors?
Only deliberately, and with a scope you can explain. The value is real and so is the discomfort: a returning visitor greeted with knowledge they do not remember providing experiences surveillance rather than service, and the threshold for that is lower than product teams assume. No published research establishes where the threshold sits, so testing and practitioner judgement are the only guides available.
Last reviewed 30 July 2026. Written and reviewed by Dhruv Dholakia, founder of AsqVox.