Speech-to-speech models
In one sentence
A speech-to-speech model is a single model that consumes audio and emits audio directly, with no transcription step in the middle, which buys latency and preserved tone at the cost of the text record a cascaded pipeline produces as a by-product.
Not to be confused with Automatic speech recognition, or Text to speech.
Definition
A speech-to-speech model hears audio and answers in audio, with no step in between that turns what you said into text.
The older arrangement chains three models together. This is one model, and it can hear that you sound annoyed.
The pipeline most systems still run is a cascade: recognize the speech as text, reason over that text, then synthesize the reply back into sound. Every handoff costs time and throws information away. A speech-to-speech model, sold variously as end-to-end audio or native audio, is a lone model, trained on audio tokens, that hears speech and answers in speech. Omni and multimodal models stretch the same idea across text, audio and sometimes vision inside one architecture.
What is gained
- Latency. Two handoffs between models disappear from the response path, and the text bottleneck goes with them.
- Paralinguistic information. How something was said, and not only what was said, survives into the reasoning step: tone, hesitation, emphasis, emotion, urgency, accent. Transcribe an annoyed sentence and a cheerful one and the words come out the same, which is exactly why a cascade cannot separate them.
- Expressive output. Sound is what the model produces, not a script it performs, so it can match tone, laugh, soften or pick up the pace.
- Overlap and timing. Turn dynamics are part of what the model learned, so interruption and backchannel come out more naturally.
The side of the ledger that tends to get skipped
- Inspectability. Nothing writes down what the model understood, because there is no text stage to write it at, so the audit record is thinner than the one a cascade produces for free. Compliance teams and anyone debugging a bad call both feel that immediately.
- Controllability. There is no text interface to reach into, which makes retrieved knowledge harder to inject, guardrails harder to enforce and output harder to constrain.
- Grounding. Retrieval is a text operation at heart, and wiring genuine retrieval into an unbroken audio route remains something teams are still working out rather than something anyone has finished.
- Cost predictability. Audio tokens are not priced the way text tokens are, and the same exchange can come out considerably more expensive.
How production systems actually settle it
- Hybrid architectures are the pragmatic norm today. Native audio carries the conversational path while a parallel transcription stream feeds logging, retrieval and guardrails.
- Nearly all of the speed and the expressiveness survive, and so does a text record somebody can audit. What you have built is more complicated than either pure design, not simpler.
Common misconception
That the cascade is now obsolete. Consider a business agent that may answer only from documents its owner uploaded and has to leave a defensible record of every call: for that job the text stage the cascade gets criticized for is doing useful work. Choose the architecture by deciding which you would rather have, a conversation that sounds alive or answers you can check.
Why it matters commercially
No other architectural change in the category comes close in scale, and nobody serious disputes where this is heading. That still leaves a practical question for anyone buying an agent to field questions about their own prices and policies: get the vendor to walk through retrieval and audit logging on a native-audio stack specifically, because the rough edges are all in that corner.
In voice specifically
Everything this architecture gains is information that only exists in speech. Tone, hesitation and urgency have no representation in a transcript, so a text-first pipeline does not lose them late. It loses them at the first step, and nothing downstream can recover them.
Visual
Cascade against native audio
The cascade's bottleneck is also its audit trail.
Statistics
Every figure carries its source and year. Vendor numbers are labelled as vendor numbers, and where no reliable figure exists this page says so rather than borrowing one.
Measured time to first token across leading realtime models: around 0.78 seconds for xAI Grok Voice, 0.82 for OpenAI gpt-realtime-1.5, 1.14 for Amazon Nova 2 Sonic, and 2.98 for Gemini 3.1 Flash Live.
0.78s to 2.98sindependentArtificial Analysis, via Softcery, April 2026, 2026 - Two cautions when citing. The clock here stops at the model's first token rather than at audible speech, so anything actually shipped sits above these figures. And with the fastest and slowest of the leading realtime models almost four times apart, picking a model can settle the sub-second question on its own.
Sub-second time to first audio is the design goal these measurements are read against.
Under 1,000 msindustry rangeIndustry working threshold, 2026 - A working convention rather than a published standard. No body sets it and no vendor is audited against it.
Past roughly 1,500 milliseconds, an exchange starts to feel broken.
1,500 msindustry rangeIndustry working threshold, 2026 - Same status as the sub-second target: a shared working line rather than a measured cliff.
The human conversational turn gap averages around 200 milliseconds.
200 msindustry rangeWidely cited conversational turn-taking baseline, 2026 - A reference point, not a target. Cite it as the human baseline rather than as a measurement of anything you built.
Nobody has published an independent test of how accurately each architecture answers from a grounded source, which is precisely the comparison a business buyer is trying to make.
-no reliable figureThe latency comparison is measured in public and the accuracy comparison is not. That asymmetry is worth naming whenever a vendor leads with the latency half.
Audio token prices cannot be laid side by side across providers, because each one tokenizes differently.
-no reliable figureCost comparisons need modeling on representative traffic rather than a reading of rate cards. Two providers can quote per-token prices that simply do not describe the same thing.
Examples
In practice
A team swaps its cascade for a realtime audio-in, audio-out API and median time to first audio drops by several hundred milliseconds. Launch is then blocked by compliance, who point out that nothing in the new stack can say reliably what the caller was understood to have asked. They settle on a hybrid: the native-audio path keeps carrying the conversation, and a second transcription stream runs beside it doing nothing but logging and retrieval. The speed is kept, the audit box is ticked, and the system is now more involved than either design it was chosen over.
The everyday version
Picture the old way as a relay of interpreters. Someone takes down your words, hands the note to a colleague who drafts a reply, and a third person reads that draft aloud to you. The new way is one person listening and replying. Quicker, and this one can hear that you are annoyed. The snag is that no note was ever taken.
Usage
Who says it
- Platform architects and AI engineers, plus a growing number of vendors who now lead a datasheet with it.
- The vocabulary has not settled. You will hear omni model, realtime audio, native audio, S2S and speech-to-speech traded as though they named one thing, so pin down which is meant before agreeing to anything.
Where it turns up
- In architecture decision records, model release notes and the platform documentation.
- In an RFP it shows up sideways, buried in clauses about audit logging, interruption handling, emotion detection and latency. Ask for sub-second response and a complete transcript trail in one document and you have specified a hybrid, whether or not you used the word. A good response says so outright.
Common misuse
- Selling it as a straight improvement. Speed and expressiveness are bought with inspectability, and a trade is not an upgrade.
- Claiming grounded retrieval on an unbroken audio route while leaving out where the retrieved text actually enters. Put that question to them plainly.
- Treating omni and speech-to-speech as one word. Omni is about handling several input types at once. Speech-to-speech names one particular route: audio in, audio out.
Questions people ask
What is the difference between a speech-to-speech model and a cascaded voice pipeline?
A cascade runs three models in a chain: speech recognition turns audio into text, a language model reasons over the text, and text-to-speech turns the reply back into sound. A speech-to-speech model does the whole thing in one pass over audio tokens. The cascade is slower and flattens tone into plain words, but it produces a transcript at the reasoning step, which is what audit logging, retrieval and guardrails attach to.
Is speech-to-speech better than a cascaded pipeline?
Not universally. It buys latency, preserved tone and more natural interruption handling, and it costs inspectability, controllability and easy retrieval grounding. For a business agent restricted to its own documents and required to leave a record behind, that text step in the cascade is partly a feature. Most production systems run a hybrid rather than choosing one side.
Can a speech-to-speech model use RAG?
Retrieval works on text, so getting it into an unbroken audio route is still an open problem rather than a finished one. The usual workaround is the hybrid: native audio carries the conversation while a second transcription stream feeds retrieval, logging and guardrails. Where a vendor claims grounding without that second stream, ask exactly where the retrieved text enters.
How fast are speech-to-speech models?
Measurements published by Artificial Analysis via Softcery in April 2026 put time to first token at around 0.78 seconds for xAI Grok Voice, 0.82 for OpenAI gpt-realtime-1.5, 1.14 for Amazon Nova 2 Sonic and 2.98 for Gemini 3.1 Flash Live. Those are model-level figures, so a deployed system lands higher. With almost four times separating fastest from slowest, model choice alone can settle whether you come in under a second.
Last reviewed 30 July 2026. Written and reviewed by Dhruv Dholakia, founder of AsqVox.