Distillation
In one sentence
Distillation trains a smaller model to imitate a larger one, producing a compact model that performs close to the original on the tasks it was trained to imitate, at a fraction of the cost and latency.
Definition
Distillation trains a small model to copy a big one. The result runs close to the original on the jobs it was taught, for far less cost and far less delay.
It transfers capability, not data. In voice, almost all of the value comes from the drop in latency.
How it works
- A large teacher model produces outputs on a set of inputs.
- A smaller student model is trained to reproduce those outputs.
- The student learns from the behavior of the teacher, including, in some approaches, the full probability distributions of the teacher rather than only its final answers.
- The student ends up substantially smaller while retaining much of the performance of the teacher on the covered task distribution.
Why the task-distribution caveat matters
- The student performs well on tasks resembling those it was distilled on and degrades outside them.
- A distilled model is therefore a specialist wearing the clothes of a generalist. It answers confidently on anything and is only reliable on what it learned.
- That makes distillation excellent for narrow pipeline stages and risky as a general replacement.
Where it fits in a voice pipeline
- Turn completion prediction, query rewriting, intent classification, guardrail screening, response compression.
- These are exactly the narrow, high-frequency tasks where latency matters most and where a frontier model is expensive overkill.
The commercial appeal in voice
- Latency reduction is the primary driver. A distilled model can serve a pipeline stage in a fraction of the time.
- Cost reduction follows, and it compounds because these stages run on every turn.
- On-device deployment becomes plausible for some stages, removing network round trips.
The legal and contractual caveat
- Model provider terms frequently restrict using outputs to train competing models.
- Distilling from a commercial API may breach those terms.
- This is a genuine constraint rather than a theoretical one, and it should be checked before the work rather than after.
Common misconception
That a distilled model is simply a smaller version of the teacher. It is a model trained to imitate the teacher on a specific distribution, and its behavior outside that distribution is not guaranteed to resemble the teacher at all.
Why it matters commercially
Distillation is one of the main techniques behind the small, fast models that make low-latency voice economically viable. For a buyer it is mostly invisible. For a platform operator it is a significant lever.
In voice specifically
The whole reason distillation shows up in voice is latency. The pipeline stages it suits run on every single turn, so a fraction of a second saved on each is heard directly by the visitor as a faster reply. In text, the same saving is barely noticed.
Where AsqVox fits
Relevant as infrastructure. The pipeline stages that run on every turn are the natural candidates, and a latency improvement there is felt directly by the visitor.
Visual
Teaching a small model to imitate a large one
Capability transferred, within a boundary. Outside it, all bets are off.
Statistics
Every figure carries its source and year. Vendor numbers are labelled as vendor numbers, and where no reliable figure exists this page says so rather than borrowing one.
Model time to first token spans roughly 0.78 to 2.98 seconds across leading realtime models, and distilled models generally target the faster end.
0.78 to 2.98sindependentArtificial Analysis, reported via Softcery, 2026 - Sub-second time to first audio is the working target. Above roughly 1,500 milliseconds, conversations feel broken. Distillation is one of the levers that moves a pipeline stage toward the faster end.
All-in production voice cost lands around USD 0.11 to USD 0.33 per minute, with model inference as a controllable component.
$0.11 to $0.33 per minuteindustry rangeIndustry-reported production voice cost range, 2026 - Distillation reduces the inference component. Because the suited stages run on every turn, the per-minute saving compounds across a conversation.
There is no published benchmark comparing distilled against full models on voice-specific pipeline tasks.
-no reliable figureGeneral distillation benchmarks exist, but none maps to turn completion, query rewriting or the other narrow voice stages where distillation is actually deployed.
Provider terms regarding distillation vary and change, so any statement about what is permitted should direct readers to current terms rather than assert a position.
-no reliable figureThe restriction on training competing models from outputs is common but not universal, and it moves. Read the live terms of the provider you intend to distill from.
There is no standard measure of how far the performance of a distilled model degrades outside its distillation distribution.
-no reliable figureThis is the exact risk that makes a distilled model unsafe as a general replacement, and there is no agreed metric for it. Treat out-of-distribution behavior as unknown.
Examples
In practice
A platform distils a small model for turn completion prediction, a task running on every turn of every conversation. The distilled model matches the teacher closely on that task and runs in a fraction of the time. Median time to first audio improves measurably across all conversations, because the saving applies to every turn rather than to a subset.
The everyday version
Distillation is training a small, fast model to copy a big, slow one. The small copy is very good at the specific jobs it was taught and unreliable outside them, while still sounding confident. It is one of the reasons voice agents got fast enough to be worth using.
Usage
Who says it
- ML researchers and engineers use it precisely.
- It appears in model release documentation, explaining how smaller variants were produced.
- It is rarely a buyer term.
Where it turns up
- In model documentation and architecture decisions.
- In provider terms of service, as a restricted activity.
Common misuse
- Treating a distilled model as a general-purpose replacement for its teacher.
- Distilling from a commercial API without checking provider terms.
- Assuming performance transfers to tasks outside the distillation distribution.
Questions people ask
What is model distillation?
Distillation trains a smaller student model to imitate a larger teacher model. The student learns to reproduce the outputs of the teacher, sometimes down to its full probability distributions, and ends up substantially smaller while keeping much of the performance on the tasks it was trained to copy. The payoff is lower cost and lower latency.
Is a distilled model just a smaller version of the original?
No. It is trained to imitate the teacher on a specific task distribution, and its behavior outside that distribution is not guaranteed to resemble the teacher at all. It performs like a specialist on what it learned and answers confidently, but unreliably, on everything else.
Why do voice pipelines use distilled models?
For latency. The stages that suit distillation, turn completion prediction, query rewriting, intent classification, guardrail screening and response compression, run on every turn, so shaving a fraction of a second off each is heard by the visitor as a faster reply. A frontier model on those stages is expensive overkill.
Is it legal to distil from a commercial API?
It depends on the terms of that provider, which frequently restrict using outputs to train competing models and do change over time. This is a real contractual constraint, not a theoretical one, so check the current terms before doing the work rather than after.
Last reviewed 3 August 2026. Written and reviewed by Dhruv Dholakia, founder of AsqVox.