Inference
In one sentence
Inference is running a trained model to produce an output, as distinct from training which builds the model, and in production it is where nearly all the cost and latency live.
Definition
Training is building the model. Inference is using it. Every time someone talks to your agent, the model runs inference to produce the answer.
You almost certainly do no training. You pay for inference, and you pay for it on every single request, forever.
Training and inference have completely different economics. Training happens once, or occasionally, and is enormously expensive. Inference happens on every request and its cost scales with usage. Most public discussion is about training costs; almost all business cost is inference.
Two phases inside a single inference call
- Prefill. Processing the entire input prompt. It is parallelizable and compute-intensive, and it determines time to first token.
- Decode. Generating the output one token at a time. It is sequential and memory-bandwidth bound, and it determines how fast the rest of the response arrives.
- These have different performance characteristics, which is why a long prompt with a short answer behaves nothing like a short prompt with a long answer.
Where inference happens
- Hosted API. The common case. Simple, no infrastructure, provider-dependent.
- Self-hosted. More control over latency, cost at scale and data handling, but it needs hardware and expertise.
- On-device. Removes the network round trip entirely, which is compelling for latency, and constrains model size severely.
What determines inference latency
- Model size and architecture, input length through prefill, and output length through decode.
- Hardware and available memory bandwidth.
- Batching. Providers batch requests for efficiency, which improves throughput and can increase the latency of an individual call.
- Queueing under load, which is invisible to the caller and shows up in the latency tail.
Optimization techniques worth knowing by name
- Quantization, reducing numerical precision to cut memory and increase speed.
- Caching of stable prompt prefixes, which removes repeated prefill cost.
- Speculative decoding, where a small model drafts tokens a large model verifies.
- To a buyer these mainly explain why providers differ on latency at similar capability.
Common misconception
That inference cost is falling to zero. Per-token costs have fallen substantially, and usage has grown faster still. The variable cost structure of a voice product is durable, not a temporary phase on the way to free.
Why it matters commercially
Inference is the recurring cost in voice unit economics and the primary determinant of latency. It is where architectural decisions translate directly into margin, and where a slow provider is heard as silence before the agent speaks.
In voice specifically
In voice the two phases of a call are audible. Prefill sets time to first token, which the listener experiences as the pause before any sound. Decode sets how fast the rest arrives, which is the difference between natural speech and a stuttering delivery. A long system prompt taxes prefill and is heard as a longer opening silence.
Where AsqVox fits
Inference cost per conversation is a direct input to voice unit economics, and prompt discipline and retrieval discipline are the controllable levers on it.
Visual
Built once, run forever
Training is a project. Inference is a bill that arrives every month.
Statistics
Every figure carries its source and year. Vendor numbers are labelled as vendor numbers, and where no reliable figure exists this page says so rather than borrowing one.
All-in production voice cost lands around USD 0.11 to USD 0.33 per minute, of which model inference is a major component.
USD 0.11 to 0.33 per minuteindustry rangeIndustry-reported production voice cost range, 2026 - A reported operating range rather than a single benchmark. Inference is one line inside it, alongside speech recognition, text to speech and telephony.
Platform base rates run around USD 0.05 per minute for Vapi, USD 0.07 for Retell and USD 0.09 for Bland.
Vapi about 0.05, Retell 0.07, Bland 0.09 per minvendor claimVapi, Retell and Bland published platform rates, 2026 - Platform-published, so read as a base rate before model, telephony and provider markups. The base rate is not the whole bill.
Independent figures show model time to first token spanning roughly 0.78 to 2.98 seconds across leading realtime models. Time to first token is set by prefill; subsequent generation speed is set by decode.
0.78 to 2.98 secondsindependentArtificial Analysis, via Softcery, 2026 - Measured in April 2026. The near four-fold spread across capable models is the reason provider choice moves opening latency more than model choice does.
Provider inference pricing changes frequently, and any published figure ages quickly.
-no reliable figureCite a rate with the month you read it. A per-token or per-minute number quoted without a date is a number you cannot rely on.
Providers do not generally publish their queueing or batching behavior, so the latency tail cannot be predicted from documentation.
-no reliable figureThis is why identical requests can vary in time to first token: the variation tracks provider load, which is invisible to the caller.
There is no published benchmark isolating prefill performance from decode performance across providers.
-no reliable figureThe two phases have different bottlenecks, but no public number separates them, so a long-prompt short-answer workload cannot be sized from vendor docs.
Examples
In practice
A team observes time to first token varying widely for identical requests. The model and the prompt are unchanged. The variation tracks provider load, which affects queueing and batching, and is entirely outside the control of the team. The remedy is either a provider with capacity guarantees or an architecture that degrades gracefully when the tail stretches.
The everyday version
Inference is the AI actually running and producing an answer. Training is building it in the first place, which someone else paid for. You pay for inference every single time someone talks to your agent, which is why the cost never stops.
Usage
Who says it
- AI and infrastructure engineers say it routinely, and treat training and inference as separate line items.
- Finance teams use it in cost modeling, because inference is the variable cost that scales with call volume.
- It appears in provider documentation and pricing.
Where it turns up
- On a spec sheet it sits next to pricing, latency, throughput, hardware requirements and deployment options.
- The useful question is not the headline rate but what the number does under load, because queueing lives in the tail.
Common misuse
- Conflating training cost with operating cost. The business pays the second, not the first.
- Assuming inference costs will fall to insignificance. Usage has grown faster than per-token price has dropped.
- Treating latency as deterministic when provider queueing makes it variable.
Questions people ask
What is the difference between training and inference?
Training builds the model and happens once or occasionally at enormous cost. Inference runs the finished model to produce an output, and it happens on every request. A business deploying a voice agent almost always does no training and pays only for inference, which is why inference is the cost that never stops.
Why is my time to first token inconsistent for the same request?
Because the model and prompt are not the only inputs. Providers batch requests and queue them under load, and both are invisible to you. Identical requests can land at different provider loads, which changes queueing and batching and moves the latency tail. Documentation will not predict it, so a provider with capacity guarantees or a design that degrades gracefully is the remedy.
What are prefill and decode?
They are the two phases of a single inference call. Prefill processes the whole input prompt, is compute-intensive and parallelizable, and sets time to first token. Decode generates the output one token at a time, is sequential and memory-bandwidth bound, and sets how fast the rest of the answer arrives. A long prompt taxes prefill; a long answer taxes decode.
Is inference getting cheap enough to ignore?
No. Per-token costs have fallen a lot, but usage has grown faster, so the total variable cost of a voice product is durable rather than disappearing. Inference remains the recurring cost in unit economics and the main determinant of latency, which is why prompt and retrieval discipline are worth the effort.
Last reviewed 3 August 2026. Written and reviewed by Dhruv Dholakia, founder of AsqVox.