What Is AI Inference?

Inference is the model answering you. Training is the expensive thing that already happened; inference is the smaller cost paid every single time anyone asks anything.

Lathic's article on how AI works covers the whole arc from training to output. This one stays inside a single request: what happens in the seconds after you press send, why the words arrive the way they do, and why that shapes everything from response speed to usage limits.

Two phases, and they are different machines

A request splits cleanly in two.

First the system reads your entire prompt. This is called prefill, and it can be done all at once, because every word of your input already exists. The research describing this puts it simply: the input tokens "are all present at the start of the inference," so they can be processed in a single pass.

Then the system writes the answer. This is decode, and it cannot be parallelized, because each new word is chosen using the words already chosen. Generation "proceeds one token at a time and the computation for each token sequentially depends on the previously generated tokens."

That is why the text streams out word by word rather than appearing at once. It is not a visual effect. It is what is actually happening.

The two phases also stress the hardware differently. Google's own engineering guidance puts it in one line: prefill is compute-bound, decode is memory-bandwidth-bound. During generation, the limiting factor is how fast data moves between memory and the chip, not how fast the chip can multiply. NVIDIA's documentation says the same thing: "the speed at which the data is transferred to the GPU from memory dominates the latency, not how fast the computation actually happens."

This is the part most explanations get wrong. Generating slowly is a traffic problem, not a thinking problem.

What a token is, and why both directions are counted

Before the model sees your text, the text is split into tokens. A token can be a whole word, part of a word, a single character or a punctuation mark.

The rules of thumb differ slightly by provider, which tells you something. OpenAI says a token is roughly four characters, or about three quarters of an English word. NVIDIA uses about 0.75 words. Anthropic says roughly 3.5 English characters. None of these is exact, and OpenAI notes that "the same text can produce different token counts depending on the model, its encoding, and the language."

Both what you send and what you get back are counted, separately. Lathic's article on tokens covers what they are; the thing to carry into this article is that they are the unit everything else is measured in.

The cache, and why long conversations get heavy

To avoid redoing work for every new word, the system stores intermediate values from the words it has already processed. This is the KV cache, and it must be held in memory for the whole response.

It grows with how long the conversation is and how many people are being served at once. That growth is the practical limit on how many requests can run together: the research behind one widely used serving system notes that the cache "is huge and grows and shrinks dynamically," and that this is what limits batch size.

There is a second, blunter reason long conversations cost more, and it is documented rather than inferred. Every turn re-sends the whole conversation. Anthropic's context-window documentation states that each turn's input "contains all previous conversation history plus the current user message," and that "everything in the request counts toward the context window: the system prompt, every message, and your tool definitions."

A twentieth message in a long thread is not one message. It is the whole thread, again, plus one message.

The two numbers you actually feel

Time to first token is not purely model time. NVIDIA's definition notes it "generally includes both request queuing time, prefill time, and network latency." Part of your wait is other people's traffic.

A longer prompt makes the first token slower, because there is more prefill to do. A longer answer takes longer for the obvious reason, one token at a time.

The industry benchmark treats these as the interactive thresholds: MLPerf's server scenario for a small model allows up to two seconds to first token and 100 milliseconds between tokens, while its interactive scenario tightens that to half a second and 30 milliseconds. Those are benchmark constraints for a particular model size rather than universal standards, but they tell you roughly where a system stops feeling responsive.

You are not alone in there

Your request is not served by itself. Providers run many users' requests through the model together, which is what makes serving affordable at all.

Batching trades latency for throughput. Google's guidance states it plainly: "as batch size increases, throughput rises, but latency eventually degrades."

The clearest evidence that immediacy itself is what costs money is in the price lists. Both major developers offer a batch mode, where you submit work and collect it later, at a standing 50% discount. Nothing about the computation is cheaper. You are giving up the right to be answered now.

Why the same question can get a different answer

Turn the randomness setting to zero and ask twice, and you can still get different text. The common explanation is that GPUs are parallel and floating-point arithmetic is imprecise. That explanation is incomplete, and the real one is more interesting.

Repeated identical matrix multiplications on a GPU actually do give bitwise identical results. What varies is the batch. Server load changes from moment to moment, so the number of requests processed together changes, and the low-level operations are not batch-invariant. As one technical analysis puts it, "from the perspective of an individual user, the other concurrent users are not an 'input' to the system but rather a nondeterministic property."

Underneath it is a fact about arithmetic: floating-point addition is not associative, so the order in which numbers are summed changes the last digits. A separate 2025 preprint measured how much that matters, though it varied evaluation batch size, GPU count and GPU version rather than testing concurrent users directly. Changing those produced up to 9% variation in accuracy on one distilled reasoning model, and the authors note that for reasoning models "minor rounding differences in early tokens can cascade into divergent chains of thought."

Your neighbors' requests perturb your arithmetic. That is the actual answer.

What it costs, structurally

Specific prices change constantly and are not worth stating. The structure is stable and tells you more.

• Input tokens and output tokens are billed as separate line items, quoted per million tokens.
• Output costs several times more than input at both major developers.
• Reading from a cached prompt is much cheaper than sending it fresh, and writing to that cache costs more than plain input. Both major developers price it the same way: a cache write is 1.25 times the base input rate, and a cache read is 0.1 times it.
• Batch mode, where you accept an answer later, costs half.
• A model's private reasoning is billed. OpenAI's documentation states that reasoning tokens "are not visible via the API" yet "occupy space in the model's context window and are billed as output tokens." Anthropic's extended thinking documentation says thinking tokens count toward the turn's output limit.

That last point is the one that surprises people. If a model works through a problem before answering, you pay for the working even though you may never see it.

Why usage gets capped

The companies state their reasons, and it is worth using their words rather than guessing.

Anthropic's developer documentation says limits exist "to mitigate misuse and manage capacity on the API." OpenAI gives three reasons: protecting against abuse or misuse, ensuring "everyone has fair access to the API," and helping "manage the aggregate load on its infrastructure."

The mechanism matters as much as the motive. Limits are metered on tokens, not on how many messages you sent, and input and output are metered separately. Anthropic's limits are requests per minute, input tokens per minute and output tokens per minute. That is the bridge between everything above and what you experience: a few very long exchanges can exhaust an allowance that hundreds of short ones would not.

Consumer products work the same way. Anthropic's help documentation says usage depends on "the length and complexity of your conversations, the features you use, which Claude model you're chatting with, and the effort level you've selected," and that "longer conversations that trigger automatic context management consume more of your usage limit."

One correction worth making, because the intuitive story is wrong. It is not true that free tiers generally cap how many messages you can send. As of September 2026 OpenAI describes free-tier text chat as unlimited, "subject to abuse-prevention safeguards," with separate limits on file uploads, image generation, voice and data analysis. The capping story is about tokens and capacity, not message counts.

What one answer costs the grid

This is where published numbers disagree most, so the boundaries matter more than the figures.

Google measured the median text prompt to its Gemini apps in May 2025 at 0.24 watt-hours, 0.03 grams of CO2 equivalent and 0.26 milliliters of water. The useful part is the breakdown: the AI accelerator actively running the model accounted for 0.14 Wh, about 58%. CPU and memory added 0.06 Wh, idle provisioned machines 0.02 Wh, and data-center overhead 0.02 Wh.

Counting only the active accelerators, which is how most published estimates are made, gives 0.10 Wh. Google says of that narrower method that it "is an optimistic scenario at best and substantially underestimates the real operational footprint of AI."

Three caveats travel with that number and are usually dropped. It excludes training, external networking and user devices. It is a median, chosen deliberately because the distribution is skewed by a minority of long, expensive prompts, which means the average prompt costs more. And Google notes the findings do not indicate future performance.

An independent bottom-up estimate published in Joule in April 2026 by Microsoft researchers put optimized frontier-scale inference at a median of 0.31 Wh per query, and found that long reasoning and agentic queries "increase energy consumption by more than an order of magnitude."

For scale, two organizations arrived independently at the same comparison. The International Energy Agency writes that "simple text queries now typically consume less electricity than running a television over the same period of time." Google's own version: the median prompt uses less energy than watching nine seconds of television.

What has definitely changed is the balance. The IEA states that "the balance of AI-related energy consumption has already shifted decisively from training to inference." It gives no percentage, and neither will this article, because no authoritative source publishes one. The widely repeated claim that inference is 80% or 90% of AI compute does not trace to a primary source.

Lathic's article on AI and the environment covers the wider picture, including data-center totals, which are a different measurement from per-query energy and should not be read as one.

What this explains

Once you see the shape of a single request, several things stop being mysterious.

The first word takes longest because prefill has to finish first. The rest arrive one at a time because each depends on the last. Long conversations get slower and more expensive because the whole thread is resent every turn and the cache grows. The answer varies slightly between runs because your request is batched with other people's. And limits exist because every one of these is paid for again, per request, forever.

Training was the thing that happened once. This is the thing that happens every time.