Home / Chapter 3 · How a model writes and what it costs
Last edited · 7 min read
The generation loop and KV cache
Generation is a loop: one forward pass of the model yields one token, which is appended to the input. The KV cache holds the attention keys and values of all earlier tokens, so each step computes only the new token. The price is GPU memory, which limits context length and the number of concurrent conversations.
In plain wordsYou are writing a letter, and with every new word you glance at notes on what you have already written. You don’t reread the letter, but you do look through the notes every time. The KV cache is those notes. They sit on the desk, that is, in GPU memory, and the longer the letter, the more room they take.
Step through with and without the cache. Compare how many tokens each step computes
What the cache holds and why it works
- In every attention layer a token has three projections of its state: query Q, key K and value V (see “Attention”). A new token compares its Q with the keys of all earlier tokens and takes a weighted sum of their values. The KV cache is the K and V of all tokens so far, kept separately for every layer and KV head.
- Q is not kept. A token’s query is needed only in the step where that token is the last one. Later tokens read only its K and V.
- Keeping them is valid thanks to the causal mask: the K and V of token i depend only on tokens 0…i, so new tokens don’t change them. This is memoisation. Without it, step n would recompute all n tokens, so generating N tokens would take work growing with N² instead of N.
- It also works the other way round: a token’s K and V depend on all tokens before it and on its position (RoPE). Changing one token invalidates the cache for everything after it. That is why a provider can reuse a cache only for an identical prefix, and why an agent only ever appends to its history (see “Prompt caching”).
How much memory it takes
- Per token: 2 (K and V) × layers × KV heads × head dimension × bytes. Llama 3.1 70B in FP16: 2 × 80 × 8 × 128 × 2 B = 327,680 B, about 0.33 MB. A full 128k tokens is 43 GB for a single conversation, against 141 GB for the weights alone.
- The architecture sets the shape of the cache. GQA keeps one K, V pair per group of query heads (MQA is the extreme case: one pair per layer). Llama 3 70B has 8 KV heads for 64 Q heads; without GQA a 128k-token cache would take 344 GB. DeepSeek’s MLA stores one compressed vector per layer: in DeepSeek-V3 that is 61 × 576 × 2 B ≈ 70 KB per token, almost 5 times less than Llama 70B, even though the model has 671B parameters.
- Sliding window: some layers see only the last W tokens, so their cache never grows beyond W. Gemma 3 has five such layers (W = 1024) for every full one, which at 128k tokens comes to about 17% of the cache of the same model with only full layers. Hybrid models replace some attention layers with SSM layers that have a fixed-size state.
- When serving an existing model, precision is the lever left: an FP8 cache takes half the memory. It is an approximation, so check quality on your own evals.
Pick a model, context length and number of conversations. See how many H100s the cache alone takes up
Formula: 2 × layers × KV heads × head dimension × bytes × tokens. Instead of K and V, MLA stores one vector of 576 numbers per layer, so there is no factor of 2. In Gemma 3 every sixth layer is full and the others keep at most the last 1024 tokens. Model weights and server overhead not included. The slider stops at 128k tokens, the longest context these models support.
Prefill and decode
- Prefill processes the whole prompt in a single pass. It multiplies the weights by a matrix of all tokens at once, so every byte of weights read serves hundreds of tokens and the GPU computes at full speed. This is where the prompt’s cache is built. Prefill sets time to first token (TTFT), and attention grows with the square of the length, so a very long prompt increases TTFT faster than linearly.
- Decode produces one token per conversation per step and multiplies the same weights by a single vector. In FP16 that is about 1 operation per byte of weights for each conversation in the batch, while an H100 needs about 300 before computation becomes the bottleneck (see “Why the GPU is idle”). Step time ≈ (weights + cache of all conversations in the batch) / memory bandwidth.
- Weights are read once for the whole batch, but each conversation’s cache separately. That is why servers pack many conversations into one step (see “Continuous batching”), why one conversation’s long context slows down the whole batch, and why providers charge several times more for an output token than for an input token.
Increase the context and the number of conversations in the batch. See when reading the cache starts to outweigh reading the weights
An illustrative model as in “Why the GPU is idle”: 8B parameters in FP16 (16 GB of weights, 128 KB of cache per token), a single H100 with 80 GB of memory and 3.35 TB/s. Step time = (weights + cache of all conversations) / bandwidth. This is an upper bound on speed, with no overheads.
The cache on the server
- PagedAttention (vLLM) splits the cache into blocks of a dozen or so tokens, like virtual-memory pages. The server doesn’t reserve space for the maximum length, and a prefix shared by many conversations is stored once. Prompt caching is the same blocks kept after the request and looked up by a hash of the prefix.
- When memory runs out, the server pauses some conversations and later either recomputes their cache or offloads it to RAM or disk and loads it back. Both cost time but don’t change the result.
- Evicting some tokens from the cache (StreamingLLM, H2O: the first, the most recent and the most attended tokens stay) saves memory but is an approximation. The model loses access to the evicted text, so quality on long tasks drops.
Check yourself
What is the KV cache, and how does prefill differ from decode?
Generation is a loop: one forward pass yields one token. Attention needs the keys and values of all earlier tokens, and thanks to the causal mask they never change, so they are computed once and kept in GPU memory. Prefill processes the whole prompt in parallel, is compute-bound and sets time to first token. Decode goes token by token and is memory-bound, because every step reads the weights and the whole cache. The cache is about 0.33 MB per token for a 70B model with GQA, so it is what caps context length and batch size. GQA, MLA, sliding windows and FP8 shrink it.
Po polsku
Generowanie to pętla: jeden przebieg modelu daje jeden token. Attention potrzebuje kluczy i wartości wszystkich wcześniejszych tokenów, a przez maskę przyczynową one się nie zmieniają, więc liczy się je raz i trzyma w pamięci GPU. Prefill liczy cały prompt równolegle, jest ograniczony obliczeniami i wyznacza czas do pierwszego tokena. Decode idzie po tokenie i czeka na pamięć, bo każdy krok czyta wagi i cały cache. Cache to ok. 0,33 MB na token dla 70B z GQA, więc to on ogranicza kontekst i batch. Zmniejszają go GQA, MLA, okno przesuwne i FP8.
Follow-up questions (4)
- What determines time to first token, and what determines writing speed?
- Time to first token depends on the queue and on how much of the prompt has to be computed outside the cache hit, because that is prefill. Writing speed depends on the size of the weights, the context length, the number of conversations in the batch and memory bandwidth, because that is decode.
- Why does a long context slow down generation if each step computes only one token?
- Because every step reads the whole cache. For a 70B model at 128k tokens that is 43 GB per step, almost a third of the weights’ size, and decode is bound by memory bandwidth.
- vLLM runs out of memory on long contexts. What do you do?
- Work out the budget: weights plus cache per token × maximum context × number of concurrent sequences. Then cap the maximum length and the number of sequences, enable an FP8 cache and prefix sharing, and if that is not enough, split the model across more GPUs. Tensor parallelism splits a GQA cache by KV head, so across at most as many GPUs as there are KV heads (8 in Llama 3.1 70B). An MLA cache is replicated on every GPU, which is why DeepSeek-style models use data-parallel attention instead.
- How do you shrink the KV cache?
- At model design time: GQA or MQA, MLA, sliding windows, SSM layers. At serving time: FP8, prefix sharing, offloading to RAM, evicting tokens. Only sharing and offloading leave the output unchanged.