How does a trained model write its answer?
In training the toy saw the whole sentence at once, the answer included, and was graded on every word in one pass. Now I type “Diary of Jane is a song by” and stop. The model has seven words and nothing after them.
So it writes the answer the only way it can: one word per pass. Read the prompt, turn the last row’s scores into one word, append it, and run again until something says stop. Three passes later the toy has written “Breaking Benjamin .”
The film walks that loop with the toy’s numbers, then opens the step everyone tunes and few can explain: how a list of probabilities becomes one word. Greedy, temperature, top-k and top-p are four answers to that question, and none of them is part of the model.
The 30-second version
The model only returns a probability for every token; a decoding rule picks one. Greedy takes the most likely token: deterministic, good for short factual answers, prone to dull loops in long text. Sampling draws from the distribution, after reshaping it. Temperature divides the logits by T before the softmax: below 1 it sharpens the distribution, above 1 it flattens it. Top-k keeps the k most likely tokens and renormalises. Top-p keeps the smallest set whose probabilities reach p, so it keeps few tokens when the model is sure and many when it is not. The truncations exist because a raw draw from the full tail picks something the model itself thinks unlikely, and over a long answer that happens almost surely.What happens in prefill, and in each decode step?
Prefill runs the whole prompt in one pass, mask on, exactly like a training pass. The seven rows that come out are the same numbers the training pass computed for them, to the last decimal, because the mask kept each row blind to everything after it. That is why one parallel training pass teaches a model that is then used one word at a time: row $i$ computes the same function in both.
Only the last row speaks. The row for “by” gives “Breaking” 0.61, and the six rows before it are thrown away, apart from the keys and values each layer keeps for later.
Then decode: append the word and run again. “Breaking” in, “Benjamin” out at 0.39; “Benjamin” in, the full stop out at 0.53. Each pass adds one word, and the newest row is the only one whose prediction is used.
The loop needs an exit. In the toy the full stop is the stop token. A chat model ends its reply with a special end-of-turn token, learned like any other word, and the server always sets a length limit in case it never comes.
Without a cache, every pass recomputes the old rows. Pass 2 runs all eight words to get one new row, pass 3 all nine: 24 rows for three words24 against 9, worked outno cache: 7 + 8 + 9 = 24 rowskeep each row’s keys and values: 7 + 1 + 1 = 9 rowsThe old rows cannot change: the mask keeps them blind to the new word., where 9 would do. Keeping every row’s keys and values is the KV cache, and the new row’s query reads them directly. That single row needs no mask at all: it has no later columns to hide. The KV Cache series starts from exactly this.
Prefill and decode also stress a GPU differently. Prefill pushes every prompt row through each weight it reads, like a training step, so on a long prompt arithmetic is the limit. A decode step pushes one row per sequence through every weight, so reading the weights is the limit, which is why servers batch many users’ decode steps together.
How does a decoding rule pick the word?
The last row hands back eleven probabilities. Greedy takes the top one: “Breaking”, every time, for everyone. Every other rule reshapes the distribution and then draws from it.
Temperature divides the scores before the softmax:
\[p_i(T) = \frac{e^{z_i / T}}{\sum_j e^{z_j / T}}\]At $T = 1$ it is the plain softmax. Below 1 every gap between scores grows, and at $T = 0.5$ “Breaking” runs away to 0.850.85, worked outBreaking’s score: 3.434, and 3.434 ÷ 0.5 = 6.868e^6.868 = 960.5; e to every halved score, added: 1,134.4960.5 ÷ 1,134.4 = 0.847At T = 1 the same sum is 30.99 ÷ 50.65 = 0.61.. Above 1 the gaps shrink: at $T = 2$ “Breaking” drops to 0.34 and “Jane” climbs to 0.22, so one draw in five says the song is by Jane.
Top-k keeps the $k$ most likely words and top-p the fewest whose probabilities reach $p$. Either way the survivors are renormalised and one is drawn:
\[p_i^{\prime} = \frac{p_i}{\sum_{j \in S} p_j}, \qquad S = \text{the kept words}\]With $k = 2$ the toy keeps “Breaking” and “Jane”: 0.71 and 0.29top-2, worked outkept: Breaking 0.612, Jane 0.255, together 0.8670.612 ÷ 0.867 = 0.706, 0.255 ÷ 0.867 = 0.294. With $p = 0.9$ it keeps three words after “by”, which reach 0.90, but five after “Breaking”, which reach 0.930.93, worked outBenjamin 0.393, is 0.314, of 0.094, “.” 0.086, a 0.039running total: 0.393, 0.707, 0.801, 0.887, 0.926The fifth word is the first to pass 0.9.. Top-p keeps more when the model hesitates, which is its whole point.
That hesitation is real. After “Breaking”, “is” holds 0.31, so a sampler writes “by Breaking is …” about one time in three. Training never showed the model its own mistakes, only true prefixes, so once a wrong word is in, it has to continue from a place it has never been. That gap between training and generation is called exposure bias.
One practical detail for batches. Generation pads short prompts at the front, so every sequence’s next word lands in the same column. Those pads now sit in the past of every real row, where the causal mask lets them through, so a padding mask adds $-\infty$ to their columns as well.
Rapid fire: can you do these from memory?
- What is computed in prefill and what in one decode step, and why are the prefill rows the same numbers as in training?
- Temperature 0.5 against temperature 2: which sharpens, and what happens to the top word's probability?
- Top-k against top-p: which one keeps more words when the model is unsure, and why is that the point?
- Without a KV cache, how many rows does the model compute to write m words after a prompt of n words?
- Why does a single decode row need no causal mask, while left-padded prompts need a padding mask?