The Transformer · 8. Inference
One word per pass, and how a word gets pickedPrefill reads the prompt in one pass; each decode pass appends one word: “Breaking” 0.61, “Benjamin” 0.39, “.” 0.53. Temperature 0.5 lifts “Breaking” to 0.85, temperature 2 drops it to 0.34; top-2 keeps 0.71 and 0.29; top-p keeps 3 words after “by” and 5 after “Breaking”.
Open video ↗Read the walkthrough
Prefill. The seven prompt rows go up in one pass with the causal mask; every layer keeps their keys and values. Only the last row's probabilities are used: “Breaking” 0.61.
The loop. Append the word, run again: “Benjamin” 0.39, then the full stop 0.53.
Greedy and temperature. Greedy takes the top word. p_i(T) = e^(z_i/T) / Σ e^(z_j/T): T = 0.5 gives “Breaking” 0.85, T = 2 gives 0.34 and “Jane” 0.22.
Top-k and top-p. Keep the k best, or the fewest that reach p, and renormalise: top-2 gives 0.71 and 0.29. Top-p 0.9 keeps 3 words after “by” and 5 after “Breaking”.
Stop. The toy's full stop ends the answer; chat models stop on an end-of-turn token or a length limit.
The old rows. Without a cache: 7 + 8 + 9 = 24 rows for three words; keeping keys and values: 7 + 1 + 1 = 9. That is the KV cache.