Can you answer these without the films?
I can follow the cache through a film and still lose a factor when someone asks me to count it at a whiteboard. This is where I put the films away and work through the questions out loud.
The levels ask for different things: state the mechanism, reason about a trade-off, then defend a design with a measurement that could change my mind. Each folded sketch gives the core of an answer; every number comes from Parts 1 to 5.
The 30-second version
I keep earlier keys and values at every layer because, with fixed weights and positions, each old row depends only on tokens up to it: appending a token cannot change it. Future queries read those keys and values, never the old queries. That saves repeated computation but costs growing storage and repeated reads during dense decode. I separate techniques that shrink stored state, share it, read less of it or produce more useful tokens per read.Level 1: can you state the mechanism?
-
Which tensors go in the cache, and why does the reuse argument hold at every layer rather than just the first?
answer sketch
I keep keys and values per layer and KV head: in deterministic inference with fixed weights and positions, an old row depends only on tokens up to it. Causal attention cannot see an appended token, and rowwise norms and FFNs preserve that independence up the stack. Old queries stay fixed too, but future queries never read them.
-
Write the cache bytes per token, then evaluate them for Llama 3 70B in BF16. Why are batch size and query-head count absent?
answer sketch
I multiply two vectors by layers, KV heads, head width and bytes per element: 320 KiB per tokenThe cached pair2 × 80 × 8 × 128 × 2 = 327,680 bytes327,680 ÷ 1,024 = 320 KiB. Batch and context multiply this payload afterwards; the 64 query heads do arithmetic but share the 8 stored KV heads.
-
What does attention compute during one decode step, and how does prefill differ? Does caching make the history scan disappear?
answer sketch
I project the new row, append its key and value, then score its query against all visible keys and mix the values, including its own. Prefill processes the prompt's causal triangle together and shares weight reads across those rows. Caching removes repeated old-row computation; dense attention still scans a growing history.
-
Count the dominant logical reads for the 70B example at batch 32 and 4,096 cached tokens per request. Which bytes can unrelated requests share?
answer sketch
The model reads 141 GB of BF16 weights for the batch, plus 42.95 GB of cache, about 183.95 GB togetherThe decode read32 × 4,096 × 327,680 ÷ 10⁹ = 42.94967296 GB141 + 42.94967296 = 183.94967296 GB. I amortise the weight read across users; their distinct histories each add cache traffic, with actual HBM traffic depending on the kernel.
-
The toy answers “Breaking Benjamin .” after “Diary of Jane is a song by”. Why does caching process nine input rows rather than ten, and what does recomputation cost?
answer sketch
Prefill already predicts “Breaking”; only “Breaking” and “Benjamin” are fed back, since the full stop ends generation. That gives 9 cached rows against 24 recomputedInput positions processedCached: 7 + 1 + 1 = 9Recomputed: 7 + 8 + 9 = 24, which counts input positions, not FLOPs or a speedup.
-
On the same 80-layer shape, compare BF16 storage for 64-head multi-head attention, 8-head GQA and a 512-wide latent plus 64 rotary numbers. Can I multiply the GQA and latent savings?
answer sketch
The payloads are 2.5 MiB, 320 KiB and 90 KiB per tokenSame layers, different representationMulti-head: 2 × 80 × 64 × 128 × 2 = 2,621,440 bytesLatent: 80 × (512 + 64) × 2 = 92,160 bytesDivide by 1,048,576 for MiB or 1,024 for KiB. MLA replaces the representation, so I cannot multiply its saving by GQA's; the latent estimate borrows DeepSeek-V2's widths on our comparison shape, not its actual layer count.
-
With four-token blocks and table [6, 1, 9], where is token position 5? Why does paging need a compatible attention kernel?
answer sketch
Integer division finds logical block 1 and the remainder gives offset 1, so table entry 1 sends me to physical block 1: “song”. The kernel must follow that mapping to gather the unchanged rows from scattered storage; allocating blocks as needed avoids maximum-length reservations.
-
A latent reconstructs keys and values. Must decode materialise every reconstructed row before attending?
answer sketch
For the content part, I absorb the key up-projection into the query, score the cached latents, then mix latents before applying the value up-projection. Part 3's matrix identities justify that rearrangement; the rotary component stays separate, and the model must be trained for the bottleneck.
Level 2: can you reason about the trade-offs?
-
At 4,096 tokens, raising the batch from 32 to 320 increases total throughput but slows each user. Explain both with one byte budget.
answer sketch
In Part 5's estimate, step time is weight bytes plus the batch's cache bytes, divided by bandwidth. More users share the fixed weight read, raising total tokens per second, but their added histories lengthen each step and every user's wait. Once cache reads dominate, throughput approaches a ceiling rather than growing with batch.
-
I switch only the cache from BF16 to FP8. What happens to capacity and per-user latency, and what must I measure?
answer sketch
The ideal payload halves to 160 KiB per tokenHalve the element size320 ÷ 2 = 160 KiB, before scale metadata, doubling token capacity before metadata. At fixed batch, only the cache term halves: weights still cost the same, so latency does not halve. I measure workload quality and kernel performance, then check latency again if I admit more users.
-
Sharing KV heads leaves the query heads intact. Does that mean the layer's answer is unchanged?
answer sketch
No: queries can weight the shared rows differently, but they lose private key and value projections. Part 3's toy changes the second head's output when it shares the first head's values; real GQA learns that compromise, so its quality result is empirical, not guaranteed by the surviving queries.
-
Keeping initial attention sinks can stabilise a sliding window. Does that restore long-range retrieval?
answer sketch
No: the initial tokens can help preserve attention behaviour, but evicted content remains unavailable for direct retrieval. I distinguish streaming stability from remembering the old document, and count local and global layers separately if the model mixes them.
-
Part 4's isolated swap clock is shorter than its recompute clock. Is that enough to choose swap as the server's preemption policy?
answer sketch
Those are ideal resource clocks, not measured latencies for the sharded model. Swap consumes a shared link and host buffers; recompute spends arithmetic and can join chunked batches, so I measure the interference with running requests. Spare arithmetic can help, but recomputation is not automatically free of stalls.
-
Two prompts contain the same long document after different introductions. Can I share that document's cached rows?
answer sketch
Matching text alone is insufficient: exact reuse needs the preceding context, positions and model state to match too. I put stable content first and variable content last, then measure exact prefix reuse rather than document overlap.
-
At 128k, a selector reads 2,048 main-cache positions per query. Have I divided both memory use and step time by sixty-four?
answer sketch
The selected main KV payload falls by 64 timesSelected positions131,072 ÷ 2,048 = 64, while the main cache stays resident. The indexer still scores history and adds work and state, so neither total traffic nor latency inherits that factor; this helps a read bottleneck without necessarily fixing capacity.
-
Five draft proposals each have conditional acceptance probability 0.8. Is the expected emitted run just the expected accepted draft prefix?
answer sketch
The accepted prefix averages 2.69 tokens, but the emitted run averages 3.69Accepted versus emitted0.8 + 0.8² + 0.8³ + 0.8⁴ + 0.8⁵ = 2.689281 + 2.68928 = 3.68928, because a correction or bonus token follows when generation continues. EOS can truncate the run, and speedup must still pay for drafting and verification.
Level 3: can you defend a design decision?
-
Our eight-H100 node fits 40 distinct 32k conversations. The product now wants the same concurrency at 128k: what changes first?
answer sketch
I reconstruct Part 2's 431.8 GB poolExplicit node budget8 × (0.92 × 80 − 2) − 141 = 431.8 GB: BF16 holds only ten 128k conversationsOccupancy at the longer context131,072 × 327,680 ÷ 10⁹ = 42.94967296 GB eachfloor(431.8 ÷ 42.94967296) = 10, so sparse reads alone cannot meet the requirement. I would add capacity if full-context behaviour must stay, or evaluate a smaller cache representation with long-context quality tests. A window changes the product's access to old content, and any admission limit needs room for generation.
-
Would I ship a four-bit KV cache for this workload? What evidence would make that a defensible choice?
answer sketch
I would trial it behind workload quality and latency checks: Part 3's ideal GQA payload at 128k is 10.74 GBPacked four-bit payload42.94967296 × 4 ÷ 16 = 10.73741824 GB, before scales and residual buffers. Quantization layout and kernels matter, and a perplexity result does not establish retrieval or reasoning quality on my traffic.
-
Should reusable prefixes stay on the GPU or move into a shared storage tier? What would decide it for me?
answer sketch
I start with exact prefix hit rate and compare measured retrieval time with rebuilding the same state. A shared tier earns its cost when reuse repays transfers, routing and contention; the transport's name alone cannot settle that comparison.
-
With a free choice of model, should I choose latent attention or GQA with FP8 and sparse reads?
answer sketch
I compare resident bytes, read traffic, quality and supported kernels separately, starting from whichever resource limits my workload. Latents replace the representation, while compatible precision and selection mechanisms can compose with it; I cannot apply MLA as a serving flag to an ordinary Llama checkpoint. I would trial the smaller resident representation for a capacity problem and reduced reads for a bandwidth problem.
-
Would separating prefill and decode improve this server? Count the handoff for a 32k prompt over a 400 Gb/s effective path before deciding.
answer sketch
The BF16 cache is 10.74 GB, taking an ideal 214.75 msOne serialized handoff32,768 × 327,680 = 10,737,418,240 bytes400 Gb/s ÷ 8 = 50 GB/s10,737,418,240 ÷ (50 × 10⁹) ≈ 0.21475 seconds on that path before overhead; sharding may spread transfers across paths. I would separate the workers only if reducing prefill interference repays transfer and coordination, measured against both latency targets at the expected load.
-
A colleague says speculation is only useful at small batch. How would I test that claim at long context?
answer sketch
I compare accepted output per full speculative cycle with ordinary decoding, including draft cost and the extra verification queries. Long contexts can hit the cache capacity limit while verification remains memory bound, so concurrency alone does not rule out gains. I would measure quality-preserving output and both clocks; a greedy match is not the rejection correction required for exact stochastic sampling.
-
Long prompts are interrupting active decodes. Would I try chunked prefill before adding separate prefill workers?
answer sketch
I would try it: Part 4 splits the prompt across batches that are already decoding, reducing each interruption. Smaller chunks also take more scheduling steps to finish the prompt, so I measure time to first token alongside the output-token gaps before choosing a size.
-
I need to move from 40 to 80 tokens/s per user. Why might maximising total tokens/s send me in the wrong direction?
answer sketch
A larger batch can raise total throughput while making every user's gap longer, so I would first test a smaller batch and check for prefill interference. To improve both rates, I need fewer bytes, more bandwidth or more useful tokens per expensive read. I judge the result by requests meeting both latency targets, Part 5's goodput, rather than throughput alone.
What should you actually walk in with?
I want to reconstruct the budget, not memorise a token-rate claim:
- Why causality preserves old rows at every layer, and why only keys and values are reused.
- 320 KiB per token, 141 GB of weights and the explicit 431.8 GB cache pool: enough to rebuild the occupancy calculation.
- Storage versus reads, and per-user speed versus total throughput.
- A proposed change, the resource it saves and the quality or latency measurement that could make me reject it.