How do I get from forty tokens a second to eighty?

I turn up the batch, watch total throughput climb, and get a complaint: every user’s answer arrives more slowly. Both observations are right. The weights are shared across the batch; each conversation brings another cache to read.

I separate four clocks and counts. Time to first token includes queueing and prefill. Time per output token is the decode gap. Throughput counts all output tokens per second. DistServe’s goodput counts requests served within both latency targets. A throughput record alone tells me little about the experience.

For scale, NVIDIA’s interactive Llama 2 70B MLPerf scenario used 450 ms to first token and 40 ms per output token: 25 tokens/s per userThe output-token target40 ms = 0.040 s; 1 ÷ 0.040 = 25 tokens/s.

They will ask Your deployment does 40 tokens a second per user and you need 80. What do you do?
The 30-second version I first check batch size, context length and whether prefill is stalling decode. Reducing the batch can shorten each step, at the cost of total throughput. To improve both, I need fewer bytes, more bandwidth or more useful tokens per read. Speculation helps only when accepted tokens repay verification and drafting costs, so I measure the bottleneck before choosing it.

Where does the batch hit a wall?

For distinct, equally long conversations, my memory-bound estimate is:

\[t\approx\frac{W+BTc}{\beta},\qquad u=\frac1t,\qquad R=\frac Bt\]

Read it as weight bytes plus batch times context times cache bytes per token, divided by aggregate bandwidth. The step takes t seconds; u is one user’s rate and R the node’s total. Llama 3 70B uses 320 KiB per tokenThe cached pair2 × 80 layers × 8 KV heads × 128 numbers × 2 bytes = 327,680 bytes327,680 ÷ 1,024 = 320 KiB in BF16. I retain 141 GB of weights and eight H100s, giving 26.8 TB/sAggregate bandwidth8 × 3.35 TB/s = 26.8 TB/s.

At 4,096 tokens and batch 32, I get 6.86 ms per step, 145.7 tokens/s per user and 4,662 totalBatch 32, worked outcache: 32 × 4,096 × 327,680 = 42,949,672,960 bytestime: (141,000,000,000 + 42,949,672,960) ÷ 26,800,000,000,000 = 0.006863794 s1 ÷ time = 145.692; 32 ÷ time = 4,662.145 tokens/s. At batch 320 it is about 21.29 ms, 47 per user and 15,033 totalBatch 320, worked outcache: 320 × 4,096 × 327,680 = 429,496,729,600 bytestime: (141,000,000,000 + 429,496,729,600) ÷ 26,800,000,000,000 = 0.021287191 s1 ÷ time = 46.977; 320 ÷ time = 15,032.514 tokens/s: more output overall, longer gaps individually. Cache traffic overtakes weights at ordinary batched conversation lengths.

I use the same budget as Part 2: 431.8 GB for the cacheThe shared node budget640 GB total − 141 GB weights − 51.2 GB headroom − 16 GB workspace = 431.8 GB. It fits 321 sequences at 4k, 40 at 32k and ten at 128kCapacity, worked outfloor(431,800,000,000 ÷ (4,096 × 327,680)) = 321floor(431,800,000,000 ÷ (32,768 × 327,680)) = 40floor(431,800,000,000 ÷ (131,072 × 327,680)) = 10. At the longest context that yields about 470 tokens/s totalTen long conversations10 × 131,072 × 327,680 = 429,496,729,600 cache bytes10 × 26,800,000,000,000 ÷ (141,000,000,000 + 429,496,729,600) = 469.766 tokens/s. The bandwidth ceilings are 19,968 tokens/s at 4k and 624 at 128kBandwidth ceilings26,800,000,000,000 ÷ (4,096 × 327,680) = 19,967.55626,800,000,000,000 ÷ (131,072 × 327,680) = 623.986, before finite weights, capacity and arithmetic intervene. Kernel overheads, communication and scheduling lower real rates further.

\[B_{\max}=\left\lfloor\frac{M}{Tc}\right\rfloor,\qquad R_{\infty}=\frac{\beta}{Tc},\qquad Wu+(Tc)R=\beta\]

The first divides pool bytes by one conversation’s cache. The second divides bandwidth by that cache. The last is a straight frontier between the two rates. I also check FLOPs against the H100 ridge of about 295 FLOPs/byte: in this example ordinary decode hits capacity first. Part 2 develops that budget.

Can one cache read produce several tokens?

I can spend spare arithmetic on a draft, then verify several positions together. The film proposes “Breaking Benjamin .” after “Diary of Jane is a song by”, using the unchanged Transformer toy. Watch which input row predicts which word: that offset matters.

\[S=QK^\top/\sqrt{d_h}+M,\qquad A=\operatorname{softmax}(S),\qquad O=AV\]

Queries meet cached keys, the causal mask hides future draft positions, and each softmax row mixes the values. Here M means the mask, not the earlier memory pool. The rest of the block gives 0.612, 0.393 and 0.532Target probabilitiesFor each finished block row: logits = h × W_U + b_UApply softmax over all eleven logits: exp(chosen logit) ÷ sum exp(all logits)Read Breaking after by, Benjamin after Breaking, and the full stop after BenjaminThe cached matrix pass and toy.greedy() agree before rounding for the proposed words. All three are target argmax matches, so greedy verification accepts them. Benjamin need not exceed one half; it only needs to beat every alternative.

I hold “by” at the verification boundary and cache the preceding six rows. A completed prefill would already supply the first probability. The toy’s ideal KV payload is 144 bytes against 384Toy KV readsBF16 pair per token: 2 × 1 × 1 × 4 × 2 = 16 bytesCombined pass: 9 × 16 = 144 bytesThree serial passes: (7 + 8 + 9) × 16 = 384 bytesLogical payload only: assumes reuse across query rows; excludes writes and draft cost; weights are also read once instead of three times. Actual kernels may reread data, and drafting, cache writes and extra query work still cost something.

Greedy matching preserves greedy output. Exact stochastic speculation instead needs the rejection correction from Leviathan and colleagues; accepting likely-looking words is insufficient. Their T5-XXL result was two to three times faster; Chen and colleagues independently reported two to two and a half times on Chinchilla 70B.

When does speculation stop paying?

I distinguish accepted draft tokens from emitted tokens. For draft length k and constant conditional acceptance chance a:

\[\mathbb E[L]=a+a^2+\cdots+a^k,\qquad \mathbb E[N]=1+\mathbb E[L]=\frac{1-a^{k+1}}{1-a}\]

L counts the accepted draft prefix. N adds a target correction or bonus token, assuming generation continues. Five proposals at eighty percent acceptance give 2.69 accepted draft tokens and 3.69 emitted tokensThe expected run0.8 + 0.8² + 0.8³ + 0.8⁴ + 0.8⁵ = 2.689281 + 2.68928 = 3.68928EOS can truncate the run; the extra one is a correction or bonus. These acceptance assumptions illustrate the accounting; they are not EAGLE measurements.

\[\text{speedup}\approx\frac{\mathbb E[N]t_{\text{ordinary}}}{t_{\text{draft}}+t_{\text{verify}}}\]

Read this as useful output times ordinary step time, divided by the whole speculative cycle. Verification uses more query positions. Once arithmetic dominates, sharing a read stops buying proportional speed. Long contexts enlarge the memory bill and cap the feasible batch earlier: MagicDec shows why gains can survive high concurrency.

Better drafts change the numerator. Medusa’s abstract reports over 2.2 times speedup for Medusa-1 and 2.3 to 3.6 times for Medusa-2. EAGLE used feature-level drafting. EAGLE-3 predicts tokens from fused multi-layer features and reports up to 6.5 times, about 1.4 times EAGLE-2.

The earlier SGLang example was 158, 244 and 373 tokens/s without speculation, with EAGLE-2 and with EAGLE-3. P-EAGLE parallelises drafting through a learned shared hidden state. Those results have workloads attached; they do not replace my own latency measurement.

What would I change first?

I name the limiting resource: pool, bandwidth or arithmetic. Grouped-query attention already reduces this model’s stored heads. An FP8 cache reduces storage and read traffic, with accuracy to check. Latent attention needs a different architecture. Sparse reading cuts traffic without necessarily freeing capacity. Paging and prefix reuse recover wasted space or work. A larger node buys memory and bandwidth, with communication costs.

Then I name the goal: one user’s latency or the node’s throughput. Smaller batches slide along the frontier. Smaller payloads and more accepted tokens per expensive read can move it.

  1. Why can total throughput rise while every user slows down?
  2. How does longer context move both the capacity wall and bandwidth ceiling?
  3. Why is the memory-bound frontier straight, and why does a real server fall below it?
  4. Why does the 3.7-token expectation include more than the accepted draft prefix?
  5. When can speculative decoding still help at large batch?