Why would a model that knows two cities write New Francisco?
Ask a diffusion language model to name a US city and its most famous bridge, let it decode as fast as it can, and every so often it answers New Francisco: the Golden Bridge. No such city exists, no such bridge exists, and the model has never seen either. It produced them anyway, and the reason is the whole subject of this part.
A diffusion language model writes on a canvas. The answer starts as a row of slots, each holding a mask, a placeholder written [M]. One forward pass runs the network once over the whole canvas and predicts a token (a word or a piece of one) for every masked slot. The sampler then commits some of those predictions, leaves the rest masked, and runs another pass.
The pitch is speed. An autoregressive model, the kind behind most chatbots, commits one token per pass, left to right. For one user, a pass costs about the same whether it commits one token or twenty: an 8B model has to read all its weights from memory either way, which takes about 4.8 ms4.8 ms, worked out8B parameters × 2 bytes in bf16 = 16 GB of weightsan H100 reads 3.35 TB/s = 3,350 GB/s16 ÷ 3,350 s ≈ 0.0048 s = 4.8 msOne pass, one full read of the weights, however many tokens it commits. on an H100. Commit more per pass and you finish sooner.
So why not commit everything in one pass? Here is the canvas this series keeps coming back to. The prompt is Name a US city and its most famous bridge, and the model has learned exactly two answers, equally often, each six tokens long:
San | Francisco | : | the | Golden | Gate and New | York | : | the | Brooklyn | Bridge.
The 30-second version
Half the time. One pass gives each slot its own prediction, San or New at 50/50 and Francisco or York at 50/50, and committing both draws each from its own coin. Four outcomes are equally likely and two of them, San York and New Francisco, are mixtures. The cheapest fix is one more pass: commit slot 1 alone, and the next pass reads it, so slot 2 becomes Francisco with certainty after San and York after New. On the full six-slot answer a smarter rule hides most of that cost: it commits the certain slots (":" and "the") together with exactly one city word, then everything else, two passes in all.Look at what a pass actually hands you. For each masked slot it gives the model’s guess for that slot on its own, its marginalMarginalThe probability of each word in one slot, taking every way the rest of the answer could go into account. Slot 1’s marginal here is San 0.5, New 0.5: half the answers the model knows start with San.. Slot 1 is San 0.5 or New 0.5. Slot 2 is Francisco 0.5 or York 0.5.
Both marginals are correct. Half the answers the model knows do start with San, and half do have Francisco second. The model is not confused about cities: it knows exactly two. What goes wrong is what the sampler does with those two numbers.
Commit both slots in the same pass and each word is drawn from its own 50/50, with no knowledge of the other draw. Two fair coins give four equally likely outcomes, and two of them are not cities, so the answer is real only 50%50%, worked outeach outcome: 0.5 × 0.5 = 0.25real: San Francisco + New York = 0.25 + 0.25 = 0.5mixed: San York + New Francisco = 0.5 of the time.
This series calls it the New Francisco problem. It is not a bug in one model. It is what happens whenever two slots that depend on each other are committed in the same pass, and every fast diffusion sampler has to live with it.
What does one more pass buy?
Commit slot 1 alone. Its coin lands on San, and slot 2 stays masked. Now run the model again. This pass reads San in slot 1, and only one answer it knows starts with San, so slot 2’s guess becomes Francisco with probability 1.
That new guess is a conditionalConditionalThe probability of a word in one slot given what is already written elsewhere, P(Francisco given San). Here it is 1: after San, only Francisco fits.: the probability for one slot given what is already written. The pass did not get smarter. It got to see the decision the first coin made.
Had the coin landed on New, the second pass would have written York. Both branches are real, so the answer is right every time, for the price of one pass, about 4.8 ms for the 8B model above. San York and New Francisco can no longer happen.
This is what an autoregressive model does at every token, and it is why such a model never writes New Francisco: each token is drawn from its conditional given everything before it. Diffusion’s speed comes from exactly the opposite move, committing several tokens before any of them can see the others.
Over the whole six-token canvas, a careful schedule needs three passes. Pass 1 commits “:” and “the”, which both answers share. Pass 2 commits one city word, a fair coin. After that every other slot is certain, so pass 3 commits Francisco, Golden and Gate together.
That is 6 tokens in 3 passes, or 2 tokens per forward2 tokens per forward, worked out6 committed tokens ÷ 3 passes = 2An autoregressive model is 1 by definition: one token per pass., with no mistakes. Tokens per forward (TPF) is this series’ unit of parallelism: committed tokens divided by forward passes.
What does sampling the slots apart throw away?
Now commit all four uncertain slots in one pass. Each of San or New, Francisco or York, Golden or Brooklyn, Gate or Bridge flips its own coin. Four coins give 2⁴ = 16 equally likely outcomes, and only two of them are answers the model knows: the canvas comes out real 12.5%12.5%, worked outfour 50/50 coins: 2 × 2 × 2 × 2 = 16 outcomes, each 1/16real: San Francisco: the Golden Gate, New York: the Brooklyn Bridge2 ÷ 16 = 0.125 = 12.5%; mixed: 14 ÷ 16 = 87.5% of the time.
In plain words: a parallel pass samples from the product of marginals, each slot’s own guess multiplied together, as if the slots had nothing to do with each other. What the model knows is the joint: the probability of whole answers, here half on each of the two real ones and zero on the other fourteen. For two slots and a canvas $c$:
\[q(x_1, x_2) = p(x_1 \mid c)\, p(x_2 \mid c) \qquad \text{against} \qquad p(x_1, x_2 \mid c)\]The left side is what one pass samples; the right side is what the model actually believes. They are equal exactly when the two slots are independent given the canvas.
How far apart are they? Measure each slot’s uncertainty by its entropyEntropyHow uncertain a distribution is, in bits: a fair coin has 1 bit, a certain outcome 0 bits, a fair choice among three log₂ 3 ≈ 1.58 bits., H. The gap between the joint and the product, measured as a KL divergence, is a quantity called the total correlation:
\[\mathrm{KL}\big(p \,\Vert\, q\big) = \sum_{i} H(x_i) - H(x_1, \dots, x_n)\]In words: add up how uncertain each slot looks on its own, then subtract how uncertain the whole answer really is. What is left is the uncertainty the slots share, the part that one coin per slot counts more than once.
On the canvas, the two city slots carry 1 bit each, but the whole answer carries only 1 bit, because one choice settles both: the gap is 1 bit1 bit, worked outeach city slot: a fair coin, 1 bit; together 1 + 1 = 2 bitsthe joint: two equally likely answers, 1 bit2 − 1 = 1 bit. The four uncertain slots carry 4 bits on their own and still 1 bit together, a gap of 3 bits3 bits, worked outfour 50/50 slots: 1 + 1 + 1 + 1 = 4 bitsthe joint: still two answers, 1 bit4 − 1 = 3 bits.
On this canvas, where every answer the model knows is equally likely, the chance of a real answer is exactly 2 to the power of minus the gap: 2⁻¹ = 50% and 2⁻³ = 12.5%2 to the minus the gap, worked outreal outcomes ÷ all outcomes = 2^H(joint) ÷ 2^(sum of H)two slots: 2 ÷ 4 = 2⁻¹ = 50%four slots: 2 ÷ 16 = 2⁻³ = 12.5%Each bit the committed slots share halves the odds.. Each bit the committed slots share halves the odds.
Notice what does not cause the error: uncertainty alone. Ask for two random digits and each slot is a 1-in-10 guess, but the digits are independent, so the product of marginals is exactly the joint and one pass is perfect. The error grows with how much the committed slots depend on each other, not with how unsure each one is.
The Fast-dLLM paper (2025) gave its own version: a list of two-word poker hands, where “high card”, “two pair”, “full house” and “straight flush” are all fine and independent sampling can produce “high house”.
How does a sampler decide what to commit?
Every diffusion sampler is a rule that reads the predictions of one pass and picks which slots to commit. Three families of rules come up again and again, and the canvas shows what each one buys.
A fixed count per pass. LLaDA (2025), the 8B masked diffusion model trained from scratch that much of this field builds on, fixes the number of passes in advance. At each pass it keeps the most confident predictions and remasks the rest, so the same number of slots is committed every time. The cost is predictable. The risk is that it commits that many slots whether or not they are safe.
A confidence threshold. Fast-dLLM (2025) commits every slot whose top prediction is above a threshold, 0.9 by default. If no slot clears it, it commits the single most confident one so that decoding never stalls. The paper backs the rule with a theorem. If every committed slot’s top prediction is above 1 − ε and (n + 1)ε ≤ 1 for n slots, then committing them together picks the same tokens as committing them one at a time, greedily.
The intuition is a short counting argument. Two slots at 0.95 each can only go wrong where one of them misses its top word, so whatever the dependence between them, the product and the joint both put at least 90%90%, worked outslot 1 misses its top word at most 5% of the time, slot 2 at most 5%both hit at least 1 − 0.05 − 0.05 = 0.90 of the timeA union bound: it holds however the two slots depend on each other. of their probability on the same top pair. High confidence leaves little room for correlation.
An entropy bound. The EB-sampler (Ben-Hamu et al., 2025) sorts the masked slots by entropy, least uncertain first, and commits the longest run from the top of that list whose entropies add up to at most a small tolerance γ once the largest one is left out: $\sum H - \max H \le \gamma$.
Why that quantity? The joint can never be less uncertain than its most uncertain slot, so the sum of entropies minus the largest one is an upper bound on the total correlation above, and the model can compute it from its own marginals. The paper reports roughly 2 to 3 times faster sampling on code and math benchmarks with no loss in accuracy. Confidence can be read as the top probability, the margin between the top two, or the entropy; the EB paper sorts by any of the three.
Run the four rules on the canvas and they come apart:
| rule | where it comes from | passes | tokens per forward | real answers |
|---|---|---|---|---|
| two slots per pass | a fixed count, as in LLaDA’s sampler | 3 | 2 | 50% |
| threshold 0.4 | Fast-dLLM’s rule, set low | 1 | 6 | 12.5% |
| threshold 0.9 | Fast-dLLM’s default | 3 | 2 | 100% |
| entropy bound, γ = 0.1 | the EB-sampler | 2 | 33 tokens per forward, worked outpass 1: “:” and “the” cost 0 bits, one city word costs 0 once it is the largest; a second would add 1 bitpass 2: the three remaining slots are certain6 tokens ÷ 2 passes = 3 | 100% |
The entropy bound finds the schedule a person would choose: the certain slots plus exactly one city word in the first pass, then everything else. Two passes instead of the autoregressive six, so about 9.6 ms instead of 28.7 ms9.6 and 28.7 ms, worked outone pass of the 8B model: 16 GB ÷ 3,350 GB/s ≈ 4.78 ms2 passes × 4.78 ≈ 9.6 ms6 passes × 4.78 ≈ 28.7 ms for the 8B model, and never a mixed answer.
Here is the wall, though. Every one of these rules reads only marginals. None of them can see whether two 50/50 slots are the two halves of a city name, which must be chosen together, or two random digits, which can be chosen apart. So a safe rule refuses both, and pays passes for safety it did not always need. An aggressive rule commits both, and sometimes writes New Francisco.
Why do fast diffusion models drift back to left to right?
If a diffusion model may commit slots in any order, which order does it actually pick? Li et al. (2026) measured it. Their score, ARness, counts how often the decoder commits the leftmost slot that is still masked: 1.0 is strict left to right, and a random order scores near zero.
Under plain confidence decoding, LLaDA-8B scores 0.73. Add Fast-dLLM, the threshold sampler above, and it rises to 0.87. Dream-7B starts at 0.92 and reaches 0.94, which the authors call “nearly indistinguishable from strict autoregressive decoding”. The faster the sampler, the more it writes like the model it was meant to replace. Random orders do score near zero, but accuracy on GSM8K and MATH-500 collapses.
Their diagnosis is the data. Ordinary text and long chain-of-thought (step-by-step reasoning written out before the answer) are sequential: each sentence follows from the one before, so the slot the model is surest about is usually the one right after what is already written. A confidence rule commits wherever the model is surest, so it follows the text’s own order. Fine-tuning LLaDA on long chain-of-thought pushes its ARness from 0.73 to 0.81.
NAP, their proof of concept, changes the data instead of the sampler. Each training example holds several independent reasoning paths for the same question, written by a stronger model at temperature 1.0, laid out side by side as separate blocks, plus a summary that reaches the answer. At inference a parallel-forced decoder spreads every pass’s commitments across all the blocks.
Independent paths are exactly the case where the product of marginals is safe: a slot in path 2 does not depend on a slot in path 1. On GSM8K with Dream-7B at 4 tokens per pass, the long chain-of-thought baseline scores 46.5% and NAP 60.9%60.9% against 46.5%, worked out60.9 − 46.5 = 14.4 points at 4 tokens per passat 1 token per pass: 83.6 − 78.0 = 5.6 pointsThe gap grows as decoding gets more parallel.. At 1 token per pass the gap is only 5.6 points: NAP’s advantage grows with the parallelism.
The catch is the scale: about 100K post-training examples, and the authors say a pre-training phase on parallel-structured data may be needed to remove the left-to-right habit entirely.
Why does the first decision weigh so much?
On the canvas, a run makes exactly one real decision: the first city word. Everything after it is forced. Kim et al. (2026) found that real decoding often works the same way: the opening carries much of the outcome.
They call it proximity bias: under confidence decoding, the slots that become confident next are the ones beside what is already committed, because “the model rapidly accumulates confidence based on local token associations”. Decoding spreads outward from wherever it started. And when everything is uncertain, the start is often the very last position on the canvas, where end-of-sequence padding goes: with nothing written yet, structure is easier to predict than content.
The test is neat. On GSM8K, they let LLaDA-8B-Instruct begin each question from 256 different first steps and finish greedily, then took runs that had ended right and runs that had ended wrong. They froze each run’s first four steps and reran the rest eight times with fresh randomness. Runs that kept a good opening were right 69.1% of the time; runs that kept a bad one, 36.1%.
The fix is small. A 5M-parameter planner, two transformer layers reading the model’s final hidden states, scores 32 candidate choices for the first step and keeps the best. Only step one is planned; greedy confidence takes over after that. A second tweak, end-of-sequence annealing, divides the end-of-sequence score by a temperature that starts at 3 and falls to 1, so the end slot stops winning the opening.
With 32 passes for a 256-token answer, 8 tokens per pass8 tokens per pass, worked out256 tokens ÷ 32 passes = 8 on average, both together take GSM8K from 46.6% to 56.8%56.8% against 46.6%, worked out56.8 − 46.6 = 10.2 points on GSM8Kaverage over GSM8K, MATH, Countdown and Sudoku: 47.6 − 44.8 = 2.8 points, and the four-task average from 44.8% to 47.6%. Sudoku moved the other way, from 71.2% to 67.0%. A planned first step and a demoted end slot buy ten points on grade-school math, which says how much of the answer was being decided in the opening.
Is the freedom to write in any order a trap?
Any order sounds strictly better than left to right. A model that may write in any order can still write left to right, so its options include everything an autoregressive model can do, and more. The Flexibility Trap paper (Ni et al.), one of ICML 2026’s two Outstanding Paper awards, found that for math and code the opposite happens.
They measured pass@kpass@kThe chance that at least one of k samples solves the problem. At large k it measures how many different solutions a model can reach, which is what reinforcement learning has to work with. on LLaDA-8B-Instruct, decoding either in confidence order or strictly left to right. With one sample, confidence order is as good or better: 40.5% against 34.5% on HumanEval. With 128 samples, left to right wins everywhere: 83.0% against 67.1%83.0% against 67.1%, worked outHumanEval, 128 samples: 83.0 − 67.1 = 15.9 points for left to rightMBPP: 78.7 − 67.1 = 11.6; MATH-500: 75.6 − 71.4 = 4.2; GSM8K: 99.0 − 97.0 = 2.0 on HumanEval, and ahead on MBPP, MATH-500 and GSM8K too. On HumanEval with 1,024 samples, 21.3% of problems are solved only in left-to-right order and 0.6% only in any order.
The reason is a handful of words. In reasoning text, connectives such as “Therefore”, “Since” and “Thus” are where the argument branches, and in left-to-right order the model’s entropy spikes there. A confidence rule defers exactly those tokens, because they are uncertain. By the time it fills them in, the tokens around them have already decided which branch the argument took, and the entropy at those forks has dropped sharply.
A toy shows how a confident token can make a hard choice. Add a third answer, St. Louis: the Eads Bridge, so the model knows three, equally often. The city slot is a 1-in-3 choice, 1.58 bits1.58 bits, worked outthree equally likely cities: log₂ 3log₂ 3 ≈ 1.585 bits. But two of the three answers end in “Bridge”, so the last slot is Bridge with probability 0.670.67, worked outBridge ends New York and St. Louis: 1/3 + 1/3 = 2/32/3 ≈ 0.67; Gate, only San Francisco: 1/3, the most confident open slot on the canvas.
Left to right meets the city first, with all three open. A confidence order commits “:” and “the”, then “Bridge”, and each time it does, San Francisco is gone before the city slot was ever looked at.
Sample at temperature 0.6, the setting of the paper’s pass@k runs, and the confident detour lands Bridge in 76%76%, worked outtemperature 0.6 raises each probability to the power 1 ÷ 0.6 ≈ 1.67, then renormalisesBridge against Gate: (2/3)^1.67 ÷ (1/3)^1.67 = 2^1.67 ≈ 3.17Bridge: 3.17 ÷ (3.17 + 1) ≈ 0.76; Gate: ≈ 0.24 of runs. The city slot is then met with 0.76 bits0.76 bits, worked outafter Bridge (76% of runs): New or St. Louis, 1 bitafter Gate (24%): San Francisco is certain, 0 bits0.76 × 1 + 0.24 × 0 = 0.76 bits, against 1.58 left to right of entropy on average instead of 1.58. San Francisco falls from 33% of answers to 24%, and in 8 samples it never appears 11.2% of the time11.2% against 3.9%, worked outconfident first: (1 − 0.24)⁸ = 0.76⁸ ≈ 0.112left to right: (1 − 1/3)⁸ = (2/3)⁸ ≈ 0.039 instead of 3.9%.
One fork costs a few points of coverage; a long proof has many forks, and the losses multiply. Two honest caveats about the toy. With a perfect model sampled at temperature 1, both orders would give the same mix of answers, so the loss lives in the sharpening and in the model’s imperfect guesses. And the paper measured the effect on real reasoning, not on cities.
The paper’s remedy is almost cheeky. If arbitrary order hurts exploration during reinforcement learning, train in left-to-right order. JustGRPO treats the diffusion model as a left-to-right policy during training only. To score token k, it feeds the prefix followed by masks and reads the model’s prediction at slot k. The probability of a whole answer is then an exact product of those predictions, the quantity that is otherwise hard to compute for a diffusion model (Part 7 explains why).
With an exact likelihood, plain GRPOGRPOGroup Relative Policy Optimization: sample a group of answers to the same question, score each, and push up the probability of answers that beat the group’s average. It needs the log-probability of each answer. works unchanged. Training is sequential, but the architecture does not change, so at inference the model still decodes in parallel. JustGRPO reaches 89.1% on GSM8K from LLaDA-8B-Instruct’s 78.6%89.1% against 78.6%, worked out89.1 − 78.6 = 10.5 points on GSM8K, and 45.1% on MATH-500.
On MBPP with the EB-sampler, its gap over LLaDA-8B-Instruct widens from +10.6% at 1 token per pass to +25.5% at the most aggressive setting the paper tested. The catches: rollouts are left to right, so each RL step is slower, and on HumanEval another method, LLaDOU, reports more (59.1% against 49.4%).
Can the order be learned, or the model trained for its own decoding?
The previous two sub-pages say the order matters and the opening matters. Four 2026 papers move from diagnosis to treatment, each from a different angle.
Learn the order. Scheduling Thoughts (Xu et al.) treats the unmasking order as a policy and keeps the denoiser frozen. They derive an upper bound on the gap between the data and what step-by-step decoding produces, written in terms of the log-probability the model gives each token at the moment it reveals it. That turns every step into a reward, and a lightweight order policy, SAS, is trained on it with GRPO.
On Sudoku with a 1B model, SAS lifts puzzle accuracy from 82.0% (the best hand-written schedule) to 91.8%, and to 97.5% after fine-tuning the model along the learned orders. On GSM8K with LLaDA-8B, it goes from 64% to 76%76% against 64%, worked out76 − 64 = 12 points on GSM8K (LLaDA-8B)Sudoku: 91.8 − 82.0 = 9.8 points. The catch: policies are trained largely per task, and the reward is computed with the true tokens in place, which leaves a gap to free-running decoding.
Make early guesses commit-ready. Temporal Self-Distillation (Xu, Miele et al.) trains a slot’s earlier predictions to match the prediction the model makes at the step it finally commits that slot, on the model’s own runs with Fast-dLLM’s threshold. Early guesses that already look like final ones clear the threshold sooner, so passes can commit more.
On Countdown it reaches about 80% accuracy in about 20 function evaluationsFunction evaluations (NFE)The number of forward passes a sampler spends on one answer. Fewer NFEs at the same accuracy means faster decoding., where the reference model needs about 90: 4.5 times4.5 times, worked outabout 90 forward passes ÷ about 20 = 4.5 fewer. On GSM8K at 30 passes, the base model is below 40% and TSD close to 80%. The authors are clear about the limit: it is faster, not smarter. Its teacher is the model’s own final guess, so it cannot add an answer the model would not reach, and on MATH-500 the base model keeps the best peak score.
My own reading of where that limit bites: on the canvas, San at 50% is not an under-confident guess, it is a real fork. No amount of self-distillation should make slot 1 lean to San before anything is written. TSD helps where a slot is uncertain only because the model has not worked it out yet.
Decode several blocks at once, and train for it. Most shipped diffusion models use block diffusionBlock diffusionThe answer is cut into blocks of, say, 32 tokens. Blocks are written left to right; inside a block, slots are unmasked in parallel. Finished blocks are cached and never change., which finishes one block before starting the next. Multi-Block Diffusion (Jin et al.) lets a running set of consecutive blocks decode together, and post-trains the model on exactly the states it meets then: several partly masked blocks at different noise levels, behind a clean prefix.
On LLaDA2-Mini, averaged over GSM8K, MATH500, MBPP+ and HumanEval+, TPF rises from 3.47 to 6.196.19 against 3.47, worked out6.19 ÷ 3.47 ≈ 1.78 times the tokens per forwardaccuracy: 81.03 − 79.95 = 1.08 points higher while accuracy goes from 79.95% to 81.03%. Combined with DMax, a decoder that can also rewrite committed tokens (Part 4 is about that), it reaches 9.34 tokens per forward for a 1.02-point drop. The catch: it needs post-training, and it is shown on math and code only.
Train on your own trajectories. On Trajectory-Aware Training (Madeira et al., with Apple) starts from a mismatch. A masked model is trained on randomly masked text but samples along a path shaped by its own commitments, and each step forgets what the previous step computed. PUMBA trains on consecutive steps of the model’s own unmasking runs. It passes each position’s last hidden state forward between steps and backpropagates through several steps at once.
Exact train-inference alignment overfits; a looser alignment helps, and longer windows help more. A 125M-parameter model on grade-school math matches the best checkpoint of a same-size autoregressive model, 55.6% against 55.3% on GSM8K. Fine-tuning LLaDA-8B, it needs up to 22% fewer passes than standard fine-tuning at matched score on the full canvas, and up to 26% fewer with block diffusion. The gains are real but modest, and so far shown with confidence samplers only.
Can the slots be sampled together instead of apart?
Every fix so far works around the product of marginals. E-MoE (Ivanov et al.) attacks the product itself. Instead of one coin per slot, it first draws a shared choice z for the whole canvas, then lets every slot draw its word given z:
\[p(x_1, \dots, x_n \mid c) \approx \sum_{z} p(z \mid c) \prod_{i} p(x_i \mid c, z)\]In words: a mixture of products. Each product on its own still treats the slots as independent, but the shared z ties them together. On the canvas, z could be “which city”: given z, every slot is certain, and one pass writes a real answer every time. That is the cheapest fix in principle: one pass, no mixing.
Where does z come from? Earlier work used a continuous Gaussian latent trained as a variational autoencoder, and the latent tended to be silently ignored (posterior collapse). E-MoE’s idea is that a mixture-of-experts network already makes a discrete shared choice on every forward pass: which experts each token is routed to. It uses those routing decisions as z, so the latent costs no extra active parameters, and the paper reports that it “remains in use throughout training without warmup schedules or auxiliary losses”.
On LM1B, generative perplexity (how surprised a reference model is by the samples; lower is better) in a single step falls from 1,433.8 to 643.8, and in two steps from 997.5 to 383.7: 2.2 to 2.6 times2.2 to 2.6 times, worked out1 step: 1,433.8 ÷ 643.8 ≈ 2.232 steps: 997.5 ÷ 383.7 ≈ 2.6032 steps: 148.6 ÷ 135.1 ≈ 1.10 lower. The advantage shrinks with more steps: at 32 steps it is 148.6 against 135.1, which makes sense, since many small steps already let the slots see each other.
The catches are scale and cost: toy data, binarized MNIST and LM1B with a small backbone, and two forward passes per training step instead of one. Whether a shared latent survives at 8B parameters is untested.
What is still open?
Put the year’s moves side by side and they share one shape: each one either makes fewer dependent slots land in the same pass, or makes the model’s guesses less dependent when they do.
| move | what it changes | the number | the catch |
|---|---|---|---|
| NAP | the data: independent reasoning paths | GSM8K 46.5% → 60.9% at 4 tokens per pass (Dream-7B) | proof of concept, ~100K examples |
| planner and end-of-sequence annealing | the first decision | GSM8K 46.6% → 56.8% (LLaDA-8B) | first step only; Sudoku fell |
| JustGRPO | trains in left-to-right order | GSM8K 89.1% (LLaDA-8B) | slower rollouts |
| SAS | a learned unmasking order | Sudoku 82.0% → 91.8% | trained per task |
| TSD | early guesses pulled toward final ones | about 4.5 times fewer passes on Countdown | faster, not smarter |
| Multi-Block | several blocks at once, trained for it | TPF 3.47 → 6.19 (LLaDA2-Mini) | post-training; math and code |
| PUMBA | trains on its own trajectories | up to 26% fewer passes (LLaDA-8B) | modest gains so far |
| E-MoE | a shared latent across slots | 2.2 to 2.6 times lower generative perplexity in 1 to 2 steps | small models only |
The wall itself is still standing. No commit rule is both fast and safe in general, because the rules read marginals and the danger lives in the dependence between slots, which the marginals do not show. A safe threshold pays for passes it did not need; an aggressive one writes New Francisco.
Three questions I would watch in 2027. Can a sampler estimate dependence directly, cheaply, from one pass, instead of bounding it from entropies? Is left to right a habit to be trained out (NAP’s bet) or the right order for reasoning that we should train in (JustGRPO’s)? And do shared latents like E-MoE’s survive at the scale of the models that shipped this year?
The other way out is to stop treating a committed token as final. If the model can notice New Francisco and rewrite it, an aggressive rule becomes much less dangerous. That is Part 4, on editing and self-correction.
Rapid fire: can you do these from memory?
- A model is 50/50 between two answers, and three of the slots you commit in one pass all hang on that choice. How often is the answer real?
- Write the product of marginals and the joint for two slots. When are they equal, and what is the gap between them called?
- Why can a confidence threshold not tell New Francisco's two slots from two random digits, and what does that cost it?
- State the EB-sampler's stopping rule, and say why its quantity bounds the error of committing a set of slots together.
- What does ARness measure, and why does training on long chain-of-thought push it up?
- What is proximity bias, and which single decision does the planner of Kim et al. choose?
- What did the Flexibility Trap paper find at pass@128 that it did not find at pass@1, and how does JustGRPO get an exact likelihood?
- How would a shared latent remove New Francisco in a single pass, and where does E-MoE get its latent?
Where can you go deeper?
- Zanlin Ni et al., The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models (ICML 2026, Outstanding Paper), arXiv:2601.15165. Project page: nzl-thu.github.io/the-flexibility-trap; code: LeapLabTHU/JustGRPO. Pass@k by decoding order, forking tokens, and JustGRPO.
- Pengxiang Li et al., Why Diffusion Language Models Struggle with Truly Parallel (Non-Autoregressive) Decoding? (preprint, 2026), arXiv:2602.23225. Code: pixeli99/NAP. ARness, the sequential-data diagnosis, and NAP.
- Jiyeon Kim et al., Early Decisions Matter: Proximity Bias and Initial Trajectory Shaping in Non-Autoregressive Diffusion Language Models (preprint, 2026), arXiv:2604.10567. Proximity bias, the frozen-opening experiment, the first-step planner and end-of-sequence annealing.
- Jiawei Xu et al., Scheduling Thoughts: Learning the Order of Thought in Diffusion Language Models (preprint, 2026), arXiv:2606.23567. Project page: jimmyxu123.github.io/SAS. The decoding-mismatch bound and the SAS order policy.
- Shijian Xu, Andrea Miele et al., Temporal Self-Distillation: Faster Inference in Discrete Diffusion Language Models (preprint, 2026), arXiv:2609.15177. Distilling early predictions toward commit-time predictions.
- Yijie Jin et al., Multi-Block Diffusion Language Models (preprint, 2026), arXiv:2606.29215. Project page: sjtu-deng-lab.github.io/mbd-lms; code: SJTU-DENG-Lab/mbd-lms. Multi-block decoding and Multi-block Teacher Forcing.
- Manuel Madeira et al., On Trajectory-Aware Training for Masked Diffusion Language Models (preprint, 2026), arXiv:2609.37974. PUMBA: training on the model’s own unmasking runs, with a carry and backpropagation through time.
- Arseny Ivanov et al., E-MoE: Enhanced Mixture-of-Experts for Non-Factorized Diffusion Language Models (preprint, 2026), arXiv:2609.37533. Code: Arseny5/e-moe. Routing decisions as a shared latent.
- Chengyue Wu et al., Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding (preprint, 2025), arXiv:2505.22618. The confidence threshold, its high-confidence theorem, and “high house”.
- Heli Ben-Hamu et al., Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmasking (preprint, 2025), arXiv:2505.24857. The EB-sampler and its error bound.
- Shen Nie et al., Large Language Diffusion Models (preprint, 2025), arXiv:2502.09992. LLaDA and its low-confidence remasking sampler.