Why can a masked diffusion model never take a word back?

A diffusion language model writes on a canvas. The answer starts as a row of slots, each holding a mask, a placeholder written [M]. One forward pass runs the network once over the whole canvas and predicts a token (a word or a piece of one) for every slot. The sampler then commits some of those predictions, writes them into their slots, and runs another pass.

The appeal is that one pass can commit many tokens, where an autoregressive model commits one. The danger is what this part is about. Here is the canvas the series keeps returning to. The prompt is Name a US city and its most famous bridge, and the model has learned exactly two answers, equally often, six tokens each:

San | Francisco | : | the | Golden | Gate and New | York | : | the | Brooklyn | Bridge.

Run it as fast as it will go and one day it hands you New Francisco: the Golden Gate.

The puzzle A fast pass wrote "New Francisco : the Golden Gate". Which single token should be changed, and how could the model know without being told?
The 30-second version Change New to San. The model can find it by reading its own draft again: predict every slot from all the other words. Slot 1 sees Francisco, Golden and Gate, three words that only ever appear in the San Francisco answer, so it predicts San with near certainty, and San is not what is written. That disagreement, if it is strong enough, is the signal. A plain masked model cannot act on it: it was only ever trained to fill masks, and once a slot is written nothing turns it back into a mask. LLaDA2.1 adds a second threshold that lets the model replace a written token when its new prediction is confident enough. The catch: when the draft splits two words a side, no slot has a majority, and the same trick finds nothing.

Where does the draft come from? Before anything is written, each slot’s prediction on its own, its marginalMarginalThe probability of each word in one slot, taking every way the rest of the answer could go into account. Slot 1’s marginal is San 0.5, New 0.5: half the answers the model knows start with San., is 50/50 in four slots: San or New, Francisco or York, Golden or Brooklyn, Gate or Bridge. The colon and the are certain.

A fast sampler sets a low confidence threshold and commits every slot that clears it. Here that is all six at once, and each 50/50 slot is drawn on its own coin. Four coins give 16 equally likely drafts16 drafts, worked outfour 50/50 slots, two choices each2 × 2 × 2 × 2 = 2⁴ = 16, and only 2 of them are real answers: 12.5%12.5%, worked outreal answers: all four San Francisco words, or all four New York words: 22 ÷ 16 = 0.125The other 14 drafts, 87.5%, mix the two cities.. The rest mix the two cities. Part 3 is about why: each slot is sampled from its own marginal, and the product of marginals is not the joint.

This part is about what happens next. In a masked diffusion model, nothing happens next. The draft is final.

The reason sits in how the model was trained. Training runs a noising process forward: take a real sentence, pick a noise level, and turn each token into [M] with that probability. Once a token has become a mask, it stays a mask at every higher noise level. In the language of Markov chains the mask is an absorbing stateAbsorbing stateA state that a random process can enter but never leave. In masked diffusion each token can jump to [M] as noise is added, and from [M] the only move is to stay [M].: you can fall in, but you can never climb out.

Generation runs that process backwards, and the mirror image of “a mask stays a mask” is “a written token stays written”. Sahoo and colleagues spell it out for the exact reverse step: if a slot is not masked, the token must remain fixed.

There is a second, quieter reason. Every visible token the network saw in training was a true token from real text. It learned to fill masks given correct context; nobody ever showed it a wrong visible token and asked it to fix one. So even if you let it, it has no habit of doubting what is written.

Put the two together and you get the trade the whole field ran into. Every speed-up for a masked model commits more tokens per pass, sooner, with less context. Each extra commitment is a chance to write a mistake, and every mistake is kept. The 2026 response has one theme: let the model take words back.

How could the model know which word is wrong?

By reading its own draft. Run one more forward pass over New Francisco : the Golden Gate and ask every slot again, this time with the other five words in place.

Slot 1 now sees Francisco, Golden and Gate. In the model’s world those three words only ever appear in the San Francisco answer, so its prediction for slot 1 is San, and confidently. Slot 2 sees New on one side and Golden Gate on the other, a context it has never met, and it is torn.

The figures in this part use a toy re-reader to put numbers on this: each slot lets the other written city and bridge words vote for their answer. Slot 1 gets three votes for San Francisco, so San at 1San at 1, worked outslot 1 sees Francisco, Golden, Gate: 3 San Francisco words3 votes of 3 = 1. Slots 2, 5 and 6 each get two votes for their own word and one against: 0.670.67, worked outslot 2 sees New, Golden, Gate2 votes for San Francisco of 3 = 0.67, so Francisco stays on top.

A real network is not a vote, but the shape is the same: the odd word out is the one whose re-read most strongly disagrees with what is written.

Notice when that evidence appears. At the moment New was committed it had probability 0.5, exactly the same as San. Nothing about New looked wrong when it was written. The evidence only exists after the other words land. So a model that wants to fix mistakes has to look again after it writes, and it has to be allowed to act on what it sees.

How does LLaDA2.1 edit a token it already wrote?

LLaDA2.1 (Ant Group’s inclusionAI team, February 2026) is a pair of mixture-of-expertsMixture of experts (MoE)A network whose feed-forward layers are split into many “experts”, with a router sending each token to only a few of them. The model holds many parameters but uses a fraction of them per token. diffusion models, LLaDA2.1-Mini at 16B parameters and LLaDA2.1-Flash at 100B. Both are block diffusion models: the answer is written in blocks, left to right, and inside a block the slots are filled in parallel.

Its change is to give every pass two jobs. The paper calls them Mask-to-Token, filling masks as before, and Token-to-Token, replacing a written token with a different one. Each has its own threshold, and both compare the same number: the probability of the model’s top choice for that slot.

\[\Gamma_t = \{\, i : x_t^i = \texttt{[MASK]} \text{ and } p_\theta(v_t^i \mid x_t) > \tau_{\text{mask}} \,\}, \qquad \Delta_t = \{\, i : x_t^i \neq v_t^i \text{ and } p_\theta(v_t^i \mid x_t) > \tau_{\text{edit}} \,\}\]

In words: $v$ is the model’s top choice for slot $i$ given the current canvas. Fill a masked slot when that choice is above the unmask threshold. Replace a written token when the top choice is a different token and is above the edit threshold. Every slot in either set takes its top choice in this pass, and everything else stays as it is.

On the draft, slot 1’s top choice is San at 1, which differs from New. Any edit threshold below 1 replaces it. Slots 2, 5 and 6 have their own word on top, so there is nothing to replace. One edit, and it rides on the same pass that re-read the draft.

How does a masked model learn to do this, given the second problem above? LLaDA2.1 trains two streams together, from continued pretraining through fine-tuning. The drafting stream is the usual one: predict the true token at masked positions. The editing stream shows the model text with random tokens swapped in and asks it to recover the originals. A data augmentation the paper calls Multi-turn Forward exposes it to more editing situations.

The figure’s last two phases show where this runs out. Take the draft San Francisco : the Brooklyn Bridge, two words from each answer. Every slot sees two votes for the other city and one for its own, so each prefers the other answer at only 0.67. With the edit line at 0.8, nothing changes, and the mixed answer is final.

Lower the edit line to 0.6 and it is worse: all four slots clear it in the same pass, and all four flip to New York : the Golden Gate, still two a side. The next pass flips them back. Replacing words in parallel has exactly the problem that writing them in parallel had: each slot decides alone.

What do Speedy and Quality mode trade?

Editing turns the threshold into a dial. If mistakes can be repaired, the unmask threshold can drop: commit rough tokens early and fix the bad ones on later passes. LLaDA2.1 ships the two ends of that dial as named modes. Speedy Mode lowers the unmask threshold and leans on edits; Quality Mode keeps conservative thresholds and needs few. The paper does not publish the threshold values behind either mode.

The unit to watch is tokens per forwardTokens per forward (TPF)Committed tokens per forward pass, averaged over a run. An autoregressive model is 1 by definition. A diffusion model that writes a 6-token answer in 3 passes is at 2. Edits do not add to it: a replaced token was already counted when it was first written.. Here is LLaDA2.1-Flash against its predecessor, averaged over the paper’s 33 benchmarks, plus one coding benchmark where parallel decoding shines:

model (100B)average scoreaverage TPFHumanEval+ scoreHumanEval+ TPF
LLaDA2.0-flash72.433.0888.416.45
LLaDA2.1-flash, Speedy72.345.9389.6313.81
LLaDA2.1-flash, Quality73.543.6489.639.18

Speedy commits 1.63 times as many tokens per pass1.63×, worked outSpeedy 5.93 TPF ÷ Quality 3.64 TPF ≈ 1.63score: 73.54 − 72.34 = 1.2 points to Quality as Quality and gives up 1.2 points of average score. On HumanEval+ the two modes score the same and Speedy still commits 1.5 times as much per pass1.5×, worked outboth modes score 89.63 on HumanEval+Speedy 13.81 TPF ÷ Quality 9.18 TPF ≈ 1.50. Quality beats LLaDA2.0 on both model sizes; Speedy scores slightly below it. For scale, the paper’s autoregressive comparison, Qwen3-30B-A3B-Instruct-2507, averages 73.09.

The headline speed, 892 tokens per second on HumanEval+ for the 100B model, is Speedy Mode with FP8 quantization. The same run without quantization does 746.66 tokens per second, and the quantized one scores 3.04 points lower on that benchmark. Speed also varies a lot by task: the same setting manages 248 tokens per second on the instruction-following benchmark IFEval.

One more lever: Multi-Block Editing lets the model revise earlier, finished blocks after it writes new ones. Under Speedy settings it lifts Flash’s average on the paper’s ablation set from 70.69 to 72.67 for 12% fewer tokens per pass12% fewer, worked outTPF without it 5.82, with it 5.14(5.82 − 5.14) ÷ 5.82 ≈ 0.117, with AIME 2025 going from 63.33 to 70.00.

The authors are candid about the limits. Very low unmask thresholds produce rough drafts, and editing only partly cures the “stuttering” they cause, repeated n-grams from slots sampled independently. Code and math tolerate Speedy well; for general chat they recommend Quality.

The toy canvas shows the same trade in miniature. Speedy (unmask line 0.4) writes everything in pass 1 and edits; Quality (0.9) writes the certain slots, then one city word, then the rest. Both stop when a pass changes nothing, which costs one checking pass at the end. For a single user a pass costs about 4.8 ms4.8 ms, worked outan 8B model in bf16: 8 × 10⁹ × 2 bytes = 16 GB of weightsan H100 reads 3.35 TB/s16 ÷ 3,350 s ≈ 4.8 ms per pass, however many tokens it writes on an H100 for an 8B model, so a checking pass is not free.

Over all 16 drafts Speedy averages 2.5 passes2.5 passes, worked out2 real drafts: write, check = 2 passes each8 drafts with one odd word: write, edit, check = 3 passes each6 two-a-side ties: write, check = 2 passes each, wrong(2 × 2 + 8 × 3 + 6 × 2) ÷ 16 = 40 ÷ 16 = 2.5 and is right in 10 of 1610 of 16, worked out2 real drafts + 8 repaired drafts = 1010 ÷ 16 = 62.5%; Quality takes 4 passes and is always right; the same fast pass with no editing is right in 2 of 16. That is 2.4 against 1.5 tokens per forward2.4 and 1.5, worked outSpeedy: 6 tokens ÷ 2.5 passes = 2.4Quality: 6 tokens ÷ 4 passes = 1.5. The numbers are a toy’s; the shape, faster with a residue of unfixable mixes, is the real one.

Why is replacing one word with another not enough?

Because the commonest mistakes of a fast parallel draft are not wrong words in the right places. They are right words in the wrong places. A slot that repeats its neighbor pushes everything after it one slot along; a word that never got written leaves everything after it one slot short.

Take a six-slot block that came out as New New Francisco : the Gate. Compared slot by slot with the real answer, 5 of 6 slots differ5 of 6, worked outdraft: New, New, Francisco, :, the, Gateanswer: San, Francisco, :, the, Golden, Gateonly slot 6 matches: 5 slots differ. A model that can only replace would need five replacements, and three of them overwrite a correct word that happens to sit one slot too far right. Re-reading slot 3 shows Francisco right after a New, which looks fine; there is no local reason to touch it.

The cure is an old idea from string matching. The Levenshtein distanceLevenshtein distanceThe smallest number of single-token edits that turn one sequence into another, where an edit is insert, delete or substitute. Spell-checkers use it; “kitten” to “sitting” is 3. between the draft and the answer is 3: delete the second New, insert a slot for Golden, substitute San for the first New.

LLaDA2.2-flash (July 2026), the 100B successor, builds exactly that into decoding. Its model card introduces two control tokens, DELETE and INSERT, “allowing diffusion decoding to edit sequence structure, remove redundant content, and create insertion slots”. The technical report in the LLaDA2.X repository lists four operations in all: keep, substitute, delete and insert. At a slot, DELETE removes the token there and INSERT opens a new masked slot just before it, which a later pass fills.

Blocks keep their size, 32 tokens in the released settings. After a round of edits the report pads the block with masks or trims its tail to restore the length, and at most one insertion is labeled per gap per round, so a long omission takes several rounds.

Where do the training labels come from? The report lines up each intermediate draft with the true text along their longest common subsequenceLongest common subsequence (LCS)The longest list of tokens that appears in both sequences in the same order, not necessarily side by side. For the draft and the answer above it is Francisco, :, the, Gate.. Tokens on it are kept; draft tokens left over between two of them become substitutions or deletions; a gap the draft is missing becomes an insertion. The labels are recomputed from each new draft, round after round.

Does it pay? The cleanest number is an ablation: the same base model fine-tuned on the same software-engineering trajectories, with and without Levenshtein editing, goes from 35.8 to 44.4 on SWE-bench Verified, 8.6 points8.6 points, worked out44.4 − 35.8 = 8.6. The report motivates it with failures that token replacement cannot repair: repeated n-grams, missing tool arguments, incomplete code edits, tool calls whose boundaries got mixed.

The released model is aimed at agents: 128K context, tool use, an agentic reinforcement-learning stage the team calls L-EBPO. Across 7 agentic benchmarks it averages 53.83 against 55.74 for Ant’s own autoregressive Ling-2.6-flash, ahead on three of them, at 1.64 times its average decoding throughput across 11 workloads (the autoregressive model running multi-token prediction with 4 draft tokens).

The report is just as clear about what is left. The two new tokens are not equally learnable: DELETE answers a visible redundancy, while INSERT needs the model to notice something that is absent and decide where it belongs, so repetitions are easier to remove than omissions are to repair. Editing also cannot guarantee valid syntax for nested JSON or multi-argument tool calls, and errors that slip through still compound over a long agent run.

Can a model be trained to fix its own mistakes?

LLaDA2.1 teaches editing on random corruption. A sharper idea is to train on the mistakes the model actually makes, and that is ProSeCo (Schiff, Belhasin and colleagues at Cornell and NVIDIA, NeurIPS 2026).

There was an older, training-free fix first. ReMDM (Wang and colleagues, NeurIPS 2025) changes only the sampler: during generation it puts some committed tokens back to [M] and lets the model fill them again, for example the ones that were least confident when they were written. That makes mistakes revisitable, but it decides what to revisit from confidence at writing time, which is exactly the number that could not tell New from San.

ProSeCo trains one network for both jobs. Each training step runs the model once on a masked sentence, as usual. It then takes the model’s own best guess at every position, a complete draft with the model’s typical errors in it, feeds that draft back in, and adds a loss for recovering the true sentence at every position. Same weights, no extra parameters; a draft with no masks in it is the cue to correct rather than fill.

At sampling time a correction loop runs between unmasking steps, every so many steps and for a set number of passes. The loop fills every mask with the current best guess so it has a whole draft to read, re-reads it, and writes the corrections back only into slots that were already written. The open slots stay masked, and the next unmasking step fills them using the corrected predictions.

On LLaDA-8B, fine-tuned for about 400B tokens, the gains show up even before any correction step runs, and grow with it:

LLaDA-8B, pass@1HumanEvalMBPPGSM8KMATH
standard masked fine-tuning58.5456.8888.8646.60
ProSeCo training, no correction steps69.5157.4191.3651.98
ProSeCo training and sampling (most compute)72.5669.3192.1955.06

That is 14 points on HumanEval14 points, worked out72.56 − 58.54 = 14.02GSM8K: 92.19 − 88.86 = 3.33; MATH: 55.06 − 46.60 = 8.46. The speed story is the one LLaDA2.1 told: write 4 to 8 tokens per unmasking step, run a correction loop every second step with up to 4 passes in it, and the paper reports about 2 to 4 times faster sampling than standard LLaDA decoding without losing accuracy.

Two details make the case. A model trained the standard way, given the very same correction loop at sampling time, fails to correct: it never learned to change a visible token. And on unconditional text generation from scratch, ProSeCo reaches a MAUVE score of 0.604, a measure of how close generated text is to real text, against 0.403 for ReMDM and 0.042 for plain masked diffusion (an autoregressive model scores 0.76 there).

The price is paid in training: every step needs a second forward pass to build the draft. The authors argue it is worth it; doubling the training steps of the standard model does not close the gap.

What if the answer’s length is not known in advance?

A masked model needs its canvas before it starts. LLaDA-style generation picks a generation length up front, and every slot must end up holding something. Too long and the model pads; too short and something gets squeezed out. Deletion and insertion repair the result, but a cleaner idea is to let length be part of the generation itself.

FlexMDM (Kim and colleagues, 2025) showed the recipe: generate by inserting masks and unmasking them, so the canvas grows where it needs to. Retrofitting LLaDA-8B into it took three days on 16 H100s and lifted GSM8K from 58% to 67%.

A2D2 (Tang, Zhu, Tao and Chatterjee, June 2026) builds on that and asks how to fine-tune such a model toward a reward. Its any-length model starts from an empty sequence. A second output head predicts, for every gap between tokens, how many tokens should be inserted there, and the sampler draws that many masks.

The editing comes in through two learned quality heads. One estimates how likely each newly written token is to be right; the other estimates whether each inserted mask will end up holding a real token. At each step low-quality tokens are put back to masks and low-quality inserted masks are removed. Fine-tuning then tilts the whole insert-and-unmask process toward high reward, with a derivation that does not need samples from the target distribution.

On LLaDA-8B-Base made any-length and fine-tuned with LoRA adapters, GSM8K rises from 35.71 to 60.96 at 128 steps, 25 points25 points, worked out60.96 − 35.71 = 25.25 over the same model instruction-tuned without A2D2, and code infilling on HumanEval-infill from 48.89 to 57.12 at 1,024 steps. The catch shows on molecules: it reaches 71.3% quality where a fixed-length baseline, GenMol, reaches 84.0%.

Set Diffusion (Arriola and Kuleshov, ICML 2026) removes the canvas instead. It generates sets of positions in any order, each set conditioned on the sets before it, and caches every set’s keys and values as soon as it is accepted. Block diffusion is the special case of fixed-size sets taken left to right. Length is flexible without an insertion network: generation simply stops once the model is confident the text has ended, and the cache grows after every step.

On GSM8K its sliding-window variant scores 66.41 against block diffusion’s 63.53, at slightly higher throughput, though the autoregressive baseline still scores 75.74. On story infilling with 110M-parameter models it beats block diffusion, 11.6 against 9.2 ROUGE-1, and decodes up to about 25% faster. Its catch is the mirror of this part’s theme: what Set Diffusion commits is cached and final. Length is flexible; mistakes are not.

What if no token were ever final?

Everything so far patches masked diffusion from outside. The other route changes the noise. In uniform-state diffusionUniform-state diffusionDiscrete diffusion whose noise replaces a token with a random token from the vocabulary instead of a mask. Fully noised text is random tokens; every slot always holds some token, and the reverse process may change any of them at any step., noising swaps a token for a random one, so there is no absorbing state. Every slot always holds some token, and generation may change any of them at any step. Editing is not an add-on; it is the process.

Masking won the last few years on perplexity, and Scaling Beyond Masked Diffusion Language Models (Sahoo and colleagues, February 2026) is the first scaling-law study to push back. Its uniform-state model, Duo, needs about 23 times the compute of an autoregressive model to match its perplexity. Yet at 1.7B parameters, fine-tuned on GSM8K and decoded left to right one token at a time, Duo scores 65.8 against 62.9 for the autoregressive model and 58.8 for masked diffusion.

The paper lists the uniform family’s advantages as self-correction and few-step sampling, and argues that perplexity is not comparable across diffusion families. Why the noise matters, and whether masking was ever the right default, is Part 5’s question.

The idea has reached production. DiffusionGemma, Google’s open-weights model converted from Gemma 4, starts each 256-token canvas from tokens drawn uniformly over the vocabulary, not from masks. Its report says tokens accepted earlier in the current canvas can still be revised; tokens in canvases already finished are frozen.

And then the honest test. Sumi (Ye and colleagues at Tohoku, June 2026) is a 7B uniform-state model trained from scratch on 1.5T tokens, fully open. Its authors gave it extra passes after the answer was written, 1, 3 or 7 more, a revision budget, and watched what it did with them.

It used them. Between 58% and 100% of revision passes overwrote at least one committed token, depending on the task. And it changed almost nothing: at most 1% of final tokens differed from the first pass, at most one answer in 30 flipped, and accuracy did not move on any task. Most overwrites were round trips, A to B and back to A. The authors’ reading: the model does not know how to improve a committed answer.

That is the toy’s flip-flop from the figure above, observed in a real 7B model. Being allowed to change any token is not the same as knowing which one to change.

Can a reward tell the sampler what to undo?

When the model cannot tell which token is wrong, ask something that can. VGB for masked diffusion (Jeon, Vuong and Tao, June 2026) brings a classical backtracking random walk into masked decoding. Its moves are unmask a slot or re-mask one, at any position, and it favors moves into partial drafts that a process verifierProcess verifierA model that scores an unfinished output: how likely is this partial draft to end in a good final answer? An outcome verifier, by contrast, only judges complete outputs. scores highly.

Why not just backtrack the autoregressive way? Because an autoregressive backtracker can only remove its latest token. To fix an early mistake it has to erase everything written after it. A masked state can re-mask the one bad slot and keep the rest.

On repairing Dyck-grammar strings, the bracket-matching kind, the masked version reaches 99.41% accuracy at an average of 29.98 moves, against 25.64% at 127.63 moves for the autoregressive one, 4.3 times fewer moves4.3×, worked out127.63 ÷ 29.98 ≈ 4.26. Started from low-reward outputs on molecule, DNA and protein design, editing beats the best method that starts over by 1.9, 18.0 and 15.7 percentage points.

The paper also has the cleanest statement of the parallel-edit problem. Take $1+2=4$. Changing the 4 to a 3 makes it valid. Changing the 4 to a 3 and the 2 to a 1 at the same time, each sensible on its own, makes it invalid again. That is New York : the Golden Gate in arithmetic.

The catch is the verifier. Training a good process verifier is expensive for long sequences and large vocabularies, the authors say, and the method has not yet been run on GSM8K, MATH500, HumanEval or SWE-bench, which they name as next. Test-time search of this kind returns in Part 7.

What is still open?

Knowing which committed token is wrong. Every method in this part can change a token; the hard part is choosing which one, and the year’s evidence on that is sobering.

Confidence at writing time cannot do it: New and San both had 0.5. Re-reading can, when the draft has a majority, but a two-a-side split gives every slot the same lukewarm doubt, and editing all of them at once just flips the mix. Sumi shows a real model with full permission to revise mostly going round in circles. ProSeCo shows a standard model given a correction loop not correcting at all. LLaDA2.2 finds deletions easier to learn than insertions, because a repeat is visible and an omission is not.

What works, works by bringing in information the bare model lacked: training on its own mistakes (ProSeCo), aligned edit labels (LLaDA2.2), a reward and a verifier (VGB, A2D2). Here is the map:

method what it can change how it decides a number the catch
LLaDA2.1 token editing any written token in the block top choice above the edit threshold 5.93 against 3.64 TPF for 1.2 points ties and stutter survive
LLaDA2.2 Levenshtein editing adds deleting and inserting slots labels from aligning drafts to the truth SWE-bench Verified 35.8 to 44.4 omissions are harder than repeats
ProSeCo written tokens, in correction loops trained on its own drafts about 2 to 4 times faster than LLaDA decoding a second forward pass per training step
A2D2 inserts slots, re-masks weak tokens learned quality heads, a reward GSM8K 35.71 to 60.96 behind fixed length on molecules
Set Diffusion the length ordered sets, cached at once GSM8K 66.41 against 63.53 committed sets are final
uniform-state diffusion any token at any step the reverse process itself Duo 65.8 against AR 62.9 on GSM8K Sumi’s revisions are round trips
VGB re-masks any slot a reward and a process verifier 99.41% against 25.64% on Dyck repair verifiers are costly to train

What would a solution look like? A signal, available during decoding and cheap to compute, that says “this token is likely wrong” and is calibrated: high on New, low on Francisco, and honest about the two-a-side tie. The LLaDA2.2 report points at one route, edits that check consistency with the already written tokens as they decode. The other route is to make wrong tokens less likely to need fixing at all, which is what Part 3’s ordering methods and Part 5’s choice of noise are about.

Things to watch in 2027: whether insertion becomes as reliable as deletion, whether uniform-state models learn to revise toward something rather than back and forth, and whether a verifier can be cheap enough to run inside every block. Part 9 puts this problem next to the other nine.

Rapid fire: can you do these from memory?

  1. What is an absorbing state, and why does it make a masked model's written tokens final?
  2. Four 50/50 slots are committed in one pass. What share of drafts is a real answer, and why does editing not rescue all of the rest?
  3. Write LLaDA2.1's two update sets in words. What number do both thresholds compare?
  4. Why can Speedy Mode lower the unmask threshold, and what does Quality Mode buy back?
  5. A draft has one repeated word near the start. Why does token replacement need many edits, and which two Levenshtein operations fix it?
  6. How does ProSeCo build the drafts it trains its corrector on, and where does the correction loop write its results?
  7. Name two ways to let the length of an answer be decided during generation.
  8. What did Sumi's revision-budget experiment find, and what does it say about uniform-state diffusion?

Where can you go deeper?

  • Tiwei Bie and others, LLaDA2.1: Speeding Up Text Diffusion via Token Editing (preprint, 2026), arXiv:2602.08676, code and models at inclusionAI/LLaDA2.X. The two thresholds, Speedy and Quality modes, the drafting and editing training streams, Multi-Block Editing.
  • Tiwei Bie and others, LLaDA2.2: Enabling Agentic Diffusion Language Models via Levenshtein Editing (technical report, 2026), in the LLaDA2.X repository, model card at LLaDA2.2-flash. Keep, substitute, delete and insert; labels from the longest common subsequence; the SWE-bench ablation.
  • Yair Schiff and others, Learn from Your Mistakes: Self-Correcting Masked Diffusion Models (NeurIPS 2026), arXiv:2602.11590, code. ProSeCo: training on the model’s own drafts, correction loops between unmasking steps.
  • Guanghan Wang and others, Remasking Discrete Diffusion Models with Inference-Time Scaling (NeurIPS 2025), arXiv:2503.00307. ReMDM: a training-free sampler that re-masks committed tokens.
  • Jaeyeon Kim and others, Any-Order Flexible Length Masked Diffusion (preprint, 2025), arXiv:2509.01025. FlexMDM: generation by inserting masks and unmasking them.
  • Sophia Tang, Yuchen Zhu, Molei Tao and Pranam Chatterjee, A2D2: Fine-Tuning Any-Length Discrete Diffusion for Adaptive Decoding (preprint, 2026), arXiv:2606.13565. Insertion and unmasking fine-tuned toward a reward, with learned quality heads.
  • Marianne Arriola and Volodymyr Kuleshov, Set Diffusion: Interpolating Token Orderings Between Autoregression and Diffusion for Fast and Flexible Decoding (ICML 2026), arXiv:2607.01775, project page. Flexible-length, any-order token sets with a cache update after every step.
  • Subham Sekhar Sahoo and others, Scaling Beyond Masked Diffusion Language Models (preprint, 2026), arXiv:2602.15014, project page. The first scaling laws for uniform-state and interpolating diffusion; Duo on GSM8K at 1.7B.
  • DiffusionGemma Team, DiffusionGemma Technical Report (2026), arXiv:2608.00146, model. Uniform noise over 256-token canvases; revision within a canvas.
  • Mengyu Ye and others, Sumi: Open Uniform Diffusion Language Model from Scratch (preprint, 2026), arXiv:2606.19005, project page, code. A 7B uniform-state model from scratch, and the revision-budget experiment.
  • Kijung Jeon, Thuy-Duong Vuong and Molei Tao, VGB for Masked Diffusion Model: Efficient Test-time Scaling for Reward Satisfaction and Sample Editing (preprint, 2026), arXiv:2606.28301, code. Reward-guided unmasking and re-masking with a backtracking random walk.