Is the work a diffusion model does on unwritten slots wasted?

Ask a language model to name a US city and its most famous bridge. Suppose it learned two answers equally often, each exactly six tokens: San Francisco: the Golden Gate and New York: the Brooklyn Bridge. This is the canvas the whole series runs on.

An autoregressive (AR) model writes the answer left to right, one token per forward passForward passOne run of the input through every layer of the network. It ends in a prediction, a probability for every word in the vocabulary, at each position the model was asked about.: six passes. A diffusion language model starts from six masked slots instead, and on every pass it predicts every masked slot at once and writes the ones it is sure of.

On pass 1, “:” and “the” are certain, so they are written. The other four slots sit at 50/50: San or New, Francisco or York, Golden or Brooklyn, Gate or Bridge. On pass 2 nothing is certain, so the model writes one city word, say San, on a coin. Now Francisco, Golden and Gate are certain, and pass 3 writes all three: six tokens in three passes, 2 per pass2 tokens per pass, worked out6 tokens ÷ 3 passes = 2An AR model is 1 per pass by definition..

The puzzle Before it writes the city, a diffusion model has already computed a 50/50 belief about the bridge. Is that wasted work, or is it part of how it decides the city?
The 30-second version It is part of the decision. A masked diffusion model is trained to predict every unknown slot, and the slot it writes attends to its neighbors' half-formed predictions. Cut those predictions out and the model gets faster but worse: on logic puzzles, 25 points worse. 2026's papers call them latent tokens, recycle them, add depth to them, and carry them across wiped text. The price is oversight: computation you never see written down is computation you cannot read.

Now count what the diffusion model computed. Pass 1 predicted 6 slots, pass 2 predicted 4, pass 3 predicted 3: 13 predictions13, worked outpass 1: all 6 slots are masked, 6 predictionspass 2: 4 still masked, 4 predictionspass 3: 3 still masked, 3 predictions6 + 4 + 3 = 13, for 6 written tokens to write 6 tokens. The AR model made 6. Seven of the diffusion model’s predictions were never written anywhere.

The bridge slot is the clearest case. Its 50/50 was computed on pass 1, computed again on pass 2, and only written on pass 3. An AR model never does this: it never predicts a slot it is not about to write.

Is that work wasted? For one user on one GPU it is nearly free. A forward pass of an 8B-parameter model must read its 16 GB of weights from memory, about 4.8 ms4.8 ms, worked out8 × 10⁹ parameters × 2 bytes (bf16) = 16 GBan H100 reads 3.35 TB/s = 3,350 GB/s16 ÷ 3,350 s ≈ 0.0048 s ≈ 4.8 msThe same for a pass that fills 1 slot or 20. on an H100, whether it predicts one slot or twenty. On a busy server the extra slots do cost compute, which is a story for Part 8.

This part asks a different question: is the computation on unwritten slots useless? 2026’s answer is no, and the year’s papers went on to recycle it, deepen it, and ask what it means for anyone trying to read the model.

What breaks if the model stops predicting the slots it will not write?

Andre He, Sean Welleck and Daniel Fried at Carnegie Mellon asked exactly that, in Reasoning with Latent Tokens in Diffusion Language Models. Their starting point: a masked diffusion model is trained to predict a distribution over every unknown token, including the ones it will not decode this step.

They call those predictions latent tokensLatent tokensThe predictions a diffusion model computes on a pass at positions it does not write on that pass. They are never output, but the position being written can attend to them, so they shape what gets written.. In the bridge example, the four 50/50 slots on pass 1 are latent tokens: computed, attended to, and then thrown away.

The ablation is clean: each masked slot may attend only to the written tokens and to itself, never to another masked slot. Then nothing about a slot depends on the other unknowns, so the model only has to run the slot it is about to write; the rest are dropped from the pass.

Trained the same way and decoded the same way (most confident slot first), here is what that costs:

taskfull model: accuracyno latent tokens: accuracytokens processed, relative
Sudoku puzzles80.2%68.1%0.54×
Zebra logic grids96.9%71.5%0.52×
Countdown arithmetic27.3%18.4%0.60×

Half the computation, and accuracy falls on every task: by 25.4 points25.4 points, worked outfull model on Zebra: 96.9%without latent tokens: 71.5%96.9 − 71.5 = 25.4 points on the Zebra puzzles. The slot being written was leaning on its neighbors’ guesses. As the paper puts it, “accurate prediction at the decoded position relies on joint reasoning about the distribution of undecoded tokens.”

Sudoku makes the reason easy to see. Whether a cell can be a 7 depends on cells that are still empty, and those depend on each other. A model that writes one cell while holding a rough guess for every other cell is doing constraint propagation in its activations. Take the guesses away and it decides each cell half blind.

The authors then turned the all-or-nothing ablation into a dial. In a semi-causal version, masked slots see each other only in decoding order, and at test time you choose how many future masked slots n the current one may see. With n = 0 you get the ablated model; with every slot visible, the full one. No retraining.

As latent tokens are added back, the generated text’s perplexity (how surprised a strong reference model is by it; lower is better) drops from about 48 to about 40, against 36.5 for a full masked model in the same comparison. On the puzzles, accuracy generally rises with the count.

One honest limit. The puzzle models are small, from 6.4M to 92.4M parameters. At that size the wall-clock speed-ups were not significant, so the paper reports tokens processed rather than seconds. The mechanism is clear; how it scales to a 100B model is not measured.

Can an autoregressive model get latent tokens too?

If latent tokens are a mechanism and not an accident of diffusion, an AR model should be able to use them. The same paper tests this, and I think it is the result that will travel furthest.

They take an AR transformer, remove its causal mask, and train it on a clean prefix followed by a masked suffix, with a loss on every masked position (multi-token prediction). At test time it still writes one token per pass, strictly left to right. But every pass also runs the unwritten suffix through the network as masks, so the next token can attend to rough guesses about everything after it.

taskplain ARAR with latent tokensmasked diffusion, confident first
Sudoku, generate a board27.4%78.3%86.4%
Sudoku, solve a puzzle5.5%53.6%80.2%
Zebra logic grids54.7%68.8%96.9%
Countdown arithmetic2.6%8.6%27.3%

Solving Sudoku goes from 5.5% to 53.6%, nearly ten timesnearly ten times, worked out53.6 ÷ 5.5 ≈ 9.7, with the same left-to-right writing order. The diffusion model, which can also choose its order, still wins every row, so order matters too. But a large part of diffusion’s edge on these lookahead tasks turns out to be the latent computation, and an AR model can have it.

It is not free: the masked suffix goes through the network on every pass. A diffusion model gets latent tokens as a side effect of its training objective; an AR model has to pay for them on purpose.

Why throw away the predictions it did not write?

Go back to pass 2 of the bridge example. Pass 1 worked out that the bridge slot is Golden or Brooklyn at 50/50. Then standard decoding, called remasking, keeps the confident predictions and resets the rest to plain masks, so pass 2 starts from [M] [M] : the [M] [M], as if pass 1 had never looked at those four slots.

Residual Context Diffusion (RCD), from Berkeley in January, asks why the rest should be discarded at all.

RCD turns each discarded prediction into a vector and feeds it into that slot’s input on the next pass. The vector is the prediction’s average token embedding: every token’s embeddingEmbeddingThe vector a model looks up for each token before its first layer. A soft embedding mixes these vectors, weighted by probabilities, so it can stand for “half Golden, half Brooklyn”. weighted by its probability. A mixing weight decides how much of the slot’s input is this residual and how much is the plain mask embedding:

\[\mathbf{r}_i = \sum_{v} p_i(v)\,\mathbf{e}_v, \qquad \alpha_i = \frac{H(p_i)}{\log V}, \qquad \mathbf{x}_i = (1-\alpha_i)\,\mathbf{e}_{\text{mask}} + \alpha_i\,\mathbf{r}_i\]

In words: slot $i$’s residual is its predicted distribution $p_i$ turned into one vector. Its weight $\alpha_i$ is the prediction’s entropyEntropyHow spread out a prediction is, in nats or bits: 0 when one token has all the probability, largest when every token in the vocabulary is equally likely. A 50/50 between two words has entropy log 2. divided by the largest entropy possible over a vocabulary of $V$ tokens, so it lies between 0 and 1. The next pass sees a blend of “masked” and “here is what I thought”.

The weight runs the way you might not expect: the more uncertain a slot, the more of its residual goes forward. The authors’ argument is that uncertain slots may hold the information that most shapes the later steps. The bridge slot enters pass 2 as half Golden, half Brooklyn, not as a blank.

Backpropagating through a chain of passes is memory-hungry, so RCD trains in two stages: a small frozen model (1.7B parameters, for SDAR) supplies the predictions that become residuals, and the bigger model learns to use them. Converting a model takes about 300 million tokens.

On SDAR-8B, a block diffusionBlock diffusionDiffusion inside fixed-size blocks that are written left to right: the model denoises one block of, say, 64 slots, keeps it, then moves to the next. Nearly every shipped diffusion LLM works this way. model with 64-slot blocks, MATH500 goes from 64.2% to 74.2% and AIME 2024 from 7.08% to 18.75%, 2.6 times2.6 times, worked out18.75 ÷ 7.08 ≈ 2.65Other configurations gain less: AIME 2025 goes from 9.79% to 16.04%. as many problems. At the baseline’s best accuracy, RCD needs up to 4 to 5 times fewer denoising steps.

The extra cost at inference depends on the model. On SDAR it is under 1%under 1%, worked outSDAR on GSM8K: 149.45 tokens/s without RCD, 148.41 with(149.45 − 148.41) ÷ 149.45 ≈ 0.7% of throughput; on LLaDA’s GSM8K run it is 12.6%12.6%, worked outLLaDA on GSM8K: 75.00 tokens/s without RCD, 65.56 with(75.00 − 65.56) ÷ 75.00 ≈ 12.6%.

The catch is in the training: the residuals come from a frozen helper model, but at test time they come from the model’s own previous pass. On-Policy Residual Learning (ORL), from Together AI and university collaborators in October, argues that this mismatch is the whole problem.

Their point: residuals decide which tokens get written and which stay open, so they decide which states the model visits next. Learn from residuals produced by some other model, or from cached trajectories, and the model trains on states it will never see. ORL regenerates its own rollouts at every optimization step, and adds an entropy-contraction loss: a penalty whenever a denoising step fails to make the open slots more certain by a set fraction.

The payoff is mostly speed, measured in tokens per forwardTokens per forward (TPF)Tokens written per forward pass, averaged over a task. An AR model is 1 by definition; the bridge example above is 2.. On a 4B diffusion model converted from Qwen3-4B, HumanEval goes from 1.8 to 11.1 tokens per forward, 6.2 times6.2 times, worked out11.1 ÷ 1.8 ≈ 6.2Across GSM8K, MATH-500, HumanEval and MBPP the gain ranges from 2.5× to 6.2×. as many. Carrying the belief forward lets the model commit sooner. Remove the entropy-contraction loss and tokens per forward drop by 28% on GSM8K.

End to end, on one H100 serving one request with the authors’ own engine, that model reaches about 1,298 tokens per second against the starting model’s 187, 6.9 times6.9 times, worked out1,298 ÷ 187 ≈ 6.9On SGLang instead of the authors’ engine the paper reports roughly 5×. faster. It also beats AR models sped up with speculative decodingSpeculative decodingA small, fast drafter proposes several tokens; the big AR model checks them all in one forward pass and keeps the ones it agrees with. EAGLE-3 and DFlash are two such drafters.: 1.74 times1.74 times, worked outORL: 1,298 tokens/s; AR with DFlash: 746 tokens/s1,298 ÷ 746 ≈ 1.74This is under the paper’s relaxed verification; under strict verification ORL is 2.26× faster but 2.4 points less accurate. faster than DFlash at matched accuracy.

It transfers cheaply, too: on LLaDA-2.1-mini, 36 H100-hours and 4.2M self-generated tokens lift GSM8K from 5.6 to 8.5 tokens per forward. On DiffusionGemma, already at 26.0 tokens per forward on GSM8K, it adds another 1.1 to 1.2 times.

The same idea, carrying a slot’s whole belief forward instead of collapsing it into a hard choice, returns in Part 5 as an answer to what the noise should be. Remember it: later in this part, it is what makes a diffusion model hard to read.

Can a model think longer without writing more tokens?

Latent tokens give a model more width: more positions to compute at once. The other knob is depth. A transformer’s depth is fixed by its layer count, so a hard slot gets exactly as many layers of computation as an easy one.

Looped Diffusion Language Models (LoopMDM), from KAIST and KRAFTON in May, loops two middle layers: the hidden states come out of the block and go back into it, S times, before moving on. Looping adds depth with no new weights. In training S is drawn at random from 1 to a maximum, so the model learns to work with any loop count.

On language modeling, a 170M-parameter LoopMDM matches the test loss of a same-size masked diffusion model without loops using 3.34 times fewer training FLOPs on LM1B, and 2.34 to 2.95 times fewer on two larger corpora. It also beats a deeper non-looped model with similar compute per step: at this scale, looping is a better way to add depth, not just a cheaper one.

On math, a 125M-parameter model trained on TinyGSM with up to 8 loops gains 8.5 points on GSM8K at 16 loops, twice what it ever saw in training, over a same-size model without loops. The loop count becomes a dial at test time: more loops, more thinking, same weights.

The dial can also turn itself. If the model stops looping once its hidden states change by less than 10% from one loop to the next, the average loop count falls from 12 to about 5, while generation quality barely moves (generative perplexity 38.53 against 39.11). Most of the loops land on the middle of the denoising process, where the canvas is half written.

What do the extra loops do? Attention from masked slots to other masked slots rises with the loop count, which the authors call suggestive, not causal. It fits this part’s story: the extra depth seems to go into the unwritten slots reasoning about each other.

LoopMDM gives every masked slot the same number of loops. ALoDLM: Adaptively Looped Diffusion Language Models, from Amazon and university collaborators in October, calls that a computation-difficulty mismatch: within one canvas, some unknown tokens are easy and some need far more computation.

In ALoDLM a recurrent core (all 28 layers of the 1.7B model; the middle 16 of the 8B) can run up to 4 times inside one denoising step. After each loop, every open slot that is confident enough (entropy below a threshold) is written and fed back as a plain token. The rest keep their hidden states, not a fresh mask, and go round again; if none qualifies, the lowest-entropy slot is written so decoding moves.

When to stop is a separate, learned decision: an exit gate scores each open slot after each loop, and the step ends when, on average, they say they are done. Training treats each slot’s exit loop as a hidden variable under a likelihood bound with a prior that favors exiting early, and trains the gate with a REINFORCE-style estimator.

At both sizes it is the strongest model in its comparison. Averaged over eleven benchmarks (knowledge, math, science and code), the 8B scores 80.3, against 78.5 for Qwen3-8B, the AR model of the same size, and 75.1 for the best other diffusion model, WeDLM-8B. The 1.7B scores 65.5 against Qwen3-1.7B’s 63.8.

It stays fast. On one B200 serving one request, at a matched 93.25% on GSM8K, ALoDLM-8B writes 612.4 tokens per second against WeDLM-8B’s 564.2, 8.5% faster8.5%, worked out612.4 ÷ 564.2 ≈ 1.085compute per token: 133.5 against 154.6 GFLOPs, 1 − 133.5 ÷ 154.6 ≈ 13.6% less, and the paper reports about 2.7 times the throughput of Qwen3-8B served with vLLM at comparable accuracy.

How much looping does it actually do? Between 1.6 and 2.34 loops per token on the 8B model, depending on the exit threshold; in that ablation the average score rises from 77.9 to 79.1 across the range. And the tokens most inclined to loop again are numbers: their average chance of stopping after the first loop is 0.369, against 0.417 across all tokens.

Can it keep reasoning after its text is wiped?

So far the hidden computation lives inside a pass or inside a step. Register Tokens for Bounded-State Reasoning in Diffusion Language Models, from Wisconsin, Microsoft Research and UC San Diego in September, pushes it further: can a diffusion model keep reasoning after the text it wrote has been cleared away?

The motivation is cost. Long reasoning means a long context, and a diffusion model attends over all of it on every pass. Instead, write the reasoning in chunks (128 tokens for math, 64 for code) and clear each chunk once it is done.

What survives is four register tokens: fixed slots in the prompt whose job is to carry the reasoning forward. After a chunk is done, one extra forward pass reads the last layer’s hidden states at those four positions. The next chunk starts from the prompt alone, with those four vectors put back in place of the registers’ input embeddings. The text is gone; four vectors of state remain.

The comparison that matters is against carrying text: the same four slots, but holding the last four tokens of the previous chunk. Registers beat that on every one of the twelve model and benchmark pairs. On LLaDA, GSM8K rises from 40.6% to 49.1%, 8.5 points8.5 points, worked outLLaDA on GSM8K: 49.1% with registers, 40.6% carrying text49.1 − 40.6 = 8.5; on Dream, MBPP coding rises from 21.4% to 40.9%, 19.5 points19.5 points, worked outDream on MBPP: 40.9% with registers, 21.4% carrying text40.9 − 21.4 = 19.5. After reinforcement learning on a long-arithmetic task, registers score 39.7 on the paper’s reward against 31.6 for carried text.

Bounded state is not free, though. Keeping the whole context still wins on math: the same register model scores 63.2% on GSM8K with full context against 48.9% with the bounded carry. What the registers buy is time. On LLaDA at 2,048 tokens of output, the bounded run takes 75.6 seconds against 260.7 with full context, 3.4 times3.4 times, worked out260.7 s ÷ 75.6 s ≈ 3.4At 3,584 tokens the gap grows to 736.3 ÷ 132.4 ≈ 5.6. faster, and the gap grows with length.

Notice what has happened. After the first chunk, the reasoning that carries the model forward exists only as four vectors that nobody reads, and the text a person could inspect has been deleted on purpose. Which brings us to the question 2026 asked next.

How much of a diffusion model’s computation can anyone read?

With AR models, safety work leans on something simple: the model writes its reasoning down. Every token of a chain of thought is a place where a person, or another model acting as a monitor, can look. How Transparent is DiffusionGemma?, from Google DeepMind in June, asks whether diffusion keeps that property.

The authors split transparency in two. Variable transparency: can we understand the model’s intermediate states? Algorithmic transparency: can we use those states to reconstruct how it reached its answer? For the first they borrow a measure from earlier work by Brown-Cohen and colleagues, opaque serial depth.

Opaque serial depthOpaque serial depthThe longest chain of computation that runs from one readable state to the next with nothing readable in between. The paper writes the model and its sampler as a circuit, marks the nodes a person can read (text in, text out), and measures the longest path that avoids them. is the length of the longest stretch of computation with no readable state inside it. Text counts as readable. For an AR model the stretch is one forward pass: each pass ends in a token, and the next pass starts from text.

For Gemma 4 (the 26B mixture-of-experts version with about 4B parameters active) the paper counts 21,235, at 256k tokens of context. The number itself is circuit depth, not seconds; what matters is the comparison.

DiffusionGemma, Google’s diffusion model converted from that same Gemma 4, writes a canvas of 256 slots over several denoising steps. After each step the canvas is tokens, and you can read them. So it should look as transparent as the AR model.

It does not, because the canvas is not all that passes from one step to the next. DiffusionGemma also passes a self-conditioningSelf-conditioningFeeding a model’s own previous prediction back in as an extra input. Here it is, for every slot, the predicted distribution turned into a soft embedding: each token’s embedding weighted by its probability. signal: for every slot, its whole predicted distribution turned into one continuous vector, each token’s embedding weighted by its probability. That vector goes around the readable canvas.

A readable state only breaks the chain if everything the next step uses passes through it. The self-conditioning vector never becomes text, so the longest unreadable path runs through every denoising step. At the worst case of 48 steps the paper counts 608,016, 28.6 times28.6 times, worked outDiffusionGemma, 48 steps: 608,016Gemma 4: 21,235608,016 ÷ 21,235 ≈ 28.6Both at 256k tokens of context; an upper bound measured on the circuit. Gemma 4’s. With adaptive stopping, which usually ends after 12 to 16 steps, the paper notes the typical depth is about a third of that.

Look at that vector again. It is a probability-weighted average of token embeddings, the same object RCD and ORL feed forward to make a model better and faster. The computation that buys quality is exactly the computation nobody can read.

The fix is simple. Squeeze the carried signal through a token bottleneck: for each slot keep only the tokens with probability at least 0.03 (or the top 8), and spread the leftover probability evenly over everything else. What passes between steps is then a short list of readable candidates per slot, at most 33at most 33, worked outeach kept token has probability ≥ 0.0334 of them would add up to at least 34 × 0.03 = 1.02 > 1so at most 33 can be kept of them: San or New, Golden or Brooklyn.

On code (Natural2Code, LiveCodeBench), competition math and GPQA, those settings match the unrestricted model’s scores. And now every intermediate state can be read, so the chain is cut at every step: opaque depth 23,571, 1.1 times1.1 times, worked outDiffusionGemma with the bottleneck: 23,57123,571 ÷ 21,235 ≈ 1.11 Gemma 4’s.

Are the candidates actually meaningful? On 800 real chat prompts from WildChat, most of the tokens that survive the bottleneck turn out to be the slot’s final token or one that lands in a neighboring slot. You can look at an intermediate step and see what the model is weighing.

What does the model do between readable states?

Readable snapshots are not a readable process, and here diffusion is genuinely different. Every slot can change on every step, so a diffusion model can run distributed algorithms across its canvas that no left-to-right writer would. The paper’s case studies show what that looks like.

It plans the length first. Asked to explain photosynthesis in two sentences, the model predicts end-of-text padding beyond about position 55 after the very first step; the answer comes out at 50 tokens.

It changes its mind. On a question about square numbers its top answer at step 4 is 9; by step 7 it is 8, the correct one, with higher confidence. Self-correction is not guaranteed: sometimes it locks in early.

It smears. At step 3 of that same problem the canvas is roughly an even mixture of a sequence of one-digit numbers and a sequence of two-digit numbers, both held at once, until step 4 collapses it to the single digits. A single token can smear too: “ bracket” spread across many positions of a docstring before settling in one.

That is the New Francisco problem seen from the inside. The canvas holds San Francisco and New York at the same time; if a sampler commits slots from both answers in one step, it writes a mixture. (Part 3 is about exactly that.)

It uses scratch work that never appears. Given a Fibonacci-like sequence in which the digit 3 has been replaced by the word “Gold”, the model often writes a 3 first, uses it to compute the later terms, and then replaces it. The 3 never appears in the final output. If you only read the answer, you would never know it was there.

It writes code skeleton first. For a longest-subarray function: the outline, then the core loop, then variables and comments. Not chronological, but close to how many programmers work.

Does any of this make DiffusionGemma harder to supervise? On monitorability, the practical test, the paper adapts an established suite of evaluations (does the chain of thought reveal an intervention, a process, a property of the outcome?) and finds DiffusionGemma about as monitorable as Gemma 4.

One caution from the same section: DiffusionGemma’s chains of thought were shorter than Gemma 4’s on every evaluation, and longer chains of thought tend to be more monitorable. Similar today is not a guarantee.

What do its activations know?

If the interesting computation lives in vectors, you need tools that read vectors. Two 2026 papers built them for diffusion models. Subliminal Clocks, from Sapienza, EPFL and NVIDIA, starts from an odd fact. Diffusion LLMs such as LLaDA and Dream are never told which denoising step they are on: there is no timestep input. Does the model know anyway?

Probes trained on the residual streamResidual streamThe running vector at each position that every layer reads from and adds its output to. Probing it means training a small model to predict some property from those vectors. of LLaDA and Dream recover the fraction of the canvas already written, with R² above 0.5 at every layer of LLaDA.

The signal is simple in shape. In each layer, the top principal component of the step-by-step mean activations captures most of their variance, typically above 90%. And it is causal: push the activations along that direction toward “later” and the model grows more confident, with lower entropy; push toward “earlier” and it grows less sure. A random push of the same size does nothing coherent.

The clock matters for quality, too. Steering Dream all the way to “nearly finished” drops its GSM8K score from 68.8% to 23.1%, which the authors trace to the model emitting end-of-text tokens too early. LLaDA stays within a few points.

DLM-Scope, from the University of Hong Kong and Alibaba’s Tongyi Lab, brings sparse autoencodersSparse autoencoder (SAE)A wide, mostly-zero dictionary trained to rebuild a layer’s activations from a few active features at a time. The features often line up with human concepts, so they are a standard tool for reading and steering a model. to Dream-7B and LLaDA-8B. The features come out interpretable, as they do in AR models, and two results are new.

First, splicing a sparse autoencoder into an early layer of a diffusion model can lower its masked-token loss; in AR models (Qwen2.5-7B and Llama 3 8B in the comparison) splicing one in raises the loss, as you would expect from an approximation.

Second, steering with those features during denoising often works better in diffusion models than the same steering does in AR models, mostly in the deep layers. The features also carry over from Dream’s base model to its instruction-tuned version, in every probed layer but the last.

Neither paper reads a diffusion model’s reasoning yet. But both show that its internal state has structure you can find, name and push on, which is the first thing interpretability needs.

What is still open?

Every pass computes far more than it writes. 2026 showed that the extra computation does real work, built ways to make it do more (recycle it, loop it, carry it across wiped text), and then asked what that does to anyone who wants to read the model. Here is what nobody knows yet.

How much hidden computation is the right amount? Latent tokens help but cost passes; ALoDLM settles at 1.6 to 2.34 loops per token; registers lose to full context on math. Nobody has a principled way to say how much computation a slot deserves, beyond training a gate to guess.

Who pays for it? For one user, unwritten predictions ride along with a weight read that happens anyway. On a server full of users the passes are limited by arithmetic, not by memory reads, and every latent token becomes real compute. The methods here are measured on single-request serving, mostly; Part 8 is about what happens when the batch fills.

How do you train it? Backpropagating through many steps is expensive, so RCD trains against a frozen helper, ORL regenerates rollouts at every optimization step, and ALoDLM trains its exit gate with a high-variance estimator. Each is a workaround for the same problem: the model’s own trajectory is part of what it has to learn from.

Can we keep it readable? The bottleneck result is the hopeful one: DiffusionGemma’s carried signal could be made readable at no measured cost. But residual context, kept loop states and register tokens are all new continuous channels, and each is another place for the chain to run unread. None of the papers that add them measures opaque depth. A register, by design, carries reasoning across a chunk boundary with nothing readable in between.

Can we read the process, not just the snapshots? Smearing, scratch values that get erased, code written skeleton first: algorithmic transparency is harder for diffusion than for AR, by the paper’s own account. Probes and sparse autoencoders are a start. And monitorability has been measured on one diffusion model, against one AR model, with chains of thought that were shorter.

Does any of it hold at scale? The latent-token evidence comes from puzzle models under 100M parameters; LoopMDM’s from models under 200M. ALoDLM at 8B and ORL on DiffusionGemma are the exceptions. The next year will show whether “think without writing” survives the jump to frontier models.

The trade underneath is simple to state. Computation you cannot read buys accuracy and speed. How much of it we want, and how much of it we insist on being able to see, is a question the field has only just started to ask out loud.

Rapid fire: can you do these from memory?

  1. A block of 32 slots is written 4 per pass. How many slot predictions does the model make, and how many per written token?
  2. What exactly does the latent-token ablation remove, and roughly what does it save and cost?
  3. How do you give an AR model latent tokens without changing its left-to-right writing order?
  4. Write RCD's residual for one slot. Which slots get the most weight on their residual, and why?
  5. Why does ORL insist that residual learning is on-policy?
  6. LoopMDM and ALoDLM both loop layers. What does ALoDLM decide per slot that LoopMDM does not?
  7. Define opaque serial depth. Why is DiffusionGemma's 28.6 times Gemma 4's, and what brings it down to 1.1 times?
  8. Name two behaviors of a diffusion model that you can only see in its intermediate canvases.

Where can you go deeper?

  • Andre He, Sean Welleck and Daniel Fried, Reasoning with Latent Tokens in Diffusion Language Models (preprint, 2026), arXiv:2602.03769. The ablation of joint prediction, latent token modulation as a speed-quality dial, and latent tokens for AR models through multi-token prediction.
  • Yuezhou Hu et al., Residual Context Diffusion Language Models (preprint, 2026), arXiv:2601.22954; project page, code. Entropy-weighted residuals from discarded predictions and the two-stage training.
  • Yifan Yu et al., On-Policy Residual Learning for Diffusion Language Models (preprint, 2026), arXiv:2610.20687. Why residual learning must be on-policy, entropy contraction, and the speed comparisons against EAGLE-3 and DFlash.
  • Sanghyun Lee et al., Looped Diffusion Language Models (preprint, 2026), arXiv:2605.26106. Looping early-middle layers, loop count as a test-time dial, and adaptive loop counts.
  • Liancheng Fang et al., ALoDLM: Adaptively Looped Diffusion Language Models (preprint, 2026), arXiv:2610.04198; project page, code. Token-adaptive latent recurrence, the exit gate and its training bound, at 1.7B and 8B.
  • Albert Ge et al., Register Tokens for Bounded-State Reasoning in Diffusion Language Models (preprint, 2026), arXiv:2609.16372; code. Reasoning across cleared chunks with a fixed-size continuous state.
  • Joshua Engels et al., How Transparent is DiffusionGemma? (preprint, 2026), arXiv:2606.20560. Variable and algorithmic transparency, opaque serial depth, the token bottleneck, the case studies and monitorability.
  • Maximo Eduardo Rulli et al., Subliminal Clocks: Latent Time Modelling in Diffusion Language Models (preprint, 2026), arXiv:2607.01774. Probing and steering the denoising progress that models are never told.
  • Xu Wang et al., DLM-Scope: Mechanistic Interpretability of Diffusion Language Models via Sparse Autoencoders (preprint, 2026), arXiv:2602.05859. Sparse autoencoders for Dream and LLaDA, steering during denoising, and stability across post-training.
  • DiffusionGemma Team, DiffusionGemma Technical Report (preprint, 2026), arXiv:2608.00146. The model whose transparency is measured above: Gemma 4 converted to diffusion with under 10% of its training tokens.