Why does RL need the probability of an answer?
Reinforcement learning is how language models learned to reason in 2025: sample a few answers to a problem, check which ones are right, and nudge the model toward them. GRPO, the recipe behind DeepSeek-R1, does exactly that. When we set out to post-train a diffusion language model with it, the code asked for one number per sampled answer that the model could not give: the log-probability of that answer.
That missing number is what this part is about. A diffusion language model writes many tokens per forward pass, and the earlier parts followed what that buys and what it breaks. Training it to reason with RL breaks something else: the quantity every policy-gradient method is built on.
The 30-second version
An AR model writes in one fixed order, so the answer's probability is a product of six next-token probabilities, and a causal mask lets one forward pass return all six: 0.5 × 1 × 1 × 1 × 1 × 1 = 0.5. A diffusion model can reach the same six tokens along 720 orders, and its probability is the sum over all of them; scoring even one order costs a forward pass per step, because a bidirectional model sees a different pattern of masks at each. What it can afford is the ELBO, an average over orders of the log of each path, which is a lower bound and exact only when every order agrees. RL cares because the policy gradient multiplies each answer's reward by the gradient of exactly this log-probability: use a loose stand-in and the update drifts away from the one you wanted.The series’ running example makes it concrete. The prompt is Name a US city and its most famous bridge, and the model has learned two answers equally often, six tokens each: San Francisco: the Golden Gate and New York: the Brooklyn Bridge. A diffusion model starts from six masked slots and fills them over a few forward passes (one run of the whole network each). On its own, slot 1 is San or New at 0.5 each, and the colon and the are certain.
RL wants the expected reward to rise. Write the model as a policy $\pi_\theta$ that samples an answer $y$, and the reward as $r(y)$. The gradient of the expected reward has a famous form:
\[\nabla_\theta\, \mathbb{E}_{y \sim \pi_\theta}\big[r(y)\big] \;=\; \mathbb{E}_{y \sim \pi_\theta}\Big[\big(r(y) - b\big)\,\nabla_\theta \log \pi_\theta(y)\Big]\]In words: sample answers, compare each reward with a baseline $b$, push up the log-probability of the answers that beat it and push down the rest, each in proportion to its margin. The log is there because $\nabla \pi = \pi\, \nabla \log \pi$: the sampling supplies the $\pi$, and what is left to compute is the gradient of the log-probability of each answer actually drawn.
GRPOGRPOGroup Relative Policy Optimization (Shao et al., 2024). For each prompt it samples a group of answers, scores them, and gives each answer the advantage (its reward minus the group’s mean, divided by the group’s spread). It needs no learned value network, which is why it became the default for reasoning RL. uses the group of answers sampled for one prompt as its own baseline: each answer’s advantage is its reward minus the group’s mean, divided by the group’s spread. It also limits how far one update can move by clipping the ratio $\pi_\theta(y) / \pi_{\text{old}}(y)$ near 1. Both pieces need the same number: the probability of the sampled answer.
For an autoregressive (AR) model that number is free. Its probability factorizes left to right, one next-token probability per position, and a single forward pass with a causal mask (each position sees only the tokens before it) returns every factor at once. For San Francisco: the Golden Gate the factors are 0.5, 1, 1, 1, 1, 1, and the product is 0.50.5, worked outSan: 0.5, the only uncertain choicethen Francisco, the colon, the, Golden, Gate: 1 each0.5 × 1 × 1 × 1 × 1 × 1 = 0.5Along a fixed order, an answer’s probability is the product of its step probabilities..
A diffusion model writing the same answer left to right gets the same product, but pays more for it. Its attention is bidirectional, so its prediction for slot 2 depends on which other slots are still masked, and every state needs its own forward pass: six passessix passes, worked outone order writes one slot per step: 6 stepseach step starts from a different pattern of masks: 1 pass each6 × 1 = 6 passes, against 1 for the AR model to score one order of a six-token answer. And left to right is only one of its orders.
Why can’t a diffusion model say how likely its answer is?
Because it can reach the same answer along many paths, and the probability of the answer is the sum over all of them. Writing one slot at a time, six slots can be filled in 720 orders720, worked out6 choices for the first slot written, 5 for the second, and so on6 × 5 × 4 × 3 × 2 × 1 = 720, and a 256-token answer in a number of orders with 507 digits507 digits, worked out256! = 256 × 255 × … × 2 × 1log₁₀(256!) ≈ 506.9so 256! ≈ 8.6 × 10⁵⁰⁶, a number with 507 digits. Real samplers also commit several slots per pass, which only adds paths. For a sampled answer $y$:
\[\pi_\theta(y) \;=\; \sum_{\tau\, \to\, y} p_\theta(\tau)\]the sum, over every denoising path $\tau$ that ends at $y$, of the probability of that path.
For the perfect model of the last figure the sum is harmless, because every order gives the same product. Write the answer bridge first: Golden 0.5, then everything else is certain, 0.5 again. A model whose predictions all come from one consistent table of probabilities agrees with itself, whatever the order.
A trained network is not such a table. Each prediction is a separate output for a separate pattern of masks, learned from randomly masked text, and nothing forces the predictions to fit together. Take a toy model that writes Golden 0.8 of the time once San is written, but writes San only 0.2 of the time once Golden is written, with San and Golden each at 0.5 on an empty canvas.
City first, the answer comes out 0.400.40, worked outSan 0.5, Francisco 1, colon 1, the 1Golden, given San: 0.8; Gate 10.5 × 1 × 1 × 1 × 0.8 × 1 = 0.40. Bridge first, the same six tokens come out 0.100.10, worked outGolden 0.5, Gate 1, colon 1, the 1San, given Golden: 0.2; Francisco 10.5 × 1 × 1 × 1 × 0.2 × 1 = 0.10. If the sampler takes each order half the time, the model writes this answer with probability 0.250.25, worked outeach order half the time0.5 × 0.40 + 0.5 × 0.10 = 0.20 + 0.05 = 0.25The probability of an answer is the average of its paths, weighted by how often the sampler takes each.: the average of its paths.
To compute that number for a real answer you would need every path, each costing a forward pass per step. Worse, the sampler a deployed model runs does not pick its order uniformly: it commits the slots it is most confident about, so its probability for an answer is yet another sum. RL needs $\log \pi_\theta(y)$ for the sampler that produced $y$, and nobody can afford it.
What is the ELBO, and how did RL use it before 2026?
What a masked diffusion model can compute cheaply is the bound it was trained on, the ELBOELBOEvidence lower bound: a quantity that is never above the true log-likelihood and, unlike it, can be estimated by sampling. Diffusion models train by maximizing one, or a weighted version of it; for masked diffusion it reduces to a weighted average of the log-probabilities the model gives to masked tokens. (evidence lower bound). For an answer $y$, with the prompt left implicit:
\[\log \pi_\theta(y) \;\ge\; \mathbb{E}_{t \sim U(0,1)}\;\mathbb{E}_{y_t}\Big[\,\frac{1}{t}\sum_{i\,:\,y_t^i = [\mathrm{M}]} \log p_\theta\big(y^i \mid y_t\big)\Big] \;=\; \mathrm{ELBO}_\theta(y)\]Here $y_t$ is the answer with each token hidden independently with probability $t$. In words: pick a masking level, hide that fraction of the answer at random, ask the model for the hidden tokens, add up the log-probabilities it gives the true ones, and average over many levels and hidings. The $1/t$ puts every masking level on the scale of a whole answer. LLaDA trains by minimizing this quantity with a minus sign in front.
Ou et al. showed what the bound means: it equals an average, over random unmasking orders, of the log-probability of the answer along each order. The truth averages the probabilities of the paths; the bound averages their logs. An average of logs is never above the log of the average (Jensen’s inequality), so the bound sits below the truth, and it is exact only when every order agrees.
In the toy the bound is ln 0.20ln 0.20, worked outaverage the logs of the two paths: 0.5 × ln 0.40 + 0.5 × ln 0.10= 0.5 × (−0.916) + 0.5 × (−2.303) = −1.609e^(−1.609) = 0.20The bound is the geometric mean of the paths, √(0.40 × 0.10) = 0.20; the truth is their arithmetic mean, 0.25., against a truth of 0.25: a gap of 0.223 nats0.223 nats, worked outln 0.25 − ln 0.20 = −1.386 − (−1.609)= ln(0.25 ÷ 0.20) = ln 1.25 ≈ 0.223A nat is a unit of log-probability with natural logs: a gap of 0.223 nats is a factor of 1.25.. And the bound itself is only ever estimated, from a handful of random maskings per answer, so even the stand-in arrives with noise.
Through 2025, RL for diffusion models was mostly a search for good stand-ins for $\log \pi_\theta(y)$. d1’s diffu-GRPO ran one forward pass with the whole answer masked and treated the token predictions as independent, a mean-field estimate. wd1 recast the objective as a weighted log-likelihood so that only the current policy’s likelihood had to be approximated. SPG sandwiched the truth between the ELBO and an upper bound. d2 estimated the likelihood of the path the sampler actually took.
At scale, the ELBO won. Ant Group’s LLaDA2.1, released at 16B and 100B parameters, describes the first large-scale RL framework built for diffusion models, EBPO: GRPO’s clipped ratio, with ELBOs standing in for log-probabilities, and a block-causal mask that computes every block’s term in one forward pass per masking level. The paper calls the ELBO a principled proxy. A proxy is still a bound, though, and an optimizer pushing on a bound is the next question.
What goes wrong when RL climbs the bound instead of the truth?
DiPOD (Jiang et al., June 2026) names it. Write the ELBO as the truth minus a gap $D$ that is never negative:
\[\mathrm{ELBO}_\theta(y) \;=\; \log \pi_\theta(y) - D_\theta(y), \qquad D_\theta(y) \ge 0\]Put the ELBO into the policy gradient, with an answer’s advantage $\hat A$, and each answer’s update becomes:
\[\hat A\,\nabla_\theta\, \mathrm{ELBO}_\theta(y) \;=\; \hat A\,\nabla_\theta \log \pi_\theta(y) \;-\; \hat A\,\nabla_\theta D_\theta(y)\]The first term is the update you wanted; the second is an error that vanishes only where the gap is flat. Near a well-trained model the bound is tight and the error small. But nothing in the update keeps the model there.
DiPOD calls what follows the double drift. First the ELBO separates from the true log-likelihood: the gap grows. Then, since the update’s error is the gap’s own gradient, the update separates from the true policy gradient. The dangerous case is a wrong answer, with a negative advantage: the cheapest way to lower its ELBO may be to widen the gap, which lowers the bound without making the answer much rarer.
The toy shows it with exact arithmetic. Read it as a choice of city and a choice of bridge word, the other words following, and it is the two-token setting DiPOD itself uses to watch the drift. Push the wrong answer New York: the Golden Gate down through its bound for four updates, and the bound falls from 0.20 to 0.04 (both as probabilities) while the truth falls only from 0.25 to 0.0880.088, worked outfour updates on the bound’s gradient: advantage −1, learning rate 1bound: 0.200 → 0.147 → 0.102 → 0.066 → 0.040truth: 0.250 → 0.203 → 0.160 → 0.121 → 0.088Computed exactly in the figure’s toy: two orders, six step probabilities..
The bound is an average of logs, so its cheapest descent is to crush the path that is already small: city first falls to 0.010, while bridge first, which carries the truth, only reaches 0.167. The model learns to disagree with itself. The second drift is visible too: the same four updates with the exact gradient bring the truth to 0.0590.059, worked outthe same four updates on the gradient of ln(truth)truth: 0.250 → 0.197 → 0.143 → 0.095 → 0.059the gap shrinks meanwhile: 0.22 → 0.06 nats, not 0.088. Once the bound is loose, its gradient points somewhere else.
DiPOD’s fix keeps the bound tight. In principle it alternates the RL update with a self-distillation step: maximize the ELBO on the model’s own samples, which pulls the orders back into agreement without changing what the model writes. A tight bound has the true gradient, since a gap sitting at its minimum, zero, has zero slope. In practice it is one extra term per update: the ELBO gradient of the batch’s own rollouts, times a small weight (0.05 in the language runs).
The gains are largest where the drift did the most damage. On LLaDA-8B-Instruct, zero-shot at length 256, Sudoku goes from 25.12% with SPG to 97.56% with SPG plus the regularizer, 72.44 points72.44 points, worked out97.56 − 25.12 = 72.44d1 on the same task, from the d1 paper: 22.10%. more; Countdown goes from 51.95% to 80.08%. FPO, which simply swaps the ELBO in for the log-likelihood, had collapsed on Countdown to 12.50%; with the regularizer it reaches 75.39%.
Math moves less: GSM8K from 84.23% to 84.91%, MATH500 from 37.80% to 40.00%. The authors suspect the fixed generation window caps the harder problems, and note that results vary substantially across random seeds. The regularizer keeps the bound honest on the model’s own samples; it does not make the true likelihood any cheaper.
How much freedom over the order should RL allow?
The Flexibility Trap (Ni et al., an ICML 2026 Outstanding Paper) asks a blunter question: is the freedom to write in any order, the very thing that makes the likelihood intractable, even helping? For reasoning, it finds, often not. Its yardstick is Pass@kPass@kThe probability that at least one of k sampled answers is correct. RL can only reinforce answers the model sometimes samples, so a high Pass@k at large k marks how far RL can take a model.: a problem the model never solves in $k$ tries gives RL nothing to reinforce.
Restricting LLaDA-Instruct to left-to-right order gives a higher Pass@k than letting it choose its own order. On HumanEval, sampling 1,024 times, 21.3% of the problems are solved only in left-to-right order and 0.6% only in free order. Sweeping the block size up from 1, which is left to right, lowers Pass@k step by step.
The mechanism is what the authors call entropy degradation. Reasoning hinges on a few forking tokens, words such as Therefore or Since where the argument can branch, and those are uncertain. A model free to choose its order postpones them and fills in the easy tokens around them; by the time it comes back, the context has already settled the fork. It explores less.
So JustGRPO drops the freedom during RL. It defines a left-to-right policy on top of the diffusion model: the probability of token $k$ is read from a pass where every earlier token is written and every later one masked. The answer’s likelihood is then an exact product, and plain GRPO applies. LLaDA-Instruct reaches 89.1% on GSM8K and 45.1% on MATH-500 at length 256.
The catch is cost: every position needs its own forward pass, where an AR model needs one for the whole answer. JustGRPO-Fast computes the ratios only at the 25% highest-entropy positions, cutting 75% of those passes. And the model still decodes in parallel afterwards: at about 5 tokens per step, its lead on MBPP over the base model grows from 10.6 points to 25.5.
Block size sits between the two extremes, and 2026 found it matters for training as much as for decoding. Block-R1 (Jiang, Qiu and Huang) shows that RL across several domains wants a different block size per domain, and that one shared size can leave a model below where it started. Its 41K-example dataset tags every example with its best training block size.
DreamReasoner-8B found the same in supervised fine-tuning for long chains of thought. Training with blocks of 32 from the start wrecked reasoning: 20.0% on AIME 2024 with blocks of 4 at decoding, against 47.1% for a model trained on blocks of 4. A curriculum, blocks of 4 first and then mixed sizes up to 32, keeps both: 73.8% at block 4 and 68.3% at block 32, 2.2 points2.2 points, worked outQwen3-8B-Thinking on AIME 2024: 76.0%DreamReasoner-8B at block 4: 73.8%76.0 − 73.8 = 2.2 below Qwen3-8B-Thinking’s 76.0%.
How do you give credit to each step of the path?
Everything so far treats the answer as one object with one probability. A diffusion model does not produce it that way: it writes it in passes, and some passes matter more than others. In our PACT paper we call the neglect of this structure trajectory blindness: methods that weight every denoising step by the answer’s single advantage cannot tell the step that won the answer from the step that wrote a certain word.
Take two rollouts of the bridge prompt with a group-relative advantage. Rewards of 1 and 0 give advantages of +1 and −1±1, worked outrewards 1 and 0: mean 0.5each is 0.5 from the mean, so the spread is 0.5(1 − 0.5) ÷ 0.5 = +1, (0 − 0.5) ÷ 0.5 = −1, and every pass inherits its answer’s score. Pass 1 writes the same colon and the in both rollouts, and is pushed up once and down once. The wrong rollout broke at a single slot, but each of its passes takes the same blame.
PACT (Potential-Anchored Consistency of Transitions) gives every intermediate state a score. Start from what RL is after when it keeps the model near a frozen reference $p_{\text{ref}}$ with a strength $\beta$: the reference’s paths, reweighted toward high reward, $p^\star(\tau) \propto p_{\text{ref}}(\tau)\, e^{r(y)/\beta}$. Then define the potential of a half-written state $x$ as the tilted reward the reference expects if it finishes from there:
\[h(x) = \mathbb{E}_{p_{\text{ref}}}\big[\,e^{r(y)/\beta} \,\big|\, x\,\big], \qquad f(x) = \log h(x)\]A classical result for Markov processes, the Doob h-transform, says the tilted target is the reference with every step reweighted by the potential it gains:
\[\log \frac{P^\star(x^{\prime} \mid x)}{P_{\text{ref}}(x^{\prime} \mid x)} \;=\; f(x^{\prime}) - f(x)\]In words: a step that raises the chance of a good finish should become more likely, by exactly the potential it gained, and a step that changes nothing should stay as the reference had it.
PACT trains the policy to satisfy this on the transitions it actually sampled. A small head, a two-layer network on the model’s pooled hidden state, predicts $f$; at a finished answer it is pinned to $r(y)/\beta$. For each sampled step the loss is the square of a consistency residual:
\[\delta_t \;=\; \underbrace{\sum_{i \in S_t} \log \frac{\pi_\theta(a_i \mid x_t)}{\pi_{\text{ref}}(a_i \mid x_t)}}_{\text{what the step did}} \;-\; \underbrace{\big(f(x_{t-1}) - f(x_t)\big)}_{\text{what it gained}}\]Here $S_t$ is the set of slots the step committed and $a_i$ the token it wrote in slot $i$; a second term pulls the head’s scores toward the observed reward. Every quantity is computed exactly: the step’s own token probabilities, read from its own forward pass, at the transition that happened. No sequence likelihood, and no importance-ratio clipping.
In the toy, with $\beta = 0.1$ as in the paper, the model ends right 0.70.7, worked outSan or New, 0.5 eachafter San it finishes right 0.9 of the time, after New 0.50.5 × 0.9 + 0.5 × 0.5 = 0.45 + 0.25 = 0.7 of the time from six masks, 0.9 after San and 0.5 after New. Writing the colon gains nothing. Writing San gains +0.25+0.25, worked outfor a 0/1 reward with chance s: f = ln(s · e¹⁰ + (1 − s)), since 1/β = 10f(after San) − f(start) = 9.895 − 9.643 = 0.251≈ ln(0.9 ÷ 0.7) = 0.251With a strong tilt, a gain in potential is the log of a ratio of chances., so the tuned model should write San 0.640.64, worked outSan: 0.5 × e^0.251 = 0.5 × 1.286 = 0.643New: 0.5 × e^(−0.336) = 0.5 × 0.714 = 0.3570.643 + 0.357 = 1: the tilt keeps it a distribution of the time instead of 0.5. Writing Golden after New loses almost everything, and the tuned model all but stops doing it.
On LLaDA-8B-Instruct, trained with LoRA adapters on the same 900 rollout batches as AGRPO (a policy-gradient method that scores sampled denoising transitions with the answer’s advantage), PACT improves the base model by 9.9, 12.0, 55.7 and 72.6 pointsthe gains, worked outGSM8K: 89.1 − 79.2 = 9.9MATH500: 44.4 − 32.4 = 12.0Countdown: 79.9 − 24.2 = 55.7Sudoku: 99.5 − 26.9 = 72.6Greedy Pass@1 at generation length 256. on GSM8K, MATH500, Countdown and Sudoku. Its four-task average, 78.2, is the best reported for this model, 3.7 points3.7 points, worked outPACT’s four-task average: 78.2the next best, DTM: 74.578.2 − 74.5 = 3.7 above the next, DTM (Discrete Tilt Matching). It reaches AGRPO’s best GSM8K accuracy in about half the computeabout half, worked outAGRPO’s best GSM8K accuracy, 85.1, after 53.4 GPU-hoursPACT first reaches 85.1 after 28.6 GPU-hours28.6 ÷ 53.4 ≈ 0.54.
The limits, stated as plainly as we would want them from anyone else. A PACT batch costs about 10%10%, worked outper batch on 16 H100s: 41.3 s against AGRPO’s 37.5 s41.3 ÷ 37.5 ≈ 1.10over 900 batches: 165 against 150 GPU-hours more than an AGRPO batch, because each scored transition needs two extra gradient passes and a reference pass. On Countdown, DTM stays ahead at length 256 (81.3 against 79.9), and the MATH500 and Sudoku margins are within about one standard error. AGRPO, wd1 and DMPO were retrained under our budget; the other rows are each paper’s own numbers. One model, four tasks.
Two choices are approximations. The order the sampler takes is drawn from the model’s own confidences but not trained, because training it lowered accuracy (87.7 against 89.1 on GSM8K). And pulling the head toward the observed $r/\beta$ makes it a value estimate rather than the exact log-potential. The tilt matters most: at $\beta = 0.5$, GSM8K drops to 86.0.
Summed along a whole path, the residuals telescope into the single residual of trajectory balance, the objective behind TraFL. Penalizing each step separately keeps errors on different steps from cancelling; in our comparison under the same rollouts, TraFL fell behind at all three tilts we tried, peaking within 300 steps and then declining.
What can a model learn without a policy gradient?
Two 2026 papers sidestep the likelihood by changing the training signal rather than estimating the likelihood better.
d-OPSD (Luo et al.) adapts on-policy self-distillation. The model samples its own answer, keeping the first correct one in up to 8 tries. The teacher is the same network with part of that final answer revealed in the masked slots: it sees its own future at every denoising step. The student learns to match the teacher’s predictions on the slots each step commits, a step-level KL divergenceKL divergenceA measure of how far one probability distribution is from another: zero when they agree, larger the more the first puts probability where the second does not. Distillation minimizes it between student and teacher predictions., with no reward-weighted likelihood anywhere.
That suffix trick is one only a bidirectional model can play: an AR teacher can only be handed extra text before the answer. On LLaDA-8B-Instruct, across four reasoning tasks, d-OPSD beats RLVR (the d1 recipe) and SFT, converging in about 10% of RLVR’s optimization steps at the same rollout cost per step. Its signal is only as good as the correct answers the model can find on its own.
Outcome routing (Lu, Zhu and Clark) post-trains DiffusionGemma-26B-A4B without any likelihood ratio. Each group of 8 rollouts is routed by its verifier outcome. Mixed groups get plain REINFORCE on the log-probabilities of the tokens actually written, with no old-policy ratio, because replaying a diffusion model in bf16 drifted by about 0.1 nat. All-fail groups learn from a teacher only when the teacher succeeded. The rest are skipped.
On tool calling, routing adds 18.31 points over its distilled starting point, even though the teacher succeeds on only 18.6% of the training prompts. The price shows up elsewhere: reasoning-trace accuracy drops 0.71 points while direct-answer accuracy on the same items rises 13.21, and a KL anchor to the distilled model restores the traces. The math result depends on the seed.
LIFT (Parashar et al.) goes back a stage, to supervised fine-tuning, which can actually hurt a diffusion model. The reason is learnability: rare tokens are nearly impossible to predict when most of the answer is masked, and common tokens are trivial when most of it is visible. LIFT trains on the easy tokens at heavy masking and on the hard ones at light masking.
It reaches up to 3 times vanilla SFT’s accuracy on AIME 2024 and 2025 (Pass@16), and comes close to d1’s RL results with roughly 500 times fewer GPU hours. Better SFT is not RL, but it decides where RL starts.
What can you do after training: search harder or sample faster?
A diffusion sampler makes explicit decisions, which slots and which tokens, so its path can be searched. UnMaskFork (Misaki and Akiba, ICML 2026) treats the unmasking trajectory as a tree and runs Monte Carlo Tree SearchMonte Carlo Tree SearchA search that grows a tree of partial solutions, finishes promising branches with quick rollouts, and spends more visits on branches whose rollouts scored well. It is how AlphaGo chose moves. over it. Its moves are deterministic: unmask the next stretch greedily with one of several models, here Dream-Coder and LLaDA. Deterministic moves can be cached, so revisiting a branch costs nothing.
At 12,288 forward passes per problem it reaches 28% on LiveCodeBench, 88% on HumanEval+ and 72% on MBPP+, against 19%, 75% and 66% for best-of-N with the same two models: 9, 13 and 6 points ahead9, 13 and 6, worked outLiveCodeBench: 28 − 19 = 9HumanEval+: 88 − 75 = 13MBPP+: 72 − 66 = 6100 problems per benchmark, Pass@1 on the held-out tests.. Raising the temperature for diversity, the AR habit, made the samples worse instead.
VGB (Jeon, Vuong and Tao) adds the move a masked model lacks: taking a token back. Guided by a reward or a process verifier, its sampler unmasks and remasks positions, a backtracking random walk over partially masked states. Its cost grows quadratically where best-of-N’s can grow exponentially as errors accumulate, and it tolerates a noisy verifier. On 9 by 9 Sudoku with a small 28.6M-parameter model, its momentum variant reaches perfect accuracy with much less compute than the baselines.
IDLM (Li et al., ICML 2026) goes the other way: fewer steps. Inverse distillation trains a fast student whose samples would make the slow teacher the optimal diffusion model, if the teacher were trained on them. The paper proves the discrete version has a unique solution and adds relaxations that keep the gradients stable, matching a 1,024-step teacher’s quality with 4 to 64 times fewer steps on OpenWebText and TinyGSM.
What is still open?
Here is the year’s map of RL for diffusion language models, by what each route uses in place of the answer’s log-probability.
| Route | Stand-in for the log-probability | Example | A number | The catch |
|---|---|---|---|---|
| The bound | ELBO from random maskings | LLaDA2.1 (EBPO) | RL at 100B | can drift from the truth |
| The bound, kept tight | ELBO plus an ELBO term on the model’s own samples | DiPOD | Sudoku 25.12% to 97.56% | still a bound; small gains on math |
| One fixed order | the exact left-to-right likelihood | JustGRPO | GSM8K 89.1% | a forward pass per token; trains a left-to-right policy |
| Credit per step | step log-ratios and a learned potential | PACT | four-task average 78.2, +3.7 | order not trained; potential is a value fit |
| No policy gradient | KL to a self-teacher that sees its own answer | d-OPSD | about 10% of RLVR’s steps | needs a correct sample |
| Route by outcome | REINFORCE on written tokens, KL to a teacher | outcome routing | +18.31 on tool calling | costs trace accuracy unless anchored |
| Better SFT | tokens chosen by learnability | LIFT | up to 3x on AIME Pass@16 | supervised, not RL |
| Search | tree search or reward-guided remasking | UnMaskFork, VGB | LiveCodeBench 28% against 19% | compute at every query |
The open problem is a cheap, unbiased policy gradient for any-order generation, and each route gives up one of those words. The ELBO is cheap but biased, and the bias can grow under RL. JustGRPO is exact, but only for a fixed order, and pays a pass per token. PACT is exact per step and cheap, but trains the tokens along the order the sampler happened to take, not the order itself.
A solution would score the sampler that actually runs, confidence-ordered with several slots per pass, for a few forward passes per answer, without a bias that grows as the model moves. It might be an estimator, a sampler whose likelihood is tractable by design, or step-level credit that also learns the order.
The evidence is narrow, too. Almost every 2026 RL paper reports LLaDA-8B-Instruct on the same four tasks, under protocols that differ in decoding steps, block size, fine-tuning and how the best checkpoint is picked. JustGRPO and PACT both report 89.1% on GSM8K under different settings; do not read that as a tie. Meanwhile the 100B models train on the ELBO, the stand-in the drift results warn about.
What to watch in 2027: RL on the shipped models (DiffusionGemma’s report already pairs RL with sampler distillation), whether tight bounds and per-step credit hold up at 100B, and a shared protocol that makes these methods comparable.
Rapid fire: can you do these from memory?
- Write the policy-gradient identity, and say in words why the log appears.
- Why does an AR model give an answer's log-probability in one forward pass, while a bidirectional diffusion model needs one pass per step to score even one order?
- Two orders give an answer probabilities 0.4 and 0.1, each taken half the time. What are the true probability and the ELBO as a probability, and when would they be equal?
- What is DiPOD's double drift, and why are negative-advantage updates the dangerous ones?
- JustGRPO fixes the order during RL. What does it gain, what does it cost, and can the model still decode in parallel?
- In PACT, what is the potential of a half-written state, and what does the consistency residual ask of each transition?
- d-OPSD and outcome routing both avoid likelihood ratios. What signal does each use instead?
- Why can best-of-N scale badly for a masked diffusion sampler, and what does VGB do differently?
Where can you go deeper?
- Haozhe Jiang and others, DiPOD: Diffusion Policy Optimization without Drifting Apart (preprint, 2026), arXiv:2606.13795, with a project page. The double drift, the self-distillation principle and the one-line ELBO regularizer; the two-token toy this part borrows.
- Zanlin Ni and others, The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models (preprint, 2026; named an ICML 2026 Outstanding Paper), arXiv:2601.15165, with a project page and code. Pass@k under fixed and free order, entropy degradation, and JustGRPO.
- Vaibhav Singh and Eugene Belilovsky, RL Post-Training Diffusion Language Models Along Their Own Generation Paths (preprint, 2026), arXiv:2610.17266. PACT: trajectory blindness, the potential, the consistency residual and the comparison with trajectory balance.
- Yifu Luo and others, Learning from the Self-future: On-policy Self-distillation for dLLMs (preprint, 2026), arXiv:2606.18195, with code. d-OPSD: the suffix teacher and step-level supervision.
- Wentao Lu, Tianyu Zhu and Jesse Clark, Outcome-Routed Distillation and Reinforcement Learning for Diffusion Language Models (DiffuLM workshop at NeurIPS 2026), arXiv:2610.19271. Likelihood-free post-training of DiffusionGemma with routing by verifier outcome.
- Shubham Parashar and others, Learnability-Informed Fine-Tuning of Diffusion Language Models (preprint, 2026), arXiv:2605.22939, with code. LIFT: which tokens to learn at which masking level.
- Yan Jiang, Ruihong Qiu and Zi Huang, Block-R1: Rethinking the Role of Block Size in Multi-domain Reinforcement Learning for Diffusion Large Language Models (preprint, 2026), arXiv:2605.11726, with code. Block size as a training variable, and the Block-R1-41K dataset.
- Zirui Wu and others, DreamReasoner-8B: Block-Size Curriculum Learning for Diffusion Reasoning Models (preprint, 2026), arXiv:2606.19257, with code. Why large training blocks hurt long chains of thought, and the curriculum that fixes it.
- Tiwei Bie and others, LLaDA2.1: Speeding Up Text Diffusion via Token Editing (preprint, 2026), arXiv:2602.08676, with code. Section 3.2: EBPO, the ELBO-based RL used at 100B.
- Kou Misaki and Takuya Akiba, UnMaskFork: Test-Time Scaling for Masked Diffusion via Deterministic Action Branching (ICML 2026), arXiv:2602.04344, with a project page. Tree search over unmasking with deterministic, cached moves.
- Kijung Jeon, Thuy-Duong Vuong and Molei Tao, VGB for Masked Diffusion Model: Efficient Test-time Scaling for Reward Satisfaction and Sample Editing (preprint, 2026), arXiv:2606.28301. Reward-guided unmasking and remasking, and why best-of-N can scale exponentially.
- David Li and others, IDLM: Inverse-distilled Diffusion Language Models (ICML 2026), arXiv:2602.19066, with a project page. Inverse distillation for 4 to 64 times fewer sampling steps.
- Siyan Zhao and others, d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning (2025), arXiv:2504.12216. diffu-GRPO and its one-pass, mean-field log-probability estimate.
- Xiaohang Tang and others, wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models (ICLR 2026), arXiv:2507.08838. A ratio-free objective that needs one likelihood approximation.
- Chenyu Wang and others, SPG: Sandwiched Policy Gradient for Masked Diffusion Language Models (ICLR 2026), arXiv:2510.09541. The ELBO below and an upper bound above.
- Guanghan Wang and others, d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimation (ICML 2026), arXiv:2509.21474. Estimating the likelihood of the path the sampler took.
- Jingyang Ou and others, Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data (2024), arXiv:2406.03736. Why the masked-diffusion ELBO is an average over orders of the log-likelihood along each.
- Shen Nie and others, Large Language Diffusion Models (2025), arXiv:2502.09992. LLaDA and the masked-diffusion training objective used throughout this part.
- Zhihong Shao and others, DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (2024), arXiv:2402.03300. Where GRPO comes from.