Why is the training loss a log, and how low can it go?
A small language model reads the cat sat on the and has to say what comes next. It answers with four probabilities: mat 0.7, floor 0.15, sofa 0.1 and moon 0.05. Then training shows it the word that really came next, mat, and charges it $-\log 0.7 = 0.357$ nats.
That charge is the training loss, the one number every language model is trained to push down. This last part takes it apart: why it is the log of a probability and nothing else, what it comes to on average, the floor no model can get under however large, which way round the gap above that floor is measured, and how a framework computes all of it from a model’s raw scores.
The 30-second version
Because 1 − q gets two things wrong that the log gets right. It barely tells a near miss from a disaster: giving the right word 0.01 or 0.0001 costs 0.99 or 0.9999, while the log charges 4.6 and 9.2. And it rewards bluffing: on that text a model charged 1 − q pays least by saying mat 100%, while under the log the cheapest answer is exactly the truth. The floor is not 0. It is the entropy of the data, 0.914 nats, a perplexity of about 2.5: even a perfect model is as unsure as a choice between two and a half words, because the text itself is. Everything above the floor is the KL divergence, the price of being wrong about the world, and it is the only part training can remove.What should a wrong guess cost?
Start from scratch. The model handed over four probabilities and the world handed back one word. Training needs a single number from that, a price: 0 when the model was sure and right, and more the less probability it gave the word that came.
Call the probability it gave the right word $q$. The first price anyone would try is what the model missed by, $1 - q$. Here that is $1 - 0.7 = 0.3$. It looks reasonable, and it fails twice.
The first failure: it hardly cares how wrong the model was. Give the right word 0.01 and $1 - q$ charges 0.99. Give it 0.0001, a hundred times more confidently wrong, and it charges 0.9999. A model that called the truth one in ten thousand pays almost exactly what a model that called it one in a hundred pays.
The log charges 4.6 and 9.2:
\[\text{price} = -\log q\]Read it as: every time the model divides the right word’s probability by 10, the price goes up by the same 2.3 nats, and every halving adds $\log 2 = 0.69$. The price never stops growing, so a confident mistake is expensive.
There is a good way to feel the number. In base 2 the price counts coin flips. A word the model gave probability 1/2 surprises it as much as one fair coin landing heads: 1 bit. A word it gave 1/4 is two heads in a row, and 1/1024 is ten. Training uses the natural log, which counts the same thing in a unit called the nat. One coin flip is 0.693 nats, and mat at 0.7 is about half a coin flip of surprise.
At $q = 0$ the price is infinite, and that is a warning, not a flaw. A model that learns by counting, and has seen my four times but never my mat, gives mat after my probability 0. The first time real text says my mat, it pays $-\log 0 = \infty$, and one infinite word makes the whole average infinite. Part 3 fixed that with a prior, which gave the same word 1/6 instead: a price of 1.79, steep but finite.
Why not charge what the model missed by?
Because a model charged $1 - q$ learns to bluff. That is the second failure, and the deeper one.
Real text does not always say mat. Say the training text has the cat sat on the twenty times, followed by mat 14 times, floor 3 times, sofa twice and moon once. The honest answer is the one the model already gives: 0.7, 0.15, 0.1 and 0.05.
Charged $1 - q$, the honest model pays 0.3 fourteen times, 0.85 three times, 0.9 twice and 0.95 once: 9.5 over the twenty sentences. A model that bluffs and says mat 100% pays 0 fourteen times and 1 the other six: 6 in all. The bluff is cheaper, so the bluff is what training would teach.
Which answer is cheapest under each price? Guess before the curves appear.
Charged $-\log q$, the bluffer pays $-\log 0 = \infty$ the first time floor comes. The honest model pays 0.357 fourteen times, 1.897 three times, 2.303 twice and 2.996 once: 18.3 over the twenty, 0.914 a word, and every other answer pays more. Under the log, the cheapest thing a model can say is the truth.
That is not luck, and it is worth seeing why the log is the only price that does it.
Write the price of the right word as a curve, $f(q)$, and ask what honesty demands. At the truth, moving a sliver of probability $\epsilon$ from one word to another must not lower the average price, or training would move it. Take $\epsilon$ from word $j$ and give it to word $k$, and the average price changes by about
\[\epsilon\,\big[\,p_k\,f^\prime(p_k) - p_j\,f^\prime(p_j)\,\big]\]That must be 0 for every pair of words, so $p f^\prime(p)$ is the same number for every word, whatever its probability. Call it $-c$. Then $f^\prime(p) = -c/p$, and the only curve with that slope that charges nothing for a sure hit is
\[f(q) = -c\,\log q\]Read it as: honesty fixes the slope of the price at every probability, and that slope belongs to the log alone. The constant $c$ is only the unit: $c = 1$ counts nats, $c = 1/\log 2$ counts bits.
Curves that look just as reasonable fail the test, each in its own direction. Squaring the miss, $(1 - q)^2$, caps a total miss at 1, so giving up a rare word is cheap. $1/q - 1$ punishes a miss without limit but more harshly than the log does. Guess what squaring teaches, then try each price.
Squaring teaches mat 0.82, floor 0.18 and nothing at all for sofa or moon: it would rather be wrong about the rare words than pay to cover them. $1/q - 1$ hedges toward uniform, 0.47, 0.22, 0.18 and 0.13. Only $-\log q$ teaches 0.7, 0.15, 0.1 and 0.05.
There are honest prices that are not logs. The Brier score, $\sum_k (q_k - y_k)^2$ with $y$ the one-hot truth, also makes the truth cheapest, but it charges the model for every word, including the ones that did not come. The derivation above assumed a price that looks only at the word that came. Honesty plus that one assumption leaves nothing but the log.
A second road leads to the same place. The probability of a whole sentence is the product of its words’ probabilities, each given the words before it, so a long sentence’s probability is tiny: 150 words at 0.5 each give $0.5^{150} \approx 7 \times 10^{-46}$, which float32 rounds to 0. The log turns that product into a sum, so each word pays its own price:
\[-\log p(w_1, \dots, w_T) = \sum_{t=1}^{T} -\log p(w_t \mid w_1, \dots, w_{t-1})\]Read it as: the surprise of a sentence is the surprise of each word, given the words before it, added up. Ask for that in general, a price under which two independent surprises add, $f(ab) = f(a) + f(b)$, and the only smooth answer is again $f = -c \log$. The 150-word sentence costs $150 \times 0.693 = 104$ nats, an ordinary number. Part 2 builds that product from a table of twenty phrases.
What does the model pay on average?
Dividing the tally by twenty gave an average price per word, and that average has a name. Write the data’s shares as $p = (0.7, 0.15, 0.1, 0.05)$ and the model’s answer as $q$. Each word $w$ comes up a share $p(w)$ of the time and costs $-\log q(w)$ when it does, so the average price weights each word’s price by how often the word comes:
\[H(p, q) = \mathbb{E}_{w \sim p}\big[-\log q(w)\big] = -\sum_{w} p(w)\,\log q(w)\]Read it as: the price of each word, times how often the word comes, added up. This is the cross-entropy, and an average of this kind is an expectation (Part 4 is about measuring one). A model that has learned nothing gives each of the four words 0.25, pays $\log 4 = 1.386$ whichever word comes, and so pays 1.386 on average.
Its exponential has a name, the perplexity: $e^{1.386} = 4$, exactly the number of words the clueless model is choosing between. A model’s perplexity is the number of equally likely words it is as unsure as.
How low can the loss go?
Set the model’s probabilities equal to the data’s, $q = p$, the honest answer, and the price is as low as it gets:
\[\begin{aligned} H(p) &= -\sum_{w} p(w)\,\log p(w) \\ &= 0.7 \cdot 0.357 + 0.15 \cdot 1.897 + 0.1 \cdot 2.303 + 0.05 \cdot 2.996 \\ &= 0.914 \end{aligned}\]That is the entropy of the data, and it is not 0. The data is uncertain on its own: six times in twenty the next word is not mat, and no model can know which six. So even a perfect one pays 0.914 nats per word, a perplexity of about 2.5.
Everything a model pays above that floor is the gap between its beliefs and the world, the Kullback-Leibler divergence:
\[D_{\mathrm{KL}}(p \,\Vert\, q) = H(p, q) - H(p) = \sum_{w} p(w)\,\log\frac{p(w)}{q(w)} \;\ge\; 0\]The clueless model’s gap is $1.386 - 0.914 = 0.472$. A model that says (0.5, 0.3, 0.15, 0.05) pays 1.005, a gap of only 0.091. The gap is 0 exactly when $q = p$.
Why can no model get under the floor? The proof fits in a picture. The log curve sits below its tangent line at 1: $\log x \le x - 1$, with equality only at $x = 1$. Write $x_w = q(w)/p(w)$ for how many times too much or too little the model gives word $w$. The terms $x_w - 1$, weighted by $p(w)$, add to $\sum_w q(w) - \sum_w p(w) = 0$, so they can be slipped into the KL for free:
\[D_{\mathrm{KL}}(p \,\Vert\, q) = \sum_w p(w)\,\big[(x_w - 1) - \log x_w\big]\]Read it as: each word’s gap between the tangent and the curve, weighted by how often the word comes. Every gap is at least 0, so the KL is too, and it is 0 only when every word sits exactly at $x = 1$, which is $q = p$.
Moon has the biggest gap on the bowl, five times its share, but it comes once in twenty. Most of the uniform model’s 0.472 is mat, given far too little, fourteen times.
MML mentions KL in a single remark (§6.4.6), as the usual way to measure how far apart two distributions are. It is not a distance in the everyday sense: swap $p$ and $q$ and the number changes, and which order you minimise changes what a model learns.
So the loss splits in two: cross-entropy = entropy + KL. The entropy belongs to the data. The KL belongs to the model, and it is the only part training can remove.
Which way round does the KL go?
Training minimises $D_{\mathrm{KL}}(p \Vert q)$: the data first, the model second. The order stops mattering only when the model can match the data exactly, and a real model never can. It is smaller than the world.
Read the forward KL, $\sum p \log (p/q)$, as an average over what the data does. It explodes wherever the data puts weight and the model puts almost none, so a model trained this way must cover everything the data does, even if that means putting weight where the data has none.
Swap the order and $D_{\mathrm{KL}}(q \Vert p)$ averages over what the model does. It explodes wherever the model puts weight and the data has none, so a model fitted that way locks onto one thing the data does and ignores the rest.
The cleanest place to see it is data with two humps and a model that can draw only one bell. Guess where training puts the bell.
The forward fit spreads one wide bell over both humps, centred at 0 with width 2.09, and its forward KL is 0.555. Its peak sits where the data has almost nothing. The reverse fit sits on one hump: its reverse KL is 0.692, close to $\log 2$, the price of ignoring exactly half the data, and its forward KL would be 10.2.
Both directions turn up in practice. Pretraining is maximum likelihood, the forward direction, and it is why a language model keeps some probability on every continuation its corpus contains. The KL penalty in RL fine-tuning runs the other way, $D_{\mathrm{KL}}(\pi \Vert \pi_{\mathrm{ref}})$ with the average taken over the model being tuned, and so does variational inference, behind the bound a variational autoencoder trains on at the end of this part.
What does training actually remove?
A real model does not hand over probabilities. It hands over one score per word, the logits $z$, and softmax turns them into probabilities: $q(w) = e^{z_w} / \sum_v e^{z_v}$. Put that into the price of the right word $c$ and the log undoes the exponential:
\[-\log q(c) = -z_c + \log \sum_{v} e^{z_v}\]Read it as: the loss is the log of the total, minus the right word’s own score. For the twenty sentences, logits of $\log 14$, $\log 3$, $\log 2$ and $\log 1$ give exactly 0.7, 0.15, 0.1 and 0.05, and the loss for mat is $\log 20 - \log 14 = 0.357$.
Now differentiate. The derivative of $\log \sum_v e^{z_v}$ with respect to one score is that score’s exponential over the total, which is softmax itself, $q(w)$. The right word’s own score adds $-1$. So
\[\frac{\partial}{\partial z_w}\big[-\log q(c)\big] = q(w) - y_w\]where $y$ is 1 for the right word and 0 for the rest. Read it as: what the model said, minus what happened. Averaged over the data, $y$ becomes the data’s shares, and the gradient of the average price is $q(w) - p(w)$, the simplest gradient in deep learning.
The same formula is why every framework’s cross-entropy takes logits, never probabilities. The sum $\sum_v e^{z_v}$ overflows float32 as soon as a logit passes about 88, so the library subtracts the largest logit from all of them first, which changes nothing in the formula. Softmax first and log second fails at the other end: a word whose logit sits 120 below the top gets probability $e^{-120}$, about $10^{-52}$, which float32 rounds to 0, and its log is $-\infty$. The logit formula gives about $-120$ directly. Guess what adding 100 to every logit does.
Now hand the four scores to gradient descent. Every step moves each one against $q(w) - p(w)$. Guess which scores rise first, then watch the red gap drain.
The price falls fast, then stops dead at 0.914 with $q = p$. That is what a falling training loss is: the red part shrinking. When a real loss curve flattens, part of the plateau is this floor, and no amount of training gets under it.
Here the log pays off once more. For a single word, the data’s share is 1 for the word that came and 0 for the rest, so the push on the right word’s score is $1 - q$. A model that gave the right word 0.01 gets a push of 0.99. Charged $1 - q$ instead, the push would be $q(1 - q)$: 0.0099, a hundred times weaker. Under that price the most wrong model would learn the slowest; under the log it learns the fastest.
One more consequence of that gradient. On a single sentence the target is one-hot, so the push on the right word never stops while its probability is below 1, and its logit keeps rising for as long as training runs. Here the data disagrees with itself six times in twenty, so the scores settle. On data where one answer is always right, nothing stops them except a brake added on purpose. Label smoothing is that brake: it trains against a target of 0.9 for the right word with the other 0.1 spread evenly over the whole vocabulary, which is what the original Transformer did.
Why is this the same as maximum likelihood?
A training run never sees $p$. It sees a corpus, and it averages the price over the words in it:
\[\frac{1}{N}\sum_{n=1}^{N} -\log q_\theta(w_n \mid \text{context}_n) \;=\; -\frac{1}{N}\,\log q_\theta(\text{corpus})\]The left side is the training loss. The right side is the negative log of the probability the model gives the whole corpus, divided by $N$: that probability is the product of the words’ probabilities, and the log turns it into the sum on the left. It is called the likelihood (MML eq. 8.17), so minimising the loss is maximum likelihood estimation, MML §8.3.1.
Here it is on the twenty sentences. Each one is charged its price, and the total is minus the log of the probability the model gives the whole corpus, so the model with the smallest total makes the corpus most likely.
The corpus average is itself only an estimate of the cross-entropy, and a minibatch of $B$ words is an estimate of that, with noise that shrinks like $1/\sqrt{B}$: four times the batch, half the noise.
Two more threads tie in here. Add a prior, a belief about the weights held before any data is seen, and the thing you minimise gains one term, $-\log p(\theta)$. That is maximum a posteriori estimation (MML §8.3.2), and with a Gaussian prior on the weights the extra term is weight decay.
And when the output is a number rather than a word, the model’s answer is a Gaussian bell centred on its prediction, and its price is the same $-\log$: $-\log \mathcal{N}(y \mid f(x), \sigma^2) = (y - f(x))^2/2\sigma^2 + \text{const}$ (MML Example 8.5). That is squared error. Squared error and cross-entropy are one idea, the negative log-likelihood, under two distributions; Part 5 is about that Gaussian.
Why does averaging first look better?
One last property of the log. Two predictions are equally likely: one gives the right word 0.1, the other 0.9. Their average probability is 0.5, which would cost 0.693. Is the average of their two prices the same?
It is not: $\tfrac{1}{2}(2.303 + 0.105) = 1.204$. The price curve bends upward, so the average of two points on it sits above the curve at their average. That is Jensen’s inequality, for any convex function:
\[\mathbb{E}\big[-\log q\big] \;\ge\; -\log \mathbb{E}\big[q\big]\]You meet it twice in practice. An ensemble that averages its members’ probabilities never has a worse log loss than its members have on average. And the same inequality, read as $\log \mathbb{E} \ge \mathbb{E} \log$, is the lower bound a variational autoencoder trains on.
How does the whole chain fit together?
Here is the series in five lines, one per step of the model’s life:
- A model’s output is a distribution: twenty tickets, fourteen of them mat (Part 1).
- A sentence is a product of such probabilities, so its log is a sum of per-word prices (Part 2).
- A prior keeps every one of them above 0, and on the weights it is weight decay (Part 3).
- The training loss is a corpus average, an estimate of an expectation with noise like $1/\sqrt{B}$ (Part 4), and for real-valued outputs Part 5’s Gaussian turns it into squared error.
- Each word’s price is $-\log q$, the one price under which the truth is the cheapest answer. Its average is the cross-entropy, entropy plus KL, and minimising it is maximum likelihood: it drains the KL until the model’s distribution matches the data’s.
For a single word, with the truth written as a one-hot vector $y$, the training step’s gradient becomes $q - y$: what the model said, minus what happened. That row is what backpropagation carries back through every layer to every weight, which Gradient Part 4 plays out step by step.
The 0.7 the model gave mat cost it 0.357 nats. Over everything the data can say, the bill is 0.914 at best, and every training step is spent on whatever the model pays beyond that.
Rapid fire: can you do these from memory?
- What does a model pay for giving the right word 0.5, 0.25 and 0.01?
- Charge a model 1 − q instead of −log q on text that says mat 14 times in 20. What does it learn to say, and why?
- Derive, in three lines, why an honest price that looks only at the word that came must be a log.
- What does (1 − q)² teach a model on the twenty sentences, and why does it drop the rare words?
- Why does a single zero in a count model make its average loss infinite, and what fixes it?
- Define cross-entropy, entropy and KL divergence in one sentence each, and say how they add up.
- A model has perplexity 4. What does that mean, and what is its loss in nats?
- Can a model's average loss go below the entropy of the data? Prove it with log x ≤ x − 1.
- Training minimises KL(p ‖ q). What does that do to a model too small to fit the data, and where does the other order show up?
- Why does every framework's cross-entropy take logits, and what goes wrong in float32 if you compute log(softmax(z)) in two steps?
- Why is minimising the average cross-entropy over a corpus the same as maximum likelihood?
- What is the gradient of the cross-entropy with respect to one score, and which scores rise first when the model starts uniform on the twenty sentences?
- Two equally likely predictions give the right word 0.2 and 0.8. Which is larger, the average of their losses or the loss of their average, and which inequality says so?
Where can you go deeper?
Most of this part is in Mathematics for Machine Learning by Deisenroth, Faisal and Ong (Cambridge University Press, 2020), free at mml-book.github.io:
- §8.3.1, maximum likelihood estimation: the negative log-likelihood (eq. 8.14), its product and sum over independent examples (eqs. 8.16 and 8.17), and Examples 8.4 and 8.5, where a Gaussian likelihood turns into least squares (eq. 8.18).
- §8.3.2, maximum a posteriori estimation: adding a prior, with Example 8.6’s Gaussian prior on the parameters.
- The remark that closes §6.4.6, on measuring how far apart distributions are with the Kullback-Leibler divergence.
MML stops short of entropy itself. For that, David MacKay’s Information Theory, Inference, and Learning Algorithms (Cambridge University Press, 2003), free at inference.org.uk, Chapter 2: the Shannon information content $\log_2 1/P(x)$, which is this part’s price counted in bits, entropy, relative entropy (the KL divergence), Gibbs’ inequality, which is why KL is never negative, and Jensen’s inequality. Chapter 33, on variational methods, works with the other order, the KL with the model first.