Is 82% better than 80%?

The model from Parts 1 to 3 is finished. It says mat 70% of the time after “the cat sat on the”, and it has learned what to say about pairs it never saw. Now you measure it: one benchmark, 100 questions. It scores 80. A rival scores 82, and the leaderboard puts the rival first.

Every number on that leaderboard is an average of coin flips. This part asks what such an average is worth, with the tools of MML’s §6.4: expected values, variances, and what happens when random things are added up.

The puzzle Model A scores 80% on a 100-question benchmark. Model B scores 82%. Is B the better model?
The 30-second version You cannot tell. Each question is a coin that lands right with probability about 0.8, so a 100-question score wobbles by √(0.8 × 0.2 / 100) = 0.04: four points. An exact copy of model A scores 82 or more 36% of the time. To tell a 2-point gap apart you need either about 3,200 questions per model, or both models graded on the same questions, so that their shared luck cancels in the gap.

What is one question worth?

One question is the smallest random thing in this story. The model gets it right or wrong, so write its score as $X = 1$ or $X = 0$. That is a Bernoulli variable with parameter $\mu$, the model’s accuracy (MML Example 6.8).

Its expected value is the probability-weighted average of its values (MML Definition 6.3):

\[\mathbb{E}[X] = \sum_x x\,p(x) = 0\cdot 0.2 + 1\cdot 0.8 = 0.8\]

In words: the balance point. Put the two probabilities on a beam at 0 and 1, and the beam balances at $\mu$. The fulcrum in the figure is draggable.

The wobble is the variance: the expected squared distance from the mean, or “the mean of the square minus the square of the mean” (MML 6.43 and 6.44). A 0-or-1 answer squares to itself, so

\[\mathbb{V}[X] = \mathbb{E}[X^2] - \mu^2 = \mu - \mu^2 = \mu(1-\mu) = 0.16\]

That gives a standard deviation of $\sqrt{0.16} = 0.4$ per question. The parabola peaks at $\mu = 0.5$: a benchmark that models half pass is the noisiest per question, and one they nearly always pass tells you almost nothing new.

Does the score settle down?

A benchmark score is the empirical mean of the answers, $\bar{x} = \frac{1}{n}\sum_i x_i$ (MML 6.41). Watch it build up one question at a time.

Early on, every question moves the score a lot. In this run it dips to 0.6 after ten questions and only then climbs back. By question 100 each new answer moves it by a hundredth at most, so it has settled near 0.8. That settling is the law of large numbers.

Settling is not arriving. A model whose true accuracy is exactly 80% scores exactly 80 on 100 questions only one run in ten (9.9%). Press new run and watch the final score land somewhere else.

How far can a rerun land?

Here is the real question. Rerun our model on fresh sets of 100 questions, again and again, and stack up the scores. Then decide about model B.

Why four points? The score is a sum of 100 answers, divided by 100. Expectations always add (MML 6.46), and for independent answers the variances add too, because the covariance terms of MML 6.48 are zero (§6.4.5). Dividing by $n$ divides the variance by $n^2$ (MML 6.51 with $A = 1/n$):

\[\mathbb{E}[\bar{x}] = \mu, \qquad \mathbb{V}[\bar{x}] = \frac{n\,\mu(1-\mu)}{n^2} = \frac{\mu(1-\mu)}{n} = \frac{0.16}{100}\]

That is a standard deviation of 0.04, four points. Nearly every rerun lands between 72 and 88, and 36% of them score 82 or more. B’s 82 is a score our own model posts about a third of the time.

The stack also takes a shape no single question has: a bell. One question is only ever 0 or 1, yet the sum of many independent ones comes out Gaussian. MML names it in a margin note: the central limit theorem. Part 5 is about that bell.

How many questions do you need?

More questions, less wobble. But how much less? Guess how many questions it takes to halve the wobble, then slide $n$.

The wobble is $\sqrt{\mu(1-\mu)/n}$: it shrinks with the square root of the number of questions. Four times the questions for half the noise: 100 give 4 points, 400 give 2, 1,600 give 1. To separate a 2-point gap with two independent question sets, the gap’s own spread, $\sqrt{2}$ times one score’s, has to come down to 1 point. That takes about 3,200 questions per model.

Real benchmarks are often small. For a model at 80%:

benchmark questions one score wobbles by
AIME (one year, both papers) 30 7.3 points
HumanEval 164 3.1 points
GPQA Diamond 198 2.8 points
MMLU 14,042 0.3 points

On a 30-problem test one problem is worth 3.3 points, and luck alone moves the score by more than two problems.

How do you compare two models fairly?

More questions are expensive. Here is a cheaper idea: grade both models on the same 100 questions, instead of a fresh 100 each. In the figure B really is 2 points better. Guess which comparison finds it.

Easy questions are easy for both models and hard ones are hard for both, so on the same questions the two scores rise and fall together. MML 6.49 says what that does to the gap:

\[\mathbb{V}[B - A] = \mathbb{V}[A] + \mathbb{V}[B] - 2\,\mathrm{Cov}[A, B]\]

In words: the gap’s noise is each score’s noise, minus twice the noise they have in common. The covariance (MML Definition 6.5) is that shared luck.

With fresh questions for each model, the gap wobbles by 5.5 points around the true 2, and B often loses. On the same questions, at a correlation of 0.75, it wobbles by 2.8. The most two models with these accuracies can agree is a correlation of 0.94, where the gap’s spread is 1.4 points.

Same models, same 100 questions each. Only the covariance changed. This is why serious comparisons report results on one shared test set, and why a paired difference is more trustworthy than two separate leaderboard numbers.

Where else does the square root show up?

Everywhere an average stands in for an expectation, and training runs on one. The training loss is an average over all the text there is. Its gradient is an average of per-example gradients, and a minibatch of $B$ examples estimates it.

The estimate wobbles by $\sigma/\sqrt{B}$, the same law as the benchmark. Quadrupling the batch halves the noise in each step, at four times the compute per step. That trade sets how large a batch is worth using.

Every average in this part ended in the same bell. Part 5 asks why that bell, the Gaussian, sits inside the model itself, in the random numbers its weights start from.

Rapid fire: can you do these from memory?

  1. A model gets 90% of a benchmark's questions right on average. What are the mean and the variance of one question's score?
  2. Why is a benchmark that models pass about half the time the noisiest per question?
  3. How large is the standard deviation of a 400-question score for a model at 80%?
  4. Rerunning the same model on fresh questions moved its 100-question score by 3 points. Is that surprising?
  5. Why does halving the noise of a score take four times the questions, not twice as many?
  6. Two models are graded on the same questions. Which term of V[B − A] makes the comparison cleaner, and why is it positive?
  7. One question scores only 0 or 1. Why does a histogram of benchmark scores come out bell-shaped?
  8. Quadrupling the batch size halves the gradient noise. What does it cost?

Where can you go deeper?

Everything here is from Chapter 6, Probability and Distributions, of Mathematics for Machine Learning by Deisenroth, Faisal and Ong (Cambridge University Press, 2020), free at mml-book.github.io:

  • §6.4.1, means and covariances: the expected value (Definition 6.3, eqs. 6.28 and 6.29), the mean (Definition 6.4), the covariance (Definition 6.5, eqs. 6.35 and 6.36) and the variance (Definition 6.7).
  • §6.4.2, the empirical mean (Definition 6.9, eq. 6.41): a benchmark score.
  • §6.4.3, three expressions for the variance, including “the mean of the square minus the square of the mean” (eqs. 6.43 and 6.44).
  • §6.4.4, sums and transformations: eqs. 6.46 to 6.49 for sums and differences, eq. 6.51 for scaling.
  • §6.4.5, statistical independence: why the covariance of independent questions is zero, and what i.i.d. means.
  • §6.6, Example 6.8: the Bernoulli distribution, its mean $\mu$ and variance $\mu(1-\mu)$ (eqs. 6.92 to 6.94).
  • §6.5, the margin note naming the central limit theorem.