Introduced on day 35
Paired comparison
Comparing two models on the same rows and looking at the rows where only one is right (or at the per-row loss difference), which has less noise than comparing two independent accuracies.
In detail
Accuracies computed on the same rows are positively correlated, so their difference is less noisy than the independent formula suggests. The discordant counts (rows only A gets right, rows only B gets right) feed McNemar's exact test, and per-row log-loss differences can be summarized by their mean and spread or bootstrapped.