S14 · paper simulation

Does the new optimizer really beat AdamW?

People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.

4stages
35 minabout
0stages committed

The brief

A simulated preprint ("OptX") claims that its optimizer improves test accuracy by 0.5 points over AdamW on a small CNN image-classification benchmark and recommends adopting it. The paper reports one run per method. OptX was tuned with a 20-trial hyperparameter search; AdamW "used default settings from the library". Your team trains similar small CNNs, so a half-point gain would matter.

What the registry supports: AdamW is the optimizer with decoupled weight decay (source: paper-loshchilov-2019-adamw). All results in this scenario are simulated; no real benchmark was run.

Stage 1 of 4

You skim the paper. It has one results table, a method section and an appendix listing the search ranges for OptX. There is no variance estimate and no statement about how AdamW's learning rate and weight decay were chosen.

Which details decide how much weight the +0.5 deserves? Select all that apply.

How many tuning trials each optimizer received, and whether tuning used the same validation split.How many independent runs per method, and what seeds, initialisation and data order were varied.Whether the test set was used for any selection (early stopping, checkpoint choice, hyperparameters).Whether the optimizer name or the authors' institution is familiar from earlier work, which makes the result easier to trust.Whether the paper explains the update rule with a convincing derivation of why it should converge faster.