Neural networks, deep learning, NLP · insight-1436

Prune weak configurations early with successive halving when each run is costly

Situation
A search over learning rate, width and regularisation needs 100 candidate configurations, and each run takes a day on a shared GPU.
Action
Start all candidates for a small fixed number of epochs, keep the top third, extend them, and repeat. Check on a few runs that early rankings predicted late rankings.
Reason
Hyperband allocates resources adaptively and stops poor configurations early, framed as a pure-exploration bandit problem, and was reported to beat Bayesian optimisation methods on several tasks.
Boundary
Searches where partial-training results correlate with final results, such as most supervised nets.

Symptom or risk

The full budget is spent finishing runs that were visibly poor after a few epochs, so you search far less of the space than you could.

Schedules with late-phase effects, for example large warmup or one-cycle runs, where early ranks mislead.

Some good slow starters will be cut; keep one wider first rung.

Changing the schedule horizon per rung without rescaling the learning-rate schedule.

Hyperband speeds up random search by adaptively allocating resources and stopping poor configurations early, and its authors report it beats popular Bayesian optimisation methods on deep learning and kernel tasks.
Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization