Neural networks, deep learning, NLP · insight-1402
Call the scheduler after the optimizer and decide whether it steps per batch or per epoch
- Evidence type
- documented fact
- Difficulty
- easy
- Roles
- ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A schedule that should warm up over 500 steps is configured through a scheduler object, and the training loop was written by combining examples from different library versions.
Action
Order each iteration as forward, backward, optimizer step, then scheduler step; print the learning rate at steps 0, 1, the warmup end and the final step, and compare with the curve you intended.
Reason
The PyTorch documentation expects the scheduler to run after the optimizer's update, and a scheduler only knows how many times it was called, not what a call means in your loop.
Boundary
Any scheduler-driven training loop, including wrappers that hide the loop.
Symptom or risk
Stepping the scheduler before the optimizer skips the first value of the schedule, and an epoch-level scheduler stepped per batch (or the reverse) compresses or stretches the whole schedule by the number of batches per epoch.
Exceptions
Schedulers driven by a validation metric are stepped at evaluation time, not per batch.
Trade-offs
A ten-line learning-rate printout costs nothing; plotting the schedule needs a dry run without data.
Pitfalls
Gradient accumulation changes how many optimizer steps happen per batch; count optimizer steps, not batches.
Lessons and concepts
Claims and their sources
The PyTorch documentation expects the learning-rate scheduler to step after the optimizer, and calling it first skips the first value of the schedule.