Neural networks, deep learning, NLP · insight-1421
Register submodules in ModuleList or ModuleDict, never in a plain Python list
- Evidence type
- documented fact
- Difficulty
- easy
- Roles
- ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A model builds a variable number of layers in a loop and stores them in a Python list or dict attribute.
Action
Use ModuleList or ModuleDict, then assert that the parameter count and the number of tensors in the state dictionary match your hand count, and that every parameter received a non-None gradient after one backward pass.
Reason
PyTorch registers submodules only when they are held as modules or in its container classes; a ModuleList registers its modules so all module methods can see them, unlike a plain list.
Boundary
Any dynamically built PyTorch model; the same pitfall exists in other frameworks with explicit registration.
Symptom or risk
Those layers are missing from parameters(), are never given to the optimizer, are not moved to the GPU or saved in the state dictionary, and the network trains with fixed random weights without an error.
Exceptions
Lists of non-parametric callables such as functions need no registration.
Trade-offs
One assertion per architecture.
Pitfalls
Checking only that the loss goes down; fixed random layers can still reduce loss somewhat.
Lessons and concepts
Claims and their sources
A PyTorch ModuleList registers the submodules it holds so that every Module method can see them, unlike a plain Python list.