Neural networks, deep learning, NLP · insight-0431
Seed every library inside each data-loader worker
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Augmentations draw randomness from NumPy, Python random or another library inside a
Dataset, and the DataLoader runs several workers.Action
Pass a
worker_init_fnthat seeds NumPy andrandomfrom the worker's PyTorch seed, and give the DataLoader a seededgenerator. Print a few augmentation parameters per worker once to confirm they differ.Reason
PyTorch seeds its own generator per worker as base seed plus worker id, and its docs warn that seeds of other libraries may be duplicated across workers. Current DataLoader worker code also reseeds Python random and NumPy per worker, so print per-worker draws to see whether your versions and libraries actually collide.
Boundary
Any multi-worker DataLoader with augmentation or sampling outside torch random functions.
Symptom or risk
If a library's generator is not reseeded per worker, workers can produce identical "random" augmentations, quietly reducing augmentation diversity and making runs hard to reproduce.
Exceptions
Pipelines that draw all randomness from torch generators.
Trade-offs
A few lines of boilerplate; reproducibility also needs fixed worker count and data order.
Pitfalls
Seeding once in the main process and assuming workers inherit different streams.
Lessons and concepts
Claims and their sources
PyTorch sets each worker's seed to base_seed plus worker_id, notes that seeds for other libraries may be duplicated across workers, and recommends seeding them in worker_init_fn.