Neural networks, deep learning, NLP · insight-1424

Train once with the inputs zeroed to prove the model uses its features

Situation
A network reaches a decent score, but its loss curve looks similar to what a constant prediction would give.
Action
Replace all inputs with zeros, keep everything else, and train for the same steps. The real model must beat this input-independent baseline by a margin larger than seed noise; if not, debug the data path.
Reason
An input-independent run isolates what the model learns from labels and sampling alone, which gives the minimum any feature-based model has to beat.
Boundary
Any supervised network, as one of the first end-to-end checks.

Symptom or risk

The model may be exploiting only the label distribution, the ordering of rows or an ID, so extra features appear to add nothing and nobody knows why.

Sequence models with positional structure, where the zero baseline may still learn from order.

One short run.

Comparing a single run against a single run when seeds differ by more than the margin.

Karpathy's recipe describes training with the inputs zeroed out as an input-independent baseline that the real model should beat.
A Recipe for Training Neural Networks