Weight decay
Also known as: L2 regularization · decay
Weight decay (a.k.a. L2 regularization) adds a small penalty proportional to the size of the weights, nudging them toward zero.
The effect:
- Simpler model — fewer, smaller weights.
- Better generalization — less capacity to memorize noise.
It’s one of the most-used regularizers in practice.
Plain version: it's a gentle tax on complexity. The model is nudged to stay simple, and simple models tend to generalize better.