Normalization
Also known as: LayerNorm · RMSNorm
Normalization (e.g., LayerNorm, RMSNorm) rescales intermediate activations so their statistics stay in a healthy range as data flows through many layers.
It’s one of the quiet heroes of deep training:
- Stability — prevents values from blowing up or shrinking away.
- Faster convergence — optimization behaves better.
- Standard — essentially every modern transformer uses a form of it.
Don't skip it conceptually: normalization is the reason very deep networks train at all. It's the handrail that keeps the signals on the road.