Mixed precision
Also known as: mixed precision training · BF16 · FP16
Mixed precision runs parts of training in lower-precision formats (FP16 or BF16) where numerical error is acceptable, while keeping critical parts in higher precision.
The payoff:
- Faster — lower precision is quicker on modern GPUs.
- Less memory — smaller numbers, smaller activations.
It’s now standard practice for large-model training.
Connect it to local AI: the same "use fewer bits where it's safe" idea underlies quantization. Mixed precision is that idea applied to *training*.