Gradient checkpointing
Also known as: activation checkpointing
Gradient checkpointing is a memory-saving trick: instead of storing every intermediate activation during training, you recompute some of them when you need them for the backward pass.
- Cost: a bit more compute (recomputation).
- Benefit: much less memory.
It’s one of the standard ways to fit larger models/batches into limited VRAM.
When you'll see it: training frameworks expose it as a flag (e.g., `--gradient-checkpointing`). Turn it on when you're out of memory but have spare compute.