Gradient accumulation
Also known as: accumulation
Gradient accumulation lets you simulate a large batch on limited memory. Instead of processing the whole batch at once, you process small micro-batches, accumulate their gradients, and do one weight update.
- Same effect as a bigger batch.
- Same memory as a small batch.
It’s one of the most useful levers when you’re memory-bound.
Rule of thumb: out of memory? Raise the accumulation steps before you cut quality. It's the cheapest way to buy back effective batch size.