Activation function

An activation function is a small non-linear step applied after each layer’s matrix math. It sounds minor, but it’s what gives a network its power to model complex, curved relationships.

Without activations, stacking layers would just collapse into one big linear transformation — no matter how deep the net, it could only learn straight-line rules.

Common ones: ReLU (fast, default for years) and its modern cousins.