Skip to content
ML Academy

Introduced on day 32

Activation function

The nonlinearity applied to a unit's weighted sum, such as sigmoid, tanh, ReLU or GELU; without it, stacked layers collapse into one linear map.

In detail

The slope of the activation matters as much as its shape, because backpropagation multiplies slopes layer by layer. Sigmoid and tanh flatten in their tails; ReLU has slope 1 for positive inputs and 0 for negative ones; GELU is a smooth ReLU-like curve common in Transformers. The last layer's output activation is chosen from the label structure (sigmoid, softmax, none), not from the hidden-layer choice.