Activation Functions: ReLU, GELU, SwiGLU
Activation functions are the nonlinearity sitting between linear layers; drop them and a deep network folds into a single linear map however many layers it stacks. The useful lens is gradient flow: sigmoid and tanh saturate and choke off gradients, ReLU fixed that by letting gradient through unchanged for positive inputs (at the cost of dying units), and modern transformers turn to smooth variants like GELU and gated SwiGLU. AI, ML, and GenAI engineer interviews probe it because the choice directly decides whether deep nets train at all and reveals whether you reason about backprop instead of memorizing names.
TL;DR: Activations are the nonlinearity that lets stacked layers express more than one big linear map. Rate them by gradient flow: sigmoid and tanh saturate at the tails and hand back near-zero gradient, which stalls deep nets; ReLU won because it passes gradient unchanged on the positive side, at the cost of units that can die. Modern transformers reach for smooth or gated variants (GELU, SwiGLU) that retain ReLU's gradient behavior while staying differentiable everywhere and being slightly more expressive.
Why nonlinearity is mandatory
Stack two linear layers, W₂(W₁x), and you get (W₂W₁)x, which is just another linear layer. Compose a thousand of them and it is still a single matrix. Depth gains nothing without a nonlinearity between the layers. The activation is what turns a deep network into a universal function approximator rather than an overcomplicated linear regression. So the question is never "should I use one" but "which one, and what does it do to the gradient."
The saturation problem with sigmoid and tanh
Sigmoid squashes inputs to (0, 1); tanh to (-1, 1). Both look reasonable until you check their derivatives. Sigmoid's derivative peaks at 0.25 and falls to near zero once the input is even moderately large in magnitude. In the backward pass, gradient gets multiplied by that derivative at every layer. With a peak of 0.25, a 10-layer network can shrink the gradient by a factor of 0.25^10 ≈ 1e-6 before architecture even gets involved. That is the vanishing-gradient mechanism, and it is why pre-2012 deep nets were nearly untrainable. tanh is a bit better (zero-centered, derivative peaks at 1.0) but still saturates flat at the tails.
Why ReLU won
ReLU is max(0, x). For positive inputs its derivative is exactly 1, so gradient passes through unchanged, no shrinking factor per layer. It is cheap (a threshold), and it produces sparse activations (many exact zeros), which can help. ReLU is the single change that made very deep networks practical, alongside better initialization.
Its failure mode is the dying ReLU: for any negative input the output and the gradient are both zero, so a unit pushed into the negative regime (often by a large gradient step or a bad bias) stops updating and stays dead forever. With a too-high learning rate you can watch a large fraction of units die. The fixes are Leaky ReLU (small negative slope, e.g. 0.01) and its learned cousin PReLU, which keep a trickle of gradient alive on the negative side.
GELU and SwiGLU in transformers
Transformers mostly do not use plain ReLU. GELU weights an input by the probability a Gaussian is below it, behaving like a smooth ReLU that lets small negative values through instead of hard-clipping them. The smoothness gives a nonzero gradient near the origin on the negative side, sidestepping the dying problem, and it empirically trains a touch better. GPT-family and BERT used GELU.
SwiGLU is the current default in many LLM feed-forward blocks (LLaMA, PaLM-style). It is a gated unit: split the projection into two halves, pass one through a Swish/SiLU activation, and multiply it element-wise by the other half. The gate modulates information multiplicatively rather than just thresholding it, which is more expressive per parameter. The cost is a third weight matrix, so implementations shrink the hidden dimension (often to about 2/3) to keep parameter count matched.
| Function | Derivative behavior | Failure / cost | Typical use |
|---|---|---|---|
| Sigmoid / tanh | saturates, near-zero at tails | vanishing gradient | gates, output layers |
| ReLU | 1 if positive, else 0 | dying units | classic CNNs, MLPs |
| Leaky ReLU / PReLU | small negative slope | extra hyperparameter | when ReLU dies |
| GELU | smooth, nonzero near origin | slightly costlier | BERT, GPT |
| SwiGLU | gated, multiplicative | extra matrix | modern LLM FFN |
Why interviewers probe this
They want to see the gradient-flow lens, not a list of curves. The strong move is to explain ReLU's win in terms of its derivative being 1 on the positive side (so it does not compound a shrinking factor through depth), then name dying ReLU and its fix, then connect to why transformers moved to GELU/SwiGLU. The held-back follow-up is often "why not just use sigmoid everywhere," and the expected answer is the 0.25^depth saturation math. Candidates who can do that tiny calculation out loud stand out.
Common misconceptions
- "More layers always means more capacity." Without a nonlinearity, stacked linear layers collapse to one linear map.
- "ReLU has no downside." Dead units that output and propagate zero forever are a real, common failure with high learning rates.
- "Newer activations give large accuracy jumps." GELU/SwiGLU over ReLU is usually a small, consistent gain; the big historical jump was sigmoid to ReLU.
- "Softmax is an activation like ReLU." Softmax is a normalization over a vector for output probabilities or attention weights, not a per-neuron hidden nonlinearity.
Key takeaways
- Nonlinearity is what makes depth meaningful; without it the whole network is one matrix.
- Sigmoid/tanh saturate and multiply gradients toward zero through depth; ReLU passes positive-side gradient unchanged.
- ReLU's price is dying units; Leaky ReLU, GELU, and gated SwiGLU keep gradient alive on the negative side.
- Pick activations by their derivative behavior under backprop, not by reputation.
Check yourself before an interviewer does. Answer from memory first.
In one line, why did ReLU make very deep nets trainable when sigmoid could not?