Quick answer: an activation function adds non-linearity to each neuron. Without it, any stack of layers collapses into a single linear model. Use ReLU (or Leaky ReLU) in hidden layers, sigmoid on the output for yes/no predictions, and softmax on the output for picking one of several classes. Smooth functions such as GELU and Swish are common in very large models. Tanh outputs −1 to 1 and, like sigmoid, suffers from vanishing gradients in deep networks.
Here is a strange fact: you could stack 100 layers of a neural network and it would be exactly as powerful as a single layer. Sounds like a scam, right? It would be, if it weren’t for one tiny ingredient called the activation function.
If you have ever opened a deep learning tutorial and seen words like ReLU, sigmoid or softmax fly by, this post is for you. No heavy maths, no fear. By the end you will know what activation functions do, why they exist, and which one to pick without googling in a panic.
Key takeaways
- An activation function is the small non-linear function each neuron applies to its weighted sum. Without it, a deep network is no more powerful than a single linear layer.
- Hidden layers: start with ReLU; switch to Leaky ReLU if many neurons die. Large transformers usually use GELU or Swish.
- Output layers: sigmoid for yes/no, softmax for many classes, and no activation for predicting a number.
- Sigmoid and tanh flatten at the ends, which causes vanishing gradients in deep networks.
What is an activation function, really?
Every artificial neuron does two jobs. First it takes its inputs, multiplies each by a weight and adds them up. Second, it passes that total through a small function that decides how strongly the neuron should "fire". That second step is the activation function.
Think of a bouncer at a club. The neuron’s total is the crowd noise outside. The bouncer (activation function) decides: too quiet, stay out; loud enough, come in; ridiculously loud, still only one person gets in at a time. Different bouncers, different rules.
Why can’t we skip it? The 100-layer trick
Multiplying and adding is called a linear operation. The awkward thing about linear operations is that combining them gives you… another linear operation. Two straight-line rules stacked together are still just one straight-line rule. So a network with no activation functions, however deep, can only draw straight lines through your data.
Real problems are not straight. Recognising a cat, translating Hindi to English or predicting engine wear all involve curves and twists. The activation function adds a bend at every layer, and those bends stacked together let the network trace almost any shape. That is the whole secret.
Layers give a network depth. Activation functions give it the ability to use that depth.
The main characters
1. Sigmoid: the classic that got old
Sigmoid squashes any number into a value between 0 and 1, which makes it look like a probability. Big positive input gives nearly 1, big negative gives nearly 0. It was the star of early neural networks because it loosely mimics a biological neuron that is either firing or quiet.
The catch: at the far ends the curve goes almost flat. Flat means the slope is nearly zero, and networks learn by following slopes. Even at its steepest, sigmoid’s slope is only 0.25. Multiply a number like that through many layers and the learning signal shrinks to almost nothing. This is the famous vanishing gradient problem. Early layers basically stop learning, like a game of telephone where the message fades before it reaches the first person.
Where it still shines: the final layer of a yes/no prediction (spam or not spam).
2. Tanh: sigmoid’s more balanced sibling
Tanh (say "tanch") squashes numbers between -1 and 1. Because its output is centred around zero, learning is usually smoother than with sigmoid. It still flattens at the ends, so it shares the vanishing gradient headache, just a little less. You will still meet it inside older recurrent networks such as LSTMs.
3. ReLU: the boring one that changed everything
ReLU stands for Rectified Linear Unit, which is a very grand name for a very simple rule: if the number is negative, output 0; otherwise pass it through unchanged. That’s it. In code it is literally max(0, x).
Why did such a plain rule take over deep learning? Two reasons. It is extremely cheap to compute (just a comparison, no exponentials). And for positive inputs its slope is exactly 1, so the learning signal does not fade as it travels back through layers. ReLU became the default after it powered the AlexNet image model that won the 2012 ImageNet competition and kicked off the modern deep learning boom.
The catch: a neuron that keeps receiving negative inputs outputs zero forever, and with zero slope it can never recover. People call this a "dead" neuron, or the dying ReLU problem. Poor thing.
4. Leaky ReLU: giving the dead a pulse
Leaky ReLU fixes the dead neuron issue with a small tweak: instead of outputting exactly 0 for negatives, it lets a tiny amount through (often 1% of the input). The neuron always has some slope, so it can always recover.
5. GELU and Swish: the smooth modern ones
ReLU has a sharp corner at zero. GELU and Swish (also called SiLU) round that corner off into a gentle curve, which often trains a little better in very large models. GELU is used in well-known language models such as BERT and the GPT family. A fun bit of trivia: Swish was discovered by researchers using an automated search that tried many candidate functions, rather than by someone sitting down and deriving it.
6. Softmax: the vote counter
Softmax is different. Instead of treating one neuron at a time, it looks at a whole group of outputs and turns them into percentages that add up to 100%. If a model is deciding between cat, dog and bird, softmax might output 85%, 10% and 5%. In a classifier it belongs in the last layer. (Transformers are the notable exception: they also use softmax inside every attention layer to decide which words to focus on.)
Quick comparison
| Function | Output range | Best used for | Watch out for |
|---|---|---|---|
| Sigmoid | 0 to 1 | Final layer, yes/no output | Vanishing gradients |
| Tanh | -1 to 1 | Older recurrent networks | Vanishing gradients |
| ReLU | 0 to infinity | Hidden layers, default choice | Dead neurons |
| Leaky ReLU | Slightly below 0 to infinity | When ReLU neurons die | One extra setting to pick |
| GELU / Swish | Smooth, mostly positive | Transformers, large models | Slightly more compute |
| Softmax | 0 to 1, sums to 1 | Final layer, many classes | Not for hidden layers |
So which one should you pick?
A beginner-safe recipe that works for a surprising number of projects:
- Hidden layers: start with ReLU. If many neurons seem dead, try Leaky ReLU.
- Output for yes/no: sigmoid.
- Output for many categories: softmax.
- Output for a number (like predicting a temperature): no activation at all, just the raw value.
- Building a transformer: follow what similar models use, usually GELU or a Swish-style function.
Try it yourself in 30 seconds
import numpy as np
x = np.array([-3, -1, 0, 1, 3])
sigmoid = 1 / (1 + np.exp(-x))
relu = np.maximum(0, x)
tanh = np.tanh(x)
print("sigmoid:", sigmoid.round(2))
print("tanh :", tanh.round(2))
print("relu :", relu)
Run it and compare. Notice how sigmoid barely changes between 3 and 100, while ReLU keeps growing. That difference is exactly why one struggles with depth and the other doesn’t.
Frequently asked questions
Can I use different activation functions in different layers?
Yes. Hidden layers commonly use one function (like ReLU) while the output layer uses another (like softmax) to match the task.
Is ReLU always the best?
No. It is the best starting point. Newer smooth functions can edge it out in very large models, but for most beginner projects ReLU is fast, simple and hard to beat.
What is a vanishing gradient in one sentence?
It is when the learning signal shrinks as it travels backwards through the layers until the early layers get almost no feedback and stop improving.
Final thought
An activation function is a tiny bend that turns a pile of straight-line maths into something that can recognise faces, translate languages and predict failures. Small switch, huge effect. Which one surprised you most: the 100-layer trick or the fact that ReLU is just max(0, x)? Tell me in the comments, and if you want a follow-up on how neural networks actually learn (backpropagation without tears), say so.
Sources and further reading
- Stanford CS231n. Neural Networks Part 1: Setting up the Architecture (sigmoid saturation, tanh, dying ReLU, Leaky ReLU).
- Krizhevsky, A., Sutskever, I. and Hinton, G. (2012). ImageNet Classification with Deep Convolutional Neural Networks (AlexNet).
- Hendrycks, D. and Gimpel, K. (2016). Gaussian Error Linear Units (GELUs).
- Ramachandran, P., Zoph, B. and Le, Q. V. (2017). Searching for Activation Functions (Swish).
- Vaswani, A. et al. (2017). Attention Is All You Need (softmax inside transformer attention).
- PyTorch. LeakyReLU documentation (default negative slope 0.01).
Want a step-by-step plan for learning machine learning?
The 90-Day Machine Learning Roadmap takes you from Python basics to a deployed ML portfolio in 13 weeks, with a plan for every day and a progress tracker. Going for your first ML or data science job? The ML Career Bundle adds an interview kit, Python practice and resume templates.
Prefer video? Watch the tutorials on YouTube · Browse all toolkits




5 responses to “Activation Functions in Neural Networks: Sigmoid, Tanh, ReLU, GELU”
[…] Activation Functions Explained Simply: Sigmoid, ReLU, Tanh, Softmax says: October 1, 2026 at 7:32 pm […]
[…] you’re optimizing a neural network or just trying to make sense of gradient descent without descending into chaos, these concepts are […]
[…] Neural networks: only when the simpler models are not enough, for example images or complex signals. (Start with activation functions.) […]
[…] Activation Functions in Neural Networks: Sigmoid, Tanh, ReLU, GELU […]
[…] Activation Functions in Neural Networks: Sigmoid, Tanh, ReLU, GELU […]