Activation Functions — Explained Simply

What Activation Functions Actually Are

You Are Hiring Staff For Your Company

Your company receives job applications. Each application has several scores — experience, education, skills. You have one employee whose job is to look at those scores and decide whether to pass the application forward to the next stage.

The simplest version is a calculator. They add up all the scores with some weighting and pass the total forward:

Decision = (experience × 2) + (education × 1.5) + (skills × 3)

This works fine. But here is the problem.

The Problem With Just Adding Things Up

No matter how many calculator employees you chain together, the final result is mathematically identical to having just one calculator do everything in a single step.

Three layers of pure addition is still just addition. You get nothing extra from the depth.

This is the fundamental problem activation functions solve.

The Solution — A Different Kind of Employee

After adding up the weighted scores, the employee asks one question:

Is this result actually meaningful or is it just noise?

If the result is positive and meaningful — pass it through unchanged. If the result is negative or meaningless — send zero. Nothing passes through.

This simple extra step is the ReLU activation function. And it changes everything.

Why This Changes Everything

Without activation functions → chain of calculators → can only learn straight lines. With activation functions → team of decision-makers → can learn curves and complex patterns.

The Four Activation Functions You Need to Know

ReLU — The Most Common One

If the input is positive → pass it through unchanged
If the input is negative → send zero

f(x) = max(0, x)

Input:   -3   -1    0    1    3    5
Output:   0    0    0    1    3    5

Output
  │              /
  │             /
  │            /
  │           /
  │──────────/
  └──────────────→ Input
  negative = 0    positive = x

nn.ReLU()   # Used in hidden layers of almost every network

Why ReLU is used in hidden layers: Simple, fast, and does not suffer from the vanishing gradient problem. Gradient passes through at full strength — either 1 or 0, never a tiny fraction.

Sigmoid — The Probability Maker

Squashes any number into a value between 0 and 1

f(x) = 1 / (1 + e^(-x))

Input:   -5     -2     0     2     5
Output:  0.007  0.12  0.5  0.88  0.993

nn.Sigmoid()   # Used in OUTPUT layer for binary classification
               # "Is this review positive? → 0.87 → 87% confident"

Why Sigmoid is NOT used in hidden layers: Maximum derivative is 0.25. Stack many layers and the gradient shrinks to almost zero — early layers stop learning. This is the vanishing gradient problem.

Softmax — The Multi-Class Probability Maker

Converts a list of numbers into probabilities that sum to 1
Used when there are more than two output classes

Raw model output:   [2.0,  1.0,  0.1]
After Softmax:      [0.66, 0.24, 0.10]  ← sums to 1.0

Positive: 66% | Negative: 24% | Neutral: 10%

# Applied automatically inside CrossEntropyLoss
# No need to add manually in PyTorch
nn.CrossEntropyLoss()

# Your Nigerian Pidgin model:
nn.Linear(hidden_size, 3)   # 3 classes: Positive, Negative, Neutral
# CrossEntropyLoss converts those 3 numbers to probabilities automatically

GELU — The Modern Transformer Activation

A smoother version of ReLU used in all modern transformers
Instead of a hard cutoff at zero — a smooth curve

nn.GELU()   # Used in BERT, GPT, LLaMA, Claude — all modern transformers

Where Each One Goes in Your Network

import torch.nn as nn

model = nn.Sequential(
    nn.Linear(input_size, 256),
    nn.ReLU(),              # Hidden layer → ReLU or GELU

    nn.Linear(256, 128),
    nn.ReLU(),              # Hidden layer → ReLU or GELU

    nn.Linear(128, 3),
    # No activation — CrossEntropyLoss handles it
)

# Rule of thumb:
# Hidden layers  → ReLU (general) or GELU (transformers)
# Binary output  → Sigmoid (or BCEWithLogitsLoss handles it)
# Multi-class    → Nothing (CrossEntropyLoss handles Softmax)
# Regression     → Nothing (just the raw number)

What Happens Without Activation Functions

# Without activation — three layers collapse into one
# Layer 1: X @ W1 + b1
# Layer 2: (X @ W1 + b1) @ W2 + b2
# Layer 3: ((X @ W1 + b1) @ W2 + b2) @ W3 + b3
# Simplifies to: X @ W_combined + b  ← identical to one layer

# With ReLU — depth becomes real
# Layer 1: relu(X @ W1 + b1)
# Layer 2: relu(relu(X @ W1 + b1) @ W2 + b2)
# Cannot be simplified — relu breaks the linear chain

The Real Words Mapped to the Story

The One Thing to Remember

Without activation functions a deep neural network is just one layer pretending to be many. Activation functions are what give depth its power — they turn a chain of calculators into a team of decision-makers that can learn any pattern.