Softmax and Cross-Entropy — Explained Simply

What Softmax and Cross-Entropy Actually Are

You Are a Judge at a Talent Competition

Three contestants have just performed. Your raw scores:

Contestant A (singing):   8.2
Contestant B (dancing):   3.1
Contestant C (comedy):    5.7

The audience wants percentages — how confident are you in each contestant relative to the others? That conversion from raw scores to percentages that sum to 100% is Softmax.

Softmax — Converting Raw Scores to Probabilities

Raw scores:    [8.2,  3.1,  5.7]
After Softmax: [0.79, 0.02, 0.19]
                79%   2%    19%   → sums to 1.0 exactly

Your sentiment model does the same:

Input: "Dis product too good"
Logits: [4.2,  -1.3,  0.8]    ← raw scores (can be any size)
         Pos    Neg    Neu

Softmax: [0.87,  0.03,  0.10]
          87%    3%     10%

→ 87% confident this review is Positive

Softmax in Code

import torch
import torch.nn.functional as F

logits        = torch.tensor([4.2, -1.3, 0.8])
probabilities = F.softmax(logits, dim=0)

print(probabilities)        # tensor([0.8718, 0.0274, 0.1008])
print(probabilities.sum())  # tensor(1.0000) — always sums to 1

Three Properties to Know

1. Largest Score Gets Most Probability

[4.2, -1.3, 0.8] → [0.87, 0.03, 0.10]
 ↑ largest           ↑ most probability

2. Amplifies Differences

Similar scores:   [1.0, 0.9, 0.8] → [0.38, 0.34, 0.28]  ← uncertain
Spread scores:    [5.0, 0.5, 0.1] → [0.98, 0.01, 0.01]  ← confident

3. Temperature Controls Confidence

def softmax_with_temperature(logits, temperature=1.0):
    return F.softmax(logits / temperature, dim=0)

# Low temp → very confident, predictable
softmax_with_temperature(logits, temperature=0.1)
# tensor([1.0000, 0.0000, 0.0000])

# High temp → uncertain, more random/creative
softmax_with_temperature(logits, temperature=2.0)
# tensor([0.6439, 0.1100, 0.2461])

This is why ChatGPT's temperature setting affects creativity. Low = focused. High = creative but sometimes wrong.

Cross-Entropy Loss — Measuring How Good Your Probabilities Are

Cross-entropy answers: how surprised were you by the correct answer?

Loss = -log(probability you gave to the correct class)

0.99 → -log(0.99) = 0.01  ← tiny loss   — very confident and right ✅
0.50 → -log(0.50) = 0.69  ← medium loss — uncertain
0.01 → -log(0.01) = 4.61  ← huge loss   — very confident and WRONG ❌

Softmax and Cross-Entropy Together

import torch.nn as nn

logits  = torch.tensor([[4.2, -1.3, 0.8]])  # Raw model output
labels  = torch.tensor([0])                  # Correct class = 0 (Positive)

loss_fn = nn.CrossEntropyLoss()   # Applies Softmax internally
loss    = loss_fn(logits, labels)
print(loss)   # tensor(0.1400) ← small — model was 87% confident and right

# If confident about WRONG class:
wrong   = torch.tensor([[-1.3, 4.2, 0.8]])
print(loss_fn(wrong, labels))   # tensor(5.5600) ← huge loss

Critical Rule — Never Double Softmax

# ❌ Wrong — double Softmax
probs = F.softmax(logits, dim=1)
loss  = nn.CrossEntropyLoss()(probs, labels)   # Softmax applied AGAIN internally

# ✅ Correct — raw logits only
loss = nn.CrossEntropyLoss()(logits, labels)   # Softmax handled internally

Where Softmax Goes in Your Network

model = nn.Sequential(
    nn.Linear(input_size, 256), nn.ReLU(), nn.Dropout(0.3),
    nn.Linear(256, 128), nn.ReLU(), nn.Dropout(0.3),
    nn.Linear(128, 3)    # Raw logits — NO Softmax here
)

# Training — CrossEntropyLoss applies Softmax internally
loss = nn.CrossEntropyLoss()(model(X_batch), y_batch)

# Inference — apply Softmax manually for readable probabilities
model.eval()
with torch.no_grad():
    logits        = model(X_test)
    probabilities = F.softmax(logits, dim=1)   # Apply here for output
    predictions   = probabilities.argmax(dim=1)

The Complete Sentiment Classifier Pipeline

"Dis product dey too good"

1. Tokenise  → ["Dis", "product", "dey", "too", "good"]
2. Embed     → each token = 256-dim vector
3. Transform → transformer layers mix the vectors
4. Classify  → Linear(768, 3) → Logits: [3.8, -2.1, 0.4]
5. Softmax   → [0.91, 0.02, 0.07] → 91% Positive ✅
6. Predict   → argmax = 0 = Positive
7. Loss      → -log(0.91) = 0.094 ← tiny, model was confident and right

The Real Words Mapped to the Story

The One Thing to Remember

Softmax converts raw model scores into probabilities that sum to 1. Cross-entropy measures how surprised the model was by the correct answer. In PyTorch CrossEntropyLoss applies Softmax internally — never apply Softmax first or you will get wrong results.