Transformers — Explained Simply

What a Transformer Actually Is

You Are Building a Translation Machine

Input:  "The bank refused the loan"  (English)
Output: "La banque a refusé le prêt" (French)

The machine needs two capabilities:

Capability 1 — Understand the input completely Read English, understand every word in context, build a deep representation of full meaning.

Capability 2 — Produce the output word by word Generate French one token at a time using the full understanding from step 1.

These two capabilities are the encoder and the decoder.

The Full Architecture

INPUT TEXT
    ↓
[Tokenisation]          → token IDs
    ↓
[Embeddings]            → each ID becomes a vector
    ↓
[Positional Encoding]   → add position information (word 1, word 2, ...)
    ↓
[Encoder Block × N]
  ├─ Multi-Head Self-Attention    (every word attends to every word)
  ├─ Add & Norm                   (residual + layer norm)
  ├─ Feed-Forward Network         (two linear layers + GELU)
  └─ Add & Norm
    ↓
[Encoder Output]        → contextualised representations of input
    ↓
[Decoder Block × N]
  ├─ Masked Self-Attention        (attends to previous output only)
  ├─ Add & Norm
  ├─ Cross-Attention              (attends to encoder output)
  ├─ Add & Norm
  ├─ Feed-Forward Network
  └─ Add & Norm
    ↓
[Linear + Softmax]      → probability over vocabulary
    ↓
OUTPUT TOKEN

The Three Components You Have Not Seen Before

1. Positional Encoding

Attention has no sense of word order. Positional encoding adds a position signal to each embedding.

class PositionalEncoding(torch.nn.Module):
    def __init__(self, d_model, max_len=5000):
        super().__init__()
        pe       = torch.zeros(max_len, d_model)
        position = torch.arange(0, max_len).unsqueeze(1).float()
        div_term = torch.exp(torch.arange(0, d_model, 2).float()
                             * (-math.log(10000.0) / d_model))
        pe[:, 0::2] = torch.sin(position * div_term)
        pe[:, 1::2] = torch.cos(position * div_term)
        self.register_buffer('pe', pe.unsqueeze(0))

    def forward(self, x):
        return x + self.pe[:, :x.size(1)]

2. Residual Connections

After every sub-layer, the original input is added back to the output.

output = LayerNorm(x + Attention(x))
output = LayerNorm(x + FeedForward(x))

Gives gradients a direct highway to early layers — prevents vanishing gradients in deep networks. This is why transformers can be 96 layers deep and still train.

3. Feed-Forward Network

Two linear layers applied after attention at each position independently.

class FeedForward(torch.nn.Module):
    def __init__(self, d_model, d_ff):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(d_model, d_ff),
            nn.GELU(),
            nn.Linear(d_ff, d_model)
        )
    def forward(self, x):
        return self.net(x)

Complete Transformer Block in PyTorch

class TransformerBlock(nn.Module):
    def __init__(self, d_model, num_heads, d_ff, dropout=0.1):
        super().__init__()
        self.attention = nn.MultiheadAttention(
            embed_dim=d_model, num_heads=num_heads,
            dropout=dropout, batch_first=True
        )
        self.ff      = nn.Sequential(
            nn.Linear(d_model, d_ff), nn.GELU(), nn.Linear(d_ff, d_model)
        )
        self.norm1   = nn.LayerNorm(d_model)
        self.norm2   = nn.LayerNorm(d_model)
        self.dropout = nn.Dropout(dropout)

    def forward(self, x):
        attn_out, _ = self.attention(x, x, x)
        x           = self.norm1(x + self.dropout(attn_out))  # Add & Norm
        x           = self.norm2(x + self.dropout(self.ff(x))) # Add & Norm
        return x

# Stack 6 blocks
model = nn.Sequential(*[
    TransformerBlock(d_model=256, num_heads=8, d_ff=1024)
    for _ in range(6)
])

Three Flavours of Transformer

Encoder-Only          → Understanding tasks
  Models: BERT, AfroXLMR
  Used for: Classification, NER, sentiment analysis
  Your paper: AfroXLMR for Nigerian Pidgin sentiment ✅

Decoder-Only          → Generation tasks
  Models: GPT-2/3/4, LLaMA, Mistral, Claude
  Used for: Chatbots, text generation, code
  Your paper phase 2: fine-tune Mistral on Nigerian text

Encoder-Decoder       → Sequence-to-sequence tasks
  Models: T5, BART, mT5
  Used for: Translation, summarisation

Scale Comparison

Model           Layers  Dimensions   Parameters
──────────────────────────────────────────────────
BERT-base       12      768          110 million
GPT-2           12      1600         117 million
AfroXLMR-large  24      768          560 million  ← you will fine-tune this
GPT-3           96      12288        175 billion
Claude          ?       ?            ?

Same architecture. More layers + dimensions + data + compute = more powerful.

BERT vs GPT — One Table

                BERT              GPT
Type:           Encoder-only      Decoder-only
Task:           Understanding     Generation
Attention:      Full (both ways)  Causal (left to right only)
Training:       Masked LM         Next token prediction
Fine-tune for:  Classification    Text generation

Your Fine-Tuning Code

from transformers import AutoModelForSequenceClassification

# 24 transformer blocks pretrained on African languages
model = AutoModelForSequenceClassification.from_pretrained(
    "Davlan/afro-xlmr-large",
    num_labels = 3   # Positive, Negative, Neutral
)

# Fine-tuning teaches the attention heads to focus on
# Nigerian Pidgin sentiment signals specifically

The Real Words Mapped to the Story

The One Thing to Remember

A transformer is stacked blocks of attention + feedforward + residual connections. Encoder-only models like AfroXLMR understand text. Decoder-only models like GPT and Claude generate text. The paper "Attention Is All You Need" introduced this architecture — you now understand every component of it.