Tokens and Tokenisation — Explained Simply

What Tokens and Tokenisation Actually Are

You Are a Translator at the United Nations

A diplomat hands you a document in a language you have never seen. Before you can translate a single word you need to break it into pieces you can work with — identify where one unit ends and the next begins.

That process of breaking text into manageable pieces before any understanding happens is tokenisation.

The Problem — Computers Cannot Read

"I love Lagos"

Computer sees: 73 32 108 111 118 101 32 76 97 103 111 115
               I     l  o  v  e     L  a  g  o  s

Just numbers representing characters. No idea where words begin or end. Tokenisation defines the units the model works with.

What a Token Actually Is

A token is not always a word:

"tokenisation" → ["token", "isation"]       → 2 tokens
"Lagos"        → ["Lagos"]                  → 1 token
"unhappiness"  → ["un", "happiness"]        → 2 tokens
"!"            → ["!"]                      → 1 token

The Three Approaches

Word Tokenisation

"I love Lagos" → ["I", "love", "Lagos"]

Problem: Nigerian Pidgin words = unknown
"E dey whine me" → ["E", "[UNKNOWN]", "[UNKNOWN]", "me"]

Character Tokenisation

"Lagos" → ["L", "a", "g", "o", "s"]

Benefit: No unknown words
Problem: 100 words = 500+ tokens — very long sequences

Subword Tokenisation — Used by ALL Modern LLMs

"Lagos"    → ["Lagos"]            ← common, stays whole
"Lasgidi"  → ["Las", "gi", "di"] ← rare, split into pieces
"running"  → ["run", "ning"]     ← split at meaningful boundary

Byte Pair Encoding — The Most Common Subword Method

Used by GPT-2, GPT-3, GPT-4, LLaMA, Mistral.

Step 1: Start with individual characters
Step 2: Count most frequent adjacent pair in training text
Step 3: Merge that pair into one token
Step 4: Repeat thousands of times

After many merges:
"ing"     → one token
"Lagos"   → one token
"Nigeria" → one token
"wetin"   → one token (if common in Pidgin training text)

The Full Pipeline

"I love Lagos"
→ tokenise → ["I", "love", "Lagos"]
→ to IDs   → [146, 2293, 14189]
→ embed    → [ [0.82,...], [0.45,...], [0.91,...] ]
→ transformer layers operate from here

In Code — Hugging Face Tokenisers

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Davlan/afro-xlmr-large")

text   = "Dis product dey too good, I go buy am again"
tokens = tokenizer.tokenize(text)
ids    = tokenizer.encode(text)

print("Tokens:", tokens)
# ['Dis', 'product', 'dey', 'too', 'good', ',', 'I', 'go', 'buy', 'am', 'again']

# Batch tokenise your entire dataset
batch = tokenizer(
    ["Dis product dey good", "E no good at all"],
    padding        = True,
    truncation     = True,
    max_length     = 128,
    return_tensors = "pt"
)

print(batch["input_ids"].shape)       # (2, 128)
print(batch["attention_mask"].shape)  # (2, 128)
# attention_mask = 1 for real tokens, 0 for padding

Special Tokens

[CLS] → Start of sequence — final embedding used for classification
[SEP] → End of sequence or between two sequences
[PAD] → Fills empty space in shorter batched sequences
[UNK] → Unknown token — rare with subword tokenisation
[MASK] → Used during BERT training — "predict what word was here"

Why This Matters For Your Nigerian Pidgin Research

English BPE on "Wetin dey happen":
→ ["W", "etin", "Ġd", "ey", "Ġhapp", "en"]  ← 6 tokens
   Random fragments the model has no context for

AfroXLMR on "Wetin dey happen":
→ ["Wetin", "dey", "happen"]  ← 3 tokens
   Each word is a known meaningful unit ✅

This is why African-language-pretrained models outperform English-only models on African text — the tokeniser alone is already doing a better job before the model reads a single word.

Include a tokenisation analysis in your paper — show how many tokens AfroXLMR uses vs English-only models on the same Pidgin sentences. Fewer tokens = better vocabulary coverage = better understanding.

The Real Words Mapped to the Story

The One Thing to Remember

Tokenisation converts raw text into numbered pieces before any understanding happens. The tokeniser's vocabulary determines how well it handles your language. AfroXLMR's tokeniser treats Nigerian Pidgin words as known meaningful units — not random fragments. This is why it outperforms English-only models on your dataset before any fine-tuning even begins.