Train, Validation, and Test Split — Explained Simply

What Train, Validation, and Test Split Actually Are

You Are a Teacher Writing an Exam

You have a bank of 1000 possible questions and three groups of people who need different sets.

The Students Studying at Home They study practice questions many times — learn from them, make mistakes, improve. These are your training samples.

You Checking Progress Mid-Term Every two weeks you quiz them on questions they have never seen — not to grade permanently but to check if your teaching is working. These are your validation samples.

The Final Official Exam Questions nobody has ever seen. The true honest measure of what was learned. These are your test samples.

The Critical Insight

If students see exam questions in advance — even once — the exam no longer measures what they learned. It measures how well they memorised those specific questions.

Same for your model. If you use the test set to make any decisions during training, the test score is contaminated.

Training set   → The model learns from this
Validation set → You make decisions based on this
Test set       → Touch ONCE at the very end. Never use to make decisions.

The Split in Numbers

Total dataset: 1000 Nigerian Pidgin reviews

Training:   700 samples (70%)  → model learns from these
Validation: 150 samples (15%)  → you monitor progress here
Test:       150 samples (15%)  → final honest evaluation

Common splits in papers:
  70 / 15 / 15  ← standard for medium datasets
  80 / 10 / 10  ← when you need more training data
  60 / 20 / 20  ← when you want more reliable evaluation

In Code

from sklearn.model_selection import train_test_split

# Step 1 — Split off test set first and PUT IT AWAY
X_temp, X_test, y_temp, y_test = train_test_split(
    reviews, labels,
    test_size    = 0.15,
    random_state = 42,
    stratify     = labels    # Keep class balance
)

# Step 2 — Split remainder into train and validation
X_train, X_val, y_train, y_val = train_test_split(
    X_temp, y_temp,
    test_size    = 0.176,    # 15% of original
    random_state = 42,
    stratify     = y_temp
)

print(f"Training:   {len(X_train)} samples")   # ~700
print(f"Validation: {len(X_val)} samples")     # ~150
print(f"Test:       {len(X_test)} samples")    # ~150

What You Do With Each Split

for epoch in range(50):

    # Training set — model learns
    model.train()
    for X_batch, y_batch in train_loader:
        optimizer.zero_grad()
        loss = loss_fn(model(X_batch), y_batch)
        loss.backward()
        optimizer.step()

    # Validation set — you make decisions
    model.eval()
    with torch.no_grad():
        val_loss = evaluate(model, val_loader)

    # Decisions based on validation:
    # Should I stop? Best model so far? Change learning rate?
    if val_loss < best_val_loss:
        best_val_loss = val_loss
        torch.save(model.state_dict(), 'best_model.pth')

# Test set — run ONCE at the very end
model.load_state_dict(torch.load('best_model.pth'))
model.eval()
with torch.no_grad():
    test_loss = evaluate(model, test_loader)
    # This number goes in your research paper

The Most Common Mistake

Looking at the test set during development.

❌ Wrong:
   Train → check test → adjust → train → check test → adjust
   Test score is now lying to you

✅ Correct:
   Train → check validation → adjust → train → check validation → adjust
   → completely done → check test ONCE → final score

Why Stratify Matters

Without stratify — random split might give:
  Training:   550 positive, 100 negative, 50 neutral  ← imbalanced

With stratify=labels — guaranteed proportional:
  Training:   420 positive, 210 negative, 70 neutral  ← correct ✅

# Always stratify when classes are imbalanced
# Your Nigerian Pidgin dataset will almost certainly be imbalanced
X_train, X_test, y_train, y_test = train_test_split(
    X, y,
    test_size    = 0.2,
    stratify     = y,      # ← critical line
    random_state = 42
)

Cross-Validation — When Your Dataset Is Small

If you only have 300 reviews, a standard split gives too few validation samples. Rotate instead:

from sklearn.model_selection import StratifiedKFold

kfold  = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = []

for fold, (train_idx, val_idx) in enumerate(kfold.split(X, y)):
    X_train, X_val = X[train_idx], X[val_idx]
    y_train, y_val = y[train_idx], y[val_idx]

    val_f1 = train_and_evaluate(X_train, y_train, X_val, y_val)
    scores.append(val_f1)
    print(f"Fold {fold+1}: F1 = {val_f1:.4f}")

print(f"Average F1: {sum(scores)/len(scores):.4f}")
# Report this in your paper

The Real Words Mapped to the Story

The One Thing to Remember

The test set is sacred. You touch it exactly once — at the very end — to get your final honest score. Every decision during training uses the validation set. The moment you use the test set to make a decision, it is no longer an honest test.