AI-generated note
Model validation — full lecture notes
Source: Data Science · Module 3 — Model Validation · Coursera · Andrew Park · 46 min lecture
Why split the data
Lesson- Goal: estimate how the model will perform on data it has never seen — the only metric that matters.
- Failure mode: if you only score on the training set, every model looks great — including the bad ones.
- Held-out principle: no decision (model choice, hyperparameters) can use the test set BEFORE the final evaluation.
- Why not just train more: training-set accuracy is bounded by the model's capacity; it tells you nothing about generalization.
- Prof.'s analogy: "using training accuracy is like grading your own essay" — 14:48 timestamp.
The three-way split
Lesson- Training set: used to fit model parameters via the learning algorithm.
- Validation set: used to tune hyperparameters and pick between models — touched many times.
- Test set: used exactly ONCE at the end to estimate real-world performance.
- Why three not two: if you tune on the test set, it becomes a second training set — you'll be overconfident about generalization.
- Common mistake: looking at test-set score and going back to tweak the model — invalidates the test set.
Typical split ratios
Lesson- 70/15/15: default for medium datasets (1k–100k samples).
- 80/10/10: common when 10–100k samples are available and CV will be used on training set.
- Big data 98/1/1: with 1M+ samples, 1% can still be 10k — enough for stable estimates.
- Small data alternative: below ~1k samples, use cross-validation instead of a fixed validation set.
Stratification
Lesson- When: use stratified sampling whenever the target has class imbalance or rare events.
- How: preserves the class ratio across train/val/test splits — prevents accidentally getting 0 positives in val.
- Tool: sklearn's StratifiedKFold or train_test_split(stratify=y).
- Continuous y: for regression, bin the target into quantiles and stratify on bins.
Cross-validation (lesson 3.3 preview)
Lesson- k-fold CV: split training set into k parts; train on k-1, validate on 1; rotate; average the k scores.
- Typical k: 5 or 10 — diminishing returns past 10 in most settings.
- Leave-One-Out: k = n; extremely high-variance estimate but unbiased; use only for very small datasets.
- Stratified k-fold: preserves class ratios in each fold — default choice for classification.
- Time-series CV: expanding-window or rolling-window splits; never train on future data.
Overfitting (timestamp 31:20)
Lesson- Definition: the model memorises training-set noise instead of learning the signal.
- Signal: training accuracy keeps rising while validation accuracy plateaus or falls.
- Causes: too few samples relative to model capacity, too many features, no regularization, no early stopping.
- Fixes: more data, simpler models, L1/L2 regularization, dropout, early stopping, data augmentation.
- Diagnosis: plot train vs val accuracy across epochs — diverging curves = overfitting.
Data leakage (advanced)
Lesson- Pre-processing trap: fitting your scaler/imputer on the FULL dataset (including val/test) — leaks distribution info.
- Fix: fit pre-processing inside a Pipeline that's cross-validated, so val data is never seen during fitting.
- Target leakage: feature derived from target (e.g., 'days since survey' when target is survey result) → unrealistic accuracy.
- Temporal leakage: training on future data to predict past — common in time-series; use proper rolling splits.
Worked example
Lesson- Setup: 10,000 samples, binary classification, 70/15/15 split.
- Counts: train = 7,000; val = 1,500; test = 1,500.
- Stratified: if class ratio is 80/20, each split keeps that ratio — train has 5,600 negatives / 1,400 positives.
- Pipeline: fit scaler on train → transform val/test using the SAME scaler; never re-fit.
- Final model: after tuning on val, retrain on train+val and report ONE test number.
