ML · Week 1 · Note 3 of 15

Feature scaling, normalization and encoding

Models see numbers, not meaning. A feature measured in millimetres shouts over one measured in metres, and a city name is not a number at all. Scaling puts numeric features on comparable footings; encoding turns categories into numbers a model can use. Both are done wrong constantly, and the mistakes are the kind interviewers ask about.

Why this note matters

Most of the work in a classical ML project is not choosing a model; it is getting the data into a form the model can learn from. Two questions come up on every tabular dataset: are the numeric columns on scales the model can handle, and how do the text categories become numbers? Get the first wrong and gradient descent crawls, or a model that works by distances (k-nearest neighbours ?, week 4) ignores most of your features. Get the second wrong and the model believes Tokyo is twice Paris. Get either one wrong in the wrong place and you leak test information into training, which is the most common way a project reports a score it cannot reproduce.

Questions that are really this note:

• “Why do we scale features? Which models need it and which do not?”
• “Standardization or min-max scaling? What about outliers?”
• “One-hot encoding versus label encoding: when is each wrong?”
• “What is the dummy variable trap?”
• “You scaled the data before splitting it. What is the problem?”
• “A categorical feature has 10,000 values. What do you do?”

In plain words

Why scale: distances, training steps, penalties

1. Distances. k-nearest neighbours ?, k-means ? (both week 4) and anything using Euclidean distance add squared differences across features. Take three customers with (age, income): A = (30, 50,000), B = (31, 51,000), C = (60, 50,100). Unscaled, $d(A, B) = \sqrt{1^2 + 1000^2} \approx 1000$ and $d(A, C) = \sqrt{30^2 + 100^2} \approx 104$. The model says C, thirty years older, is ten times closer to A than B is. Income’s large numbers swamped age. Standardize both (say age has a spread of 10 years and income a spread of 10,000) and $d(A, B) \approx 0.14$, $d(A, C) \approx 3.0$: B is closest, as it should be. The norms note made this point about distance; this is where it bites.

2. Gradient descent. With features of very different scales the loss surface is a long, narrow valley rather than a round bowl. The gradient points mostly across the valley, not along it, so a learning rate small enough not to overshoot the steep direction makes almost no progress along the flat one. Fig. 1 shows the zigzag. Scaling makes the bowl round, and the same learning rate works for every parameter. This is why neural networks ? normalize their inputs and, inside, the numbers flowing between their layers.

3. Penalties and interpretation. In week 3 you will add a penalty to the loss for big weights (regularization ?), using the L1 or L2 norm from the norms note. Such a penalty charges for the size of each weight, equally. A feature measured in millimetres needs a weight a thousand times smaller than the same feature in metres to have the same effect, so the penalty hits them differently for no reason. Scaling makes the penalty fair. The same scaling makes coefficients comparable, as the linear regression note showed.

And the models that do not care, which interviewers love to ask about:

The scaling methods

Use the column of ages $(20, 30, 40, 50, 60)$, mean 40, standard deviation 14.1, as the example throughout.

Standardization (z-scoring). Subtract the mean, divide by the standard deviation:

The result has mean 0 and standard deviation 1, whatever the original units. Values are not bounded: an outlier at 200 would become 11.3. It is the default for linear models and for the inputs of the week-4 and week-5 models (SVMs, PCA, neural networks), because those methods assume roughly zero-centred, unit-spread inputs and because it preserves outliers as outliers.

Min-max scaling. Map the smallest value to 0 and the largest to 1:

Bounded, which some methods want (pixel intensities, inputs to a sigmoid, distance computations where you want every feature to have the same maximum influence). Fragile with outliers: one value of 1,000 in the income column sends every normal income to 0.01 or so. Also, new data can fall outside [0, 1].

Robust scaling. Subtract the median, divide by the interquartile range (75th percentile minus 25th):

Median and quartiles barely move when a few values are extreme, so the bulk of the data keeps a sensible spread and the outliers stay far out where a model can treat them as unusual. Use it when the column has heavy tails you do not want to delete.

Log transform. Replace $x$ by $\log(1 + x)$ (the $1 +$ handles zeros) for columns that span orders of magnitude and pile up near zero: income, prices, counts, populations, word frequencies. It is not a rescaling but a reshaping: it turns a long right tail into something closer to a bell, which linear models fit much better. The logarithms note explained why multiplicative quantities belong on a log scale; here is the picture.

Which one, in one table:

Fit on train, apply everywhere

In scikit-learn the two steps are literally two methods: fit learns the statistics, transform applies them, and fit_transform does both on the training set. The discipline is: fit_transform on train, transform only on everything else. The mistake of fitting on the full dataset before splitting is a form of data leakage: information from the test rows (their mean, their range, their categories) has shaped the features the model trained on. The damage is usually small for a scaler and large for things like target encoding, but the rule is the same, and reviewers and interviewers check for it.

The tool that makes the rule automatic is a Pipeline: chain the preprocessing steps and the model into one object, call fit on the training data once, and every step fits on train and transforms downstream correctly. That stays true later when you train many times on different splits of the same data (cross-validation, week 3): the pipeline refits every step on each training part. A ColumnTransformer routes numeric columns to a scaler and categorical columns to an encoder. The code section has the complete pattern; use it for everything in Project 0.

One more reason the rule matters: deployment. The production system must scale a new row with the training mean and standard deviation, so the scaler is part of the model and is saved with it. A pipeline object saved to disk carries both; a hand-written (x - x.mean()) / x.std() in a notebook does not, and the first production row is scaled by itself into NaN.

Encoding categories

Nominal versus ordinal. A nominal feature has categories with no order (city, colour, user ID). An ordinal feature has categories with a meaningful order but not necessarily equal gaps (small / medium / large; poor / fair / good / excellent). The encoding follows the type.

One-hot encoding turns a nominal feature with $k$ categories into $k$ binary columns, exactly one of which is 1 per row. A linear model then learns one weight per category, which is a separate additive effect for each: the model can learn that Tokyo adds 80 and Paris adds 20 with no assumption linking them. Trees handle one-hot columns too, though many tree libraries now accept categories directly and do better with them. Two details:

• The dummy variable trap. The $k$ columns always sum to 1, which is the intercept’s constant column. For a linear model with an intercept that is perfect collinearity: $X^\top X$ is singular and the coefficients are not unique (the normal equation fails). Drop one column; the dropped category becomes the reference, and each remaining coefficient reads as “difference from the reference”. Models with the week 3 weight penalty, and the decision trees of week 4, do not need the drop.
• Unseen categories. Production will send a city that was not in training. The encoder must be told what to do: handle_unknown="ignore" outputs all zeros (the reference category’s effect), which is a reasonable default. Without it, the first unknown value crashes the service.

Ordinal encoding maps ordered categories to integers that respect the order: small = 0, medium = 1, large = 2. Linear models then assume equal gaps between levels, which may or may not be right; trees only use the order and are happy. The pitfall interviewers probe is using integer codes for a nominal feature (sometimes called label encoding, after the LabelEncoder that is meant for targets): London = 0, Paris = 1, Tokyo = 2 tells a linear model that Tokyo is “two Londons” and that Paris sits exactly between. Trees can partly recover by splitting repeatedly, but it is still arbitrary.

Binary features (yes/no) are a single 0/1 column; one-hot would create two redundant columns. Cyclic features such as hour of day or month have an order that wraps around: hour 23 is next to hour 0. Encoding them as $\sin(2\pi h/24)$ and $\cos(2\pi h/24)$ puts them on a circle so that 23 and 0 are neighbours; it is a small trick that shows up in time-series interviews.

Many categories: targets, hashes, embeddings

Target encoding. Replace each category by the mean of the target among training rows with that category: the postcode becomes “average price in this postcode”. One column, very informative, and very easy to get wrong. If you compute the mean using the row’s own target, the feature contains the answer and the model overfits spectacularly; rare categories with one row become perfect predictors. The fixes are to compute the encoding out-of-fold (each row’s value from the other folds) and to smooth rare categories towards the global mean. scikit-learn’s TargetEncoder does both. This is the leakage problem of the previous section in its most dangerous form, and “how would you encode a high-cardinality feature without leaking?” is a real interview question.

Hashing. Apply a hash function to the category string and use the result modulo a fixed number of buckets (say 1,024) as a one-hot index. No category list to store, unseen values handled automatically, fixed memory. The cost is collisions: two unrelated postcodes may share a bucket. Used at scale in ad systems and in the original scikit-learn FeatureHasher.

Embeddings. Give each category a learned vector of, say, 8 or 32 numbers, stored in a lookup table of shape $(\text{categories} \times \text{dimension})$, and train the table along with the model. Categories that behave similarly end up with similar vectors, so the model generalizes across them; rare categories borrow strength from their neighbours. This is a one-hot encoding followed by a weight matrix, learned end to end (a one-hot row times the table picks out one row, as the matrices note observed). It needs a neural network ? and enough data, and it is the native way deep models handle categories: user IDs in recommenders, and tokens in language models.

Scaling the target, and missing values

The target. For regression on a positive, skewed target (prices, incomes, durations), training on $\log y$ turns multiplicative errors into additive ones: a 10% miss costs the same on a cheap house as on an expensive one, instead of the expensive house dominating the loss. Predict, then apply $e^{(\cdot)}$ to get back to the original units, and compute your reported metric in those units. Standardizing the target is also common for neural networks so their last step starts at a sensible scale. For classification, the labels are categories and are not scaled; they are integer-coded, and the loss handles the rest.

Missing values. Models cannot multiply NaN. The standard approach is imputation: fill numeric gaps with the training median (robust to outliers) or mean, and categorical gaps with the most frequent category or an explicit “missing” category. Fit the imputer on the training split, like a scaler. Add a binary missing indicator column when missingness might carry meaning (a blank income field often does). Tree libraries such as XGBoost and LightGBM handle missing values natively by learning which way to send them at each split, which is one more reason they are the default for messy tabular data.

Code

Type these out. The pipeline example is the template to copy into Project 0.

The four scalers by hand, on a clean column and an outlier column

Each scaler is one line of arithmetic. Run them on a tidy column (ages) and on a column with one huge value (incomes).

import numpy as np

age = np.array([20.0, 30.0, 40.0, 50.0, 60.0])
# one outlier
income = np.array([20_000.0, 25_000.0, 30_000.0, 35_000.0, 1_000_000.0])

def standardize(x):
    # subtract the average, divide by the spread
    return (x - x.mean()) / x.std()

def minmax(x):
    # smallest value -> 0, largest -> 1
    return (x - x.min()) / (x.max() - x.min())

def robust(x):
    # a quarter of the values are below q1
    q1 = np.percentile(x, 25)
    # three quarters are below q3
    q3 = np.percentile(x, 75)
    # centre on the middle value, divide by the middle half's width
    return (x - np.median(x)) / (q3 - q1)

# print 2 decimals, never 1e+05 style
np.set_printoptions(precision=2, suppress=True)

print(standardize(age))
print(minmax(age))
print(robust(age))

# the four normal incomes are squashed together by the outlier
print(standardize(income))
print(minmax(income))        # even worse
print(robust(income))        # the normal incomes keep their spread
# the log pulls the outlier in: 1,000,000 becomes 6
print(np.log10(income))

# Output:
# [-1.41 -0.71  0.    0.71  1.41]
# [0.   0.25 0.5  0.75 1.  ]
# [-1.  -0.5  0.   0.5  1. ]
# [-0.52 -0.51 -0.49 -0.48  2.  ]
# [0.   0.01 0.01 0.02 1.  ]
# [-1.  -0.5  0.   0.5 97. ]
# [4.3  4.4  4.48 4.54 6.  ]

Training steps with and without scaling

The second reason to scale, in numbers. Two features, one between 0 and 1 and one between 0 and 1,000, and a simple training loop that adjusts the weights step by step (gradient descent ?, week 2). Watch which step sizes work on the raw features and which on the scaled ones.

import numpy as np

# random numbers that come out the same every run
rng = np.random.default_rng(0)
n = 200
# feature 1: values between 0 and 1
x1 = rng.uniform(0, 1, n)
# feature 2: values between 0 and 1000
x2 = rng.uniform(0, 1000, n)
# a table with 200 rows and 2 columns
X = np.column_stack([x1, x2])
noise = rng.normal(0, 0.1, n)             # small random wobble
# the true rule the model has to find
y = 3 * x1 + 0.002 * x2 + noise

def train_and_score(X, y, learning_rate, steps=5000):
    # one weight per feature, starting at 0
    w = np.zeros(X.shape[1])
    b = 0.0
    for step in range(steps):
        # prediction minus truth, for every row
        error = X @ w + b - y
        # move each weight a little downhill
        w = w - learning_rate * 2 * X.T @ error / n
        # move the bias a little downhill
        b = b - learning_rate * 2 * error.mean()
    final_error = X @ w + b - y
    # the mean squared error after training
    return np.mean(final_error ** 2)

# standardize each column separately
X_scaled = (X - X.mean(axis=0)) / X.std(axis=0)

# hide the overflow warnings when a run blows up
with np.errstate(all="ignore"):
    for learning_rate in [0.000001, 0.0001, 0.01, 0.1]:
        raw = train_and_score(X, y, learning_rate)
        scaled = train_and_score(X_scaled, y, learning_rate)
        print(f"step size {learning_rate:<9}  raw MSE {raw:>10.4f}   scaled MSE {scaled:>8.4f}")

# Output:
# step size 1e-06      raw MSE     1.4845   scaled MSE   8.0557
# step size 0.0001     raw MSE        nan   scaled MSE   1.1347
# step size 0.01       raw MSE        nan   scaled MSE   0.0098
# step size 0.1        raw MSE        nan   scaled MSE   0.0098
# Raw features: only the tiniest step size avoids blowing up (nan), and then
# training crawls.
# Scaled features: step size 0.01 or 0.1 reaches the noise level (about
# 0.01).

Nearest neighbours with and without scaling

The first reason, distances, on a real dataset. A nearest-neighbour ? model labels a new example by looking at the closest training examples, so whichever column has the biggest numbers decides what “closest” means.

from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

# 178 wines, 13 measurements each, y = which of 3 growers
X, y = load_wine(return_X_y=True)
# column 0: around 10 to 15
print("largest alcohol value:", X[:, 0].max())
# column 12: in the hundreds and thousands
print("largest proline value:", X[:, 12].max())

X_train, X_test, y_train, y_test = train_test_split(
    # stratify: keep the same mix of growers in both parts
    X, y, test_size=0.3, random_state=0, stratify=y)

# vote among the 5 closest training wines
raw_model = KNeighborsClassifier(n_neighbors=5)
raw_model.fit(X_train, y_train)
# fraction of test wines it gets right
raw_accuracy = raw_model.score(X_test, y_test)

scaled_model = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=5))
# learns the scaling from X_train, then stores the scaled wines
scaled_model.fit(X_train, y_train)
# scales X_test the same way, then votes
scaled_accuracy = scaled_model.score(X_test, y_test)

print("accuracy without scaling:", round(raw_accuracy, 2))
print("accuracy with scaling:   ", round(scaled_accuracy, 2))

# Output:
# largest alcohol value: 14.83
# largest proline value: 1680.0
# accuracy without scaling: 0.72
# accuracy with scaling:    0.96
# Unscaled, distances are decided almost entirely by proline; the other 12
# measurements barely count.

The complete pattern: ColumnTransformer and Pipeline

A table with numbers, a missing value and a category column, prepared and modelled in one object. This is the template to copy into Project 0.

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

# A small table of houses. np.nan marks a size that is missing.
houses = pd.DataFrame({
    "size":  [1.0, 1.5, 2.0, 1.2, np.nan, 2.4, 0.9, 1.8],
    "age":   [10, 5, 1, 30, 12, 2, 40, 8],
    "city":  ["London", "Paris", "Tokyo", "Paris", "London", "Tokyo", "London", "Paris"],
    "price": [200, 300, 380, 230, 260, 450, 150, 340],
})
X = houses[["size", "age", "city"]]      # the three feature columns
y = houses["price"]                      # the answer column
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=0)

# Number columns: fill gaps with the training median, then standardize.
number_steps = Pipeline([
    ("fill_gaps", SimpleImputer(strategy="median")),
    ("scale", StandardScaler()),
])
# The city column: one 0/1 column per city. A city never seen in training
# becomes all zeros.
city_step = OneHotEncoder(handle_unknown="ignore")

# Send each group of columns to its own steps, then put the results side by
# side.
prepare = ColumnTransformer([
    ("num", number_steps, ["size", "age"]),
    ("cat", city_step, ["city"]),
])
model = Pipeline([
    ("prepare", prepare),
    ("line", LinearRegression()),
])

# every median, mean, spread and city list is learned from X_train only
model.fit(X_train, y_train)
# the same learned numbers are applied to X_test, then the line predicts
predictions = model.predict(X_test)
print(predictions.round(1))
# the columns the line actually sees
print(model.named_steps["prepare"].get_feature_names_out())

# Output:
# [146.2 398.8]
# ['num__size' 'num__age' 'cat__city_London' 'cat__city_Paris'
#  'cat__city_Tokyo']

Leakage from fitting the scaler on all the data

What goes wrong when the scaler is allowed to see the test rows, on test data that has drifted.

import numpy as np
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(0)
# 100 rows, 1 column, centred on 50
train = rng.normal(50, 10, size=(100, 1))
# the test data comes from a shifted world: centred on 80
test = rng.normal(80, 10, size=(20, 1))
# train and test stacked into one table
everything = np.vstack([train, test])

wrong_scaler = StandardScaler()
# WRONG: learns its mean and spread from the test rows too
wrong_scaler.fit(everything)
right_scaler = StandardScaler()
# RIGHT: learns from the training rows only
right_scaler.fit(train)

wrong_mean = wrong_scaler.transform(test).mean()
right_mean = right_scaler.transform(test).mean()
print("average scaled test value, wrong way:", round(wrong_mean, 2))
print("average scaled test value, right way:", round(right_mean, 2))

# Output:
# average scaled test value, wrong way: 1.7
# average scaled test value, right way: 3.12
# The wrong way hides most of the shift (1.7 instead of about 3), so the
# test data looks more
# familiar than it is. In production the scaler is frozen, and the shift
# hits you in full.

Where you will meet this again

• Week 3, penalties. Ridge and Lasso (regularization ?) only make sense on standardized features, for reason 3 above.
• Week 4, distance-based models. kNN ?, k-means ? and PCA ? all break on unscaled features; trees do not care. The week-4 case study in Project 0 uses the Pipeline pattern from the code section end to end.
• Week 6, scaling inside a network. LayerNorm ? and its relatives standardize the numbers flowing between layers on the fly, for the Fig. 1 reason: a round bowl trains faster and tolerates a larger learning rate. Weight initialization schemes exist for the same reason.
• Week 7, tokens as categories. A language model’s vocabulary of 50,257 tokens ? is a nominal feature with 50,257 values. One-hot would be absurd; the model keeps an embedding ? table of shape $50{,}257 \times 768$ and looks up one row per token. That table is the learned encoding from the high-cardinality section. The “unknown category” problem is solved at the tokenizer level by byte-pair encoding, which guarantees every string can be split into known pieces.
• Week 7, positions on a circle. The original transformer told the model where each word sits using sines and cosines of the position, the same trick as encoding hour-of-day on a circle.

The full interview answer

The usual prompt is “how do you prepare features for a model?” or the more pointed “why scale, and how do you handle categoricals?”. Here is a complete answer in about ninety seconds.

If they push further, be ready for:

• “Does scaling change a linear regression’s predictions?” With the exact normal equation and no regularization, no; coefficients rescale but predictions are identical. With gradient descent (speed) or a penalty (which weights are shrunk), yes.
• “Why not label-encode cities for a random forest?” It imposes an arbitrary order; trees can partially undo it with multiple splits but it is still worse than one-hot or native categorical handling for most libraries.
• “What goes wrong with target encoding and how do you fix it?” The row’s own label leaks into its feature; compute out-of-fold and smooth towards the global mean.
• “A test row has a category never seen in training.” Encoder configured to ignore (all zeros) or map to an “other” bucket; for embeddings, a reserved unknown row.
• “What is an embedding, in these terms?” A learned, dense encoding of a categorical variable: one-hot followed by a weight matrix, trained end to end.

Practice questions

Answer each one out loud before opening it.

Practice problems

Paper first, then check. Solutions are hidden under each problem.

By hand

Code

import numpy as np
from sklearn.preprocessing import StandardScaler, MinMaxScaler

class MyStandardScaler:
    def fit(self, X):
        # one average per column, learned from X
        self.mean_ = X.mean(axis=0)
        self.scale_ = X.std(axis=0)      # one spread per column
        # returning the object itself is the scikit-learn habit
        return self

    def transform(self, X):
        return (X - self.mean_) / self.scale_

    def inverse_transform(self, Z):
        # undo: multiply back, add back
        return Z * self.scale_ + self.mean_

class MyMinMaxScaler:
    def fit(self, X):
        self.min_ = X.min(axis=0)
        self.range_ = X.max(axis=0) - X.min(axis=0)
        return self

    def transform(self, X):
        return (X - self.min_) / self.range_

    def inverse_transform(self, Z):
        return Z * self.range_ + self.min_

rng = np.random.default_rng(0)
X_train = rng.normal(50, 10, size=(100, 3))
X_test = rng.normal(55, 12, size=(20, 3))

pairs = [(MyStandardScaler(), StandardScaler()), (MyMinMaxScaler(), MinMaxScaler())]
for mine, theirs in pairs:
    # both learn from the training rows only
    mine.fit(X_train)
    theirs.fit(X_train)
    assert np.allclose(mine.transform(X_test), theirs.transform(X_test))
    round_trip = mine.inverse_transform(mine.transform(X_test))
    # scaling then unscaling gives back the original
    assert np.allclose(round_trip, X_test)
print("both scalers match scikit-learn")

# Output:
# both scalers match scikit-learn

import numpy as np
from sklearn.preprocessing import OneHotEncoder

# one column
cities = np.array([["London"], ["Paris"], ["Tokyo"], ["Paris"], ["London"], ["Tokyo"]])

# all three city columns
full_encoder = OneHotEncoder(sparse_output=False)
full = full_encoder.fit_transform(cities)
# drops the first city (London)
dropped_encoder = OneHotEncoder(sparse_output=False, drop="first")
dropped = dropped_encoder.fit_transform(cities)

for name, H in [("full one-hot", full), ("drop first", dropped)]:
    # put the intercept column of ones in front
    X = np.column_stack([np.ones(len(H)), H])
    G = X.T @ X
    # how many truly independent columns there are
    rank = np.linalg.matrix_rank(G)
    print(name, "| columns:", X.shape[1], "| rank of X^T X:", rank)
    if rank < X.shape[1]:
        print("   singular: one column is a sum of the others, the dummy variable trap")
    else:
        print("   invertible: the normal equation has one answer")
# Why not just call np.linalg.inv and see if it fails? Rounding errors often
# let inv return
# a matrix of huge, meaningless numbers instead of raising an error. The
# rank is the honest test.

# Output:
# full one-hot | columns: 4 | rank of X^T X: 3
#    singular: one column is a sum of the others, the dummy variable trap
# drop first | columns: 3 | rank of X^T X: 3
#    invertible: the normal equation has one answer

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import TargetEncoder
from sklearn.metrics import r2_score

rng = np.random.default_rng(0)
n = 2000
category = rng.integers(0, 500, n)       # 500 categories, about 4 rows each
# pure noise: the target has NOTHING to do with the category
y = rng.normal(size=n)
cat_train, cat_test, y_train, y_test = train_test_split(
    category, y, test_size=0.3, random_state=0)

# Naive encoding: each category's average target over the training rows,
# computed INCLUDING each row's own answer.
totals = {}
counts = {}
for c, target in zip(cat_train, y_train):
    # .get(c, 0.0): the total so far, or 0 if c is new
    totals[c] = totals.get(c, 0.0) + target
    counts[c] = counts.get(c, 0) + 1
overall_mean = y_train.mean()

def naive_encode(cats):
    values = []
    for c in cats:
        if c in counts:
            values.append(totals[c] / counts[c])
        else:
            # a category never seen in training
            values.append(overall_mean)
    # one column, as scikit-learn expects
    return np.array(values).reshape(-1, 1)

naive_train = naive_encode(cat_train)
naive_test = naive_encode(cat_test)
naive_model = LinearRegression().fit(naive_train, y_train)
naive_train_r2 = r2_score(y_train, naive_model.predict(naive_train))
naive_test_r2 = r2_score(y_test, naive_model.predict(naive_test))
print("naive   train R^2:", round(naive_train_r2, 2), "  test R^2:", round(naive_test_r2, 2))

# Proper: TargetEncoder encodes each training row without using that row's
# own answer, and smooths rare categories.
encoder = TargetEncoder(target_type="continuous", random_state=0)
proper_train = encoder.fit_transform(cat_train.reshape(-1, 1), y_train)
proper_test = encoder.transform(cat_test.reshape(-1, 1))
proper_model = LinearRegression().fit(proper_train, y_train)
proper_train_r2 = r2_score(y_train, proper_model.predict(proper_train))
proper_test_r2 = r2_score(y_test, proper_model.predict(proper_test))
print("proper  train R^2:", round(proper_train_r2, 2), "  test R^2:", round(proper_test_r2, 2))

# Output:
# naive   train R^2: 0.33   test R^2: -0.33
# proper  train R^2: 0.0   test R^2: -0.0
# naive: train R^2 clearly above 0 (it 'learned' noise from its own
# answers), test R^2 below 0.
# proper: both about 0, which is the truth: the category carries no
# information.

import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import LinearRegression
from sklearn.neighbors import KNeighborsClassifier

rng = np.random.default_rng(0)
n_train = 160
n_test = 40
# Feature 1 is a sensor that drifts upward after the training period;
# feature 2 is stable.
drift = np.concatenate([rng.normal(10, 1, n_train), rng.normal(14, 1, n_test)])
stable = rng.normal(0, 1, n_train + n_test)
X = np.column_stack([drift, stable])
# the class depends only on the stable feature
label = (stable > 0).astype(int)
# a number to predict, for the linear model
price = 2 * drift + stable + rng.normal(0, 1, n_train + n_test)

# the first 160 rows: the past
X_train = X[:n_train]
# the last 40 rows: the future
X_test = X[n_train:]

# (a) Leaky: the scaler learns from ALL rows, including the future test rows.
leaky_scaler = StandardScaler()
leaky_scaler.fit(X)
leaky_train = leaky_scaler.transform(X_train)
leaky_test = leaky_scaler.transform(X_test)

# (b) Clean: the scaler sits inside a pipeline and learns from the training rows only.
clean_knn = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=5))
clean_line = make_pipeline(StandardScaler(), LinearRegression())

# Plain linear regression: scaling only rescales the weights, so the leak
# changes nothing.
leaky_line = LinearRegression()
leaky_line.fit(leaky_train, price[:n_train])
leaky_predictions = leaky_line.predict(leaky_test)
clean_line.fit(X_train, price[:n_train])
clean_predictions = clean_line.predict(X_test)
print("linear regression predictions identical:", np.allclose(leaky_predictions, clean_predictions))

# Nearest neighbours compare distances, so the scaler's numbers matter.
leaky_knn = KNeighborsClassifier(n_neighbors=5)
leaky_knn.fit(leaky_train, label[:n_train])
leaky_accuracy = leaky_knn.score(leaky_test, label[n_train:])
clean_knn.fit(X_train, label[:n_train])
clean_accuracy = clean_knn.score(X_test, label[n_train:])
print("nearest neighbours, scaler fit on all rows:", leaky_accuracy)
print("nearest neighbours, scaler inside pipeline:", clean_accuracy)

# Output:
# linear regression predictions identical: True
# nearest neighbours, scaler fit on all rows: 0.975
# nearest neighbours, scaler inside pipeline: 0.825
# The leaky scaler already knew where the drifted test rows sit, so the
# nearest-neighbour score looks
# better than it will be in production. The clean, lower number is the
# honest one.

Common mistakes

All sources

Everything linked above, in the order to use it, plus one reference.

1. Google ML Crash Course, Normalization and One-hot encoding (reading, 20 min total). Start here.
2. scikit-learn, Importance of feature scaling and Compare the effect of different scalers on data with outliers (runnable examples).
3. scikit-learn, Common pitfalls: data leakage (reading, 15 min).
4. scikit-learn user guide, Preprocessing data, with the OneHotEncoder, TargetEncoder and ColumnTransformer pages (documentation).
5. Kaggle Learn, Feature Engineering (hands-on, for week 4).
6. Raschka, Machine Learning with PyTorch and Scikit-Learn, chapter 4 code: “Building Good Training Datasets: Data Preprocessing”. The chapter is the reading for this note.

One-minute revision