Reference · 24 entries
scikit-learn, as the notes use it
Every scikit-learn name that appears in a code block on this site has an entry here, and clicking the name in the code brings you to it. Each entry says what it does in plain words, which note teaches the idea behind it, and shows a small example with its real output.
from sklearn.… import … · .fit(X, y) and .predict(X) · .coef_ and .intercept_ · .fit_transform(X) and .transform(X) · .score(X, y)
load_diabetes, load_iris, load_wine · fetch_california_housing
LinearRegression · Ridge · KNeighborsClassifier · DummyRegressor and DummyClassifier
StandardScaler · MinMaxScaler · OneHotEncoder · TargetEncoder · SimpleImputer · PolynomialFeatures
The one pattern: fit, then predict
scikit-learn is the standard Python library for classical machine learning. Every model and every preprocessing step in it is an object with the same few methods, so once you know fit, predict and transform you can read any scikit-learn code.
§ from sklearn.… import …
scikit-learn is imported as sklearn and split into modules by job: linear_model for linear models, preprocessing for feature changes, metrics for scores, and so on. You import the one class or function you need from its module.
The idea behind it is taught in What machine learning is: data, model, loss, optimizer (week 1).
Call it as from sklearn.<module> import <Name>
from sklearn.linear_model import LinearRegression # a model
# a preprocessing step
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_squared_error # a score
import sklearn
print(sklearn.__version__)output1.7.2
- The package is installed as scikit-learn (pip install scikit-learn) but imported as sklearn.
- Class names are written in CapitalCase (LinearRegression) and are made with (); function names are lower_case (train_test_split) and are called directly.
§ .fit(X, y) and .predict(X)
Every model follows the same two steps. fit(X, y) is training: it looks at the inputs X and the right answers y and sets the model's parameters. predict(X) is inference: it uses those parameters to guess answers for inputs, including ones it has never seen. X is always 2-D, one row per example and one column per feature; y is 1-D, one answer per row.
The idea behind it is taught in What machine learning is: data, model, loss, optimizer (week 1).
Call it as model = SomeModel(settings)
model.fit(X_train, y_train)
predictions = model.predict(X_new)
import numpy as np
from sklearn.linear_model import LinearRegression
# 3 houses, 1 feature (size): shape (3, 1)
X = np.array([[1.0], [1.5], [2.0]])
y = np.array([200.0, 300.0, 380.0]) # 3 prices: shape (3,)
model = LinearRegression() # 1. make the model (knobs not set yet)
model.fit(X, y) # 2. train: set the knobs from the data
print(model.predict([[1.8]])) # 3. predict for a new houseoutput[347.33333333]
- X must be 2-D even with one feature: [[1.0], [1.5]], not [1.0, 1.5]. A 1-D X gives the error 'Expected 2D array, got 1D array instead'.
- fit returns the model itself, so model = LinearRegression().fit(X, y) makes and trains it in one line. That is why you will see that form often.
- Calling predict before fit raises a NotFittedError.
§ .coef_ and .intercept_
After fit, a model stores what it learned in attributes whose names end with an underscore. For a linear model, coef_ holds the weights (one per feature) and intercept_ holds the bias. The trailing underscore is a convention meaning 'this was learned from data, it does not exist before fit'.
The idea behind it is taught in What machine learning is: data, model, loss, optimizer (week 1).
Call it as model.coef_ model.intercept_
import numpy as np
from sklearn.linear_model import LinearRegression
X = np.array([[1.0], [1.5], [2.0]])
y = np.array([200.0, 300.0, 380.0])
model = LinearRegression().fit(X, y)
print(model.coef_) # one weight per feature
print(model.intercept_) # the bias boutput[180.]
23.333333333333428
- coef_ is always an array, even with one feature, so the slope is model.coef_[0].
§ .fit_transform(X) and .transform(X)
Preprocessing steps (scalers, encoders, imputers) do not predict; they change the data. fit learns what they need from the training data (for a scaler: each column's mean and spread). transform applies that change. fit_transform does both in one call and is used on the training data only; new data and test data get transform alone, so they are changed with the training data's numbers.
The idea behind it is taught in Feature scaling, normalization and encoding (week 1).
Call it as step.fit_transform(X_train)
step.transform(X_test)
import numpy as np
from sklearn.preprocessing import StandardScaler
X_train = np.array([[1.0], [2.0], [3.0]])
X_test = np.array([[4.0]])
scaler = StandardScaler()
# learn mean 2 and spread, then apply
print(scaler.fit_transform(X_train).ravel())
# apply the SAME numbers to new data
print(scaler.transform(X_test).ravel())output[-1.22474487 0. 1.22474487]
[2.44948974]
- Calling fit_transform on the test set is the classic leak: the test data then influences the preprocessing. Fit on train, transform everything else.
- transform returns a new array; it does not change X in place.
§ .score(X, y)
A shortcut that predicts on X and compares with y in one call. For a regression model it returns R² (1 is perfect, 0 is no better than guessing the average); for a classifier it returns accuracy, the fraction of right answers.
The idea behind it is taught in Linear regression (week 1).
Call it as model.score(X_test, y_test)
import numpy as np
from sklearn.linear_model import LinearRegression
X = np.array([[1.0], [1.5], [2.0], [2.5]])
y = np.array([200.0, 300.0, 380.0, 470.0])
model = LinearRegression().fit(X, y)
print(round(model.score(X, y), 4)) # R² of the line on these four housesoutput0.9982
- Which number score returns depends on the model type. When you need a specific metric, call it explicitly from sklearn.metrics instead.
Built-in datasets
Small real datasets that ship with the library, used throughout the notes so that every example runs without downloading anything by hand.
§ load_diabetes, load_iris, load_wine
Small datasets that come inside the library. With return_X_y=True each one returns two arrays: X, the features (one row per example), and y, the answers. diabetes: 442 patients, 10 measurements, y is a disease-progression score (a number). iris: 150 flowers, 4 measurements, y is the species 0, 1 or 2. wine: 178 wines, 13 chemical measurements, y is the grower 0, 1 or 2.
The idea behind it is taught in What machine learning is: data, model, loss, optimizer (week 1).
Call it as X, y = load_diabetes(return_X_y=True)
from sklearn.datasets import load_diabetes, load_iris, load_wine
for load in [load_diabetes, load_iris, load_wine]:
X, y = load(return_X_y=True)
print(load.__name__, X.shape, y.shape, y[:5])outputload_diabetes (442, 10) (442,) [151. 75. 141. 206. 135.]
load_iris (150, 4) (150,) [0 0 0 0 0]
load_wine (178, 13) (178,) [0 0 0 0 0]
- Without return_X_y=True you get a dictionary-like object; then the arrays are data.data and data.target, and data.feature_names lists the column names.
§ fetch_california_housing
A larger real dataset: 20,640 districts of California from the 1990 census, 8 features (median income, house age, rooms, location and so on), and y is the median house value in units of 100,000 dollars. It is downloaded once on first use and cached.
The idea behind it is taught in Linear regression (week 1).
Call it as X, y = fetch_california_housing(return_X_y=True)
from sklearn.datasets import fetch_california_housing
data = fetch_california_housing()
print(data.data.shape, data.target.shape)
print(data.feature_names)
print(data.target[:3]) # 4.526 means $452,600output(20640, 8) (20640,)
['MedInc', 'HouseAge', 'AveRooms', 'AveBedrms', 'Population', 'AveOccup', 'Latitude', 'Longitude']
[4.526 3.585 3.521]
- The first call needs an internet connection.
Splitting the data
Tools for holding data back from the model so that you can measure it honestly. The idea is taught in the first ML note; these are the functions that do it.
§ train_test_split
Shuffles the rows and cuts them into a training part and a test part. test_size=0.2 keeps 20% back for testing. random_state fixes the shuffle so you get the same split every run. It returns four arrays, always in this order: X_train, X_test, y_train, y_test.
The idea behind it is taught in What machine learning is: data, model, loss, optimizer (week 1).
Call it as X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
import numpy as np
from sklearn.model_selection import train_test_split
X = np.arange(20).reshape(10, 2) # 10 examples, 2 features
y = np.arange(10) # 10 answers
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=0)
print(X_train.shape, X_test.shape)
print(y_test) # which rows went to the test setoutput(8, 2) (2, 2)
[2 8]
- The order of the four results is easy to get wrong: both X parts first, then both y parts.
- For classification with rare classes, pass stratify=y so both parts keep the same class proportions.
§ KFold and cross_val_score
Instead of one train/test cut, cross-validation cuts the training data into k equal parts (folds), trains k times, each time holding a different fold out, and gives you k scores. Their average is a steadier estimate than one split. KFold describes how to cut; cross_val_score runs the k trainings and returns the k scores.
The idea behind it is taught properly in week 3, in “Train, validation, test and cross-validation”.
Call it as cross_val_score(model, X, y, cv=KFold(5, shuffle=True, random_state=0))
from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import KFold, cross_val_score
X, y = load_diabetes(return_X_y=True)
folds = KFold(n_splits=5, shuffle=True, random_state=0)
# R² on each held-out fold
scores = cross_val_score(LinearRegression(), X, y, cv=folds)
print(scores.round(3))
print(round(scores.mean(), 3))output[0.332 0.46 0.537 0.522 0.595]
0.489
- The model you pass is copied and refitted for every fold; the model object you passed stays untrained.
- Any preprocessing must be inside the model (a pipeline) or it leaks across folds.
Models
The models the notes use. Each one is a different rule for turning features into a prediction, but all of them are used the same way: make it, fit it, predict with it.
§ LinearRegression
Fits the best straight line (with many features, the best flat surface): prediction = w1·x1 + w2·x2 + … + b, with the weights chosen to make the mean squared error on the training data as small as possible. It has no settings you need to choose.
The idea behind it is taught in Linear regression (week 1).
Call it as LinearRegression().fit(X, y)
import numpy as np
from sklearn.linear_model import LinearRegression
X = np.array([[1.0], [1.5], [2.0]])
y = np.array([200.0, 300.0, 380.0])
model = LinearRegression().fit(X, y)
print(model.coef_, round(model.intercept_, 2))
print(model.predict([[1.8]]).round(2))output[180.] 23.33
[347.33]
- It is ordinary least squares: no penalty on large weights. For that, see Ridge.
§ Ridge
Linear regression with a penalty for large weights. The setting alpha says how strong the penalty is: 0 is plain linear regression, bigger values pull the weights toward zero. Used when there are many features or features that move together.
The idea behind it is taught properly in week 3, in “Regularization: L1, L2 and early stopping”.
Call it as Ridge(alpha=1.0).fit(X, y)
import numpy as np
from sklearn.linear_model import LinearRegression, Ridge
X = np.array([[1.0], [1.5], [2.0]])
y = np.array([200.0, 300.0, 380.0])
for model in [LinearRegression(), Ridge(alpha=0.1), Ridge(alpha=10)]:
model.fit(X, y)
print(type(model).__name__, model.coef_.round(1))outputLinearRegression [180.]
Ridge [150.]
Ridge [8.6]
- The penalty depends on the size of each weight, so features must be on the same scale first, or the penalty hits some features harder than others.
§ KNeighborsClassifier
Classifies a new example by finding the k training examples closest to it (by straight-line distance) and taking a vote among their labels. fit just stores the training data. The setting n_neighbors is k.
The idea behind it is taught properly in week 4, in “kNN and k-means: learning from similarity”.
Call it as KNeighborsClassifier(n_neighbors=5).fit(X, y)
import numpy as np
from sklearn.neighbors import KNeighborsClassifier
X = np.array([[1, 1], [1, 2], [8, 8], [9, 8]]) # two clusters of points
y = np.array([0, 0, 1, 1]) # their labels
model = KNeighborsClassifier(n_neighbors=3).fit(X, y)
print(model.predict([[2, 1], [8, 9]]))output[0 1]
- Because it compares distances, a feature measured in large numbers (salary) drowns out one in small numbers (age) unless you scale the features first.
§ DummyRegressor and DummyClassifier
Baselines packaged as models: they ignore the features and always predict one thing. DummyRegressor(strategy='mean') always predicts the training average; 'median' the training median. DummyClassifier(strategy='most_frequent') always predicts the most common class.
The idea behind it is taught in What machine learning is: data, model, loss, optimizer (week 1).
Call it as DummyRegressor(strategy="mean").fit(X, y)
import numpy as np
from sklearn.dummy import DummyRegressor, DummyClassifier
X = np.zeros((4, 1)) # the features are ignored anyway
mean_model = DummyRegressor(strategy="mean").fit(X, [10, 12, 14, 40])
print(mean_model.predict(X[:1])) # always the training mean
majority = DummyClassifier(strategy="most_frequent").fit(X, [0, 0, 0, 1])
print(majority.predict(X[:1])) # always the most common classoutput[19.]
[0]
- A dummy model still needs an X to fit and predict, only to know how many rows there are.
Preprocessing: changing the features
Steps that change the input columns before the model sees them: rescaling numbers, turning categories into numbers, filling gaps, adding powers of a feature. They learn what they need from the training data with fit and apply it with transform.
§ StandardScaler
Rescales each column to have mean 0 and standard deviation 1: subtract the column's average, divide by its spread. After it, a value of 1.5 means '1.5 spreads above average' in every column, so columns measured in different units become comparable.
The idea behind it is taught in Feature scaling, normalization and encoding (week 1).
Call it as StandardScaler().fit_transform(X_train)
import numpy as np
from sklearn.preprocessing import StandardScaler
# salary, age
X = np.array([[50_000, 25], [80_000, 35], [110_000, 45]], dtype=float)
scaler = StandardScaler().fit(X)
print(scaler.mean_) # the averages it learned
print(scaler.transform(X).round(2))output[8.0e+04 3.5e+01]
[[-1.22 -1.22]
[ 0. 0. ]
[ 1.22 1.22]]
- Fit it on the training data only, then transform the test data with it.
- It uses the population standard deviation (divides by n), the same as np.std by default.
§ MinMaxScaler
Rescales each column so that its smallest training value becomes 0 and its largest becomes 1, with everything else in proportion.
The idea behind it is taught in Feature scaling, normalization and encoding (week 1).
Call it as MinMaxScaler().fit_transform(X_train)
import numpy as np
from sklearn.preprocessing import MinMaxScaler
X = np.array([[10.0], [20.0], [40.0]])
scaler = MinMaxScaler().fit(X)
print(scaler.transform(X).ravel())
# bigger than anything in training: above 1
print(scaler.transform([[50.0]]).ravel())output[0. 0.33333333 1. ]
[1.33333333]
- One extreme value squashes every other value into a tiny range near 0.
- New data outside the training range lands outside 0 to 1.
§ OneHotEncoder
Turns a column of categories into several 0/1 columns, one per category. 'red', 'green', 'blue' becomes three columns, and each row has a single 1 in the column of its category. Models need numbers; this is the safe way to give them categories without inventing an order.
The idea behind it is taught in Feature scaling, normalization and encoding (week 1).
Call it as OneHotEncoder(handle_unknown="ignore").fit_transform(X_cat)
import numpy as np
from sklearn.preprocessing import OneHotEncoder
# one column, 4 rows
colors = np.array([["red"], ["green"], ["blue"], ["green"]])
encoder = OneHotEncoder(handle_unknown="ignore", sparse_output=False)
encoder.fit(colors)
print(encoder.categories_)
print(encoder.transform(colors))
print(encoder.transform([["purple"]])) # unseen category: all zerosoutput[array(['blue', 'green', 'red'], dtype='<U5')]
[[0. 0. 1.]
[0. 1. 0.]
[1. 0. 0.]
[0. 1. 0.]]
[[0. 0. 0.]]
- Without handle_unknown='ignore', a category that never appeared in training raises an error at prediction time.
- By default it returns a sparse matrix (mostly zeros stored compactly); sparse_output=False gives a normal array for printing.
§ TargetEncoder
Replaces each category by the average answer (target) for that category in the training data, for example each city by its average house price. Useful when a column has thousands of categories and one-hot would make thousands of columns. It mixes in the overall average for rare categories and uses cross-fitting internally so a row never sees its own answer.
The idea behind it is taught in Feature scaling, normalization and encoding (week 1).
Call it as TargetEncoder(target_type="continuous").fit_transform(X_cat, y)
import numpy as np
from sklearn.preprocessing import TargetEncoder
city = np.array([["A"], ["A"], ["B"], ["B"], ["C"], ["C"]] * 5)
price = np.array([100, 120, 300, 320, 200, 220] * 5, dtype=float)
encoder = TargetEncoder(target_type="continuous", random_state=0)
encoder.fit(city, price)
print(encoder.transform([["A"], ["B"], ["C"]]).ravel().round(1))output[110.1 309.9 210. ]
- Computing the category averages by hand on the whole dataset leaks the answers into the features; the encoder's internal cross-fitting exists to avoid exactly that.
§ SimpleImputer
Fills in missing values (NaN) in each column, with the column's training median, mean, or most frequent value.
The idea behind it is taught in Feature scaling, normalization and encoding (week 1).
Call it as SimpleImputer(strategy="median").fit_transform(X_train)
import numpy as np
from sklearn.impute import SimpleImputer
X = np.array([[1.0], [np.nan], [3.0], [100.0]])
print(SimpleImputer(strategy="median").fit_transform(X).ravel())
print(SimpleImputer(strategy="mean").fit_transform(X).ravel())output[ 1. 3. 3. 100.]
[ 1. 34.66666667 3. 100. ]
- The median is the safer default: one extreme value moves the mean a lot.
§ PolynomialFeatures
Adds powers of each feature as new columns: with degree=3, the single column x becomes the columns 1, x, x², x³. A straight-line model given these columns can draw a curve, because its prediction is now w1·x + w2·x² + w3·x³ + b. A higher degree means a curve that can bend more times.
The idea behind it is taught in Linear regression (week 1).
Call it as PolynomialFeatures(degree=3).fit_transform(X)
import numpy as np
from sklearn.preprocessing import PolynomialFeatures
x = np.array([[2.0], [3.0]]) # one feature, two rows
# columns: 1, x, x^2, x^3
print(PolynomialFeatures(degree=3).fit_transform(x))output[[ 1. 2. 4. 8.]
[ 1. 3. 9. 27.]]
- The number of columns grows very fast with several features (all products of features are added too).
- High degrees give huge numbers (x¹² for x = 10 is a trillion), so scale the feature first.
Chaining steps together
Real code rarely uses one step on its own. These tools glue preprocessing and a model into a single object, so that the right thing happens to training data and to new data automatically.
§ make_pipeline and Pipeline
Glues several steps and a model into one object that behaves like a single model. Calling fit on the pipeline runs fit_transform on each step in turn with the training data, then fits the model on the result. Calling predict runs transform on each step (with what it learned in fit) and then predicts. So you cannot forget a step on new data, and the test data never influences the preprocessing. make_pipeline names the steps for you; Pipeline lets you name them yourself.
The idea behind it is taught in Feature scaling, normalization and encoding (week 1).
Call it as model = make_pipeline(step1, step2, final_model)
import numpy as np
from sklearn.pipeline import make_pipeline, Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression
X = np.array([[1.0], [1.5], [2.0]])
y = np.array([200.0, 300.0, 380.0])
# scale, then fit a line
model = make_pipeline(StandardScaler(), LinearRegression())
# scaler learns mean and spread, then the line is fitted
model.fit(X, y)
# 1.8 is scaled the same way, then predicted
print(model.predict([[1.8]]).round(2))
print(list(model.named_steps)) # the names make_pipeline chose
# Pipeline: the same thing, but you choose the step names
same = Pipeline([
("scale", StandardScaler()),
("line", LinearRegression()),
])
same.fit(X, y)
print(same.named_steps["line"].coef_.round(2))output[347.33]
['standardscaler', 'linearregression']
[73.48]
- Everything except the last step must be a transformer (have transform); the last can be a model.
- To reach a step after fitting, use named_steps with its name, as in the last line.
§ ColumnTransformer
Applies different steps to different columns and puts the results side by side: for example, scale the number columns and one-hot encode the category columns. You give it a list of (name, step, which columns) triples. get_feature_names_out lists the output columns.
The idea behind it is taught in Feature scaling, normalization and encoding (week 1).
Call it as ColumnTransformer([("num", StandardScaler(), ["size"]), ("cat", OneHotEncoder(), ["city"])])
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
df = pd.DataFrame({"size": [1.0, 1.5, 2.0], "city": ["A", "B", "A"]})
prep = ColumnTransformer([
# numbers: scale
("num", StandardScaler(), ["size"]),
# categories: one-hot
("cat", OneHotEncoder(sparse_output=False), ["city"]),
])
print(prep.fit_transform(df).round(2))
print(prep.get_feature_names_out())output[[-1.22 1. 0. ]
[ 0. 0. 1. ]
[ 1.22 1. 0. ]]
['num__size' 'cat__city_A' 'cat__city_B']
- Columns you do not list are dropped unless you pass remainder='passthrough'.
Scoring predictions
Functions that compare the true answers with the predictions and return one number. They always take the true answers first.
§ mean_squared_error
The average of the squared differences between the true answers and the predictions. True answers come first.
The idea behind it is taught in What machine learning is: data, model, loss, optimizer (week 1).
Call it as mean_squared_error(y_true, y_pred)
from sklearn.metrics import mean_squared_error
y_true = [200, 300, 380]
y_pred = [200, 295, 390]
print(mean_squared_error(y_true, y_pred)) # (0 + 25 + 100) / 3output41.666666666666664
- It is in squared units (squared dollars). Its square root, the RMSE, is back in the original units and easier to say out loud.
§ mean_absolute_error
The average size of the misses, ignoring whether each was too high or too low.
The idea behind it is taught in What machine learning is: data, model, loss, optimizer (week 1).
Call it as mean_absolute_error(y_true, y_pred)
from sklearn.metrics import mean_absolute_error
# (0 + 5 + 10) / 3
print(mean_absolute_error([200, 300, 380], [200, 295, 390]))output5.0
- Unlike MSE it does not punish one big miss extra, so it is less affected by outliers.
§ r2_score
How much of the error of 'always guess the average' the predictions remove: 1 is perfect, 0 is no better than the average, below 0 is worse.
The idea behind it is taught in Linear regression (week 1).
Call it as r2_score(y_true, y_pred)
from sklearn.metrics import r2_score
y_true = [200, 300, 380]
print(round(r2_score(y_true, [200, 295, 390]), 3)) # close to the truth
average = sum(y_true) / len(y_true)
print(round(r2_score(y_true, [average] * 3), 3)) # just the average: 0output0.992
0.0
- Here the 'average' is the average of y_true itself, so R² on a test set compares against the test mean, not the training mean.