Calibration for Binary Classification¶
Setup¶
pip install ydf -U
Why Calibrate?¶
After training a binary classification model (e.g., predicting whether an email is SPAM or NOT_SPAM), the predicted probabilities are often not representative of the actual observed rates of the positive label.
For example, a binary classifier might output a predicted probability of 30%, but the true rate of observing the SPAM label at that predicted rate could be 45% or just 22% (i.e. not 30%).
Correcting this mismatch is referred to as calibration.
When should you care?¶
You should consider calibrating your binary classification model if your workflow meets any of the following conditions:
- You use predicted probabilities as direct decision thresholds: If you trigger down-stream actions based on a specific probability (e.g., "If spam probability > 85%, auto-delete the email"), you need those numbers to reflect true historical frequencies.
- Your outputs feed into expected value calculations: If your model's probability \(p\) is used to calculate risk or financial outcomes (e.g., \(\mathrm{Expected Value}=p\times\mathrm{Value}\)), uncalibrated probabilities will severely bias your financial projections and lead to poor resource allocation.
- You are combining or comparing multiple models: If you have multiple models feeding into a single decision engine, they must speak the "same language." Uncalibrated models cannot have their probabilities compared or combined reliably.
- You are using models prone to overconfidence: Heavy modern classifiers (like deep trees or deep learning architectures) are notorious for pushing predictions close to
0.0or1.0without actual statistical justification.
Note:
Calibration will not improve your model's ranking metrics (like ROC-AUC) because YDF uses isotonic regression, which strictly preserves prediction ordering. If your only goal is to rank items (e.g., recommendation feeds), calibration is likely unnecessary overhead.
This tutorial walks through how to calibrate a binary classifier in YDF as well as how you assess calibration information included in the evaluation results.
Binary classification model¶
For calibration, let's start by training a binary classification model on a real dataset. We'll use the classic "Adult" dataset for this tutorial.
# Load necessary libraries
import ydf # Yggdrasil Decision Forests.
import pandas as pd # To load and inspect the dataset.
# Download a classification dataset and load it as a Pandas DataFrame.
ds_path = "https://raw.githubusercontent.com/google/yggdrasil-decision-forests/main/yggdrasil_decision_forests/test_data/dataset"
train_ds = pd.read_csv(f"{ds_path}/adult_train.csv")
test_ds = pd.read_csv(f"{ds_path}/adult_test.csv")
The data is already partitioned into train and test splits. However, when calibrating a model it is advised that you reserve an additional held-out portion of data specifically for this purpose.
# We will reserve 10% of the training data for calibration.
cal_ds = train_ds.sample(frac = 0.10, random_state=0)
train_ds = train_ds.drop(cal_ds.index)
The task is binary classification: predict whether an individual's income is >50k or <=50k based on other numerical and categorical features.
This dataset also contains missing values, which YDF handles automatically.
# Note: task=ydf.Task.CLASSIFICATION is the default, so it's optional here.
model = ydf.RandomForestLearner(label="income").train(train_ds)
Train model on 20513 examples Model trained in 0:00:02.491828
The evaluation plots for a binary classification model include a reliability diagram. This diagram reveals how well the predicted probabilities reflect the frequency of the true label. Ideally, this curve is a \(45^\circ\) line between \((0,0)\) and \((1,1)\). The error band around the curve is proportional to the amount of data used to determine the reliability at that point (specifically, a Beta-binomial error model).
Two calibration-related metrics are also included: the MCE and ECE. These two metrics summarize how well the actual curve matches the ideal.
evaluation = model.evaluate(test_ds)
evaluation
Evaluation of classification models
- Accuracy
- The simplest metric. It's the percentage of predictions that are correct (matching the ground truth).
Example: If a model correctly identifies 90 out of 100 images as cat or dog, the accuracy is 90%. - Confusion Matrix
- A table that shows the counts of:
- True Positives (TP): Model correctly predicted positive.
- True Negatives (TN): Model correctly predicted negative.
- False Positives (FP): Model incorrectly predicted positive (a "false alarm").
- False Negatives (FN): Model incorrectly predicted negative (a "miss").
- Calibration Metrics
- Two metrics that measure how well the model's predicted probabilities reflect the actual likelihood of the positive class.
- ECE (Expected Calibration Error): The difference between the average predicted probability and the actual fraction of positives, averaged across all bins.
- MCE (Maximum Calibration Error): The maximum difference between the average predicted probability and the actual fraction of positives, across all bins.
Interpretation: A well-calibrated model should have low ECE and MCE. - Calibration Plot (Reliability Diagram)
- A graph that plots the actual fraction of positives against the average predicted probability for each bin.
Interpretation: A well-calibrated model should have points lying close to the diagonal line (where actual fraction = average predicted probability). - Threshold
- YDF classification models predict a probability for each class. A threshold determines the cutoff for classifying something as positive or negative.
Example: If the threshold is 0.5, any prediction above 0.5 might be classified as "spam," and anything below as "not spam." - ROC Curve (Receiver Operating Characteristic Curve)
- A graph that plots the True Positive Rate (TPR) against the False Positive Rate (FPR) at various thresholds.
- TPR (Sensitivity or Recall): TP / (TP + FN) - How many of the actual positives did the model catch?
- FPR: FP / (FP + TN) - How many negatives were incorrectly classified as positives?
Interpretation: A good model has an ROC curve that hugs the top-left corner (high TPR, low FPR). - AUC (Area Under the ROC Curve)
- A single number that summarizes the overall performance shown by the ROC curve. The AUC is a more stable metric than the accuracy. Multi-class classification models evaluate one class against all other classes.
Interpretation: Ranges from 0 to 1. A perfect model has an AUC of 1, while a random model has an AUC of 0.5. Higher is better. - Precision-Recall Curve
- A graph that plots Precision against Recall at various thresholds.
- Precision: TP / (TP + FP) - Out of all the predictions the model labeled as positive, how many were actually positive?
- Recall (same as TPR): TP / (TP + FN) - Out of all the actual positive cases, how many did the model correctly identify?
Interpretation: A good model has a curve that stays high (both high precision and high recall). It is especially useful when dealing with imbalanced datasets (e.g., when one class is much rarer than the other). - PR-AUC (Area Under the Precision-Recall Curve)
- Similar to AUC, but for the Precision-Recall curve. A single number summarizing performance. Multi-class classification models evaluate one class against all other classes. Higher is better.
- Threshold / Accuracy Curve
- A graph that shows how the model's accuracy changes as you vary the classification threshold.
- Threshold / Volume Curve
- A graph showing how the number of data points classified as positive changes as you vary the threshold.
| Label \ Pred | <=50K | >50K |
|---|---|---|
| <=50K | 6964 | 448 |
| >50K | 888 | 1469 |
How well the model is calibrated can be assessed from the Expected Calibration Error (ECE), Maximum Calibration Error (MCE), and the Reliability Diagram. Detailed descriptions of each of these are given in the provided glossary. Note that from the Reliability Diagram we can see that the model is poorly calibrated. This is evident in the fact the curve is significantly different from the ideal (below).
Let's run YDF's calibration routine to modify the model's prediction such that they better match the observed frequencies in the data.
Calibrate¶
A model has a calibrate method that accepts a dataset who's labels the model's predictions are calibrated.
The method also requires a ydf.CalibrationConfig.
There are two hyperparameters used to perform the calibration and another hyperparameter for deployment.
Here we will set a parameter that controls the amount of smoothing (z_threshold) as this results in the best fit.
Here we'll calibrate on specially reserved dataset that is 10% of the original dataset.
# Calibrate the model in place.
model.calibrate(
cal_ds,
config=ydf.CalibrationConfig(z_threshold=1.0),
)
Let's re-evaluate on the test set to see if the reliability diagram has improved.
evaluation = model.evaluate(test_ds)
evaluation
Evaluation of classification models
- Accuracy
- The simplest metric. It's the percentage of predictions that are correct (matching the ground truth).
Example: If a model correctly identifies 90 out of 100 images as cat or dog, the accuracy is 90%. - Confusion Matrix
- A table that shows the counts of:
- True Positives (TP): Model correctly predicted positive.
- True Negatives (TN): Model correctly predicted negative.
- False Positives (FP): Model incorrectly predicted positive (a "false alarm").
- False Negatives (FN): Model incorrectly predicted negative (a "miss").
- Calibration Metrics
- Two metrics that measure how well the model's predicted probabilities reflect the actual likelihood of the positive class.
- ECE (Expected Calibration Error): The difference between the average predicted probability and the actual fraction of positives, averaged across all bins.
- MCE (Maximum Calibration Error): The maximum difference between the average predicted probability and the actual fraction of positives, across all bins.
Interpretation: A well-calibrated model should have low ECE and MCE. - Calibration Plot (Reliability Diagram)
- A graph that plots the actual fraction of positives against the average predicted probability for each bin.
Interpretation: A well-calibrated model should have points lying close to the diagonal line (where actual fraction = average predicted probability). - Threshold
- YDF classification models predict a probability for each class. A threshold determines the cutoff for classifying something as positive or negative.
Example: If the threshold is 0.5, any prediction above 0.5 might be classified as "spam," and anything below as "not spam." - ROC Curve (Receiver Operating Characteristic Curve)
- A graph that plots the True Positive Rate (TPR) against the False Positive Rate (FPR) at various thresholds.
- TPR (Sensitivity or Recall): TP / (TP + FN) - How many of the actual positives did the model catch?
- FPR: FP / (FP + TN) - How many negatives were incorrectly classified as positives?
Interpretation: A good model has an ROC curve that hugs the top-left corner (high TPR, low FPR). - AUC (Area Under the ROC Curve)
- A single number that summarizes the overall performance shown by the ROC curve. The AUC is a more stable metric than the accuracy. Multi-class classification models evaluate one class against all other classes.
Interpretation: Ranges from 0 to 1. A perfect model has an AUC of 1, while a random model has an AUC of 0.5. Higher is better. - Precision-Recall Curve
- A graph that plots Precision against Recall at various thresholds.
- Precision: TP / (TP + FP) - Out of all the predictions the model labeled as positive, how many were actually positive?
- Recall (same as TPR): TP / (TP + FN) - Out of all the actual positive cases, how many did the model correctly identify?
Interpretation: A good model has a curve that stays high (both high precision and high recall). It is especially useful when dealing with imbalanced datasets (e.g., when one class is much rarer than the other). - PR-AUC (Area Under the Precision-Recall Curve)
- Similar to AUC, but for the Precision-Recall curve. A single number summarizing performance. Multi-class classification models evaluate one class against all other classes. Higher is better.
- Threshold / Accuracy Curve
- A graph that shows how the model's accuracy changes as you vary the classification threshold.
- Threshold / Volume Curve
- A graph showing how the number of data points classified as positive changes as you vary the threshold.
| Label \ Pred | <=50K | >50K |
|---|---|---|
| <=50K | 7001 | 411 |
| >50K | 912 | 1445 |
Both the metrics and reliability diagram show substantial improvements while leaving the quality metrics unchanged. The calibration hyperparameters are detailed here and you may need to tune them to your use case.
Technical Note¶
Normal isotonic regression fitting for calibration will perfectly align the data used for calibration with the ideal reliability diagram (i.e. the \(45^\circ\) diagonal). However, the method used in YDF uses isotonic regression as a starting point for further smoothing and interpolation. Thus, the performance of the calibration on the data used for calibration will not be perfectly calibrated. See here for a detailed explanation for why this behavior is expected.
evaluation = model.evaluate(cal_ds)
evaluation
Evaluation of classification models
- Accuracy
- The simplest metric. It's the percentage of predictions that are correct (matching the ground truth).
Example: If a model correctly identifies 90 out of 100 images as cat or dog, the accuracy is 90%. - Confusion Matrix
- A table that shows the counts of:
- True Positives (TP): Model correctly predicted positive.
- True Negatives (TN): Model correctly predicted negative.
- False Positives (FP): Model incorrectly predicted positive (a "false alarm").
- False Negatives (FN): Model incorrectly predicted negative (a "miss").
- Calibration Metrics
- Two metrics that measure how well the model's predicted probabilities reflect the actual likelihood of the positive class.
- ECE (Expected Calibration Error): The difference between the average predicted probability and the actual fraction of positives, averaged across all bins.
- MCE (Maximum Calibration Error): The maximum difference between the average predicted probability and the actual fraction of positives, across all bins.
Interpretation: A well-calibrated model should have low ECE and MCE. - Calibration Plot (Reliability Diagram)
- A graph that plots the actual fraction of positives against the average predicted probability for each bin.
Interpretation: A well-calibrated model should have points lying close to the diagonal line (where actual fraction = average predicted probability). - Threshold
- YDF classification models predict a probability for each class. A threshold determines the cutoff for classifying something as positive or negative.
Example: If the threshold is 0.5, any prediction above 0.5 might be classified as "spam," and anything below as "not spam." - ROC Curve (Receiver Operating Characteristic Curve)
- A graph that plots the True Positive Rate (TPR) against the False Positive Rate (FPR) at various thresholds.
- TPR (Sensitivity or Recall): TP / (TP + FN) - How many of the actual positives did the model catch?
- FPR: FP / (FP + TN) - How many negatives were incorrectly classified as positives?
Interpretation: A good model has an ROC curve that hugs the top-left corner (high TPR, low FPR). - AUC (Area Under the ROC Curve)
- A single number that summarizes the overall performance shown by the ROC curve. The AUC is a more stable metric than the accuracy. Multi-class classification models evaluate one class against all other classes.
Interpretation: Ranges from 0 to 1. A perfect model has an AUC of 1, while a random model has an AUC of 0.5. Higher is better. - Precision-Recall Curve
- A graph that plots Precision against Recall at various thresholds.
- Precision: TP / (TP + FP) - Out of all the predictions the model labeled as positive, how many were actually positive?
- Recall (same as TPR): TP / (TP + FN) - Out of all the actual positive cases, how many did the model correctly identify?
Interpretation: A good model has a curve that stays high (both high precision and high recall). It is especially useful when dealing with imbalanced datasets (e.g., when one class is much rarer than the other). - PR-AUC (Area Under the Precision-Recall Curve)
- Similar to AUC, but for the Precision-Recall curve. A single number summarizing performance. Multi-class classification models evaluate one class against all other classes. Higher is better.
- Threshold / Accuracy Curve
- A graph that shows how the model's accuracy changes as you vary the classification threshold.
- Threshold / Volume Curve
- A graph showing how the number of data points classified as positive changes as you vary the threshold.
| Label \ Pred | <=50K | >50K |
|---|---|---|
| <=50K | 1617 | 104 |
| >50K | 220 | 338 |
Notice that the reliability diagram is near the ideal, but clearly diverges especially in the reliability diagram's lowest-populated bins. Performance on other datasets may have an even larger divergence. Again, tune the calibration parameters until you attain a reliability diagram and set of metrics suitable for your application.