Ensembles & Boosting

Bagging, random forests, and gradient boosting (XGBoost) — still the winners on tabular data. Built from zero, starting with why a committee of mediocre models beats one clever one.

intermediate#ensembles#boosting#xgboost#random-forest

Start here — one model, or a committee?

Everything you've built so far has been one model: one decision tree, one regression line, one classifier. You trained it, you measured it, and whatever accuracy it got, that was that.

This page is about a different move, and it's one of the most reliably useful tricks in all of machine learning: don't build one model. Build a crowd of them and let them vote.

That crowd is called an , and each member of it is a .

The whole page in one sentence

A single model is wrong in its own particular way. A different model is wrong in a different particular way. If you average enough of them, the individual mistakes point in random directions and largely cancel out — while the thing they all agree on, the actual signal, survives.

Think of it like guessing how many jellybeans are in the jar:

At a fair, everyone writes down a guess. Almost every individual guess is badly wrong — some wildly high, some wildly low. But take the average of a few hundred guesses and it lands startlingly close to the truth. Nobody in the crowd knew the answer. The crowd did. Ensembles are that effect, engineered on purpose.

Why this topic still matters in 2026

Deep learning owns images, audio, and text. But on tabular data — spreadsheets, transactions, medical records, log tables, the kind of data most companies actually have — gradient-boosted trees are still, routinely, the thing that wins. Learning this is not history; it's the tool you'll reach for most often outside of deep learning.

This page adapts to you

It starts from zero. Flip the Depth switch at the top for formal notation and derivations — they open automatically once you've finished the prerequisite (Classical Models).

Why a committee wins — the errors cancel

Before any algorithm, let's establish why this works at all, because it's not obvious. How can a pile of mediocre models beat a good one?

Three coin-flippy voters, by hand

Imagine three classifiers. Each one is right 70% of the time — barely better than guessing. Crucially, assume they make their mistakes independently (when one is wrong, that tells you nothing about the others).

We take the majority vote: whatever 2 out of 3 say, we go with. When is the majority right?

All three right: $0.7 \times 0.7 \times 0.7 = 0.343$
Exactly two right (three ways this can happen — which one is wrong): $3 \times 0.7 \times 0.7 \times 0.3 = 0.441$
Total: $0.343 + 0.441 = \mathbf{0.784}$

The committee is right 78.4% of the time. Three 70% models made a 78% model, and not one of them got any smarter.

Push it further with the same arithmetic and 15 such voters reach 95.0%. From members who are barely better than a coin flip.

Try to recall

Three models each 70% accurate combine into a 78% majority vote. What single assumption did that entire calculation rest on?

Hint: Reread the word in bold in the first paragraph of the example.

See the cancellation happen

Run this. It simulates a committee of guessers and measures how far off their average is as the committee grows:

Python · runs in your browser

What this does: Simulates a crowd guessing a number, where each guesser is off by a random amount. It measures how wrong the crowd's AVERAGE is as the crowd grows from 1 to 100 members, and compares that to the theoretical prediction of spread divided by the square root of M. Watch the error fall by half every time the committee quadruples — that is error cancellation, measured.

The chart below is that formula, plotted. Each curve is the variance of the ensemble as members are added, for three different levels of member-to-member correlation:

Why decorrelating the members is everything— interactive, drag & zoom

Loading chart…

Variance of the ensemble (relative to a single member) as members are added, from the formula rho + (1-rho)/M. Uncorrelated members drive variance to zero. At rho = 0.5 — roughly where plain bagged trees sit — half the variance never goes away no matter how many trees you add. Random forests exist to drag that flat line down.

You build an ensemble of 500 models, but every model is trained identically on the same data with the same algorithm. What do you gain over a single model?

Bagging — squeezing many datasets out of one

So members must differ. But you only have one dataset. Train the same algorithm on it twice and you get the same model twice.

The fix is a beautifully cheeky statistical trick: fake having more datasets by resampling the one you have.

stands for bootstrap aggregating, and it is exactly those two steps — bootstrap, then aggregate.

A is the engine. You draw $N$ rows from your $N$ -row dataset with replacement: after picking a row, you put it back so it can be picked again.

Think of it like drawing names from a hat and putting each one back:

Five names in a hat: A, B, C, D, E. Draw one, write it down, drop it back in, and repeat five times. You might get C C D E A — C twice, and B never showed up at all. Do this over and over and each round gives you a slightly different five-person "dataset" built from the same five people. That is a bootstrap sample.

Bootstrapping a five-row dataset by hand

Dataset: rows A B C D E. Draw 5 times with replacement:

Bootstrap sample	Rows drawn	Left out (out-of-bag)
1	C, C, D, E, A	B
2	A, E, E, B, B	C, D
3	E, C, B, E, B	A, D

Two things to notice, and both are load-bearing:

Every sample is different. Train a decision tree on each and you get three genuinely different trees — different splits, different mistakes. That is the decorrelation we needed.
Every sample leaves some rows out. Sample 1 never saw row B. So we can test that tree on B for free — B is honest held-out data for that particular tree. Those left-out rows are called (OOB) rows.

Python · runs in your browser

What this does: Draws bootstrap samples from a tiny five-row dataset so you can see rows repeat and other rows get left out, then measures — for datasets of 5, 100, and 1000 rows — what fraction of rows actually make it into a bootstrap sample. It converges to the famous 63.2%, matching the formula printed beside it.

Try to recall

Bagging averages many models trained on resampled data. Does it mainly reduce bias or variance — and what does that tell you about which base learner to choose?

Hint: Think about what averaging does and does not change about a systematic error.

Random forests — forcing the trees to disagree

Bagging gets us most of the way. But it has the exact weakness the correlation formula predicted.

Here's the problem. Suppose one feature is a really strong predictor. Then every bagged tree, no matter which bootstrap sample it got, will pick that feature for its first split — it's just too good to pass up. The trees end up looking like near-copies of each other. Correlation $\rho$ stays high, the variance floor stays high, and adding trees stops helping.

The fix is almost rude

At every single split, don't let the tree look at all the features. Show it only a small random handful and make it choose the best split from those. Sometimes the dominant feature simply isn't on the menu, and the tree is forced to find a different — and genuinely different — way to be useful.

That deliberate handicap is the entire idea of a .

Think of it like a medical panel where each doctor sees a different subset of the test results:

Give every doctor the full chart and they'll all fixate on the one alarming number and reach the same conclusion — you've consulted five people and gotten one opinion. Deliberately give each doctor a different partial view and they're forced to reason from different evidence. Individually each is now slightly worse informed. Collectively the panel is far better, because it finally contains real disagreement to average over.

The counterintuitive bit worth sitting with

You are making each individual tree worse on purpose. A tree restricted to a random subset of features is a weaker tree than one allowed to see everything. You accept that loss because it buys a bigger win: the trees stop agreeing, $\rho$ drops, and the ensemble average improves. Trading individual quality for collective diversity is the core bargain of ensembling.

Let's watch the whole progression — one tree, then bagging, then a random forest — built from scratch:

Python · runs in your browser

What this does: Builds a tiny decision-tree learner from scratch, then compares three things on the same noisy 2-D problem: one deep tree on its own, 40 bagged trees, and 40 random-forest trees that may only look at one feature per split. Accuracy climbs at each step. The n_feat argument is the only difference between bagging and the forest.

import numpy as np
rng = np.random.default_rng(0)

# --- a toy 2-D problem: the label is "inside the ring", with 12% of labels flipped (noise)
def make(n, noise=0.12):
    X = rng.uniform(-3, 3, size=(n, 2))
    y = (X[:, 0]**2 + X[:, 1]**2 < 4.0).astype(float)
    flip = rng.random(n) < noise
    return X, np.where(flip, 1 - y, y)

def clean(n):                      # a clean, un-flipped test set = the real truth
    X = rng.uniform(-3, 3, size=(n, 2))
    return X, (X[:, 0]**2 + X[:, 1]**2 < 4.0).astype(float)

# --- a tiny CART tree. n_feat = how many features it may consider per split.
def grow(X, y, depth, n_feat):
    if depth == 0 or len(y) < 6 or y.std() == 0:
        return ("leaf", y.mean())
    best = None
    for f in rng.choice(X.shape[1], size=n_feat, replace=False):   # <- the forest's twist
        for thr in np.quantile(X[:, f], [0.2, 0.35, 0.5, 0.65, 0.8]):
            m = X[:, f] <= thr
            if m.sum() < 3 or (~m).sum() < 3:
                continue
            sse = y[m].var() * m.sum() + y[~m].var() * (~m).sum()  # split quality
            if best is None or sse < best[0]:
                best = (sse, f, thr)
    if best is None:
        return ("leaf", y.mean())
    _, f, thr = best
    m = X[:, f] <= thr
    return ("node", f, thr, grow(X[m], y[m], depth-1, n_feat), grow(X[~m], y[~m], depth-1, n_feat))

def predict(tree, X):
    out = np.empty(len(X))
    for i, x in enumerate(X):
        t = tree
        while t[0] == "node":
            t = t[3] if x[t[1]] <= t[2] else t[4]
        out[i] = t[1]
    return out

Xtr, ytr = make(300)
Xte, yte = clean(3000)
acc = lambda p: ((p > 0.5) == (yte > 0.5)).mean()

print(f"one deep tree           : {acc(predict(grow(Xtr, ytr, 8, 2), Xte)):.3f}")
for name, n_feat in (("bagging  (both feats)", 2), ("random forest (1 feat)", 1)):
    preds = np.zeros(len(Xte))
    for _ in range(40):
        idx = rng.integers(0, len(Xtr), size=len(Xtr))     # bootstrap sample
        preds += predict(grow(Xtr[idx], ytr[idx], 8, n_feat), Xte)
    print(f"{name} : {acc(preds / 40):.3f}")

Reading that result honestly

The jump from a single tree to bagging is large; the extra step from bagging to the forest is small here. That's expected and worth understanding: this toy problem has only two features, so "pick 1 of 2 at random" is about as much decorrelation as is available. On a real dataset with 50 or 500 columns, restricting each split to $\sqrt{p}$ features is a far more aggressive intervention, and the gap between bagging and random forests widens accordingly.

Why does a random forest restrict each split to a random subset of features, instead of always letting trees use the best one?

Boosting — a team where each member fixes the last one's mistakes

Bagging and random forests build their members in parallel: every tree is trained independently and none of them knows the others exist. It works, but there's something a little wasteful about it — tree number 40 repeats a lot of work that trees 1 through 39 already did.

Boosting asks the obvious follow-up question. What if each new model were told what the previous ones got wrong, and asked to focus only on that?

Fix the leftovers, over and over

Start with a prediction so simple it's almost a joke — say, "predict the average, always." Now look at what's left over: for each example, how far off were you? Train a small model to predict those leftovers. Add a fraction of it to your prediction. Look at the new, smaller leftovers. Train another small model on those. Repeat a few hundred times. Each new model is a tiny, targeted patch on the errors that survived every patch before it.

Those leftovers have a name: the . And the tiny models are usually or trees just a few levels deep.

Think of it like a student redoing only the problems they got wrong:

Blindly re-reading the whole textbook (bagging) helps a bit. Marking exactly which problems you missed, drilling only those, then re-testing and drilling whatever you still miss — that's boosting. Each pass is small and cheap because it only targets what's still broken.

Three boosting rounds, with real arithmetic

Four data points. Feature $x = 1, 2, 3, 4$ and target $y = 2, 4, 10, 12$ .

Round 0 — the dumbest possible start. Predict the overall mean for everyone: $(2+4+10+12)/4 = 7$ .

$x$	1	2	3	4
target $y$	2	4	10	12
prediction	7	7	7	7
residual	−5	−3	+3	+5

Round 1 — fit a stump to the residuals. The best single split is between $x=2$ and $x=3$ . Left side residuals average $(-5 + -3)/2 = -4$ ; right side average $(3+5)/2 = +4$ .

Now the crucial move: we do not add the whole correction. We multiply it by a small factor — here $0.5$ :

Left: $7 + 0.5 \times (-4) = 5$
Right: $7 + 0.5 \times (+4) = 9$

New residuals: $-3, -1, +1, +3$ . Every one is smaller than before.

Round 2 — same procedure on the new residuals. Left average $-2$ , right average $+2$ :

Left: $5 + 0.5 \times (-2) = 4$
Right: $9 + 0.5 \times (+2) = 10$

Round 3: left $\to 3.5$ , right $\to 10.5$ .

Track the left-hand prediction across rounds: $7 \to 5 \to 4 \to 3.5$ , closing in on the true left-group mean of $3$ . The right-hand one runs $7 \to 9 \to 10 \to 10.5$ , closing in on $11$ . Each round halves the remaining error rather than jumping straight to the answer — that is what shrinkage buys you. Small, cautious steps generalize far better than one confident leap.

Boosting is doing something you have seen before, wearing an unfamiliar costume. Each round computes an error signal, then takes a small step to reduce it — exactly the loop in the visualization below. Only here the "step" adds a whole tree instead of nudging a number:

Gradient Descent— interactive, try itOpen in lab →

Click anywhere to drop a new starting point.

Learning rate0.15

Momentum0.70

Brighter = higher loss. Watch how a high learning rate overshoots, and how momentum powers through the small bumps toward a minimum.

Gradient descent: measure the slope of the error, take a small step downhill, repeat. Boosting is this same loop, with one difference — instead of stepping in parameter space, each step adds a small tree, and the learning rate is the shrinkage ν. Turn the learning rate up here and watch it overshoot; boosting with too large a ν misbehaves in exactly the same way.

Here is the algorithm assembled, chasing a sine curve with nothing but one-split stumps:

Python · runs in your browser

What this does: Gradient boosting written from scratch. It repeatedly finds the single best yes/no split on the CURRENT residual and adds 30% of that correction to the running prediction. The printed error drops from 0.2945 to 0.0036 over 50 rounds, and the plot shows the staircase of stumps converging onto a smooth sine curve.

import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(0, 6, 24)
y = np.sin(x)

def best_stump(x, r):
    """The single yes/no split that best explains the residuals r."""
    best = None
    for thr in (x[:-1] + x[1:]) / 2:
        left = x <= thr
        mL, mR = r[left].mean(), r[~left].mean()
        sse = ((r[left] - mL)**2).sum() + ((r[~left] - mR)**2).sum()
        if best is None or sse < best[0]:
            best = (sse, thr, mL, mR)
    return best[1], best[2], best[3]

lr = 0.3                      # shrinkage: only take 30% of each correction
F = np.zeros_like(y)          # the ensemble's prediction, starts at zero
snaps = {}
for m in range(1, 51):
    residual = y - F                              # what we still get wrong
    thr, mL, mR = best_stump(x, residual)         # fit a stump TO THE RESIDUAL
    F = F + lr * np.where(x <= thr, mL, mR)       # add a small piece of it
    if m in (1, 5, 50):
        snaps[m] = F.copy()
        print(f"after {m:>2} rounds  mean squared error = {((y - F)**2).mean():.4f}")

plt.figure(figsize=(5.5, 3.2))
plt.plot(x, y, "k--", lw=2, label="target: sin(x)")
for m, col in zip((1, 5, 50), ("#7c9fe0", "#e0a458", "#d97757")):
    plt.step(x, snaps[m], where="mid", color=col, label=f"{m} stumps")
plt.legend(fontsize=8); plt.title("Boosting: each stump fixes what's left over")
plt.tight_layout()

The chart below is that cell's output, plotted interactively — drag and zoom to see how a pile of crude step functions becomes a smooth curve:

Fifty one-split stumps, added together— interactive, drag & zoom

Loading chart…

The exact output of the code cell above. One stump is a single step and captures almost nothing. Five stumps sketch the shape. Fifty stumps trace the sine closely — mean squared error falls 0.2945 to 0.0832 to 0.0036. No single member is more than one yes/no question.

Try to recall

In boosting, what is each new tree actually trained to predict — and why is that different from bagging?

Hint: Look at what goes into the tree as its target, not its input.

Boosting reduces error by fitting each new model to the residuals. Which error component does it primarily attack, and how does that differ from bagging?

Bagging vs boosting — the comparison worth memorizing

	Bagging / Random Forest	Boosting
How members are built	In parallel, independently	Sequentially, each on the last one's errors
What each member is trained on	A bootstrap sample, original targets	The full data, the current residuals
Base learner	Deep trees (low bias, high variance)	Shallow trees / stumps (high bias, low variance)
Mainly reduces	Variance	Bias
More members → overfitting?	Essentially no	Yes — must be tuned or early-stopped
Sensitive to noisy labels	Fairly robust	Sensitive — it will chase the noise
Parallelizable	Trivially	Only within a single tree
Typical accuracy on tabular data	Very good	Usually the best
Tuning effort	Almost none	Real, but worth it

The single most important row in that table

Boosting can and will overfit if you keep going. Every round is another correction fitted to whatever error remains — and eventually the only error remaining is random noise in your training labels, which boosting will happily memorize. Bagging has no such failure mode. In practice this means you must hold out a validation set and use early stopping: keep adding trees only while validation error is still improving. You'll see this happen in the lab at the bottom of this page.

XGBoost and the modern boosters

Gradient boosting as described above is a clean idea, but a naive implementation is slow and overfits readily. is the engineering effort that turned it into the tool people actually deploy.

Two families of improvement, and the distinction matters:

Better statistics. An explicit penalty on tree complexity, a second-order approximation of the loss, column subsampling borrowed from random forests, and a principled rule for handling missing values.

Better engineering. Cache-aware data layout, compressed column blocks, out-of-core computation for data larger than RAM, and split-finding parallelized across cores. This is why it's fast, and speed is a real feature — it lets you tune properly.

The knobs that actually matter, roughly in order

n_estimators + learning_rate — the core trade-off. Low rate (0.01–0.1) with many rounds, chosen by early stopping, is the standard recipe.
max_depth — usually 3–8. Deeper is not better here; boosting wants weak learners.
subsample and colsample_bytree — around 0.5–0.9. Row and column sampling, borrowed straight from bagging and random forests.
min_child_weight, reg_lambda, gamma — reach for these when you're still overfitting after the above.

Here's what all of this looks like in real library code — the version you'd actually write at work:

Python · needs a GPU — run on Colab

import xgboost as xgb
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

X_tr, X_val, y_tr, y_val = train_test_split(X, y, test_size=0.2, random_state=0)

# Random forest: almost no tuning needed, a strong baseline in one line
rf = RandomForestClassifier(n_estimators=500, max_features="sqrt", n_jobs=-1)
rf.fit(X_tr, y_tr)
print("RF  :", rf.score(X_val, y_val), " OOB-style feature importances:", rf.feature_importances_[:5])

# Gradient boosting: low learning rate, many rounds, stopped early by the validation set
clf = xgb.XGBClassifier(
    n_estimators=2000,        # an upper bound — early stopping picks the real number
    learning_rate=0.05,       # the shrinkage ν from the additive-model formula
    max_depth=4,              # shallow, weak learners on purpose
    subsample=0.8,            # row sampling  (bagging's idea)
    colsample_bytree=0.8,     # column sampling (the random forest's idea)
    early_stopping_rounds=50, # stop once validation error stops improving
)
clf.fit(X_tr, y_tr, eval_set=[(X_val, y_val)], verbose=False)
print("XGB :", clf.score(X_val, y_val), " trees actually used:", clf.best_iteration + 1)

LightGBM and CatBoost

Two siblings you'll meet constantly. LightGBM grows trees leaf-wise rather than level-wise and bins features into histograms — usually the fastest option on large datasets. CatBoost handles categorical features natively via ordered target statistics, so you can skip most one-hot encoding, and it tends to need the least tuning. All three implement the same core algorithm from this page; pick on data shape and convenience, not on theory.

When ensembles win — and when they don't

Reach for boosted trees when

The data is tabular — rows and columns, mixed numeric and categorical.
Features are heterogeneous and on wildly different scales (trees don't care about scaling at all).
The dataset is small-to-medium: thousands to low millions of rows.
You need decent results fast, without architecture design or a GPU.
Feature importances or SHAP-style explanations are part of the deliverable.

Reach for something else when

The input is images, audio, or raw text — structure that convolutions or attention exploit and trees cannot.
You need to extrapolate beyond the range of the training data; trees output leaf averages and are flat outside what they've seen.
The signal is a smooth linear trend — a linear model will be simpler, faster, and better.
Latency is brutal and you cannot afford to evaluate hundreds of trees per prediction.

Explain it yourself

Explain to a friend why a crowd of mediocre models beats one good model, and then explain the difference between bagging and boosting using the jellybean-jar and the student-redoing-wrong-problems pictures — no formulas. If you can't say why a random forest deliberately handicaps each tree, that's the section to reread.

Recap — the key ideas

An ensemble combines many models; it works only because their mistakes are independent enough to cancel. Three 70% voters make a 78% committee; fifteen make a 95% one.
Averaging $M$ members cuts variance to $\sigma^2/M$ — but only if they're uncorrelated. With correlation $\rho$ a floor of $\rho\sigma^2$ never goes away, no matter how many members you add.
Bagging manufactures diversity by training each member on a bootstrap sample (drawn with replacement, so each holds about 63% of the rows and leaves 37% out-of-bag as free validation data). It reduces variance, so bag deep, high-variance trees.
Random forests = bagging + restricting each split to a random subset of features. Each tree gets individually worse; $\rho$ drops; the ensemble gets better.
Boosting builds members sequentially, each fitted to the residuals of those before it, scaled by a shrinkage factor $\nu$ . Fitting residuals is gradient descent in function space — which is what makes it work for any differentiable loss.
Bagging reduces variance and won't overfit as members are added. Boosting reduces bias and will overfit — so use a validation set and early stopping.
XGBoost / LightGBM / CatBoost add explicit regularization, second-order optimization, and serious engineering. On tabular data they remain the strongest default in 2026.

Practice — and how to make it stick

Learn it the way that actually works

• Retrieval practice: before scrolling back, try to state from memory what bagging randomizes, what random forests additionally randomize, and what boosting fits each new tree to. Struggling to recall beats re-reading.
• Spaced repetition: mark this topic complete to add it to your Review queue, resurfacing right before you'd forget.
• Interleaving: mix these with problems from Classical Models and Evaluation & Metrics rather than doing them in a block — a forest is only as good as the validation protocol you judge it with.

Start with the lab. It demonstrates the one boosting failure mode you must be able to recognize on sight:

Practice labPython · runs in your browser · Ada can see your terminal

Your task: Watch boosting overfit, then stop it in time. Run the starter as-is: train error falls forever, but test error bottoms out around round 29 and then climbs — the ensemble has started memorising the noise in the training labels. Then do the TODOs. (1) Set LR = 0.05 and rerun: which round is best now, and is the best test error better or worse? (2) Set LR = 1.0: what happens, and why? (3) Print how many rounds you would keep if you early-stopped at the best test error.

editor

import numpy as np
import matplotlib.pyplot as plt
rng = np.random.default_rng(0)

x_tr = np.linspace(0, 6, 40)
y_tr = np.sin(x_tr) + rng.normal(0, 0.45, size=40)   # noisy training labels
x_te = np.linspace(0, 6, 200)
y_te = np.sin(x_te)                                  # the clean truth

def best_stump(x, r):
    best = None
    for thr in (x[:-1] + x[1:]) / 2:
        left = x <= thr
        mL, mR = r[left].mean(), r[~left].mean()
        sse = ((r[left] - mL)**2).sum() + ((r[~left] - mR)**2).sum()
        if best is None or sse < best[0]:
            best = (sse, thr, mL, mR)
    return best[1], best[2], best[3]

LR, ROUNDS = 0.3, 300          # TODO: try LR = 0.05, then LR = 1.0
F_tr, F_te = np.zeros_like(x_tr), np.zeros_like(x_te)
tr_err, te_err = [], []
for m in range(ROUNDS):
    thr, mL, mR = best_stump(x_tr, y_tr - F_tr)      # fit the residual
    F_tr += LR * np.where(x_tr <= thr, mL, mR)
    F_te += LR * np.where(x_te <= thr, mL, mR)
    tr_err.append(((y_tr - F_tr)**2).mean())
    te_err.append(((y_te - F_te)**2).mean())

best_round = int(np.argmin(te_err)) + 1
print(f"train error keeps falling : {tr_err[0]:.3f} -> {tr_err[-1]:.3f}")
print(f"test  error bottoms out at round {best_round} (={min(te_err):.3f}), then rises to {te_err[-1]:.3f}")

plt.figure(figsize=(5.5, 3.2))
plt.plot(tr_err, label="train error")
plt.plot(te_err, label="test error")
plt.axvline(best_round - 1, ls=":", color="#888")
plt.xlabel("boosting rounds"); plt.ylabel("mean squared error")
plt.legend(fontsize=8); plt.tight_layout()

terminal

Press Run (⌘/Ctrl+Enter) to execute.

Ask Ada — she can read your terminal

Stuck, or hit an error? Ask Ada on the right — she can see your code and the terminal output.

Then work through these:

By hand: four points with targets $y = 1, 3, 8, 10$ . Compute the round-0 mean prediction, the residuals, the best single split, and the round-1 predictions with shrinkage $\nu = 0.5$ . Check that every residual shrank.
From scratch: extend the random-forest cell to record the ensemble's accuracy after each tree is added, and plot accuracy versus number of trees. Confirm it flattens out and never turns upward — the property boosting does not have.
Measure the correlation directly: train 30 bagged trees and 30 random-forest trees, collect each tree's predictions on the test set, and compute the average pairwise correlation of their errors. You should find the forest's $\rho$ is meaningfully lower — the mechanism from the variance formula, observed rather than asserted.
On real data: take any tabular dataset, and race a tuned random forest against a tuned XGBoost model with early stopping. Log how much tuning effort each needed to get within 1% of its best score — that effort gap is as important a finding as the accuracy gap.
Read the source: skim §15.2 of The Elements of Statistical Learning and find the correlation formula from this page in its original form.

Next: learn to tell whether any of these models is actually good, in Evaluation & Metrics.