Feature Engineering
Turning raw data into features models can learn from — encoding, scaling, and leakage. Taught from zero, then built up to the honest fit-on-train-only workflow that keeps your numbers real.
Start here — what this is really about
A model cannot see the world. It cannot see a house, a customer, or a Tuesday. It only ever sees a table of numbers, and it learns whatever patterns happen to live in that table.
So somebody has to decide what goes in the table. Somebody has to look at a messy pile of reality — a street address, a timestamp, the word "blue" — and turn it into columns of numbers that carry the useful information and leave out the noise.
That job is feature engineering, and it is the step where most real projects are won or lost. Not the model choice. Not the learning rate. The table.
A model can only learn from what you put in front of it. Feature engineering is the craft of putting the right things in front of it — and, just as importantly, of not accidentally handing it the answer.
Your friend has to guess the price of a house they cannot see. If you say "it's nice," they have nothing to work with. If you say "110 square metres, three bedrooms, built 1998, ten minutes from the station," they can make a real estimate. Same house, same reality — but the second description is engineered to carry price-relevant information. A model is that friend, and your feature columns are everything you're allowed to say.
This page starts from zero and assumes nothing beyond the idea that a model learns from data. Flip the Depth switch at the top to reveal the formal notation, derivations, and edge cases — and it opens automatically once you've completed the prerequisite, Machine Learning. Nothing is hidden for good; the deeper material sits behind "Go deeper" panels so you can open it the moment you're curious.
What a feature actually is
Picture a spreadsheet. One row per thing you care about, one column per fact about that thing.
A is one of those columns. One row — all the features for a single example — is that example's . The whole table stacked together is the , and the thing you're trying to predict is the .
Here is one raw house listing, exactly as a database might hand it to you:
address: 12 Rue Lafayette, Lyon
listed: 2024-03-19 08:41
size: 110 m²
bedrooms: 3
heating: gas
sold_for: €310,000
A model cannot consume any of that directly. So we make decisions, one field at a time:
- size → 110.0 — already a number. Keep it.
- bedrooms → 3.0 — already a number. Keep it.
- listed → 2024, month 3, and hour 8 — a timestamp is not a number a model can use, but the year, the month, and the hour of day are. Three columns out of one field.
- heating: gas → 1, 0, 0 — the word "gas" becomes three 0/1 columns, one per possible fuel (gas, electric, oil). We'll see why in a moment.
- address → distance to city centre: 2.4 — the street name itself is useless, but the location behind it is gold. This is the step that needs a human who knows the problem.
- sold_for → 310000.0 — this is not a feature at all. It is the target, the thing we predict.
The finished row: [110.0, 3.0, 2024.0, 3.0, 8.0, 1.0, 0.0, 0.0, 2.4] with target 310000.0.
Notice how much of that was judgement, not arithmetic. Nothing forced us to extract "hour" or "distance to centre" — we chose to, because we believe they matter.
The address field was thrown away, but a new column was created from it. What does that tell you about feature engineering?
Hint: Think about what the model can and cannot use.
Scaling — putting every feature on the same ruler
Here's a problem that bites everyone once. Suppose your table has two columns:
- size in square metres, running from about 40 to 250.
- price in euros, running from about 100,000 to 900,000.
Both are perfectly sensible numbers. But one of them is roughly four thousand times bigger than the other, and a great many models take that literally.
Lots of algorithms measure how far apart two examples are by adding up the differences across all columns. If one column is measured in units that produce huge numbers, its differences swamp everything else in the sum. The model then behaves as if the other columns barely exist — not because they don't matter, but because their numbers are small.
A recipe calls for 2 of flour and 3 of salt. Two tonnes of flour and three grams of salt is a very different dish from two grams and three tonnes. The numbers 2 and 3 are meaningless until you fix the units. A model reading unscaled columns is stuck with exactly that ambiguity — and it resolves it badly, by simply following the biggest numbers.
Five houses, described by size (m²) and price (€). We want the house most similar to H1 — a modest 100 m², €300,000 flat.
| House | Size (m²) | Price (€) |
|---|---|---|
| H1 (our query) | 100 | 300,000 |
| H2 | 200 | 302,000 |
| H3 | 105 | 350,000 |
| H4 | 150 | 260,000 |
| H5 | 95 | 400,000 |
Measure similarity the obvious way — straight-line distance across both columns:
- H1 to H2: size differs by 100, price by 2,000. Distance = .
- H1 to H3: size differs by 5, price by 50,000. Distance = .
So the algorithm declares H2 — a 200 m² house, twice the size — to be H1's nearest neighbour, and considers the near-identical 105 m² H3 to be twenty-five times further away. The size column contributed essentially nothing: is a rounding error next to .
The fix is not a better algorithm. It's better columns.
The fix is . The standard move is to convert each column into "how many standard deviations above or below its own average is this value" — a . Do that and every column arrives centred on 0 with a spread of about 1, so they all get an equal vote.
You train a k-nearest-neighbours classifier on a table where one column is annual income in euros and another is number of children. Without scaling, what happens?
Categorical features — turning words into numbers
Half your columns probably aren't numbers at all. heating = gas. country = Portugal. plan = premium. These are , and they need converting before any model can touch them.
The naïve conversion is to number the categories: blue → 0, green → 1, red → 2. This is called , and for most models it quietly corrupts your data.
The moment you write red = 2 and blue = 0, you have told a linear model three things you never meant: that green sits between blue and red, that red is twice green, and that the gap from blue to green equals the gap from green to red. None of that is true about colours. The model has no way to know you didn't mean it — so it fits those invented relationships.
The player wearing 10 is not twice the player wearing 5, and the player wearing 7 is not "between" them in any meaningful sense. The numbers are name tags, not quantities. Label encoding hands a model name tags and lets it do arithmetic on them.
The honest conversion is : one new column per category, with a single 1 marking which one this row is.
Five rows with a single colour column: red, blue, green, blue, red.
Step 1 — list the distinct categories, in some fixed order: blue, green, red. That's three categories, so we will create three columns.
Step 2 — give each row a 1 in its own column and 0 in the others:
| Row | colour | is_blue | is_green | is_red |
|---|---|---|---|---|
| 1 | red | 0 | 0 | 1 |
| 2 | blue | 1 | 0 | 0 |
| 3 | green | 0 | 1 | 0 |
| 4 | blue | 1 | 0 | 0 |
| 5 | red | 0 | 0 | 1 |
Step 3 — sanity-check: every row sums to exactly 1, because every row is exactly one colour.
Now no ordering is implied. Blue is not less than red; they are simply different columns, and the model learns a separate weight for each — which is precisely the freedom we wanted.
Why is one-hot encoding safe for a linear model when label encoding is not?
Hint: Think about what a linear model does with a single numeric column.
Making new features — where domain knowledge pays
Everything so far was translation: getting existing information into a usable shape. This section is creation — building columns that were not in the data at all, because you know something about the problem that the model does not.
A model can in principle learn that price-per-square-metre matters, by discovering the relationship between the price column and the size column. But that takes data and capacity. Handing it a price_per_m2 column directly means it starts from the answer instead of having to derive it. Every good engineered feature is a shortcut you already know and are gifting to the model.
You could hand someone whole onions and carrots and a knife. They'll get there. But if you already know the recipe, handing them a bowl of neatly diced vegetables gets a better dinner, faster, with fewer accidents. Engineered features are prepped ingredients.
Three moves cover most of what you'll actually do:
1. Combine columns. Ratios and differences often carry the real signal: price / size, debt / income, days_since_last_purchase. Two columns the model has to relate for itself, collapsed into one it can use immediately.
2. Unpack the structured fields. A timestamp is not one feature, it is many: hour of day, day of week, month, is-it-a-holiday. A postcode hides latitude, longitude, population density, and distance to the centre.
3. Fix skewed distributions. Money, populations, and view counts are almost always right-skewed — a mass of small values with a long thin tail of enormous ones. That tail dominates squared-error losses and distance calculations. Taking a logarithm pulls it back in.
The log transform. Take five house prices: €120k, €240k, €310k, €480k, €1,600k. On the raw scale the top house sits €1,120,000 away from its nearest neighbour — a squared-error loss will obsess over that single mansion and effectively ignore everything else. After log1p, the same gap becomes 1.2, comparable to the other gaps. The model can now care about all five houses at once.
Cyclical time. Encode "hour of day" as the plain number 0–23 and you have told the model that 23:00 and 00:00 are 23 units apart — as distant as any two hours can be — when in reality they are one hour apart. The fix is to place each hour on a circle:
- Convert the hour to an angle around a full circle: 23:00 becomes , or 345°.
- Store the point's two coordinates on that circle, and of the angle, as two columns.
- Now 23:00 sits at (−0.259, 0.966) and 00:00 at (0.0, 1.0) — a distance of 0.26 apart, while 12:00 lands on the far side of the circle. Midnight and 11pm are neighbours again, exactly as they should be.
You encode the month as a single column with values 1 to 12. What has this told the model that is not true?
Leakage — the bug that makes your model look brilliant, then fail
This is the most important section on this page. Everything above affects how well your model works; this affects whether you can believe your own numbers at all.
is the introduction of information about the target that should not legitimately be available to learn from. It does not announce itself with an error. It announces itself with excellent results — which is exactly why it survives so long.
If a model's accuracy is far better than you or any domain expert expected, your first hypothesis should not be "great model." It should be "where is the leak?" Suspiciously good results are a bug report, not a celebration.
You study for an exam with the marked answer key beside you and score 98% on the practice paper. You feel ready. Then you sit the real exam without the key and score 51%. Nothing was wrong with your studying method — it was measured under conditions that will never exist again. A leaky model has been studying with the answer key open, and its validation score is the practice paper.
Leaks come in two families, and they need different fixes.
Family 1: target leakage — a feature that already knows the answer
A column contains information that only exists because of the outcome, or only became available after it.
A team builds a model to predict which patients will develop pneumonia. Cross-validated accuracy: 97%. Extraordinary.
Then someone checks the columns. One of them is antibiotic_prescribed.
Of course it is 97% accurate. Antibiotics are prescribed because a doctor already diagnosed pneumonia. The feature is not predicting the outcome; it is a downstream consequence of it, recorded in the same row.
The killer question is always the same: at the moment I would actually use this model, would this value exist? For a patient walking into A&E, unwell but undiagnosed, antibiotic_prescribed is empty. The feature is worthless in production and it was worth 97% in the test — which is precisely the shape of a leak.
Other members of the same family, all real:
account_closed_datein a churn model — only ever filled in for customers who churned.number_of_customer_service_calls_this_monthin a model that predicts a problem at the start of the month.- Any average, count, or ranking computed over the whole dataset, including rows from the future.
Family 2: train–test contamination — the preprocessing did the leaking
This one is nastier, because every column is legitimate and the leak lives in your code.
You standardize your features. To do that you need each column's mean and standard deviation. If you compute them over the whole dataset before splitting into train and test, then those means carry information from the test rows — and your model has been tuned, however faintly, on data it was supposed to have never seen.
Treat the test set as data that has not happened. Every number your pipeline computes — means, standard deviations, category vocabularies, min and max, target encodings, which features to keep — must be computed from the training rows alone, then simply applied to the test rows. That one discipline is the whole of this section.
The scariest version is feature selection. Suppose you have thousands of candidate features, and you keep the ones most correlated with the target — measured on all your data — and then cross-validate. This looks careful. It is catastrophically wrong, and it is famous enough to have its own section in The Elements of Statistical Learning.
Here is how badly it fails. Below, every feature is pure random noise and every label is a coin flip. There is genuinely nothing to learn — the honest answer is 50%.
In that experiment the features were random noise and the labels were coin flips. So where did the 83% come from?
Hint: Ask what the selection step saw that the test folds were supposed to hide.
You standardize your features using the mean and standard deviation of the entire dataset, then split into train and test. What is the consequence?
The workflow that keeps you honest
Put together, the whole discipline is an ordering:
| Order | Step | The rule |
|---|---|---|
| 1 | Split first | Before anything else. By time or by group if either applies. |
| 2 | Explore the training set only | Look at distributions, spot the skew, find the outliers — on train. |
| 3 | Fit the transforms on train | Means, vocabularies, fill values, target encodings — all from train. |
| 4 | Apply them to both | Same frozen parameters for train and test. Never re-fit on test. |
| 5 | Select features inside the loop | Selection is a fitted step too. It belongs inside cross-validation. |
| 6 | Touch the test set once | At the very end, to report a number. Not to make decisions. |
The practical way to enforce steps 3 to 5 is not discipline but tooling: bundle every transform and the model into a single pipeline object, and fit that object. Then the correct thing happens automatically, in every fold, forever.
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
numeric = ["size_m2", "bedrooms", "distance_to_centre"]
categorical = ["heating", "city"]
prep = ColumnTransformer([
("num", Pipeline([("impute", SimpleImputer(strategy="median")),
("scale", StandardScaler())]), numeric),
("cat", OneHotEncoder(handle_unknown="ignore"), categorical),
])
# The whole thing is ONE estimator, so cross_val_score re-fits every
# transform inside each fold. The medians, the means, the standard
# deviations and the category vocabulary are learned from that fold's
# training rows only — the leak from the section above is impossible.
model = Pipeline([("prep", prep), ("clf", LogisticRegression(max_iter=1000))])
scores = cross_val_score(model, X_train, y_train, cv=5, scoring="roc_auc")
print("honest CV AUC:", scores.mean().round(3), "+/-", scores.std().round(3))- Scaling, imputing, or encoding before the train/test split.
- Selecting features on the full dataset, then cross-validating the survivors.
- A random split on time-ordered data, or on data with repeated entities.
- Target encoding a high-cardinality column without smoothing and without holding out.
- Fitting a new encoder on the test set, so a category maps to a different column than it did in training.
- Celebrating a suspiciously high score instead of investigating it.
Explain to a friend, without any formulas, why a model that scores 97% in testing can be worthless — and what the rule fit on train only actually means in practice. If you cannot describe the two different families of leakage, that is the section to reread.
- A model only ever sees a table of numbers. Feature engineering is deciding what goes in that table — and it usually matters more than which model you pick.
- Scaling puts every column on the same ruler, so a column measured in euros cannot drown out one measured in bedrooms. Distance-based and gradient-trained models need it; trees do not.
- Categorical values become numbers via one-hot encoding, which invents no ordering. High-cardinality columns need target encoding, hashing, or learned embeddings instead.
- New features are where domain knowledge pays: ratios, unpacked timestamps, log transforms for skew, and cyclical sin/cos encoding for anything that wraps around.
- Leakage is information in your features or your split that would not exist at prediction time. It shows up as unexpectedly good results, not as an error.
- The one habit that prevents most of it: split first, fit every transform on the training rows only, apply to both — and put it all inside a pipeline so the correct order is the automatic one.
Practice — and how to make it stick
Three research-backed habits, built into this platform:
• Retrieval practice: try to answer each exercise below from memory before scrolling back up — pulling an answer out beats recognizing it on the page.
• Spaced repetition: mark this topic complete and it joins your Review queue, resurfacing right before you would have forgotten it.
• Interleaving: mix these with problems from Evaluation & Metrics and Machine Learning rather than grinding feature engineering alone — messier practice, sturdier memory.
- Hunt a leak. Take any dataset you have and list every column. For each one, ask the killer question: at the moment I would actually use this model, would this value exist? Anything you hesitate on is a suspect.
- Break it deliberately. Fit a scaler on the full dataset, then split and evaluate. Then do it correctly. Measure the gap. Doing the wrong thing on purpose, once, is the fastest way to never do it by accident.
- Feel the cardinality wall. One-hot encode a column with 5,000 categories and look at the resulting matrix shape and memory. Then try target encoding with smoothing and compare.
- Beat a baseline with features alone. Fix the model and the hyperparameters, then improve the score only by adding engineered columns — ratios, date parts, logs. This is the single most transferable ML skill there is.
Try it right here — edit and run the code, and if you get stuck or hit an error, ask Ada on the right: she can see your code and terminal output.
Next, learn how to measure a model honestly once the features are right: Evaluation & Metrics. Or see which models actually need all this scaling in Classical Models.