Software Engineering for ML

Start here — the code that isn't the model

You've written a script. It trains something, it prints a number, it works. Then one of these happens:

• A week later you change one line to try an idea, the number gets worse, and you can't get the old number back.
• A teammate runs your script and gets a different answer. Neither of you knows who is right.
• Your model quietly trains on the test set for three days and nothing crashes, because nothing can crash — the code is doing exactly what you typed.

None of those are machine-learning problems. They're software problems, and they are the ones that actually eat your weeks.

That is not a metaphor someone invented for a lesson: a well-known Google paper on real production ML systems makes exactly this point, with a diagram in which the "ML code" is a small black box surrounded by a sprawl of data collection, configuration, serving, and monitoring infrastructure.Sculley et al. 2015

Software engineering is often defined exactly that way: programming is writing the code; engineering is everything that keeps it alive afterwards.SWE at Google

There are four habits, and they build on each other:

1. Version control — never lose a working state, ever again.
2. Environments — make "it works on my machine" true on every machine.
3. Tests — catch the silent, non-crashing bugs that ML is full of.
4. Structure — arrange code so that changing it later is cheap.

1. Version control — a time machine for your work

Right now, your protection against breaking working code is probably a folder full of train_final.py, train_final2.py, train_final_ACTUALLY.py. That approach fails at the exact moment you need it: you know what the files are, but not why any of them changed, or which one produced the number in your notes.

The tool everyone uses is Git.Pro Git Four words cover most of it:

• A repository (repo) — the project, plus its whole history.
• A commit — one snapshot, with a message.
• A branch — a parallel line of work.
• A merge — folding that work back in.

The picture that makes branching click

A repository's history is not a straight line — it's a graph. Each commit points back at the commit it came from, so a branch is just a chain that split off, and a merge is a commit with two parents.

Feel it — how Git names a snapshot

Git identifies every piece of content by a hash: a short fingerprint computed from the bytes themselves. Change one character anywhere and the fingerprint changes completely. This cell computes a real Git object id — the number it prints is genuinely what git hash-object prints for that file.

2. Environments — making "it works on my machine" true everywhere

Your code doesn't run on Python. It runs on Python plus a specific set of libraries at specific versions, on a specific operating system. Change any of those and the same script can produce a different number, or no number at all.

Dependencies are managed with two ideas:

• A virtual environment — one project, one sandbox. Without it, upgrading a library for project A silently breaks project B.
• Pinning — writing versions down, exactly.

Reading a version constraint

Library versions follow semantic versioning. The statement numpy==1.26.4 decodes as:

• 1 — the MAJOR number. It increases only when a release makes backward-incompatible changes to the public API; code written for major version 1 may simply break on version 2.SemVer 2.0.0
• 26 — the MINOR number. It increases when new features are added in a backward-compatible way. Upgrading should not break you; your existing calls keep working.
• 4 — the PATCH number. It increases for backward-compatible bug fixes only. This is the safest upgrade there is.
• The dots are just separators, and each part is compared as a number, not text — which is why version 1.26 comes after 1.9, even though the string "1.9" looks bigger alphabetically.
• Why it matters for ML: pinning torch==2.3.1 rather than torch is the difference between a paper's code running in two years and it not running at all. Major-version bumps are where defaults change silently — the exact class of change that alters your loss curve without raising an error.

The other half of reproducibility — randomness

Even with a perfect environment, ML code is deliberately random: initial weights, shuffling, dropout, augmentation. Unseeded randomness means two runs of identical code give different numbers — and then you cannot tell whether your change helped or you just got a luckier shuffle.

3. Tests — the safety net that lets you move fast

Here is the thing that makes ML code unusually dangerous: most bugs don't crash. A wrong axis, a leaked statistic, a label that's off by one — the code runs, the loss goes down, and you get a number. It's just the wrong number, and nothing anywhere will tell you.

An assertion is the atom, and a unit test is an assertion with a name and a setup.

The bug you cannot see, and the test that catches it

Data leakage is the classic silent ML bug. Standardizing your features using the mean and standard deviation of the whole dataset feels harmless. It isn't: the test set's statistics have leaked into training, your reported score is optimistic, and nothing errors.

Why a test suite pays for itself — the arithmetic

You might reasonably ask whether all this is worth it on a small project. It is, and you can see why with one line of probability. If each change you make has a probability $p$ of breaking something, then after $n$ changes:

• $p$ — the chance that any single change breaks something. Even a careful engineer is not at zero; call it 2 in 100.
• $1 - p$ — the chance one change is fine. The complement: everything that isn't a break.
• $(1 - p)^n$ — the chance all $n$ changes are fine. The power means "multiply that chance by itself $n$ times," which is valid when the changes are roughly independent. This is the term that collapses: repeatedly multiplying by a number below 1 drives it toward zero fast.
• $1 - (\cdot)$ — flip it back around: "not all fine" is the same as "at least one broke." Computing it this way is far easier than adding up every way one, two, or seventeen changes could break.
• Why it matters: safety per change is not the right thing to look at — safety compounds against you. At $p = 0.02$, a hundred changes gives you an 87% chance that something is broken. The question is never "will I introduce a bug" but "how long until I notice."

4. Types and shapes — making code say what it means

Python will happily let you pass a list where an array was expected, or a (4, 1) array where a (4,) array was expected. NumPy's broadcasting then quietly produces a result of a completely different shape — and the arithmetic works, so you get a number.

Type hints do the same job for ordinary types, and they cost nothing at runtime:

def split(data: np.ndarray, frac: float = 0.8) -> tuple[np.ndarray, np.ndarray]:
    """Split data into (train, test) along the first axis.

    Args:
        data: array of shape (n_examples, ...).
        frac: fraction of examples that go to train, in (0, 1).

    Returns:
        (train, test), together containing every original example exactly once.
    """
    n = int(len(data) * frac)
    return data[:n], data[n:]

5. Structure — from one notebook to a project

Notebooks are excellent for exploring and terrible for keeping. Cells run out of order, state hides in memory, half the logic exists only in a cell someone deleted. The moment a piece of code is worth running twice, it should move into a file.

A layout that scales from a weekend project to a lab:

myproject/
├── pyproject.toml        # dependencies + project metadata
├── requirements.lock     # exact versions, generated — commit this
├── README.md             # what this is, how to run it, in that order
├── .gitignore            # data/, checkpoints/, __pycache__/, .env
├── configs/
│   └── baseline.yaml     # hyperparameters live here, not in the code
├── src/mypkg/
│   ├── data.py           # loading, splitting, preprocessing
│   ├── model.py          # architecture
│   └── train.py          # the training loop + a CLI entry point
├── tests/
│   ├── test_data.py      # the leakage and split tests from above
│   └── test_model.py     # shape tests, overfit-one-batch
└── notebooks/
    └── explore.ipynb     # exploration only; nothing depends on it

Four rules do most of the work:

• Config, not constants. A learning rate buried on line 87 is invisible; in configs/baseline.yaml it's a knob, and the config file gets logged alongside the results so you know what produced them.
• Functions, not scripts. Code inside a function can be imported, tested, and reused. Code at module top-level runs the instant anything imports it.
• Notebooks import, never define. Let explore.ipynb call from mypkg.data import split. The moment a notebook defines something important, that thing exists in exactly one person's kernel.
• A README that starts with the command. The first thing a reader needs is the line that runs it. Everything else is second.

6. Working with others — review, CI, and the loop that runs itself

Everything above is done alone. These last two habits are what make the work shared.

A pull request is a change offered for review rather than pushed straight to main. A code review is what happens there, and its most underrated benefit isn't bug-catching: it's that a second person now knows how that code works.SWE at Google

CI is the automation. On every push, a machine checks out your branch and runs the checks:

on every push:
  1. install the locked dependencies        # environment reproduces?
  2. ruff check .                           # lint: unused imports, dead code, bugs
  3. ruff format --check .                  # format: consistent style, no debate
  4. mypy src/                              # types: contradictions before runtime
  5. pytest -q                              # tests: everything you have claimed

A linter and a formatter round it out. The formatter's real value is social: nobody argues about line length in a review ever again, and diffs contain only real changes. Run both through a pre-commit hook so they fire before the commit rather than after the CI failure.

Practice — and how to make it stick

1. Do it for real, today (20 minutes). Take your messiest existing script. git init, commit it as-is, then make three commits with messages that say why. Add a .gitignore for data and checkpoints. You now have a project you cannot lose.
2. Pin it. Run pip freeze > requirements.lock in that project, commit it, and write a one-line README section showing the command that reproduces your last result.
3. Write the two tests that matter. For any preprocessing function you have: a shape test and a leakage test (the property trick from above — change only the test set, assert the training output doesn't move).
4. Break something on purpose. Introduce a subtle bug — swap an axis, drop a factor of 2 — and check whether any test you own catches it. Whatever slips through tells you exactly which test to write next.
5. Overfit one batch. On any model you have, train on 8 examples until the loss approaches zero. If it can't, you've found a bug — and you found it in a minute instead of a week.

Try it right here — edit and run the code, and if you get stuck or hit an error, ask Ada on the right: she can see your code and terminal output.

Next, put these habits to work on real code: Python & NumPy for the arrays every test above asserts on, then Reproducing Results — which is this page applied to somebody else's paper.