Object Detection

Start here — what object detection really is

Forget the acronyms for a moment. Show a photo to an image classifier and it answers one question: "is there a dog in this picture?" — a single label for the whole image. That's useful, but it's not enough for a self-driving car, which needs to know there is a pedestrian, and they are right there, two meters ahead.

Object detection answers two questions at once, for every object in the picture:

1. What is it? (a class label — dog, car, person)
2. Where is it? (a box drawn tightly around it)

That second question — the where — is the whole new idea. Everything else on this page is machinery for drawing good boxes and then cleaning up the mess when the model draws too many.

The two things a detector outputs for each object

For every object it finds, a detector emits a bounding box plus a class label. Pinning down that box is called localization.

How good is a box? Intersection over Union (IoU)

The moment you predict boxes, you need a way to score them. "Right or wrong" is too crude — a box can be almost perfect, or roughly right, or wildly off. We want one number from 0 to 1 that says how well a predicted box overlaps the true box. That number is IoU.

Now the same idea in symbols. For a predicted box $A$ and a ground-truth box $B$:

• $A$ and $B$ are the two rectangles — the guess and the truth.
• $A \cap B$ is the intersection: the little rectangle where they overlap. The $\cap$ symbol means "the part in both."
• $A \cup B$ is the union: all the area either one covers. The $\cup$ symbol means "the part in either."
• $\lvert \cdot \rvert$ means "the area of" — literally how many pixels that region contains.
• Why it matters: IoU is the yardstick behind almost every detection decision. It decides whether a prediction counts as a hit, and (next section) which duplicate boxes to throw away. Get comfortable with it — it reappears constantly.

Here it is as six lines of code you can run and poke at:

Too many boxes: Non-Maximum Suppression (NMS)

Here is a problem you did not see coming. A detector does not politely emit one box per object — it fires off many overlapping guesses for the same dog, each slightly shifted, each with its own confidence. Left alone, your "dog detector" reports five dogs where there is one.

That hushing procedure is Non-Maximum Suppression, or NMS. It is the reason a well-built detector reports one box per object instead of a pile.

Where do the candidate boxes come from? Anchors and box regression

We have been assuming boxes appear from somewhere. Where? Early detectors ran a classifier at every position and scale — astronomically slow. The trick that made detection practical is the anchor.

Each of those pre-placed reference rectangles is an anchor. The network never outputs raw pixel coordinates; it outputs small offsets that stretch and shift the anchor onto the object. Learning a gentle correction is much more stable than regressing absolute positions.

Two families of detectors

Faster R-CNN, briefly

A Region Proposal Network slides over the CNN's feature map and proposes object regions from anchors. Each proposal is cropped to a fixed size (RoI pooling) and passed to two heads — one for the class, one to refine the box. It made region proposals learnable and end-to-end, and introduced anchors.

YOLO, briefly

YOLO (You Only Look Once) divides the image into a grid; each cell predicts a fixed number of boxes with an objectness score and class scores, all in one shot. Reframing detection as a single regression problem is what made real-time detection practical.

DETR, the modern reframing

DETR removes anchors and NMS entirely by treating detection as set prediction: a Transformer decoder emits a fixed set of object queries, each becoming at most one box, matched to the ground truth by the Hungarian algorithm. Because each query is trained to claim a distinct object, there are no duplicates to suppress. It is the basis for RT-DETR and Grounding DINO.

Scoring a whole detector: precision, recall, and mAP

IoU scores one box. But how good is the detector overall, across a whole test set? Two familiar quantities combine into the standard metric.

A prediction is a true positive if it matches a real object (right class, IoU above threshold), a false positive if it matches nothing, and a false negative is a real object the detector missed. From these:

• $TP$ (true positives) — correct detections. The ones you got right.
• $FP$ (false positives) — boxes that hit nothing. Hallucinated objects.
• $FN$ (false negatives) — real objects you failed to find. Misses.
• Precision reads as "of everything I flagged, what fraction was real?" — it punishes hallucinations.
• Recall reads as "of everything really there, what fraction did I catch?" — it punishes misses.
• Why it matters: these two trade off. Lower the confidence bar and you catch more objects (recall up) but also invent more junk (precision down). A detector is only trustworthy if it holds both high at once.

Sweep the confidence threshold and plot precision against recall — the area under that curve is the Average Precision for a class. Average it over every class (and, for COCO, over the ten IoU thresholds) and you get mAP, mean Average Precision — the number every detection paper reports. See Evaluation & Metrics.

Running a modern detector

In practice you rarely write any of this — you fine-tune a pretrained model. Deep-learning libraries like PyTorch need a real GPU, so this cell opens in Colab rather than running in your browser:

from ultralytics import YOLO

model = YOLO("yolo11n.pt")             # pretrained, tiny
results = model("street.jpg")           # inference — anchors, NMS, all handled

for box in results[0].boxes:
    cls  = model.names[int(box.cls)]
    conf = float(box.conf)
    x1, y1, x2, y2 = box.xyxy[0].tolist()
    print(f"{cls} {conf:.2f} at ({x1:.0f},{y1:.0f})-({x2:.0f},{y2:.0f})")

Best practices

• Start from a pretrained detector and fine-tune on your data — almost never train from scratch.
• Match your anchors and input resolution to the object sizes you actually care about.
• Use strong augmentation (mosaic, mixup) — it is a big lever for detectors specifically.
• Evaluate at multiple IoU thresholds (COCO's mAP@[.5:.95]), not just at 0.5.

Practice — and how to make it stick

Start with this hands-on lab. It runs in your browser — finish the TODO so the near-duplicate box is suppressed, then check your answer against the NMS worked example above.

Then go deeper, roughly in order of difficulty:

1. By hand: two boxes [0,0,10,10] and [5,5,15,15]. Compute their IoU on paper, then verify with the iou function above.
2. From scratch: you already implemented IoU and NMS here — now compute mAP@0.5 yourself on a tiny labeled set and compare to a framework's reported number.
3. Fine-tune: train YOLO on a small 3-class dataset you annotate (e.g. in Roboflow), and compare a one-stage vs a two-stage detector on speed and accuracy.

Next: extend boxes to pixel-precise masks in Instance Segmentation.