Knowledge Base

Vectors, matrices, and the operations that underlie every model.

Derivatives, chain rule, and gradients — the language of learning.

Distributions, expectation, and reasoning under uncertainty.

Gradient descent, momentum, and the algorithms that fit models.

Array programming, broadcasting, and vectorized computation.

The Python you actually need under pressure — idioms, the standard library, and common traps.

Reasoning about time and space so you can spot the efficient solution — and prove it.

Pixels, convolution, filtering, and classical operators with OpenCV.

Entropy, KL divergence, and cross-entropy — the currency behind every learning objective.

Floating point, conditioning, and the numerical-stability tricks that keep ML code from exploding.

Loading, cleaning, joining, and transforming tabular data with pandas.

Communicating data and results clearly with Matplotlib (and friends).

Git, testing, environments, typing, and structuring code others can build on — the SWE bar the labs expect.

Arrays, hashmaps, stacks, queues, heaps, trees, and graphs — and when each one wins.

Sorting, searching, recursion, and the algorithmic toolkit behind most interview problems.

Two pointers, sliding window, dynamic programming, and graph traversal — the patterns that unlock most problems.

The second coding round: implement a layer, a loss, and a full training loop in NumPy/PyTorch from a blank file.

Thinking in arrays — broadcasting, einsum, and turning loops into fast vectorized tensor code.

Deliberate, timed practice with the in-browser lab — build speed and correctness under interview pressure.

Supervised learning, bias–variance, and the core ML workflow.

The multilayer perceptron: layers, activations, and forward passes.

Reverse-mode autodiff: how gradients flow through a network.

Tensors, autograd, nn.Module, and the training loop — the framework the labs actually run on.

Decision trees, SVMs, k-NN, and linear/logistic regression — and when a simple model wins.

Turning raw data into features models can learn from — encoding, scaling, and leakage.

Clustering, PCA, and dimensionality reduction — finding structure without labels.

Bagging, random forests, and gradient boosting (XGBoost) — still the winners on tabular data.

Precision, recall, ROC/AUC, calibration, and honest model evaluation and validation.

Dropout, weight decay, augmentation, and fighting overfitting.

BatchNorm, LayerNorm, residual connections, and weight init — what makes deep networks trainable at all.

Convolutions, pooling, and the architectures that see.

Recurrent networks, LSTMs, and sequence modeling — the pre-Transformer way to handle order.

Self-attention, multi-head attention, and the Transformer block.

Pretraining, fine-tuning, and reusing learned representations.

How GPUs actually run tensor math — threads, memory hierarchy, and where speed comes from.

fp16/bf16, loss scaling, and the numerical stability tricks that make big training runs survive.

Where the memory and the milliseconds go — activation memory, checkpointing, and profiling tools.

Streaming, sharding, and cleaning trillions of tokens without starving the GPUs.

Splitting one training run across many GPUs and nodes — collectives, NCCL, and synchronization.

The three axes of splitting a model and batch across devices — and when to use each.

Sharding parameters, gradients, and optimizer states so trillion-parameter models fit.

KV caching, continuous batching, speculative decoding — serving tokens fast and cheap.

int8/int4, GPTQ/AWQ, distillation — shrinking models for cheaper, faster inference.

Experiment tracking, config management, checkpointing, and shipping models reliably.

The whole journey in plain language — from raw text to a helpful assistant, and the three stages that get us there.

How words become the integer tokens a model actually reads: characters, words, and Byte-Pair Encoding.

Turning each token into a learned vector, and why similar meanings end up close together.

The one deceptively simple objective — predict the next token — that teaches a model grammar, facts, and reasoning.

How every token looks at every other token to decide what matters — the core idea of the Transformer.

Stacking attention with MLPs, residual connections, and normalization into the repeating block of every LLM.

Assemble a tiny working GPT end to end and train it to babble Shakespeare — the whole architecture in one file.

Where trillions of training tokens come from, how they're filtered and deduplicated, and why data quality is everything.

How performance improves predictably with parameters, data, and compute — and how to spend a training budget.

Running the objective across thousands of GPUs for months — optimizers, schedules, parallelism, and stability.

Teaching the raw base model to follow instructions by fine-tuning on curated prompt–response demonstrations.

Learning a model of what humans prefer, so the assistant can be optimized toward helpful, honest, harmless answers.

Aligning the model to human preferences with reinforcement learning (PPO) or the simpler direct approach (DPO).

Making models honest and harmless at scale — RLAIF, Constitutional AI, and what 'alignment' really means.

How we measure capability and safety — benchmarks, held-out evals, red-teaming, and their real limitations.

Running a trained model efficiently — KV caching, quantization, batching, and serving at scale.

Corners, edges, SIFT/ORB descriptors, and geometric matching.

Localize and classify objects: YOLO, Faster R-CNN, DETR.

Per-object masks with Mask R-CNN and modern successors.

Dense per-pixel labeling with U-Net and encoder–decoders.

ViT, patch embeddings, and Swin — attention for images.

Keypoints, multi-object tracking, and dense motion estimation.

Contrastive and masked pretraining: SimCLR, MAE, DINOv2.

SAM, CLIP, Grounding DINO, and open-vocabulary perception.

GANs, VAEs, and diffusion — learning to synthesize images.

Stereo, monocular depth, point clouds, and camera geometry.

Neural and explicit radiance fields for novel-view synthesis.

A repeatable method for tearing through a paper — the three-pass read, and how to extract the real idea.

Mapping a research area, tracing citations, and keeping up without drowning.

Turning a paper into working code — the single most valuable RE skill, and the messiest.

Framing a hypothesis, choosing baselines and controls, and running experiments you can trust.

Isolating what actually caused the improvement — the ablation table that makes a result believable.

Seeds, variance, error bars, and significance — telling a real effect from noise.

Why won't it train? Systematic debugging of loss curves, gradients, data, and shapes.

Turning results into a clear paper, blog post, or report — the skill Anthropic calls out explicitly.

How strong researchers pick what to work on — impact, tractability, and asking better questions.