Knowledge BaseTraining at Scale

FSDP, ZeRO & Sharded Training

Sharding parameters, gradients, and optimizer states so trillion-parameter models fit.

advanced#fsdp#zero#sharding#deepspeed
Full write-up in progress

This topic is on the roadmap and its detailed page — theory, math, code, quizzes and projects — is being authored. Its metadata, prerequisites and links are ready below.