An interactive report making the case that backpropagation is subject to the memory wall: the algorithm was popularized in 1986 on machines whose balance (peak FLOP/s ÷ memory bytes/s) was 0.05–0.25 FLOP per byte, and it now runs on accelerators at 150–600 FLOP per byte. Its per-example structure (read every weight twice, ~6 FLOPs per weight, write an update; keep every activation until the backward pass) was arithmetic-bound then and is memory-bound now, so modern training depends on batching, gradient accumulation, activation checkpointing, FlashAttention, kernel fusion, mixed precision, ZeRO sharding and pipeline schedules to pay the difference. The argument uses back-of-the-envelope roofline heuristics in the style of Reiner Pope's blackboard lecture on the Dwarkesh podcast.
Read it: https://yaroslavvb.github.io/backprop-memorywall/
| Path | Contents |
|---|---|
index.html |
The report. Static page; the numbers in the prose are computed live from the same model that drives the figures. |
assets/js/model.js |
The cost model: per-token FLOPs and bytes of a training step, machine balance, T_min, per-step fixed traffic, the batch-size squeeze, the capacity ledger. |
assets/js/data.js |
Hardware catalogue (Cray-1 to Blackwell and TPU v7), network presets (1986 family trees to DeepSeek-V3), 1980s CPS/CUPS table. Each entry carries a status (verified, estimate, announced) and a note on its derivation. |
assets/js/fig-*.js |
The six interactive figures (D3): balance drift, roofline explorer, bytes per token, capacity ledger, activation high-water animation, batch window. |
research/ |
Research notes with primary-source URLs: the 1980s machines (hist-machines), the 1986 papers and reverse-mode AD (backprop-1986), the tricks catalogue with quantified costs and reported MFUs (tricks-catalogue), the memory-wall literature with exact quotes (memory-wall-literature), every heuristic in the transcript (transcript-heuristics), and a red-team critique of the thesis (red-team). Each .md has a machine-readable .json twin. |
backprop_memory_wall.md, backprop_memory_wall.py, fig_backprop_memory_wall.png |
The earlier analytical note and its Python model, from which the JavaScript model's constants are taken. |
dwarkesh-matx.txt |
Transcript of the Dwarkesh Patel / Reiner Pope episode (29 April 2026) the heuristics come from. |
Every hardware entry in data.js carries a status. The 1976–1990 machines, the A100, H100, B200, TPU v5p, Cerebras WSE-3 and Rubin figures were checked against primary sources (brochures, datasheets, vendor pages) during the research pass; the remaining modern entries (other GPUs, TPUs, CPUs of 1993–2023) are vendor datasheet figures as commonly reported and are marked secondary. A planned research pass on the 1993–2026 balance timeline and on modern training-run breakdowns did not complete (API rate limits), so the training-run numbers in the report are taken from the primary papers cited in research/tricks-catalogue.md and research/red-team.md (Llama 3, DeepSeek-V3, PaLM, GPT-3, Korthikanti et al.). The transcript note and the red-team note include their own source-check logs; the other notes' verification passes were cut short by the same rate limits, so treat their [verified] tags as the researching agent's reading of the source, not an independent second check.
python3 -m http.server 8000
# open http://localhost:8000/
No build step. D3 and KaTeX load from cdnjs; fonts from Google Fonts.
For a network with P parameters visited per micro-batch and F FLOPs per token, a micro-batch of T tokens moves b_w · P bytes of weights (2 reads plus a gradient read-modify-write: 12 B/param at bf16 with an fp32 accumulator) plus a · d · L · T bytes of inter-kernel activation traffic (a ≈ 200), so its operational intensity is I(T) = F·T / (b_w·P + a·d·L·T). The step is compute-bound only when I(T) ≥ β = peak / bandwidth, which gives T_min → β·b_w/6 ≈ 2β tokens per weight visit (about 600 on an H100) and a pole at d_min = a·β/72 below which no batch helps. Per optimizer step the Adam state adds ~30 B/param and data-parallel all-reduce ~4 B/param, giving a second floor; the critical batch size gives the ceiling; their geometric mean is the time-optimal batch. Saved activations cost 34·d bytes per token per layer (Korthikanti et al. 2022) and grow with T, which is the tension the report is about.
Text and figures: CC BY 4.0. Code: MIT.