Measurement records
Soup's repository publishes its measurement records in a benchmarks/ directory. They are the working records kept while each result was built and checked, not a report written afterwards, so they keep the failures, the readings that turned out wrong and the numbers that were measured and then discarded, in the order those things happened. This page indexes every record in that directory at the v0.78.0 tag: the question each one asked, the hardware and software it states, its headline result with the caveat it carries, and whether it passed, failed or was never a gate.
The site says its performance numbers are measured rather than claimed, and that is only worth something if a reader can find the measurement. This page is the map. It is not a second copy of the results: the record is the authority, and where this page and a record differ, the record wins. Upstream keeps its own index in benchmarks/README.md. At the tag that index still describes the batch-control record as "results pending", while the record itself contains the completed second attempt, so the record wins there too. The layer streaming records are the evidence behind the layer streaming preprint, whose concept DOI is 10.5281/zenodo.21771064.
The directory holds 84 records, that README, a harness/ folder of 185 files (149 Python scripts, 11 shell scripts, two PowerShell scripts, eight Markdown notes, one notebook and 14 data and settings files) and one timing script beside the records. (At v0.77.2 it held fifty-five records and a harness/ of 95 files: 89 Python scripts, two PowerShell scripts, one shell script, two Markdown notes and one notebook.) At v0.75.2 it held fourteen records and six harness scripts; v0.76.0 added 24 records, v0.77.0 added 12, v0.77.1 added 3 (two gate records and the pre-registered protocol of one of them), v0.77.2 added two records and seven harness files, and v0.78.0 added 29 files and 90 harness files (below). No gate ran for v0.77.2 and the preprint is unchanged, and the preprint is unchanged at v0.78.0 too. Three records that were already there changed between v0.77.1 and v0.77.2, and their links on this page moved to v0.77.2 (the H100 record moves again to v0.78.0, below): the free Colab T4 record (a third, uncapped session was added), the resident-VRAM record (three rows) and the 8x H100 record (a dated clarification under its finding 9). Upstream's README index changed too. Every other earlier link on this page stays pinned to the tag its record was first published at. Every layer streaming figure published up to v0.72.4 came from one laptop. The later records add other hardware: a borrowed 8x H100 box, a free Colab T4, machines belonging to contributors, and, since v0.76.0, a second laptop with an RTX 5070 and two NVMe drives that carries the disk tier records. v0.77.1 adds two more machines, an RTX 3060 Laptop 6 GB under WSL2 and an RTX 3090 on Windows. The two records added in v0.77.2 are an RTX 5090 observation and a CPU-only investigation, and the T4 record gains a third session on a free Colab card.
How to read this page
- Each record is linked at the tag, at the top of the file. Read it to the end before quoting a number from the middle. Several leave a wrong passage standing with its correction beside it or further down, on purpose, so a figure lifted from the middle may be one the same file later withdraws. Where a record does this, its entry below says so, and a figure a record later withdraws is not quoted here.
- The status words are the record's own. "Not a gate", "partial pass" and "gate failed" are quoted from the files, not softened or sharpened.
- Units are as the record prints them. Most gate records use decimal GB, the H100 record prints MiB, the pinned-memory record prints KiB, the MLX record uses GB as MLX reports them, the QuEST records report loss in nats, the CUDA graph record prints MiB, and the 14B-on-8-GB record writes powers of two in GiB. Nothing here is converted.
- "Bit-exact" is always two claims. The forward pass (logits, compared with
torch.equal) and the backward pass (every LoRA gradient tensor) are measured independently, and above roughly 165 MiB per NF4 layer they disagreed before the repair. Wherever this page says exact, it names the half, and the quantisation where the record states one. The per-model ledger has all four cells for every row. - A correctness reference always matches the numerics under test. A streamed NF4 run is compared with a resident NF4 run, never with resident bf16, whose quantisation error would hide a real defect.
- A throughput figure is read with the clock it was taken at. The laptop card's boost clock varies about 13% between sessions, so the records quote the SM clock beside throughput and compare against a GEMM ceiling measured in the same session. Two rows taken at different clocks are compared through that ceiling fraction, not through their raw tok/s, which is how the NF4 record reads its two 8B rows, adding that a changed memory plan makes even the fraction not like-for-like.
- A new machine is a new baseline, not a comparison. The records from the RTX 5070 laptop say they are the first baseline on that card and compare nothing against a stored number from the RTX 3050. This page keeps them apart: no figure from one machine is set beside a figure from the other, and a record's number is only compared with another number from the same record. The two v0.77.1 records are two more machines (an RTX 3060 Laptop 6 GB and an RTX 3090), and the same rule applies to them.
- Derived figures are labelled as arithmetic. "One million tokens takes about 2.3 hours" (at the pre-repair 119.6 tok/s) is division, not a wall-clock run.
At a glance
| Record | Belongs to | Machine | Status as the record states it |
|---|---|---|---|
| Streaming path | v0.72.0 | RTX 3050 Laptop 4 GB | Both gates pass |
| NF4 streaming | v0.72.2, plus a post-repair row for v0.76.0 | RTX 3050 Laptop 4 GB, and a second one for the added row | Gate 1 passes (6 of 6), gate 2 complete; post-repair row added, read with its clock |
| Breadth | v0.72.3 | RTX 3050 Laptop 4 GB | Six gates, all pass |
| Preference losses | v0.72.4 | RTX 3050 Laptop 4 GB | Gate 1 passes 14 of 14, gates 2 and 3 pass 10 of 11 |
| What bounds a streamed step | v0.73.0 | RTX 3050 Laptop 4 GB | Task 1 answered, task 2 answered for the kernel and blocked for the shipped flag |
| Measuring the VRAM fit | v0.73.1 | RTX 3050 Laptop 4 GB | Found the pre-flight under-predicting at long sequence |
| Leg 2 of soup ship | v0.73.2 | RTX 3050 Laptop 4 GB, CPU for model runs | Working record, no verdict line |
| 8x H100 | The v0.72.4 line, repair in v0.73.0 | 8x H100 80 GB | External-hardware gate, defect repaired and re-gated, dated corrections |
| Free Colab T4 | v0.73.1 | Colab Tesla T4 | Run completed, correctness not gated, not a gate; a later run recorded a small-model forward comparison, and a third session (#999, v0.77.2, uncapped) measured throughput against the forecast bracket |
| A10G | #395, Soup 0.73.3 | AWS A10G 23 GB | Under-prediction does not reproduce, hypothesis not supported |
| L4 | #655 | NVIDIA L4 container | Measurement supporting a design decision |
| M4 Max | v0.74.0 | MacBook Pro M4 Max 128 GiB | Partial pass, optimizer step not validated |
| 8 GB M1 | #23 | Apple M1 8 GB | Five runs completed, one fixture failed, not a gate |
| Disk tier, asynchronous reader | #971, v0.76.0 | RTX 5070 Laptop 8 GB | Measured, with one negative result and one confound that changed the headline |
| Disk tier, direct I/O | #974, v0.76.0 | RTX 5070 Laptop 8 GB | The warm regression it was filed over does not exist position-matched; the cold target is met |
| Disk tier, two-drive striping | #1536, v0.76.0 | RTX 5070 Laptop 8 GB, two NVMe | SHIP (sequence 4) |
| Two-drive read rate | The striping track | RTX 5070 Laptop 8 GB, two NVMe | No formal verdict |
| Two-drive sustained load | The striping track | RTX 5070 Laptop 8 GB, two NVMe | NO VERDICT, twice |
| Drive throttle diagnostic | The striping track | RTX 5070 Laptop 8 GB, two NVMe | NO VERDICT |
| Batch control | The striping track | RTX 5070 Laptop 8 GB | Complete on the second attempt; batch 3 disagrees with the model on one drive |
| What bounds a streamed step, RTX 5070 | v0.76.0 | RTX 5070 Laptop 8 GB | Three sections measured |
| Layer-major micro-batching, step 0 | Not a feature | RTX 5070 Laptop 8 GB | G1 and G2 pass, G3 and G4 no verdict |
| 14B on an 8 GB card | #901, v0.76.0 | RTX 5070 Laptop 8 GB | Reproduced, explained, fixed |
| Pinned arena packing | #1702, v0.76.0 | RTX 5070 Laptop 8 GB | Memory saving confirmed; no speed difference measurable |
| Expert streaming gate | #2008, v0.77.0 | RTX 5070 Laptop 8 GB and CPU | PASS (27 of 27) |
| Moonlight-16B | #1914, #2029, v0.77.0 | RTX 5070 Laptop 8 GB | Both arms COMPLETE |
| FP8 shard gate | #2061, v0.77.0 | CPU only | G1, G2 and G3 pass |
| FP8 source loss check | #2061 | RTX 5070 Laptop 8 GB | FAIL on two pre-registered limits |
| Kimi K2 admission | #2107, v0.77.0 | CPU only | G1a fails as registered; G1b, G1c, G2 and G3 pass |
| K2-shaped step | #2008 | RTX 5070 Laptop 8 GB | COMPLETES (random weights, shape only) |
| K2-shaped step, longer sequences | A probe | RTX 5070 Laptop 8 GB | NO VERDICT |
| K2 layer on 24 to 32 GB cards | A note | None | Arithmetic only |
| stream_recompute | #2152, v0.77.0 | RTX 5070 Laptop 8 GB | Measured on one laptop |
| NF4 scale cache | #1892, v0.77.0 | RTX 5070 Laptop 8 GB | Confirmed on one laptop |
| Resident VRAM estimate | #2049, #2277; v0.77.1, rows 40 to 42 in v0.77.2 | RTX 3060 Laptop 6 GB, WSL2 | 42 rows, coefficients fitted on rows 1 to 39; no model of 7B or more, no mixture-of-experts model, no medium or selective row |
| Selective attention checkpointing | #2050, v0.77.1 | One RTX 3090, Windows | 1 of 4 comparisons confirmed; benchmark-only, nothing in soup train uses it |
| Unsloth 4-bit weight dtypes | #2301, v0.77.2 | RTX 5090 | A load-and-inject observation, no training peak |
| Fast-LoRA CPU bf16 | #2187, v0.77.2 | Ryzen 7 7840HS, CPU only | An investigation: one seed fails on one torch build; no kernel or criterion change |
| Streaming against DeepSpeed, re-measured | v0.78.0 | One H100 80 GB SXM | All three parts measured; the earlier ratio does not hold (1.56x against ZeRO-3 CPU offload); streaming slower than every DeepSpeed arm that keeps weights on the GPU |
| Forward-only streamed inference | #2509, v0.78.0 | RTX 5070 Laptop 8 GB | G1 and G2 pass, one run per arm, forward only |
| BCO on a T4 | #2766, v0.78.0 | Kaggle Tesla T4 | Four cells pass, two-layer fixture, one step |
| IPO on a T4 | #2766, v0.78.0 | Kaggle Tesla T4 | Default-setting cells failed (fp16 scaler skips); a harness-only scale let eight cells update |
| NF4 on a T4 | #842, v0.78.0 | Kaggle Tesla T4 | Aliasing reproduced on a synthetic fixture; the full-model negative control did not fire |
| NF4 on Blackwell | #776, #331 | RTX 5070 Laptop 8 GB | "Not a gate"; the post-#1143 row, one 0.5B model |
| Resident 4-bit fp32 cast | #2609, v0.78.0 | One RTX 4090 | PARTIAL against its 70% bar |
| 8-bit checkpointing VRAM | #2860, v0.78.0 | RTX PRO 5000 Blackwell, capped | A gate record, no verdict line |
| Distillation teacher VRAM | #2909, v0.78.0 | RTX PRO 5000 Blackwell, capped | A gate record, no verdict line |
| Chunked distillation | #722 | Shared H200 | A memory-for-time trade, not a speedup |
| Larger models on an 8 GB laptop | v0.78.0 | RTX 5070 Laptop 8 GB; one cloud L40S | "Not a gate"; a capacity record |
| Selective projection retention | #2162 | RTX 3090 capped at 8 GiB | INCONCLUSIVE for QKVO-only; measurement only, no option exists |
| Kimi K2, resident | #2104, v0.78.0 | CPU only | PASS, tiny K2-shaped checkpoint |
| Flash-Attention and Liger | #378 | Shared H200 | Ratios of five-run medians; changes no product code |
| Fast-LoRA, two CUDA readings | #837, #838 | T4; RTX 3060 Laptop | The readings disagree; no speed verdict |
| Packed LoRA GPU tests | v0.78.0 | RTX 5070 Laptop 8 GB, WSL2, shared | One GPU-test row; nothing is a speed measurement |
| QuEST capacity | #674 | RTX 3050 Laptop 8 GB | Eight updates completed; no parity, quality or efficiency claim |
| GRPO on T4 cards | #342 | Kaggle Tesla T4 | "Not a gate"; no evidence for or against the H100 failure |
| AutoDistill student | #580 | RTX 3090 | All cases pass; no peak, quality or workflow gate |
| MLX dense full fine-tune fixture | #1992 | Apple M4 16 GiB | Tiny fixture; no large-model claim |
| MLX SimNPO check | #2005 | Apple M6 24 GB | FAIL; the code it measures is not in a tagged release |
| ONNX export compatibility | #2676 | CPU only | Released export packages pass on a legacy-compatible fixture copy |
| quant-check smoke | #2677 | Shared H200 and CPU | Functional smokes, not a speedup or quality evaluation |
| CUDA graph decoding | v0.76.0 | RTX 5070 Laptop 8 GB | Every criterion passed |
| soup bench train contract | #836, v0.76.0 | RTX 5070 Laptop 8 GB, on battery | The contract holds; the number does not repeat |
| Tool-call discrimination | #405, v0.76.0 | Apple M4 Max | No verdict line; the revised fixture is accepted |
| Arithmetic discrimination | #1192, v0.76.0 | Apple M1 32 GB | No verdict line |
| Common-sense discrimination | #1192, v0.76.0 | Apple M1 32 GB | No verdict line |
| MMLU-style discrimination | #1192, v0.77.0 | Apple M1 32 GB | No verdict line |
| soup ab, burn-in | #1227 | CPU simulation | Superseded by #1265 |
| soup ab, mixture | #1265 | CPU simulation | Accept rule superseded by #1418 |
| soup ab, accept rule | #1418, v0.76.0 | CPU simulation | The record behind the current accept rule |
| QuEST W4A4 | #674 | RTX 5090 Laptop 24 GB class | Gate failed |
| QuEST mixed route | #674, v0.76.0 | 2x RTX 3090 | Gate passed, evaluation only |
| QuEST post-fit | #674, v0.76.0 | Not stated here | Remains above the zero-margin parity target |
| QuEST development follow-up | #674, v0.77.0 | RTX 3090 | Neither replica passes the zero-margin parity gate |
| LoRA-FA | #725, v0.76.0 | Micro-benchmark | No verdict; not a VRAM claim |
| MoE expert coverage | A probe (the feature it describes shipped in v0.77.0) | RTX 5070 Laptop 8 GB | Measured: memory only |
| GPU test runs | A log | RTX 5070 Laptop and H100 | Not a gate |
Read the last column first. The directory holds a failed gate, a result that stayed above its target, a partial pass, probes that ended with no verdict, designs superseded by a later record, a negative result for a hypothesis, the gate that found a shipped formula failing its own contract, and several records that say outright that they are not gates. They are in this index because an index of the flattering ones would be worth nothing.
Layer streaming, as it was built
The first four records are the gates that built the feature, all measured on one machine: an RTX 3050 Laptop with 4 GB of VRAM (4.29 GB usable), 16.9 GB of host RAM, NVMe and Windows 11. (The v0.72.2 record has since gained one row from a second RTX 3050 Laptop, marked where it appears.) Windows matters when reading them. Its driver spills into shared host memory instead of raising an out-of-memory error, so a run completing is not evidence that its configuration fits, which is why peak VRAM is reported beside every throughput figure.
v0.72.0: the streaming path
- File:
gate-v0.72.0-layer-streaming.md. Belongs to v0.72.0, and has been in a tagged release since v0.72.4. - Question: Does streaming the frozen base one decoder layer at a time give the same numbers as holding it resident, and what throughput does a 4 GB card sustain doing it?
- Hardware and stack: RTX 3050 Laptop 4 GB (compute capability 8.6, driver 591.44), 16.9 GB RAM, NVMe, Windows 11. torch 2.5.1+cu121, peft 0.18.1, accelerate 1.12.0, Python 3.10.
- Result: On SmolLM2-135M in bf16 the streamed forward is bit-exact against the resident model (logits, maximum absolute difference 0.0) and a 100-step loss curve is identical. Qwen2.5-3B in bf16 trains in 2.15 GB where the same model trained resident runs out of memory on that card.
- Status: "Both gates PASS", in the record's words.
- Caveats: The 3B row used a pageable store, so it is a lower bound. (Since v0.76.0 the RAM store is page-locked in power-of-two arenas rather than tensor by tensor, see the 14B record; the 3B bf16 row was not re-run with it.) The numbers are Windows and pessimistic against Linux, and nothing was measured above 3B. The only valid streaming-versus-resident overhead is 1.43x, at 0.5B, because the resident 1.5B row spilled into host memory and its number is discarded.
- Read to the end: Near the end, "After the gates" records what a passing gate did not catch: no test had built a real trainer from the streamed model, so every real run would have died at trainer construction, and a shard cache could silently stream stale weights. A paragraph that first said the transfer was fully hidden was corrected in place on 2026-07-27 and then explained row by row. Treat those row-level explanations as the author's readings at the time, and read the probe record below, where a mechanism was actually measured, before reusing one. The file also records that three third-party figures drafted for the write-up were wrong when checked against source, which is why the defensible claim is 1.3B to 3B on a 4 GB card and not "nobody has done this".
- On this site: Layer streaming, measured numbers and the paper.
v0.72.2: NF4 streaming
- File:
gate-v0.72.2-nf4.md. Belongs to v0.72.2 (its own heading says "now the v0.72.2 slot", because the gates ran while NF4 was planned as v0.72.1), and has been in a tagged release since v0.72.4. - Question: Can the base be quantised to NF4 offline and streamed, still matching a resident NF4 model, and how large a model does that bring onto 4 GB?
- Hardware and stack: RTX 3050 Laptop 4 GB, 16.9 GB RAM, NVMe, Windows 11. torch 2.5.1+cu121, bitsandbytes 0.49.2, transformers 4.57.6, peft 0.18.1, trl 0.19.1, accelerate 1.12.0, Python 3.10.8.
- Result: On SmolLM2-135M the streamed NF4 forward is bit-exact against a resident NF4 model (logits, maximum difference 0.0), gradients are non-zero on all 30 layers and a 25-step loss curve matches. Llama-3.1-8B in NF4 then runs on the 4 GB card from a 3.60 GB page-locked store.
- Status: "GATE 1 PASS (6/6, bit-exact vs resident NF4). GATE 2 complete", in the record's words.
- Caveats: The file carries two 8B rows, a first measurement from a throwaway spike and a later reproduction through the shipped code, and the reproduction is the figure the release notes, the README and this site quote: 119.6 tok/s at 952 MHz and 3.32 GB (since v0.76.1 upstream's README prints it with that clock, beside the post-#331 row), on a 3.60 GB page-locked store. Two caveats travel with the published figure. It was measured at 952 MHz before the NF4 gradient repair of v0.73.0, and the 3.32 GB peak predates the large-layer streaming change of v0.74.0, which moved the embedding and output head into the pinned store. The H100 cross-check, a median 113.00 tok/s at a 3,397 MiB peak (the H100 record prints MiB, where the laptop records print decimal GB), is a reproduction on other hardware and not a re-measurement of the laptop figure.
- The post-repair row (added for v0.76.0): the same record gained a section, "Post-#331 re-measurement", dated 2026-09-19 (the record recovers the date from the machine's file times, because the run printed none). It is Llama-3.1-8B in NF4, streamed, batch 1, sequence 512, 50 measured steps after 10 warm-up, on a different RTX 3050 Laptop 4 GB under Windows 11, measured by @umran666 on their own laptop (#1103): 208.6 tok/s at 1935 to 1957 MHz, 2.40 GB peak allocated (2.66 GB reserved), a 5.70 GB pinned store, one run, with the slowest step kept in the data. It ran on development code between v0.75.0 and v0.76.0, not on the tag. The record's own instruction is "Do not compare the two tok/s figures directly": the old row was taken at 952 MHz and the new one at 1935 to 1957 MHz, the memory plans differ (the store grew by exactly the embedding and the output head while the peak fell), and the stack differs (torch 2.5.1 against 2.6.0, transformers 4.57.6 against 5.16.1). It attributes the remaining difference to none of the repair, the memory change or the stack. The record reads the pair this way, and since v0.76.1 upstream's README does too: 59% of the same-session GEMM ceiling against 68% is about 14% slower per unit clock, so the raw 1.74x gain is the clock, not a speedup, and the changed memory pattern means even that fraction is not like-for-like. The record also kept its pre-registered expectation that the new row would come in below 119.6 tok/s: it came in above, and the record writes the outcome beside the expectation and explains the excess by the clock instead of editing the expectation away.
- Read to the end: The gate found the v0.72.1 defect, where streamed adapters were saved with an extra key segment and reloaded as the untuned base. Two of its own checks were nearly vacuous until fixed, because an adapter initialised to zero makes the adapter path do nothing. A first TFLOPS reading was held back and corrected: the ceiling tracks the GPU clock, so a fraction of a ceiling means nothing without the clock it was taken at.
- On this site: Layer streaming, measured numbers and why NF4 also made the 3B row faster.
v0.72.3: breadth
- File:
gate-v0.72.3-breadth.md. Belongs to v0.72.3, and has been in a tagged release since v0.72.4. - Question: Can streaming widen to more architectures, batches above 1, gradient accumulation, resume and a disk tier without silently training the wrong thing, and can the VRAM pre-flight predict a peak well enough to refuse a run that will not fit?
- Hardware and stack: RTX 3050 Laptop 4.29 GB, 16.9 GB RAM, NVMe, Windows 11, Python 3.10. torch 2.5.1+cu121, transformers 4.57.6, peft 0.18.1, trl 0.19.1, bitsandbytes 0.49.2.
- Result: Each of the six added architectures (
mistral,gemma,gemma2,gemma3_text,phi,phi3) streams with forward logits equal to the same checkpoint loaded resident, in an unquantised and an NF4 arm, but on tiny from-config CPU float32 fixtures, not real checkpoints and not the RTX 3050. The peak-VRAM predictor came within 0.85% of ten real runs on two models and never under-predicted on that grid. Raising the batch beat accumulating at the same effective batch by a measured 2.52x, and accumulation is neutral per token. - Status: All six gates are recorded as PASS, gate 1 as 14 of 14.
- Caveats: The predictor was fitted at sequences of 256 and 512 only, and the v0.73.1 record later found it under-predicting at long sequence. Refusing a run that is too big is policy applied to a measured demand: Windows spills instead of raising, so the record does not claim it was verified by observing an out-of-memory error. End-to-end
soup train --resumecould not be demonstrated on that box, because torch 2.5.1 makes transformers refuse every resume and astream_layers: falsecontrol fails identically, so only the streaming-specific half is verified. The speed difference between the RAM tier and the disk tier was deliberately unmeasured and no figure was claimed. Only the disk tier's correctness is shown, with logits equal to both the RAM tier and a resident run. (Since measured in v0.76.0, on a different machine: see the disk tier records.) No tok/s exists for the six added architectures. - Read to the end: The record keeps its own over-claims. A first reading of the
soup doctor --diskcost was wrong and is corrected, and a GEMM probe that swung by a factor of two at the same reported clock was diagnosed rather than accepted. - On this site: Scaling a streaming run.
v0.72.4: preference losses
- File:
gate-v0.72.4-preference-losses.md. Belongs to v0.72.4, and has been in a tagged release since v0.72.4. - Question: Can DPO, ORPO, SimPO and KTO run over a streamed base without a second copy of the weights? DPO's reference model has to be taken from the same streamed base, which this record did with the adapters disabled, because a second instance would defeat the feature, and a passing loss curve cannot detect that. (The record's stack was trl 0.19.1. Since TRL 0.29 DPO's reference is a frozen adapter snapshot, which costs adapter-sized memory more; the record was not re-measured, and its figures stand for the stack it states.)
- Hardware and stack: RTX 3050 Laptop 4.29 GB, 16.9 GB RAM, NVMe, Windows 11, Python 3.10.8. torch 2.5.1+cu121, transformers 4.57.6, peft 0.18.1, trl 0.19.1. The record states that CI resolves a much newer stack, so it asserts the properties the shipped tests depend on rather than TRL's internals.
- Result: On a synthetic 365.2M-parameter fixture (24 layers, vocabulary 260, sequence 64, batch 1), streamed DPO peaked at 0.914x the streamed SFT peak, while a control that forced a real second model instance added +730.44 MB against 730.44 MB of weights, which is exactly one copy.
- Status: Gate 1 (DPO) is recorded as PASS 14 of 14, and gates 2 and 3 (ORPO, SimPO, KTO) as PASS 10 of 11. The one miss is a pre-flight check on the SFT path at -1.7%, which the record attributes, with evidence, to a fixture far outside the range the formula was fitted on.
- Caveats: Read 0.914x as "no second copy", not as DPO being cheaper than SFT: the SFT arm computes its loss over all 64 positions while the DPO arm splits them into prompt and completion, so its logits tensor is smaller. The control is the half that carries the result. The cost is 1.52x the layer reads per step, free in memory and not in time. The fixture is not a real checkpoint and not Llama-3.1-8B, DPO's loss matches a resident DPO run exactly (difference 0.0, on a 4-layer CPU float32 model), and no tok/s was measured for any preference loss. KTO is not reference-free and needs a batch size of 2 or more.
- Read to the end: The file opens with a table of TRL versions that is wrong, and a correction directly beneath it says so and was left in place on purpose. The lesson the record draws is that a version bound derived by reading source is a hypothesis, and the experiment that tests it is constructing the object. It also lists three measurement attempts it marks INVALID so they are not repeated.
- On this site: Preference losses over layer streaming, and why 0.914x means no second copy.
The probe that retracted an explanation
One record measured what had only been inferred, and its result changed what the preprint and this site may say about why a streamed step takes as long as it does.
v0.73.0: what bounds a streamed step
- File:
probe-v0.73.0-what-bounds-streaming.md. Belongs to v0.73.0 (its header names the v0.73.0 working tree at commit 57c18e5), and has been in a tagged release since v0.73.1. - Question: What actually bounds a streamed step, and what happens when Cut Cross-Entropy is switched on beside streaming?
- Hardware and stack: RTX 3050 Laptop 4 GB, 16.9 GB RAM, NVMe, Windows 11. torch 2.5.1+cu121, bitsandbytes 0.49.2, transformers 4.57.6, peft 0.18.1, Python 3.10.8.
- Result: On Llama-3.1-8B in NF4 at batch 1 and sequence 512, deleting every host-to-device byte (6.864 GB per step) makes the step 1.44% faster. The compute stream is blocked on a copy for 8.4 ms of a 4190 ms step, and the step runs at 71.3% of the same-session, shape-matched GEMM ceiling at 952 MHz. Removing the NF4 dequantisation instead buys 9.80%. These are the measurements that retracted the explanation version 2 of the preprint gave for the H100 replication, and the record limits that explanation to very short steps, where the fixed transfer volume dominates. Its own figures put that boundary near 128 tokens per step, and the published configuration is not there.
- Status: "TASK 1 ANSWERED", and task 2 "ANSWERED for the kernel, BLOCKED for the shipped flag".
- Task 2, Cut Cross-Entropy: A hand-wired Cut Cross-Entropy kernel raised the usable microbatch from 1 to 3 for +9.6% throughput on that card, but Soup's own
training.use_cut_cecould not engage on the stack measured, for three separate import and packaging reasons, and the kernel's backward is not reproducible against itself, so the bit-exactness standard cannot apply to it. - Caveats: Those blockers were measured on the v0.73.0 stack (transformers 4.57.6). Soup has required transformers 5.x since v0.74.0, and the record does not speak to that stack. It is one card on Windows. The step decomposition is Llama-3.1-8B in NF4 only, and the Cut Cross-Entropy correctness check ran on SmolLM2-135M in NF4. bf16 streaming, sequences beyond 512 and the disk tier were not measured, and the H100's own bottleneck was never instrumented, so the record makes no claim about it.
- Read to the end: The premise was corrected before the first measurement, because the brief had assumed a 128-token step. Section 10 lists what was not measured. The record gives its own one-sentence reading of the laptop result in section 6; this page quotes the measurements, not that reading, and offers no mechanism of its own.
- On this site: What version 3 of the paper adds and the laptop number reproducing on other hardware.
The pre-flight and the release gate
Two records test the instruments rather than the feature: the VRAM pre-flight that decides whether a streaming run may start, and the scorers behind the soup ship verdict.
v0.73.1: measuring the VRAM fit
- File:
gate-v0.73.1-measured-vram-fit.md. Belongs to v0.73.1 (issue #349), and has been in a tagged release since v0.73.1. - Question: The streaming pre-flight predicts peak VRAM from a formula and promises it never under-predicts. Can that prediction be replaced by a measurement, and does the promise hold?
- Hardware and stack: RTX 3050 Laptop 4 GB (4.294 GB total, 3.45 GB free at rest), Windows 11, NVMe. torch 2.5.1+cu121, transformers 4.57.6, trl 0.19.1, peft 0.18.1, Python 3.10.
- Result: Measured through the real
soup trainon SmolLM2-135M (streamed, bf16, batch 1), the formula over-predicts at sequence 4352 (1.081x) and under-predicts at 5120 (0.934x) and at 6144 (0.787x, under by 21%). The ten-run grid behind the "never under-predicts" contract varied batch and never sequence length, so it could not have seen this. - Status: This is the gate that found a contract failing, and its headline is "the formula under-predicts at long sequence".
- Caveats: The mechanism is "NOT established" in the record's words. The release added an opt-in measured probe,
training.stream_vram_probe, which applies totask: sftonly. The probe is not validated for preference losses, where one shape is not a validation. - Read to the end: Three readings were withdrawn during the work, and two of them looked like the headline result. They are left in with their refutations, and none of them is a result. The record also corrects a draft claim about the probe on a preference loss, which had compared two different shapes.
- On this site: The v0.73.1 release page and the long-sequence under-prediction in the pre-flight.
v0.73.2: leg 2 of soup ship
- File:
gate-v0.73.2-leg2-scoring.md. Belongs to v0.73.2, and has been in a tagged release since v0.73.2. - Question: Do four reported defects in the scorers behind
soup ship's regression leg reproduce (#357, #346, #355 and #317), and does the repair hold? - Hardware and stack: RTX 3050 Laptop 4 GB on Windows 11 with Python 3.10, but every model run was on the CPU (a 135M pair at 256 new tokens), and the record says no GPU was needed and none is claimed. The baseline is shipped v0.73.1.
- Result: A stub answering every item correctly in a boxed-letter style scored 0.000 on two multiple-choice suites. A stub naming the right tool 40 of 40 times, one closing brace short, scored 0.000 on the tool-call suite. Two models with byte-identical leg-2 scores, one of them refusing every benign request, got the same SHIP verdict. (Since v0.76.0 these suites are re-scaled; the figures above are on the v0.73.2 fixture. See the re-scaled scorers.)
- Status: A working record with no gate verdict line.
- Caveats: The 0.423 to 0.731 and 0.225 figures in the issues are the H100 record's, on an 8B, and are not re-measured here. The noise floor measured on this box is 0.0000 because CPU greedy decode is deterministic, and the H100's 0.015 and 0.020 spreads are explicitly not re-claimed. The source issue's floor is n=3 on one model and one dataset, so it sizes an effect and does not calibrate a threshold. The record also explains why the preprint is scoped and not amended: its leg-2 numbers were measured with the earlier scorer, are correct as measured, and should not be compared across the boundary.
- Read to the end: A "66 failures" scare turned out to be a read of a half-written file and was withdrawn. A control that varied nothing is recorded as such, and a review finding was checked against three implementations and partly rejected.
- On this site: The v0.73.2 release page and
soup ship.
Hardware the maintainer does not own
Everything above was measured on the one laptop, except the single post-repair row noted under v0.72.2. These records are the exceptions, and each says which of its claims depend on that.
8x H100: external validation
- File:
gate-h100-validation.md(linked at v0.78.0 because two dated annotations were added in that release, saying its DeepSpeed comparison, STEP 3, is superseded as a comparison by a newer record; a dated clarification was added to its finding 9 in v0.77.2; no measured number in it changed). Measured on a checkout made just after the v0.72.4 tag, which still reported itself as 0.72.4 (the defect it found is present across v0.72.0 to v0.72.4), with the repair shipped in v0.73.0, and has been in a tagged release since v0.73.0. It is a long file, about five thousand lines. - Question: Does the method hold on someone else's hardware, at model sizes where a resident reference exists to compare against, which a 4 GB card cannot hold?
- Hardware and stack: 8x NVIDIA H100 80 GB HBM3 over PCIe with no NVLink, 503 GB RAM, Ubuntu 24.04.3, driver 590.48.01. torch 2.13.0+cu130, bitsandbytes 0.50.0, transformers 4.57.6, trl 0.26.2, peft 0.20.0.
- Result: The forward (logits,
torch.equal) matched a resident reference of the same numerics at 0.5B, 8B, 14B, 32B and 72B. The backward (every LoRA gradient tensor) matched at 0.5B, 8B and 14B in NF4, and above about 165 MiB per NF4 layer it was wrong until the repair: only 8 of 256 gradient tensors on 32B and only 8 of 320 on 72B matched the resident reference. After the repair it re-gated at 256 of 256 on 32B and 320 of 320 on 72B, each against a control arm that reproduced the defect in the same process. - Status: An external-hardware gate, with its status block saying the defect is repaired and re-gated.
- Caveats: Nowhere does the record call 72B exact in the backward before the repair, and "bit-exact at 72B" on its own is a forward statement.
- Also in the record: The laptop headline reproduced at a median 113.00 tok/s with a 3,397 MiB peak (n=5), with no explanation attached on this site; a DeepSpeed ZeRO-3 comparison, which the record frames as not faster than DeepSpeed but built for a different situation, one card that is too small (superseded as a comparison in v0.78.0: against a DeepSpeed set up from its documentation streaming ran at 1.56x, a much smaller gap than STEP 3 reported, and slower than every arm that keeps the weights on the GPU); and a convergence comparison against resident runs. The external validation page carries those figures with their caveats.
- Read to the end: The opening summary explains the H100 replication by host-to-device transfer. That was measured and refuted afterwards, by the probe record above, and the file carries dated corrections (2026-08-13) beside each place it appeared, with the original text left standing. It opens with a per-model exactness ledger giving the forward, the backward, the quantisation, MiB per layer and the reference for every row, with unmeasured cells marked "not tested" instead of left blank, because two readers took an unqualified "bit-exact" to cover both halves. The defect is also filed upstream, as bitsandbytes issue 2034. Its Reproducing section now links six of its scripts under
harness/, three of them marked as reconstructions rather than the scripts as run; the rest are still unpublished (tracked as #379). Its measured figures are unchanged. - Finding 9 and the 6.56x (added in v0.77.2, #2271, credit @xiemocun for raising #2224): the finding "Turning pinning off costs 6.56x throughput" now carries a dated clarification (2026-11-05). "Asserted" there means checked and reported, not required to pass: in STEP 8 the page-locked arm was timed while it was wrong (only 8 of 256 gradient tensors matched, on all three repetitions) and the pageable arm was exact (256 of 256), so the 6.56x is the ratio of those two medians, in upstream's words "a defective fast arm over a correct slow one". The same word appears at STEP 8 and in two later cost tables, where the shipped arm is likewise the wrong one. No number in the record changes, and this page keeps 6.56x as a historical timing from before the repair, not as a cost of pinning off under today's code. The product's own
stream_pin: falsemessage and upstream's performance document still print a "worth up to 6.56x" sentence at v0.77.2, which this site does not repeat. - On this site: Validation on hardware we do not own, the paper and what to re-run after v0.73.0.
Free Colab T4: an 8B run
- File:
run-t4-colab-free-tier.md. Belongs to v0.73.1 (the pre-Ampere fp16 repair, #385 and #387), and has been in a tagged release since v0.73.1. It is linked at v0.77.2 because the third session below was added to it in that release; the v0.77.0 copy does not have it. - Question: Does the streaming path run to completion on a pre-Ampere (Turing) card? The repair had only been verified using fp16 on an Ampere card, which establishes the plumbing and says nothing about Turing kernels.
- Hardware and stack: A free-tier Google Colab Tesla T4 (sm_75, 15.6 GB), with the process capped at 4.00 GB through
torch.cuda.set_per_process_memory_fractionin the first two sessions (the third had no cap). Driver and library versions were not recorded for the first run. Llama-3.1-8B-Instruct in NF4, fp16, batch 1,max_length: 256, LoRA r=8. The second run (below) printed soup-cli 0.75.0, torch 2.11.0+cu128, transformers 5.16.1, peft 0.20.0 and bitsandbytes 0.50.2. - Result (first session): Peak VRAM was 2.91 GB against a predicted 3.02 GB, so the pre-flight over-predicted by 3.8%, the safe direction. Seven steps completed and the adapter was written with 128 of 128 tensors non-zero. The cap was shown to bite: a deliberate 4.29 GiB allocation was refused.
- Status: "RUN COMPLETED. Correctness NOT gated", and the record's own index calls it not a gate.
- A later run of the notebook (recorded by the commit of 12 September 2026): the record now says its section 4 "has since been run and recorded". The streamed forward matched the resident model exactly (
torch.equaltrue, maximum difference 0.0) on SmolLM2-135M-Instruct in fp16, unquantised, forward only. The same run re-measured the 8B NF4 section with a 5.70 GB store, a different memory plan from the first run (the embedding and head had by then moved into the pinned store). Both runs are kept as captured; the later peak is in the record and this page does not set it beside the first run's. - A third session, uncapped (#999, added to the record in v0.77.2, credit @VanshSharmaPES, #2230, and for the
soup bench trainrepair it needed, #2216): on 2026-10-26, a free Colab Tesla T4 (sm_75, 15,360 MiB, driver 580.82.07, power limit 70 W, SM clock maximum 1590 MHz, 2 vCPUs, 13.61 GB RAM, Python 3.13.16) with soup-cli 0.77.0, torch 2.11.0+cu130, transformers 5.20.2, peft 0.21.2, trl 0.29.1 and bitsandbytes 0.50.2, plus two repairs from #2216 that the harness applied at launch, each only after checking that 0.77.0 lacked it (the first line of every log says so), because 0.77.0'ssoup bench traincould not run a 4-bit config. No process cap and nostream_vram_override. The same Llama-3.1-8B NF4 configuration (streamed, fp16, batch 1,max_length: 256,stream_buffers: 2, LoRA r=8, gradient accumulation at its schema default of 4), measured withsoup bench train --steps 28 --warmup 8, which is 20 counted optimizer steps per run, as runs A, B, A, B in fresh processes after one unmeasured priming run. Two data shapes, 32 rows each, and both belong to the result. Shape A, the proof notebook's own rows (37.5 tokens per sequence on average): total 45.49 and 44.87 tok/s (21.53 and 21.24 supervised), peak VRAM allocated 1.51 GB. Shape B, the same prompts with answers cut atmax_lengthso every sequence is exactly 256 tokens: total 232.93 and 233.09 tok/s (214.96 and 215.11 supervised), peak 1.93 GB. "Total" counts every position of every counted batch, padding included, which is what the pre-flight's bracket counts. The two runs of a shape agree to 1.4% (A) and 0.07% (B). Against the bracket published for the earlier session of the same configuration (288 to 423 tok/s) shape B is 0.81x of the low end and shape A is 0.16x of it; the record's reading is that "the bracket over-predicted, and never bound", an upper bound on this card and not a forecast. Shape B's peak is under the roughly 1.97 GB the pre-flight panel printed for batch 1 at 256 tokens (an over-prediction of 2%); shape A's shorter sequences make its peak not comparable. The clock is "a likely part of the gap, not shown as a cause": the busy SM clock's median was 750 MHz during shape B and 1530 MHz during shape A, and the board sat near its 70 W limit, but throttle reasons were not logged. What it does not establish, in the record's own list: one session on one Colab host (the repeats show spread inside it, not variation between free-tier hosts); clock and power observed, not controlled; and how the bracket's short startup probe relates to a sustained, power-limited rate was not measured. It is not a 4 GB result and is not set beside any RTX 3050 row. Its harness isharness/issue999_t4_bracket_check.ipynb(it installssoup-cli[train]==0.77.0and asserts that version) and its raw output is underresults/issue999-t4-colab/. - Caveats: The first two sessions make no throughput claim, and the pre-flight panel's forecast in them is a compute bound derived from a GEMM probe, not a measurement; the record marks the first run's forecast line stale (#714), and a corrected probe printed a different bracket in the later run. The third session, above, is the one that put a measured rate against a bracket. Exactness on Turing is shown for the forward of one small fp16 model and nothing more: not the 8B, not NF4 and not the backward, because non-zero gradients prove gradients flowed and not that they were correct. The first run is one run on one seed with no repeats, and its library versions are missing, so it is not exactly reproducible. The notebook install note that follows describes the notebook's own outputs, not the third session. The seven-step loss series is not evidence of learning. The notebook installs
soup-cli[train]==0.75.0from PyPI and capshuggingface-hubbelow 1.32.0, so its recorded outputs are from 0.75.0, not from the v0.76.0 tag; the notebook is unchanged at v0.76.1. - Read to the end: The run noticed by accident that the pre-flight read the device's free VRAM (15.10 GB), not the 4.00 GB cap the run was held to. On capped hardware the pre-flight is therefore not what enforces the real budget, which is the case
training.stream_vram_overrideexists for. The record says its list of what it does not establish is longer than its results on purpose. - More T4 records, added in v0.78.0: five other T4 records exist (NF4 aliasing, BCO and IPO, Fast-LoRA on one block, GRPO): four are a synthetic fixture or a single block, and the GRPO record runs real Qwen2.5-0.5B and 7B (NF4) checkpoints on synthetic prompts; none is a streamed-training throughput; they are listed below and on the free GPU tier page.
- On this site: An 8B on a free Colab T4 and running it yourself on a free Colab T4.
A10G: the VRAM fit on a second stack
- File:
gate-395-second-stack-vram.md. Filed against issue #395 and measured against Soup 0.73.3, and has been in a tagged release since v0.74.0. - Question: Does the long-sequence under-prediction from the v0.73.1 record reproduce on a second GPU and software stack, and is the leading hypothesis, a switch to the attention math path that materialises the full attention matrix, the cause?
- Hardware and stack: NVIDIA A10G 23.0 GB (AWS
g5.xlarge), Ubuntu 22.04, driver 580.126.09, torch 2.13.0+cu130, transformers 5.16.1, trl 0.29.1, peft 0.20.0, Python 3.10. The record says every number was measured in that configuration and nothing is extrapolated. Its transformers and peft versions are exactly the floor Soup now requires and its trl is one patch above that floor, whereas the laptop's are below it. - Result: It does not reproduce. The prediction over-estimates by a flat 1.162x to 1.169x from sequence 2048 to 6144 on SmolLM2-135M and Qwen2.5-0.5B (bf16, batch 1, LoRA r=8), against 0.934x at 5120 and 0.787x at 6144 on the RTX 3050.
- Status: A negative result for a hypothesis, published with a positive control.
- Caveats: The math path is exactly as expensive as the hypothesis needs, but SDPA does not select it on this stack. The hypothesis is therefore not supported here, and it is not refuted for the RTX 3050, whose Windows and torch 2.5.1 combination is where a math fallback is plausible and untested. The record says nothing about the RTX 3050 itself. Its accounting finds the retained-copy charge in the formula absent on this stack, which narrows issue #327 by one stack without explaining it, and every row is batch 1.
- Read to the end: A back-solved coefficient from an earlier revision was withdrawn once the instrument to measure it directly was run, and a method note records the harness trap behind the first attempt.
- On this site: The long-sequence under-prediction in the pre-flight and what was measured in v0.74.0.
L4: pinned host memory accounting
- File:
gate-655-pinned-store-accounting.md. Filed against issue #655, measured by Tristan Grech on 2026-09-10 UTC, and has been in a tagged release since v0.75.0. - Question: Does a pinned CUDA host allocation consume
/dev/shm, that is, does the streaming pinned-store path need a/dev/shmfree-space pre-flight? - Hardware and stack: NVIDIA L4, driver 570.195.03, PyTorch 2.9.1+cu128, CUDA runtime 12.8, Python 3.12.3, an Ubuntu 24.04.3 container on a RunPod host with kernel 6.14.0-33-generic.
- Result: Two fresh-process pinned 4 GiB allocations each added exactly 4 GiB to
RssShmemwith no change toRssAnonor/dev/shmusage, and the pageable control added exactly 4 GiB toRssAnonwith no change toRssShmem. - Status: A measurement record whose stated conclusion is a design decision: keep the
/dev/shmfree-space check out of the pinned-store path. - Caveats: It is one stack. It does not establish the driver's internal implementation, the behaviour on other drivers or kernels, or reclaimability, and it does not guarantee that a full streaming run fits. Host-wide counters can include other tenants, so the small background changes in those columns are not attributed to the allocation. The optional streaming test near 55% of host RAM was not run and no out-of-memory condition was induced.
- Read to the end: The original issue comment is reproduced without edits, including the host-counter drift, at the maintainer's request. The tables are in KiB, and the exact probe script is included in the file.
- On this site: What was measured in v0.75.0.
M4 Max: Qwen4-Exp, a partial pass
- File:
gate-qwen4-ple-m4-max.md. Dated 2026-08-31, belongs to v0.74.0 (theqwen4_expstreaming architecture), and has been in a tagged release since v0.74.0. - Question: Do the Qwen4-Exp and oQ decoder mapping and the read-only external N-gram (PLE) table work on Apple Silicon, and can the full production checkpoint be trained there?
- Hardware and stack: MacBook Pro M4 Max with 128 GiB unified memory and an external SSD. macOS, Python 3.12, PyTorch 2.13.0. The checkpoint is an oMLX/oQ affine bundle (
model_type=qwen4_exp, 48 decoder layers, 106.29 GB of source safetensors). - Result: All 1,167 expected tensors mapped, with none missing, unexpected or mis-shaped. The oQ affine decoder vectors matched MLX at 4, 5, 6 and 8 bits. The tiny native Qwen4 resident-versus-streamed gate was bit-exact on CPU for rows, logits, loss and LoRA gradients, and both MPS variants passed. The production checkpoint then reached "Training started!", but the one-step run stopped without completing an optimizer step.
- Status: A partial pass. The record marks checkpoint discovery, oQ decoding, expert mapping, cache construction, tiny CPU and MPS parity and production setup as PASS, a complete optimizer step on the 176.9B-parameter checkpoint as NOT VALIDATED, and Qwen4 resident-versus-streamed bf16 parity on CUDA as PENDING. Its own line: "Do not market this record as proof that the full model is trainable on a 128 GiB Mac."
- Caveats: It also says it must not be read as a throughput, peak-memory or production-trainability claim. It does not attribute the stop to host-RAM exhaustion, and the low-level cause is unresolved.
- On this site: What was measured in v0.74.0 and which architectures stream.
8 GB M1: MLX SFT
- File:
run-m1-8gb-mlx-sft.md. Filed against issue #23, which asked for exactly this and had been open since v0.25.0 because CI has no Apple Silicon, and has been in a tagged release since v0.75.0. - Question: Does
MLXSFTTrainerWrapperrun end to end against a real MLX runtime, and where is the memory ceiling on an 8 GB machine? - Hardware and stack: Apple M1, 8 GB unified memory, 8 cores, macOS 26.6.2 (arm64). Python 3.12.14, mlx 0.32.2, mlx-lm 0.31.3. The config is
backend: mlx,task: sft, LoRA r=8, batch 1,max_length: 512, on 48 synthetic chat rows for one epoch. - Result: The model the shipped
llama3.1-8b-sft-mlxrecipe names (mlx-community/Llama-3.1-8B-Instruct-4bit) trained on an 8 GB M1 under the smaller config above: 48 iterations, with the adapter written and reloadable (the record also carries a time and a peak-memory figure; this site publishes no MLX speed or memory figure). The 48 rows at batch size 1 withgradient_accumulation_steps: 1are 48 optimizer steps, and the harness now exits non-zero if rows times epochs does not equal the iteration count. The recipe's own settings (batch size 2,max_length: 2048, LoRA r=16, 3 epochs) were not run, and its catalog description says M2+ 16GB, so the record's wording that the recipe fits an 8 GB M1 is stronger than what was measured. A model too large for memory does not fail there, it pages, so an allocation-failure pre-flight would not fire. - Status: "FIVE RUNS COMPLETED, ONE FIXTURE FAILED. This is not a gate."
- Caveats: There is no correctness claim: the loss falls and the adapter reloads, and on 48 repeated rows a falling loss is memorisation. The LoRA is query and value only under
target_modules: auto. Throughput is an order of magnitude and not a benchmark, since re-running one model with the same config gave a rate 3.5 times different, so this page quotes none. It is one box with one run per rung, and DPO and GRPO are untouched. - Read to the end: The failed fixture and the corrections are left in. The small model the issue recommends ships legacy weight file names and cannot be loaded. A tok/s column that mixed two metrics was caught in review, and the author's own prediction that the 8B would not fit was wrong, which is why the row exists.
- On this site: The MLX backend and what was measured in v0.75.0.
Layer streaming on a second laptop (v0.76.0)
The records in this section were measured on a second laptop, which the records call the new dev box: an RTX 5070 Laptop GPU (Blackwell, sm_120, 8 GB), an Intel i9-14900HX, 31.7 GB of DDR5 RAM, two Samsung PM9B1-class NVMe drives (954 GB each) and Windows 11 Pro, on torch 2.14.0+cu130, transformers 5.17.0, peft 0.20.0 and bitsandbytes 0.50.2. The first of them says it is the first baseline on that card and compares nothing against a stored RTX 3050 number, and this page does the same.
Most of them share one fixture, and it is the first thing to remember about every number below: a synthetic Llama-3.1-70B-shaped NF4 store with random weights (80 layers, a 32k head, 36.39 GB on disk, larger than the machine's 31.7 GB of RAM), read through the disk tier at sequence 512 (batch 1 unless an entry says otherwise). It measures bytes moved and time. It says nothing about quality, no real 70B checkpoint was trained, and "70B-shaped" is a statement about tensor sizes. Units are decimal GB unless a record writes a power of two in GiB.
Disk tier: the asynchronous reader (#971)
- File:
gate-971-async-nvme-source.md. Belongs to #971, shipped in v0.76.0. The file changed at v0.77.0 (see Read to the end). - Question: The disk tier used to read each layer synchronously, on the compute thread, through a memory map. Does a background reader help, cold and warm, measured against a same-day control instead of a number from another session?
- Hardware and stack: The second laptop (driver 616.92, PCIe 5.0 x8 negotiated), with Soup at v0.75.0 plus the first five tasks of #971.
- Result: Cold, on the synthetic 70B-shaped store, the synchronous control stepped in 100.35 s and 92.25 s (early and late in the run) and the background reader in 48.09 s (run first) and 30.28 s (run last). The same configuration was 1.59x apart by run position, which is why the record states the cold gain as 2.1x to 3.1x depending on which position-matched pair is taken, and not as one number. Warm, with a 4.14 GB Mistral-7B NF4 store held in the page cache, the background reader was slower than the synchronous one it replaces, 1.20x in one run order and 1.03x with the order reversed.
- Status: "MEASURED, 2026-09-14, with one negative result and one confound that changed the headline", in the record's words.
- Caveats: Synthetic shape and timing only. What first looked like a read-ahead depth effect is consistent with the page cache warming across blocks, and the record publishes it as the ordering result it is. The box was shared with other sessions running test suites. The next record revisits the warm result and finds the regression it was filed over does not exist position-matched. The cold numbers were superseded by that record too, because its reader is the one that ships: cite this record for the controls and the warm-order lesson, not for 48.09 s and 30.28 s as a headline.
- Read to the end: It was revised on 2026-09-14 after a whole-branch review. Six claims its own data did not support were corrected in place, with the times, and nothing was deleted. At v0.77.0 it gained two dated corrections (2026-10-12) that supersede the record's explanation for the embedding entry (#975); its stall figures and its materiality ruling stand. Read the top, not a section.
- On this site: The disk tier.
Disk tier: direct I/O (#974)
- File:
gate-974-disk-tier-direct-io.md. Belongs to #974, shipped in v0.76.0. - Question: Is the warm regression that #974 was filed over real, and what bounds the cold read?
- Hardware and stack: The second laptop. Two fixtures: the 4.14 GB Mistral-7B NF4 cache (warm, it fits the page cache) and the 36.39 GB synthetic 70B-shaped cache (cold, larger than RAM).
- Result: The record finds the cold read bounded by buffered I/O: every buffered read shape topped out at 1.2 to 2.9 GB/s on this drive, while unbuffered reads of the same layers reached 3.5 to 5.65 GB/s. The reader that ships therefore reads each layer's data as sector-aligned byte ranges through direct I/O, four in parallel. Cold and position-matched, the 70B-shaped step took 16.2 to 17.3 s at both run positions (16.175 s and 17.311 s), against 110.5 s for the synchronous reader in the same slot and 34.8 to 45.6 s for the background reader it replaces. The read was 62 to 64% of the step, and peak VRAM was 4.378 GB. In the record's words the roughly 20 s target "is met".
- Status: "the warm regression #974 was filed over does not exist position-matched", in the record's words.
- Caveats: Synthetic shape, timing only, one machine. Windows only: the Linux
O_DIRECTand macOSF_NOCACHEpaths were read hard by reviewers and executed by none, and CI only exercises the buffered fallback. One synchronous-control block starved itself of memory and is reported, not built on. A first cold series was killed by another session on the box and discarded. The record measures the default of four parallel ranges only. - Read to the end: Two passages were corrected in the file: a finding in section 2 is corrected in section 8a, and an explanation of a shell defect in section 7 turned out to be a second session killing the probes. Read the top.
- On this site: The disk tier.
Disk tier: two-drive striping (#1536)
- File:
gate-two-drive-striping.md. Belongs to #1536, shipped in v0.76.0 (SOUP_LAYER_STREAM_STRIPE_DIRS). - Question: Does writing decoder layer i of the shard cache to drive i mod 2 speed up the cold step, under a decision rule committed before the striped cache existed?
- Hardware and stack: The second laptop: drive C: on a CPU root port, drive D: behind the chipset over a DMI 4.0 x8 link.
- Result: Sequence 4, three rounds of three timed steps after one warm-up each. Striped steps took 10.050, 9.971 and 9.929 s (median 9.971 s) against 17.552, 17.431 and 17.438 s on one drive in the same rounds, which the record gives as 1.746x, 1.748x and 1.756x. The striped and single-root caches held byte-identical tensors (2,405 of them).
- Status: "SHIP (2026-09-30, sequence 4)", in the record's words.
- Caveats: Synthetic 70B shape, direct I/O, cold. Exactly two drives in this one layout, batch 1, sequence 512. Three-step bursts only: the sustained probe did not show that the ratio holds for minutes. The two arms differ in read-ahead depth (2 on one drive, 3 on two), a confound the record argues is small and does not measure. The
soup trainsetup path (the wiring, the default depth and the NVMe checks) was not exercised. Three earlier sequences ended with no verdict (two on lost AC power, one on the rule's own validity row) and do not count. - Read to the end: A separation in the first-step loss that was noticed after sequence 2 did not survive.
- On this site: The disk tier.
Two-drive read rate
- File:
probe-rtx5070-two-drive-read.md. A probe under a rule committed on 2026-09-28, before the run. - Question: Does a second NVMe add read bandwidth: the #974 unbuffered read primitive on C: and D:, alone and both at once?
- Status: "NO FORMAL VERDICT", in the record's words. The rule's validity row fired in all three runs, and three of its further questions are ambiguous by their own rules. The record says the two-drive rate it records is a measurement, not a verdict, and that striping was instead gated on the training step, in the striping record.
- Caveats: A defect in the rule's own bound was disclosed, not corrected. No rate from this record is quoted here as a capability.
- Read to the end: The validity bound had been derived from a best-readings summary, and a count of "seventeen arms" was corrected to twenty-four. The status line was updated on 2026-10-01; until then it read "Results pending".
Two-drive sustained load
- File:
probe-rtx5070-two-drive-sustained.md. A probe under a rule committed before either sequence ran. - Question: Does the striping gate's 1.75x hold under sustained reading (each arm one process reading for about six minutes, three interleaved rounds), and not only in the gate's three-step bursts?
- Result: No verdict. In two rounds of the second sequence the late-phase ratio read 1.28x and 1.41x, which the record says is not a verdict, and the third round's striped arm was void twice (Windows Search's indexer read more than the 1 GB foreign-reader bar during it). The first sequence stopped when the laptop lost AC power and hibernated; its one complete round read 2.07x, also not a verdict. In the striped arms that reached it, both drives settled at a lower read rate than in the bursts for the rest of the arm. The record names no cause, and neither does this page.
- Status: "NO VERDICT (section 2 row 2), in both sequences", in the record's words.
- Caveats: The striping gate's 1.75x is therefore a figure for three-step bursts, and whether it holds for minutes is not shown. The record could not read drive temperature.
Drive throttle diagnostic
- File:
probe-rtx5070-drive-throttle.md. A diagnostic under a rule committed before the run, asking whether either drive slows down by itself under sustained reading. - Status: "NO VERDICT (section 2 row 1), 2026-10-02", in the record's words. The first position was void twice (Windows Search's indexer read over the 1 GB bar), so no second-drive or both-drives arm ran.
- Caveats: Its two void attempts are descriptive only, and this page quotes nothing from them. It exists in this index because a probe that produced no data is a result about the probe.
Batch control
- File:
probe-rtx5070-batch-control.md. A probe under a rule committed on 2026-10-01, before the run; it is also the control arm for the layer-major experiment below. - Question: On the cold 70B-shaped step at sequence 512, does throughput scale with batch 1, 2 and 3 as the model predicts (the step is the larger of the read and a compute floor that grows with the tokens), on one drive and on two striped drives, and what peak VRAM does each batch need on the 8 GB card?
- Result: Attempt 2 (2026-10-07) completed after one re-run of a void arm. Batches 2 and 3 fit: peak allocated VRAM was 4.378 GB at batch 1, 5.345 GB at batch 2 and 6.312 GB at batch 3, and the largest batch that fits is 3, the largest one run. On one drive the model predicted a near-free batch; batch 2 matched it and batch 3 did not, and the record says so ("DISAGREES AT b3"). On two striped drives both batches matched the model's "partial" class.
- Status: Attempt 1 ended in "NO VERDICT"; attempt 2 completed, and rows 1 to 3 of the rule did not fire.
- Caveats: Synthetic shape and the cold disk tier. The code is the head of the striping pull request, not byte-for-byte the shipped reader on one drive. Windows Search and other processes were running, and the first arm of round 0 read 9.4% slower for a reason the record does not explain. No tok/s from it is quoted here, because they are not a training throughput for a 70B.
- Read to the end: The upstream index row for this record still says "results pending"; the record contains the completed attempt, so the record wins.
What bounds a streamed step on the RTX 5070
- File:
probe-rtx5070-what-bounds-streaming.md. Dated 2026-09-12, with a section added after #974. - Question: The same question as the v0.73.0 probe, asked as the first baseline on the second laptop, plus the shipped disk tier warm and cold.
- Hardware and stack: The second laptop. The model is Mistral-7B-Instruct-v0.3 (an unsloth mirror, 112.5 MB per NF4 layer), not Llama-3.1-8B, because Llama-3.1 is gated and the box had no Hub token.
- Result: Cold, the synchronous disk tier as it then shipped took 124.5 s per step (spread 93.6 to 158.1 s) on the 36.39 GB synthetic store. Building that store surfaced a sharder defect on Windows: every source file stayed memory-mapped for the whole pass and each mapping charged commit for the file's whole size, so sharding a large checkpoint died with an access violation (#926). It is fixed in v0.76.0 (#934), which keeps at most two source files mapped at once. After the direct-I/O reader, the later section measured the cold step at 17.0 to 17.2 s at both sequence 512 and sequence 256, while the read-free compute time fell from 6.4 s to 3.8 s.
- Status: "THREE SECTIONS MEASURED", in the record's words: the RAM tier on a 7B NF4, the shipped disk tier on the same 7B with the page cache warm, and the shipped disk tier cold.
- Caveats: Mistral-7B, not the model of the RTX 3050 records. Three of the four ablation arms compute garbage by design, so they time and prove nothing else. The GEMM ceiling moved about 10% inside the session and clocks were point-sampled. The record reads its own ablations in several ways; this page quotes its measurements and none of those readings, and offers no explanation for the step.
- Read to the end: A passage about the Windows commit limit (sections 13 to 15) was wrong and was corrected on 2026-09-13: a memory mapping charges commit privately. Read the top.
Layer-major micro-batching, step 0
- File:
probe-rtx5070-l2l-step0.md. A probe under a rule committed on 2026-10-02 and amended once before any run of its two fixtures. - Status: "G1 and G2 PASS, G3 and G4 NO VERDICT", in the record's words.
- Caveats: This is not a Soup feature. Layer-major micro-batching is L2L's schedule (Pudipeddi et al. 2020), which upstream's records name as the prior art for streaming layers through a small device memory. Nothing in
src/implements it at v0.77.0, and no result from the probe is quoted here.
14B on an 8 GB card (#901)
- File:
gate-901-14b-on-8gb.md. Belongs to #901, fixed in v0.76.0. - Question: #901 reported that a 14B NF4 run on an 8 GB card dies inside
SFTTrainer.__init__with an out-of-memory error. Is it memory? - Hardware and stack: The second laptop (an RTX 5070 Laptop 8 GB with 31.7 GB of RAM). The fixture is a synthetic Qwen2.5-14B shape (14.77 B parameters, random weights).
- Result: It was not memory. A failed page-lock of the 9.93 GB store left a stale CUDA error that the first kernel launch reported as
CUDA error: out of memorywith 7.3 GB free. The page-lock itself failed because torch's caching host allocator rounds every pinned request up to the next power of two, so pinning the store one tensor at a time cost 1.73x its size. After the two fixes (drain the stale error; pin in power-of-two arenas) the same config completes through the realsoup train: the store is page-locked as 10 GiB (10.737 GB, 1.081x), peak VRAM was 2.944 GB allocated and 3.244 GB reserved at batch 1 and sequence 384 against the pre-flight's 3.39 GB prediction, and 18 optimizer steps finished with exit 0. - Status: "REPRODUCED, EXPLAINED, FIXED, 2026-09-15", in the record's words.
- Caveats: "No throughput and no quality claim is made here." The weights are random and the loss means nothing. This is the memory half only of the 14B-on-8-GB reference benchmark: the weights are random, so it says nothing about a real checkpoint. This page does not carry the earlier records' page-locked ceiling figures, which contradict each other.
- Read to the end: The author's first hypothesis, memory, was refuted by a 3 KB allocation failing, and the packer rule was revised on 2026-09-15 after three reviews.
- On this site: The disk tier.
Page-locked arena packing (#1702)
- File:
gate-1702-arena-packing.md. Belongs to #1702, shipped in v0.76.0. - Question: The pinned-arena planner also plans first-fit decreasing and returns it when it page-locks fewer bytes. Does the packed layout change the step time?
- Hardware and stack: The second laptop under a 4,000,000,000-byte per-process allocator cap. Qwen3-8B NF4 on the RAM tier, LoRA r=16 on q and v, batch 1, 512-token blocks, five warm-up and ten measured steps, five seeds.
- Result: The packed plan page-locks 6,442,450,944 bytes (three 2 GiB arenas) where the in-order plan page-locks 8,589,934,592 (four), for a 6,073,035,760-byte store of 1,154 tensors. Peak VRAM did not change (2.807 GB allocated, 3.047 GB reserved, in both plans), and three deterministic optimizer steps were byte-identical between the two plans (870 of 870 digests). The speed series completed 40 of 40 runs.
- Status: "memory saving confirmed, strict control passed, speed series run: NO DIFFERENCE measurable at +/-15.1%", in the record's words.
- Caveats: The claim is memory only. The 8 GB card runs under a software cap, which does not bound the CUDA context, the driver or page-locked memory. It is one model, and the record does not show that a page-lock refusal becomes less likely.
Mixture-of-Experts streaming and the new knobs (v0.77.0)
v0.77.0 added twelve records. Ten of them belong to layer streaming: eight to the Mixture-of-Experts work and two to new opt-in keys. They were measured on the same RTX 5070 Laptop 8 GB as the section above, or on that machine's CPU only, on development code before the v0.77.0 tag, and none of them names a person. Upstream's benchmarks index calls the RTX 5070 box a new baseline, so nothing in these records is set beside a figure from the 4 GB laptop. Several are correctness or capacity records on tiny models or random weights, their entries say so, and none of them is a throughput claim. The MoE work is described on Streaming a Mixture-of-Experts base and the keys on Scaling a streaming run. The other two v0.77.0 records are the mini_mmlu re-scale and a QuEST follow-up.
Expert streaming: the block gate (#2008)
- File:
gate-expert-streaming.md. A gate under a rule committed before the first run. - Question: Does streaming the routed experts one at a time give the same numbers as transformers' resident experts module on a real MoE?
- Hardware and stack: The RTX 5070 Laptop 8 GB and its CPU, Windows 11, torch 2.14.0+cu130, transformers 5.17.0, bitsandbytes 0.50.2. Three layers (0, 7 and 15) of
allenai/OLMoE-1B-7B-0924, which has 64 experts, fed random hidden states. - Result:
torch.equalheld in 27 of 27 checks (three layers, three cells, three tensors: the output, the input gradient and the routing-weight gradient), a maximum absolute difference of 0.0, in CPU float32, CUDA bf16, and CUDA bf16 with NF4 experts against the dequantised control. Each routed-to expert was read exactly twice, once in the forward and once in the backward (one block, no checkpoint recompute). - Status: "PASS", in the record's words, on a first run and again on a re-run after review.
- Caveats: One MoE block, not a whole model, and no DeepSeek-V3 weights. Its wall seconds are not a timing.
Moonlight-16B on an 8 GB card (#1914, #2029)
- File:
probe-rtx5070-moonlight-16b.md. A probe under a rule committed before any run. - Question: Does a LoRA step of Moonlight-16B-A3B, a
deepseek_v3base that does not fit the card resident, run under layer streaming and under expert streaming, and at what peak memory? - Hardware and stack: The RTX 5070 Laptop 8 GB with 31.7 GB of RAM, torch 2.14.0+cu130, transformers 5.17.0.
- Result: In
soup train, the same LoRA config peaked at 1.0 GB of GPU memory withper_expertagainst 3.1 GBfused(NF4 experts against bf16, different cache tiers), at about 2.7x the step time, one run each. The record's controlled pair reads 1.9 GB against 3.9 GB at about 2x. - Status: "both arms COMPLETE", in the record's words, and its check through
soup traincompletes too. - Caveats: Development code on a branch before the tag. The harness fed random token ids, and the fused store had just been written, so part of it was likely in the page cache and its reads are not cold-disk reads. The two
soup trainruns train numerically different bases. No quality or convergence claim, and this page quotes no tokens-per-second figure from it.
FP8 block-quantised sources: the shard gate (#2061)
- File:
gate-fp8-shard.md. A gate on CPU only, run on 2026-10-19 under rules written before any run. - Question: Does sharding a DeepSeek-V3 FP8 block-quantised checkpoint by dequantising it produce the weights an independent dequantisation produces, and does a model streamed from it behave like a resident model holding those weights?
- Hardware and stack: The RTX 5070 laptop's CPU only (CUDA was not available to the gate), 31.7 GiB of RAM, torch 2.14.0+cu130, transformers 5.17.0.
- Result: G1: the dequantisation equals an independent per-block reference. G2: on a tiny DeepSeek-V3, the model streamed from its FP8 copy matches a resident model holding the same dequantised weights bit for bit (logits, loss, LoRA gradients, both expert layouts), and the two caches hold the same tensors, NF4 included. G3: on Moonlight-16B-A3B converted to the format by the project, the 27-layer NF4 cache (30.60 GB) came within 0.0002% of the pre-flight's estimate.
- Status: "G1 PASS, G2 PASS, G3 PASS", in the record's words, CPU only.
- Caveats: No GPU. The Moonlight checkpoint is the project's own conversion, not an official FP8 release, and the record claims nothing about Kimi K2 or a 1T run.
FP8 sources: the GPU loss check (#2061)
- File:
probe-rtx5070-moonlight-fp8-source.md. A probe with limits set before the runs. - Question: Does LoRA on an FP8-block source of Moonlight-16B-A3B train like the bf16 source?
- Hardware and stack: The RTX 5070 Laptop 8 GB, torch 2.14.0+cu130, transformers 5.17.0.
- Result: Both arms completed. Two of the record's pre-registered limits failed, the per-step loss on a short row and the step time, and the others passed.
- Status: "FAIL (T2_max_step, T6_step_time)", in the record's words.
- Caveats: Speed parity and per-step loss parity are not shown, so nothing on this site says an FP8 source trains like its bf16 source. The record says its rule commit was made locally before either arm ran and never pushed, so "written before any run" cannot be checked from public history. Its sections 6 and 7 are analysis done after the verdict and change nothing; this page quotes neither.
Kimi K2: the admission gate (#2107)
- File:
gate-kimi-k2-admission.md. A gate on CPU only, run on 2026-10-21 and re-run after a review round. - Question: Can layer streaming admit Kimi-K2-Base and -Instruct through transformers' own
DeepseekV3Config, accepting a config only where it computes what K2's own code computes? - Hardware and stack: The RTX 5070 laptop's CPU only, 31.7 GiB of RAM, torch 2.14.0+cu130, transformers 5.17.0.
- Result: A tiny K2-like checkpoint (the published config at tiny dimensions) streams bit for bit like a resident
DeepseekV3ForCausalLMon the same weights: logits, loss and LoRA gradients,fusedandper_expert, from a bf16 checkpoint and from its FP8 block copy. The field-by-field config check fails as registered on one of 43 fields: transformers addsrope_thetaandrope_typetorope_scaling, as it does for any DeepSeek-V3 config. Over the published Base index, with no weights downloaded, the disk pre-flight sizes each layer of the cache exactly as an independent count from the shapes does. - Status: "G1a fails as registered on transformers' rope_scaling normalisation; G1b, G1c, G2 and G3 pass", in the record's words.
- Caveats: "No GPU, no K2 weight download, no K2 or 1T training run", in the record's words, and no NF4 model parity, since those tests run only on CUDA. The review round that hardened the guard is recorded after the results.
A step through a Kimi K2-shaped model (#2008)
- File:
probe-rtx5070-giant-moe-step.md. A probe; its second run followed an addendum rule committed before it ran. - Question: Can one LoRA training step run through a store built in Kimi K2's shape on the 8 GB card?
- Hardware and stack: The RTX 5070 Laptop 8 GB, with the store on NVMe.
- Result: In a measurement harness (not
soup train) on one RTX 5070 Laptop, a LoRA step through a model built in Kimi K2's shape with random weights (shape only; nothing was trained on K2) peaked at 4,723 MiB of VRAM and took 14.5-18.5 minutes. - Status: "COMPLETES", both runs, in the record's words.
- Caveats: "Every number in this record is 'random weights, shape only'", and the record calls the step "a capacity demonstration, not a training regime". Its first run used the model's own routing and, in the record's words, is not a K2 step time. Nothing here is about quality, convergence or a real checkpoint, and this page quotes no other figure from it.
The K2-shaped step at longer sequences
- File:
probe-rtx5070-giant-moe-seq-scaling.md. Pre-registered before any run. - Status: "NO VERDICT", run on 2026-10-23: one of its validity gates failed in three of four runs.
- Caveats: No ratio and no rate from it is quoted here.
A K2 layer on a 24 to 32 GB card
- File:
note-k2-layer-on-24-32gb-cards.md. A note, not a gate. - Status: "Arithmetic only: nothing here was built or measured", in the note's words.
- Caveats: An estimate about a design that is not built. This page quotes none of its figures.
stream_recompute (#2152)
- File:
gate-stream-recompute.md. - Question: Does keeping streamed layers' activations, instead of running the checkpoint recompute, shorten a step without changing the gradients?
- Hardware and stack: The RTX 5070 Laptop 8 GB (Blackwell), an i9-14900HX, Windows 11, torch 2.14.0+cu130, bitsandbytes 0.50.2, transformers 5.17.0. Mistral-7B-Instruct v0.3 in NF4 with double quantisation, layer-streamed, on the RAM tier unless a row says otherwise.
- Result: On the RAM tier a Mistral-7B NF4 step took 22% / 17% / 8.5% off the step time at 512 / 1024 / 2048 tokens, with no gain on a read-bound disk tier and slower at a high VRAM fill. Under deterministic algorithms the loss and the LoRA gradients were bit-identical to the recompute, which compares two streamed modes with each other, not a streamed run with a resident one.
- Status: "MEASURED on one laptop", in the record's words.
- Caveats: One laptop, one model,
task: sft. Sections 1 to 4 ran on an experiment branch, and the micro-benchmarks behind sections 1 and 2 stay there; this page quotes nothing from those two sections.
NF4 scale cache (#1892)
- File:
gate-1892-nf4-scale-cache.md. The decision rule was written before the first arm ran. - Question: Does
stream_nf4_scale_cacheshorten a streamed step end to end? - Hardware and stack: The RTX 5070 Laptop 8 GB with 34.1 GB of RAM, a development commit reporting soup-cli 0.76.0, torch 2.14.0+cu130, bitsandbytes 0.50.2, transformers 5.17.0. Mistral-7B NF4 streamed from a pinned RAM store, batch 1 at 512 tokens, five AB/BA pairs.
- Result: The measured window was shorter with the cache on, by a margin that met the rule. This page quotes no step-time figure for the key, as upstream's release body quotes none. A companion idea (
--train-threads 1on top of the cache) showed no gain and was dropped. - Status: "CONFIRMED on one laptop", in the record's words.
- Caveats: One laptop, one model, one shape. "Exactness is not shown here", in the record's words; that rests on upstream's tests.
Resident VRAM and one selective-checkpoint trial (v0.77.1)
v0.77.1 added three record files: two gate records and the pre-registered protocol of the second. Neither record is a throughput figure for soup train, neither was measured on a 4 GB laptop, and neither is set beside the rows on the other streaming pages. The first is about the resident (not streamed) pre-flight; the second is a benchmark-only trial of a change to a streamed run that nothing in soup train uses.
The shared VRAM estimate (#2049)
- File:
gate-2049-resident-vram.md, with every row inevidence/vram-2049-resident.json. Link the top of the file. Both are linked at v0.77.2: the record had 39 rows at v0.77.1 and has 42 now. - Question: How close does the one peak-VRAM estimate that
soup train's pre-flight,soup profile, the MCPprofiletool andsoup plannow share come to the peak of a resident run, astorch.cuda.max_memory_allocated()reads it? - Hardware and stack: An RTX 3060 Laptop GPU 6 GB (torch reports 6.442 GB), WSL2 on Windows 11, driver 610.60, torch 2.14.0+cu130, transformers 5.17.0, trl 0.29.1, peft 0.20.0, bitsandbytes 0.50.2, Python 3.12, on a development commit of
mainrather than the tag. The harness runssoup train's own setup and two training steps per row, because AdamW creates its state inside the first optimizer step, and each row in a process of its own. - Result: 42 resident
sftruns over seven dense models from 135M to 3B parameters (vocabularies of 32,000 to 151,936), at up to 4,096 tokens and batch 4. Rows 1 to 39 existed at v0.77.1; rows 40 to 42 were added for #2277 (short full fine-tunes of SmolLM2-135M at 64, 128 and 256 tokens, batch 1, where the optimizer step and not the forward and backward pass sets the peak: 2.754 to 2.757 GB whatever the length). Estimate over measured peak, with the estimate leaving out the CUDA context and allocator slack thatmax_memory_allocated()does not see: 0.994 to 1.064 for bf16 LoRA (20 rows), 0.915 to 1.086 for 4-bit (11), 0.969 to 1.164 for 8-bit (3) and 0.970 to 1.127 for full fine-tuning (8). A test holds every row to an allowed band for its kind of run. - Since v0.78.0: this record describes the estimator of v0.77.x (nine terms). v0.78.0 changed the estimate (an 8-bit checkpointing term, three distillation terms, padded SFT priced by its longest row, cuBLAS workspaces on a streamed run), and two newer records are listed below. "Calibrated on 42 rows" stays true of this record and is not a statement about the v0.78.0 estimator.
- Status: A gate record with no verdict line beyond that test.
- Caveats: In the record's words, "the coefficients are fitted on rows 1 to 39, so the bands above show how well they fit them, not the error on a run they were not fitted to": read the ratios as fit, not as an error bar on your run, and note that the three rows added in v0.77.2 are not among the 39 the coefficients are fitted on. One
adamw_torch_fusedrun (64 x 1) is reported in the record's text but is not among the 42 rows of its evidence file: the estimate, which charges a fused step no optimizer-step scratch, came out at 0.960 of that run's measured peak, just under the band, and the record says a fused figure fitted on one row would be a fixed constant the estimate does not take (decided on #2338). Seven rows allocated more than the 5.37 GB the card had free and ran on because WSL2 falls back to shared system memory; their peaks are real allocations, and the record says that on a native Linux driver those runs would have run out of memory. Only checkpointing off andfullwere measured; no model of 7B or more, no mixture-of-experts model, no fused loss, no Unsloth run and no streamed run is among the rows (the separate Unsloth observation added in v0.77.2 is described below, and it measured no training peak). The two streamed records the issue names,gate-v0.73.1-measured-vram-fit.mdandgate-395-second-stack-vram.md, do not calibrate the resident total. Upstream's printed name for the range in which the pre-flight refuses a run it says does not fit is "dense models of 3B and below on one 6 GB card". - Credit: The release notes credit the single estimate and its calibration to @abdulwaarith0 (#2146), and the short full fine-tune rows and the optimizer-step term to @abdulwaarith0 as well (#2277, #2338).
- On this site: The hardware-fit calculator. A streamed run keeps its own pre-flight (Scaling a streaming run).
Selective attention checkpointing of streamed NF4 (#2050)
- Files:
gate-2050-selective-checkpoint.mdand its protocol, written before the first Mistral-7B timing arm, with the rule and the source hashes frozen before the first full-size run. - Question: On a layer-streamed NF4 run, does saving the frozen Q/K/V (QKV) or Q/K/V/O (QKVO) base-projection outputs of every decoder layer, instead of recomputing them in the backward pass, shorten the measured step?
- Hardware and stack: One RTX 3090 on Windows, at its unchanged 350 W power limit. Mistral-7B-Instruct-v0.3 (a public mirror at a pinned revision) in NF4 with double quantisation, pinned RAM tier, two buffers, batch 1, LoRA on q and v at r=16,
PagedAdamW8bit, deterministic SDPA math with TF32 off. Each arm is a fresh process of 10 warm-up and 50 synchronised measured steps; each comparison is five AB/BA pairs, and there are four comparisons (sequence 512 and 2048, QKV and QKVO). - Result: The pre-registered rule asks for a median paired reduction of at least 2%, lower bounds above zero for both the ordinary 95% and the family-adjusted interval, exactness and a per-pair memory gate. One of four comparisons met it: saving Q/K/V/O at sequence 2048 cut the measured window by a median 3.141% (95% interval 3.027% to 3.432%) while holding 1.342 GB of retained outputs, with a largest allocated peak of 4.990 GB against 3.690 GB for full recompute. Q/K/V at sequence 2048 was inconclusive (median 1.916%, below the 2% rule). At sequence 512, Q/K/V was inconclusive, with one pair far slower that stays in the result, and Q/K/V/O was slower in all five pairs (NO GAIN). All 14 full-size gate and profile processes matched every logit, loss, all 128 LoRA gradient tensors and all 128 updated parameter tensors exactly across two updates.
- Status: "The predeclared rule passed for QKVO/seq2048", in the record's words, which add that "a separate opt-in/predictor follow-up can evaluate these qualified cases".
- Caveats: Benchmark-only: the installer is a harness helper, there is no Soup config field, and nothing in
soup trainuses it. A reduction here is less measured-window wall time in this harness, not a speed-up of asoup trainrun, and the trade in the confirmed cell is 1.342 GB of retained outputs for about 3.1% less time. The record states that its results "do not establish default flash-attention performance, performance on other GPUs, or a safe production default". It did not isolate the cause of the short-sequence timing variation or of the sequence-512 slowdown, and this page offers none; it also says the other three RTX 3090s in the machine were inventoried but their activity was not checked, so contention from the host cannot be ruled out. Five pairs give a small, discrete bootstrap sample, and the record says the bounds describe this experiment and do not guarantee population coverage. - Credit: The release notes credit the selective-checkpointing measurement to @Shutaru (#2161).
Added in v0.77.2
The release says it plainly: no gate ran for v0.77.2 and the preprint is unchanged. The directory still changed. Two records are new, three that were already there changed, seven harness files were added and one upstream measurement was published without a record file. None of this is a v0.77.2 gate, none is a throughput figure for the 4 GB rows, and the rule above holds: a record's number is compared only with another number from the same record.
- Three records changed. The free Colab T4 record gained an uncapped third session (its entry). The resident-VRAM record gained rows 40 to 42 and a finding about the optimizer step (its entry). The 8x H100 record gained a dated clarification under finding 9 (its entry). Their links are pinned to v0.77.2.
stream_loss_chunk, measured once and not in a record file (#1693). Upstream's performance documentation and issue #1693 hold a peak-VRAM table for one streamed step with and withouttraining.stream_loss_chunk: Llama-3.1-8B NF4, batch 1, LoRA onq_projandv_proj, one RTX 3060 Laptop 6 GB under WSL2, two passes. It found that a chunked step peaks in the backward of the last decoder layer and not in the loss, which is why the pre-flight now charges the larger of the two phases (#2290, by @ybai08).benchmarks/has no file for it, so this site prints no figure from the table. It covers separate q/k/v/o and gate/up/down projections only; fusedgate_upprojections, MoE layers and LoRA on MLP modules are not covered. The harness that produced it,stream_loss_chunk_vram.py, is listed under the scripts.
Unsloth 4-bit weight dtypes (#2301)
- File:
unsloth-4bit-weight-dtypes.md, with the observation inevidence/unsloth-4bit-weight-dtypes.json. - Question: What dtype are the input embedding and the output head under Unsloth 4-bit loading? The shared VRAM estimate has to price them, and it had priced them at fp32 as it does for Transformers.
- Hardware and stack: An NVIDIA RTX 5090 (compute capability 12.0, driver 570.211.01), Python 3.11.15, torch 2.8.0+cu128, Unsloth 2026.10.10, transformers 5.17.0, peft 0.21.2, bitsandbytes 0.50.2, with
unsloth/Meta-Llama-3.1-8B-bnb-4bit. Measured on 2026-10-30 on a development commit ofmain, not on the tag. - Result: After
FastLanguageModel.from_pretrainedand again afterget_peft_model(r=16, the seven projection targets), the input embedding and the output head are bothtorch.bfloat16, 2 bytes an element, shape 128256 by 4096, and they do not share storage. All seven targets received adapters in all 32 layers, 224 modules. - Status: An observation, not a gate.
- Caveats: In the record's words it is "a real weight load and LoRA injection, not a training-peak measurement": no forward or backward pass was run. It is one model on one stack, and it observes bf16 there, not every Unsloth model or a future version. Unsloth disabled xformers on a warning about the card's compute capability; the load and injection completed. The activation and logits coefficients remain uncalibrated for Unsloth, which still warns outside the calibrated range. The record also carries a table of the floors the estimator computes for a Llama-3.1-8B-shaped config; that is arithmetic a unit test pins, not a measurement, and this page quotes none of it.
- Credit: The release notes credit the Unsloth targets and 4-bit weight dtypes in the VRAM estimate to Ryzn (@ryznren, #2376).
- On this site: The hardware-fit calculator.
Fast-LoRA CPU bf16 (#2187)
- File:
fast-lora-cpu-bf16-2026-10-29.md, with the raw rows inrecords/fast-lora-cpu-bf16-20261029.jsonand the recorderrecord_fast_lora_cpu_bf16.pyunderharness/. - Question: The bf16 MLP test of the Fast-LoRA kernel (its gradient must stay within twice PEFT's own error against a float64 oracle) left its bound on seed 4 in the original report (#2187). Does it reproduce on a CPU, and does it depend on the instruction path or on the torch build?
- Hardware and stack: An AMD Ryzen 7 7840HS (8 cores, 16 logical CPUs) on Windows, Python 3.12.15, PEFT 0.21.2, transformers 5.19.0, one intra-op thread, and two separate environments, torch 2.8.0+cpu and torch 2.14.1+cpu. No GPU.
- Result: On torch 2.8.0+cpu seed 4 fails in the native (AVX512),
defaultandavx2modes. The failing gradient ismlp.up_proj.lora_B.default.weight(shape 48 by 4), at a ratio of 2.16 against the test's allowed 2, and the other eleven seeds pass. On torch 2.14.1+cpu all twelve seeds pass in all three modes, "but the seeded bf16 input changes between torch versions", so the record says the observation "does not establish that the kernel's numerical behavior improved on an identical input". - Status: An investigation. In the record's words, "no kernel or criterion change is proposed by this measurement record".
- Caveats: One CPU and one operating system. The capability overrides are the torch and ATen settings the issue asked for; independent BLAS dispatch was not constrained or inspected. The issue's original report, on a CUDA 12.8 build of torch 2.8.0, put the ratio at about 3.2; this host's CPU build reproduces the seed and the bound violation but not that exact figure. "No CUDA, performance, memory-saving or training-quality result is claimed." Read it as context for why the release body still says the correctness half of
training.fast_lorais open (#792): it is not a statement that Fast-LoRA is wrong in bf16, and not a statement that a newer torch fixes anything. - Since v0.78.0: #2546 changed the kernel: the SwiGLU LoRA input gradient is now added to the base gradient in fp32 and rounded once, where, upstream says, the earlier bf16 rounding lost an ulp in one gradient and left the bf16 MLP parity test failing on one seed. The record itself is unchanged and does not measure the changed kernel; this page does not say the bound violation is fixed or that the test now passes on a given torch build. The correctness half of
training.fast_lorais still listed as open upstream (#792), and the CUDA readings are below. - Credit: The release notes list the record among Nurkhan Esenbek's (@kok-o) benchmark, test and docs records (#2354).
- On this site: LoRA quality for what
fast_loradoes and does not claim.
Added in v0.78.0
v0.78.0 added 29 files to benchmarks/*.md (25 records, three pre-registered protocols and one evidence inventory), taking the directory to 84 records beside the README, and grew harness/ from 95 to 185 files. Count them as ls benchmarks/*.md at each tag: 56 files at v0.77.2 and 85 at v0.78.0, README included. The release body's own count of new records does not reproduce by that count (the directory gained 29), and this page does not use it. Three records that were already there changed (the last entry), and the README index changed with them. As before: none of these is a throughput figure for the 4 GB rows, a record's number is compared only with another number from the same record, and the status words are the records' own. Most were measured on machines that are not the 4 GB laptop: an RTX 5070 Laptop 8 GB, Kaggle and Colab T4 cards, an RTX 4090, shared H100 and H200 hosts, an RTX PRO 5000, an RTX 3090 and CPU-only fixtures.
Layer streaming against DeepSpeed, re-measured (STEP 3 of the H100 record)
- File:
gate-h100-deepspeed-compare.md, with its protocol underharness/ds_compare/PROTOCOL.md, committed before the first rental. - Question: Does layer streaming's advantage over DeepSpeed ZeRO-3 CPU offload reported at STEP 3 of the H100 record hold when DeepSpeed is set up from its own documentation, and where does streaming stand against DeepSpeed arms that keep the weights on the GPU?
- Hardware and stack: One rented H100 80 GB SXM (2026-11-07), EPYC 9684X, 1,511 GB RAM, torch 2.9.1+cu128, transformers 5.22.2, peft 0.21.2, deepspeed 0.19.9 with
cpu_adambuilt, soup-cli 0.77.2. Llama-3.1-8B-Instruct in bf16, LoRA r=8, batch 1, 256 steps, median of three. - Result: Status "all three parts MEASURED". Layer streaming at 61.99 tok/s and 2,921 MiB against ZeRO-3 with CPU parameter offload in a plain script at 39.79 tok/s and 4,979 MiB: 1.56x the throughput against a pre-registered threshold of 2.64x (the gate reads LOWER), at 1.70x less peak VRAM, a much smaller gap than STEP 3 reported. The record's first paragraph: "Layer streaming is slower than every DeepSpeed arm that keeps the weights on the GPU, at every setting that was run, and it is faster than DeepSpeed ZeRO-3 with CPU parameter offload by only 1.4x to 1.7x". At realistic row lengths the comparison with the fastest DeepSpeed arm stays reversed. Pinning the store was the one large effect in its ablation (throughput fell 65.6% with
stream_pin: false). - Caveats: One host; the record's header calls it a draft with two items marked pending (a tokenizer-file comparison and invoice totals); absolute throughput moved about 2.3x between two H100 hosts, so quote ratios; the cause of the gap to STEP 3 was not isolated (the Soup path, the library versions and the host all changed together); no run below 16 GiB, so the regime where only streaming and ZeRO-3 offload fit was not measured.
- On this site: External validation, which carries the table and every rider.
Forward-only streamed inference (#2509)
- Files:
gate-2509-stream-inference-throughput.mdand its protocol, committed before the first timed run (three source files frozen by sha256). - Result: On one RTX 5070 Laptop 8 GB (Windows 11), forward only, random input ids, one run per arm: G1 PASS (Mistral-7B NF4, pinned RAM tier, batch 1 x 512, 1,854.8 tok/s against a rule of at least 972) and G2 PASS (the best Qwen2.5-14B NF4 cell on the disk tier, batch 4 x 2048, peak 4.149 GiB allocated, 1,186.3 tok/s against at least 500).
- Caveats: Timing only, no quality claim, "a new baseline, never compared with RTX 3050 numbers"; the rule's 486.0 tok/s reference is a training step of that card, so forward throughput must not be set beside training rows. The first Mistral cell was still falling after two warm-ups, and another user's CPU-only job overlapped part of the run. No generation and no command exist.
- On this site: Layer streaming, forward-only.
BCO and IPO on a T4 (#2766)
- Files:
bco-streaming-t4.mdandipo-streaming-t4.md, by @here-2007. - Result: One Kaggle Tesla T4 (sm_75, FP16, torch 2.11.0, TRL 0.29.1), a two-layer vocabulary-64 random Llama, one optimizer step per cell. BCO: four cells passed (
bcoandpreferencewithpreference_loss: bco, NF4 and none, batch 2). IPO: the four default-setting cells failed (loss 25.0, NaN gradient norm); the diagnostic controls recorded the skipped first update and an unchanged adapter. With the scaler's initial scale lowered in the harness only, all eight cells applied an update, and, for the directipospelling without quantization (one streamed and one resident run, up to 12 attempts), the default setting skipped four updates and applied the fifth at scale 4096. - Caveats: Execution, update and adapter persistence only; "they do not measure throughput, large-model capacity or CUDA numerical parity", and no peak-VRAM claim. BCO's CUDA rows used batch 2; IPO's batch 1. CPU controls compare loss and every LoRA gradient exactly with an untied head and nonzero adapters. Neither record's check counts are printed here.
- On this site: The v0.78.0 section of the preference page.
NF4 streaming on a T4 (#842) and NF4 on Blackwell (#776, #331)
- Files:
gate-842-nf4-streaming.md(Kaggle T4, plus a Blackwell server card in a MIG slice) andnf4-on-blackwell.md("Status record, 2026-11-07. Not a gate"). - Result: The Blackwell record is the post-#1143 row: on one RTX 5070 Laptop (sm_120, bitsandbytes 0.50.2) the shipped streaming path matched a resident NF4 model on logits, LoRA gradients and a 20-step loss curve for Qwen2.5-0.5B-Instruct at sequence 64; the #776 difference is at most one ulp of the output dtype where bitsandbytes picks its fused kernel, "not a Soup defect", filed as bitsandbytes#2121; the #331 aliasing cause still reproduces on bitsandbytes
main. The T4 record's four-layer synthetic aliasing probe reproduced the old wrong gradient on the T4 and showed the repair removing it; its full-model arm was bit-exact with and without the repair ("the full-model negative control did not fire"). It also times one projection against a cached dense matrix: a comparison arm, not a shipped option, and no end-to-end claim. - Caveats: One 0.5B model, one sequence length and one card for the Blackwell row; sm_100, sm_121 and older architectures not measured; the harness does not itself prove which kernel ran for a given layer. The T4 record's layers are synthetic.
- On this site: Layer streaming, the exactness scope and External validation.
Resident 4-bit and the fp32 cast (#2609)
- File:
gate-2609-kbit-fp32-ab.md, pre-registered. - Result: On one rented RTX 4090 (450 W power cap), Qwen2.5-7B-Instruct NF4, LoRA r16, batch 8 x 512, gradient checkpointing, three seeds: unprofiled median step 2.2562 s (default) to 1.6949 s (24.9% shorter) with the embeddings, norms and head kept in bf16, and allocator peak 3.8 GiB lower (17,425 to 13,520 MiB), with the loss inside the tolerance written before the run. The record's verdict that the fp32 cast is the whole cause is PARTIAL (63.7% of the elementwise gap removed against a 70% bar; the three fp32 logits tensors, 7,128 MiB, are untouched).
- Caveats: Not measured: 8-bit, tied embeddings, other model families and batch shapes, another card, a 400 W cap, bf16 plus a fused loss path. The record is step 1 and changed no code; #2683 is the change. The two arms are an A/B on soup-cli 0.77.2 code, with the cast back to bf16 applied by the harness in the second arm: it is not a v0.78.0 against v0.77.2 run, and the shipped cast-back restores only tensors that loaded as bf16. It is a resident run, not a streamed one.
- On this site: The scope note on layer streaming and speed and memory.
The VRAM estimate: 8-bit with checkpointing, and the distillation teacher
- Files:
gate-int8-checkpointing-vram.md(#2860) andgate-distill-teacher-vram.md(#2909), gate records with tables and no verdict line. - Result: The 8-bit record has 47 two-step
soup trainruns on an RTX PRO 5000 Blackwell held to 12.18 GB by an allocator cap because the card was shared: pretrained 8-bit rows with full checkpointing read 0.637 to 0.933 of their measured peak before the new term (one model 1.003) and 1.048 to 1.147 after. The distillation record has 38 two-step runs with a tiny Qwen2 student and itself or Qwen2.5-1.5B-Instruct as teacher: estimate over measured peak 1.001 to 1.134 for token-level distillation, 0.111 to 0.445 before the new terms. - Caveats: Three pretrained models of 1.5B and below for the 8-bit term (plus random-weight rows kept for completeness); one student architecture and one real teacher for the distillation term,
uld_strategyandminillm_enablednot measured; one card;gradient_checkpointing: mediumis outside the calibrated range and only warns. Upstream's limits for the estimate as a whole are in the release's known limitations (tracked as #3021). Neither record promises a bound on your run. - On this site: The hardware-fit calculator.
Chunked distillation on an H200 (#722)
- File:
distill-chunked-h200-2026-11-10.md; kernel by @kok-o (#743), measurement by @AubakirovArman (#2686). - Result: A shared H200, Qwen2.5-0.5B-Instruct student with a frozen Qwen2.5-1.5B-Instruct teacher, batch 2 x 2,048 tokens, eight runs in an interleaved order: chunk size 128 with checkpointing lowered the total allocated peak from 31.656 to 21.400 GiB (32.4% less) and made the mean step 32.8% longer by medians. "This is a memory/time tradeoff, not a training speedup."
- Caveats: A shared card, "not a dedicated-device throughput ceiling"; step time excludes data loading and setup; a synthetic repeated text, no quality claim; chunking does not remove the full-model logits, so the prototype's ratio is not a prediction of total VRAM. The tok/s are not printed as a rate.
- On this site: Modality II for
distill_chunk_sizeanddistill_checkpoint.
Larger models on an 8 GB laptop (a capacity record)
- File:
run-big-models-8gb-laptop.md: the index calls it "Not a gate"; the record says of itself "This is a capacity record. It is not a throughput benchmark and it makes no model-quality claim for Mistral Large." - Result: Laptop: an RTX 5070 Laptop 8 GB with 31.7 GB of RAM and two NVMe drives. Three rank-8 q/v LoRA updates through every decoder layer at 1 x 128 tokens had a sampled whole-device peak of 3,002 MiB for Qwen3-32B NF4 and 4,598 MiB for Mistral-Large-Instruct-2407 NF4 (122.6 billion parameters, 88 layers), and 5,602 MiB for Mistral at 1 x 512; each saved adapter reloaded in another process and reproduced the held-out losses. Separate single runs through
soup train(3,042, 4,388 and 6,998 MiB) used a local integration tree and show reloaded logits that are not bit-identical to the training process's own (70.5 to 84.3% of elements differ after the cast; upstream traced the effect on a 0.5B model to LoRA's precision under autocast and did not re-measure it at these sizes, #2816). - Three kinds of numbers, never mixed: earlier recorded, re-run (Qwen3-32B only: identical losses and adapter hash, n = 1 and not the three planned, and a step time that differed for a reason the record says nobody has found; no step time is quoted here), and
soup train. The Mistral laptop figures are "from earlier recorded runs, not re-run with the runner that produced them". The record also holds a cloud run, on an L40S under a 6.25 GiB allocator cap and not the laptop: a 300-step Qwen3-32B Banking77 fine-tune measured on the full 3,080-item test split (69.578% to 82.825%, a paired bootstrap interval of +11.883 to +14.643 points, on the MMLU and GSM8K subsets neither general check fell, and both paired intervals have a lower bound of 0.0, so no change is shown in either direction), with the training process holding 5,052 MiB against 20,127 MiB for a resident control. That control reached 83.182% on the same split, 0.357 points higher than the streamed run (a paired interval of +0.032 to +0.682). - Caveats: No Mistral quality claim (its one pre-registered quality gate ended 22 of 32 against 22 of 32, not met); the Banking77 result is one task, one model, one seed and one rank-8 q/v adapter, possibly in the model's pretraining data (not tested), trained on Soup's own NF4 of the bf16 weights and evaluated on the published NF4; sampled peaks are lower bounds; the 16,384-token cloud run "is not a laptop result". Step times in it are diagnostic and are not quoted here. Nothing in it shows a 4 GB or 6 GB card working.
- On this site: Scaling a streaming run.
Selective projection retention (#2162)
- Files:
gate-2162-auto-qkvo.md,gate-2162-mlp-retention.md, the pre-registered protocol andissue2162-evidence-inventory.md. - Result: Mistral-7B NF4 on an RTX 3090 capped at 8 GiB, pinned RAM, batch 1 x sequence 2048, 50 measured updates after 10 warm-ups per process. Keeping only the Q/K/V/O projections against native
stream_recompute: autois INCONCLUSIVE: the primary reduction fell just under its frozen 2% threshold (so it is not a result to quote as a speedup), scoped dispatch did not reliably improve, and "no control or secondary result is promoted into its place". A hand-chosen layout that also keeps the MLP gate and up projections passed its own frozen rule at a higher allocated peak, and sits outside the original QKVO-only scope. - Caveats: Measurement only: the record "added no production field or default", and no retention option exists. The design changed between stages after earlier results were known. This page prints no percentage as a speedup. The evidence inventory says the bulk evidence is kept outside Git, and without it both audit commands report
numbers-only.
Kimi K2, resident (#2104)
- File:
gate-kimi-k2-resident.md: status PASS (2026-11-12, CPU only). - Result: A seeded tiny K2-shaped FP8 checkpoint loaded through the resident route at float32 reproduced the final-position logits and the loss that K2's own pinned
modeling_deepseek.pyproduced, atrtol=1e-5, atol=1e-6, on three fixed inputs (lengths 8, 16, 33); the record's controls fell outside the tolerance and three source mutants were caught. - Caveats: "No GPU, no real K2 weights, no K2 training run, no throughput." A regeneration of the checkpoint under another CPU dispatch changed it, so its weights are pinned by hash.
- On this site: Streaming a Mixture-of-Experts base.
Flash-Attention and Liger on an H200 (#378)
- Files:
flash-liger-h200-2026-11-10.mdand its pre-registered protocol; by @AubakirovArman (#2691). "A second Hopper measurement": it "does not reproduce or replace the earlier H100 operating point, and changes no product code or default". - Result: One shared NVIDIA H200, Llama-3.1-8B resident bf16 with LoRA r16, batch 4, real
SFTTrainerWrapper.setupandTrainer.train, sequence lengths 1,024 to 8,192, checkpointing off and on, four arms (default SDPA, FlashAttention 2, Liger, both), five fresh-process repetitions per feasible cell. Ratios of five-run medians to SDPA: Liger alone 1.197 to 1.229x with 21.6 to 54.7% less allocated peak; FlashAttention 2 alone 0.929 to 0.989x with about the same peak; both together 1.096 to 1.195x, below Liger alone in every feasible cell. Eight cells ran out of memory in the probe (checkpointing off at 4,096 and 8,192 tokens, all four arms) and are kept as not run. - Caveats: "Ratios of five-run medians on this model, fixture, stack and shared card, not universal speedups or a training-quality recommendation." The SDPA baseline already ran cuDNN flash attention, so FlashAttention 2 gaining nothing is a statement about that baseline. The card was shared (an unrelated process held 49,850 MiB), so an out-of-memory cell is "not an unconditional H200 capacity limit". Tokens per second count input tokens and exclude loading and the data loader. The existing Liger path disables Soup's logits-based rewind recorder, because its fused loss exposes no logits. No held-out quality evaluation, and no FlashAttention 3 figure.
- On this site: Backends for
use_ligeranduse_flash_attn, and speed and memory.
Fast-LoRA: two CUDA readings that disagree (v0.78.0)
training.fast_lora now has two CUDA readings, and they do not agree on direction. Both are one block on one card, neither is an end-to-end result, and upstream's reference still says no speedup is claimed, as does this site.
- The Q/K/V harness, recorded once and with no record file. The index row of
fast_lora_qkv_bench.pyinbenchmarks/README.mdchanged from "Not yet run on a GPU: there is no record for it" to "Recorded once, on one card": two passes of three processes per cell on an RTX 3060 Laptop (6144 MiB, sm_86) atmain294e2ae49, posted on issue #838 on 2026-10-31 and completed on 2026-11-01.fasterin all 8 cells in both passes, fast over plain 0.8798 to 0.9779, peak allocated memory 32 to 37 MB lower on the fast arm, with no void, crashed or out-of-memory row. The results live in two GitHub issue comments and that index row, not in a record file; the README calls it "a new baseline, not comparable with other cards, and no end-to-end claim", and the four NF4 cells are to be run once more now that the dtype-boundary fix (#2383) is in. - The SwiGLU MLP block on a T4 (
fast-lora-mlp-cuda-t4-2026-11-08.md, #837, by @capyriver). One synthetic Llama-3.1-8B-shaped MLP block (N=2048, d=4096, f=14336, r=64, adapters on all three projections) on one Colab T4, no language model loaded, bf16 emulated and not native, NF4 with bf16 compute, one fresh process per dtype. Median forward plus backward: Fast-LoRA 595.26 ms against plain PEFT 582.25 ms in emulated bf16, and 673.56 ms against 652.18 ms in NF4. The record's verdict: "no measurable difference" in time, with no noise threshold fixed beforehand, so "there is no speed verdict". Fast-LoRA's peak allocated memory was higher in both dtypes (1,306.2 and 1,128.9 MiB against 745.8 and 720.4 MiB for plain PEFT), "this run did not capture an allocation breakdown that explains the extra peak, so its cause remains unknown". Outputs matched plain PEFT exactly. The record's own evidence file "was assembled by hand from that run's Colab notebook output; the original raw harness JSON was not retained". - How to read the two: the two readings disagree, so neither "Fast-LoRA is faster" nor "Fast-LoRA saves memory" follows (the T4 peak is higher). The QKV range is one card, eight cells and an index row only.
- On this site: Speed and memory and LoRA quality.
Packed LoRA kernels, GPU tests (v0.78.0)
- File:
packed-lora-gpu-tests-2026-11-08.md: one GPU-test run on an RTX 5070 Laptop 8 GB under WSL2, "the card was shared with other jobs; nothing below is a speed measurement". - Result: Tiny random 4-bit models: loss within 5.6e-5 (bf16) and 6.0e-6 (fp16) of the stock forward, worst LoRA gradient error against the stock step 1.0 to 2.4% in bf16 and 0.3 to 1.5% in fp16. On Qwen2.5-1.5B-Instruct (four rows of up to 96 tokens), gradient cosine to the stock step 0.99899 to 0.99939 in exact mode and 0.99409 to 0.99550 in the default mode. The planner's estimate for a 3,000 MiB budget at 8 x 512 tokens, 2,800 MiB, against a measured torch peak of 2,837 MiB.
- Caveats:
soup_cli.kernels.packed_lorais a library: it has no config key andsoup traindoes not read it, so nosoup trainclaim follows. The gradient figures are for tiny random models and for Qwen2.5-1.5B on four rows.
QuEST on an RTX 3050 (#674)
- File:
quest-674-rtx3050-capacity.md, by @capyriver (#2715). - Result: In the record's words: "Eight of eight optimizer updates completed at batch size 1 on 32 synthetic ChatML rows (45 maximum formatted tokens), without OOM." The record says its allocator counters are unexplained and claims no reliable process-specific peak: "This is not a parity, quality, or efficiency result."
- Caveats:
ahxt/LiteLlama-460M-1Twithquantization_aware: questandlora.r: 0on an RTX 3050 Laptop 8 GB; Soup reported W4 and A4 paths that are "neither pure W4A4 nor packed INT4". This page publishes no peak from it.
GRPO on T4 cards, and the H100 non-finite note (#342)
- Files:
run-342-kaggle-t4-grpo.md("Not a gate") andrun-342-h100-nonfinite-workflow.md. - Result: Four fp16 GRPO arms on Kaggle T4 cards, on synthetic prompts whose reward "did not score answer correctness": Qwen2.5-0.5B ran 24 steps with every loss and gradient norm finite; Qwen2.5-7B in NF4 with eight generations ran out of CUDA memory before the backward; with two generations it ran 16 finite steps, and a longer run reached step 400 with one watchdog trip at step 62. The H100 note is a source audit and a set of CPU controls, and in its words "the historical incident still lacks a captured failing batch".
- Caveats: "These results provide no evidence for or against the original H100 BF16 failure in #342." The first three arms call TRL's trainer directly with Soup's watchdog attached by hand; no root cause is determined.
AutoDistill student on CUDA (#580)
- File:
autodistill-student-cuda-2026-11-02.md, by @kok-o (the release body credits #2443). - Result: One RTX 3090: the record's CUDA-marked cases and its whole numerical and tiny-student test file all pass with none skipped, on a tiny Llama (vocabulary 8, hidden 16, one layer), float64 on the GPU cases.
- Caveats: "No peak-memory, pretrained-model quality or complete AutoDistill workflow gate was run." It adds no training task.
Apple Silicon: a dense full fine-tune fixture, and a record whose code is not in the tag
mlx-full-ft-1992.md(record, #1992, #2818): an Apple M4 16 GiB and a tiny two-layer Llama, twelve cases covering updates, parameter changes and exact reload. The record says "no large-model memory or throughput claim follows from this tiny fixture" and that the instrumented peaks include test order and prior allocations, so it cannot show a memory saving from freezing; the larger-model memory figures in it are projections, not measurements. This page prints no MLX peak, time or memory figure from it.run-2005-mlx-simnpo-m6.md(record, #2005): an Apple M6 24 GB, SmolLM2-360M-Instruct, verdict FAIL on its pre-registered held-out forgetting check on all three seeds. The code it measures is not in a tagged release (#2646): at v0.78.0backend: mlxwithtask: unlearnis refused at config load with "task='unlearn' is not supported on backend=mlx: there is no MLX unlearning path. Use backend: transformers." (soup trainexits 1,soup doctor --configexits 2; run on the v0.78.0 binary). Read it as a failed check of unreleased code, not as a capability. It "does not establish erasure, privacy protection or compliance".
ONNX export compatibility (#2676)
- File:
onnx-export-compatibility-2026-11-11.md, by @AubakirovArman (#2695). CPU only, two tiny randomly initialised GPT-2 and Llama fixtures. - Result: The released Optimum and optimum-onnx packages do not import under Transformers 5.19.0, Optimum 1.16.0 fails and an unmerged optimum-onnx pull request fails GPT-2 export; the released packages in a fresh export-only environment pass
soup export --validate-onnx(maximum absolute error 5.96e-08 for GPT-2 and 7.45e-08 for Llama atatol1e-4) on a legacy-compatible copy of the fixture, "The checkpoint's config and tokenizer must be readable by Transformers 4.57.6". Four corrupted graphs were refused. - Caveats: Not a model-quality, large-model, cached-generation or Windows or macOS result; "Soup's export backend/default has not changed". Upstream's documentation says to install only
soup-cli[onnx]in a separate export environment, not[train]and[onnx]together.
soup eval quant-check performance smoke (#2677)
- File:
quant-check-performance-smoke-2026-11-10.md, by @AubakirovArman (#2689). In its words, "functional smokes ... not a speedup benchmark or a model-quality evaluation. Three toy prompts are insufficient to establish retained model quality." - Result: On an H200 (shared), Qwen2.5-0.5B-Instruct in fp16 against a locally exported bitsandbytes NF4 copy, three prompts, both 3 of 3, the NF4 allocated peak below fp16's, exit 0; on identical tiny CPU checkpoints a latency difference can appear in so short a measurement and the CUDA memory fields are null.
- Caveats: The record's tests and hashes describe an earlier tree. It prints a total of a full CPU suite run, which this site does not republish.
Records that changed in v0.78.0
gate-h100-validation.md(the 8x H100 record) gained two dated annotations (2026-11-07), in its summary and at STEP 3: the DeepSpeed comparison is "superseded as a comparison" by the re-measurement above, the original rows being kept as the record of what was run. Its link on this page moves to v0.78.0.gpu-test-runs.mdgained one row (2026-11-08): the packed-LoRA GPU tests on an RTX 5070 Laptop 8 GB under WSL2, shared with other jobs (above). Nothing in it is a speed measurement.README.md, the index, gained 29 rows and the harness rows, and itsfast_lora_qkv_bench.pyrow changed (above). It still prints the old DeepSpeed headline in thegate-h100-validation.mdrow, next to the new row that says the comparison is superseded, so the index disagrees with itself there: the records win.
Decoding and the benchmark command (v0.76.0)
CUDA graph decoding
- File:
gate-v0.76.0-cuda-graph-decode.md. The pre-declared gate that decided whethersoup infer --cuda-graphsandsoup bench infer --cuda-graphsship. It calls itself a new baseline on the RTX 5070 box and says it is not comparable to any RTX 3050-era number. - Question: The experiment behind the flag reported its result with the process pinned to one CPU core, and its own caveat was that the graph path had never been measured unpinned, which is how users run it. Does the flag still win unpinned, under criteria fixed before any run?
- Hardware and stack: The RTX 5070 Laptop 8 GB, torch 2.14.0+cu130, transformers 5.17.0, Windows. Qwen2.5-1.5B-Instruct resident in FP16, batch 1, greedy, an 82-token prompt and 128 new tokens.
- Result: On one RTX 5070 Laptop, unpinned, Qwen2.5-1.5B in FP16, the slowest run with graphs decoded 2.9x faster than the fastest run without (2.902x for the full model, 2.763x with one unmerged LoRA adapter). That is a per-run worst-case bound, pre-declared as the criterion, and not an average. Every full-model run with graphs decoded at 60.1 to 62.7 tok/s and every full-model run without at 13.5 to 20.7; with the unmerged adapter the runs with graphs read 49.2 to 60.3 and the runs without 8.1 to 17.8. The unpinned baseline is the unstable arm, and pinned to one core the same box measured 31.3 and 65.6 tok/s. Token ids were identical on all 84 requests, reserved VRAM rose by 42 MiB (40 MiB with the adapter), and
soup inferover 24 prompts end to end gave identical responses with and without graphs. - Status: "Every criterion passed", in the record's words.
- Caveats: Opt-in and experimental, torch 2.14 or newer, resident unquantised Qwen2 or Llama models only. Qwen2 was measured; Llama is accepted by the validator and covered by a tiny-model GPU test, not by this measurement. It is batch 1, greedy, 128 tokens, FP16, on Windows. The record says the absolute unpinned numbers may read low and must not be quoted as what a user gets.
soup chatandsoup servewere not measured, nor quantised or offloaded models, nor throughput at a temperature above 0, nor long--max-tokens(each key-value update costs extra traffic, unmeasured). - On this site:
soup benchand the CUDA graph flag.
soup bench train: the contract
- File:
gate-836-bench-train-contract.md. Acceptance item 7 of #836 asked for a published reference run ofsoup bench train; this is it, and the record says the useful half is not the reference number. - Question: Does the command report what it says it reports, refuse when the measurement cannot mean anything, and not fabricate an absent value?
- Hardware and stack: The RTX 5070 Laptop 8 GB, on battery at a 50 W cap, with two small models and one sequence shape each.
- Result: Across thirteen runs of a byte-identical config, the config hash, the supervised and total token counts,
max_memory_allocated,max_memory_reservedand all three checks were identical to the byte, while the reported throughput ranged from 376.4 to 915.3 supervised tokens/s (a 2.43x spread, coefficient of variation 35.3%). The spread is additive rather than multiplicative: across four shapes whose step time spans 26x, the fastest and slowest repeat of each differ by a roughly constant 0.17 to 0.28 s per step, which is the number a future comparison has to clear. One shape completed while allocating 12,879.61 MB on an 8,547 MB card, because the Windows driver spilled to host memory, so the reportedmemoryis the allocator's number and not a fit. - Status: "the contract holds; the number it reports does not repeat", in the record's words.
- Caveats: "This is not a throughput claim": one card, on battery, at a 50 W cap, with no quality claim of any kind. Four refusals exit 1.
- Read to the end: Two hypotheses for the spread (peer processes, CPU frequency) were refuted by the record's own covariates, one contaminated run is kept, and a field that timed the clock after the run was replaced by one that samples it while the run is busy. There is an addendum dated 2026-09-23.
- On this site:
soup bench.
Re-scaled scorers behind soup ship (v0.76.0 and v0.77.0)
Three bundled suites behind the regression leg of soup ship had reached a ceiling: a strong reference model scored 1.000 on them, so a regression could not show up as a drop. v0.76.0 re-scales mini_tool_call, mini_arithmetic and mini_common_sense to 40 rows each, and v0.77.0 re-scales mini_mmlu too. The bundled scorer revision moves from 1 at v0.75.2 to 5 at v0.76.0 and 7 at v0.77.0, and a stored --baseline stamped at an older revision draws a warning instead of a silent comparison. The tool-call record was measured on an Apple M4 Max and the other three on an Apple M1 with 32 GB, so scores are never compared across the four records. None of them has a verdict line.
Tool-call discrimination (#405)
- File:
gate-v0.76.0-tool-call-discrimination.md. - Question:
mini_tool_callhad reached a ceiling: the strong reference chose the right tool on every shipped row. Which replacement leaves the ceiling without losing the weak model's floor? - Hardware and stack: An Apple M4 Max (128 GB, macOS 26.5.2, MPS), Python 3.12.2, torch 2.12.1, transformers 5.12.1, checkpoint-native dtype, greedy decoding, a 256-token cap. The strong reference is Qwen2.5-7B-Instruct and the weak one SmolLM2-135M-Instruct.
- Result: The v0.73.2 fixture scored 1.000 (40/40) on the strong model and 0.075 (3/40) on the weak one. A first 32-row candidate scored 0.969 (31/32) and was rejected as still pinned near the ceiling. The accepted fixture keeps 16 legacy positive tool calls and adds 16 direct-answer decisions and 8 tool selections with close distractors; it scored 0.850 (34/40) and 0.050 (2/40). A direct-answer row passes only when the whole trimmed output is the literal
NO_TOOL, so a fencedNO_TOOLfails, while positive JSON tool calls tolerate fences. The two scorer repairs from #346 stay as controls. The scorer revision moves from 1 to 2 in this record. - Caveats: A follow-up run on Mistral-7B-Instruct-v0.3 in NF4 missed 12 rows on the
NO_TOOLaxis: 10 were real over-calls and 2 were outputs that began withNO_TOOLand then explained, so about 17% of those misses come from the deliberately strict packaging. At that point the record also notes the strong model was still at 1.000 on several other suites; it records the rails rather than widening the issue.
Arithmetic discrimination (#1192)
- File:
gate-v0.76.0-arithmetic-discrimination.md. - Question:
mini_arithmetichad measured 1.000 (36/36) on the strong reference in an earlier record. Does a revised 40-row fixture leave the rails on models that do not sit on one? - Hardware and stack: An Apple M1 with 32 GB, checkpoint-native dtype, greedy decoding, a 256-token cap. The strong reference is Qwen2.5-1.5B-Instruct, because Qwen2.5-7B-Instruct did not run on that machine; the weak one is SmolLM2-135M-Instruct.
- Result: The first 24 single-step rows are kept and 16 multi-step or large-operand rows replace the other 12. No new answer appears as a standalone token in its own question, so a model that echoes the prompt cannot score. Qwen2.5-1.5B scored 0.800 (32/40) and SmolLM2 0.375 (15/40), the weak model solving 15 of the 24 kept rows and none of the 16 new ones. Half of the strong model's eight misses are truncation at the 256-token cap and not arithmetic, and the cap is the one
soup shipuses. The scorer revision moves from 3 to 4 in this record. - Caveats: The strong reference is a 1.5B model, so the record shows the fixture leaves the rails for it and not for a 7B model. The run predates #1335, so its safety and over-refusal rows are on the earlier refusal classifier. The weak model's other suites differ by one or two rows from the earlier record, which the record puts down to machine drift between an M1 and an M4 Max.
Common-sense discrimination (#1192)
- File:
gate-v0.76.0-common-sense-discrimination.md. - Question:
mini_common_sensehad measured 1.000 (24/24) on the strong reference, and its answer key was lopsided: 14 of 24 answers were B, so a model that only ever answered "B" scored 0.583. Does a revised 40-row fixture fix both? - Hardware and stack: An Apple M1 with 32 GB on MPS, checkpoint-native dtype, greedy decoding, a 256-token cap, software versions not captured. The strong reference is Qwen2.5-1.5B-Instruct and the weak one SmolLM2-135M-Instruct.
- Result: The final fixture keeps the 24 original rows, appends 16 rows selected from a 24-row candidate pool, and swaps two options in four original rows, so no letter answers more than 30% of the suite (the key is now A 11, B 11, C 12, D 6). Qwen2.5-1.5B scored 0.850 (34/40) and SmolLM2 0.275 (11/40). That is chance level and not a skill level: uniform guessing over these rows expects 12/40 = 0.300, and a constant "A" or "B" scores 0.275, exactly SmolLM2's score. The scorer revision moves from 4 to 5.
- Caveats: As in the arithmetic record, the strong reference is a 1.5B model, so nothing is shown about a 7B model. The four candidate rows rejected because the strong model chose A on facts it plainly knows are read as a letter preference, which is a reading and not a measured bias.
MMLU-style discrimination (#1192)
- File:
gate-v0.76.0-mmlu-discrimination.md. New at v0.77.0 despite its name; the release body credits @AaronProbha18 for it (#2113). - Question:
mini_mmluhad measured 1.000 (26/26) on the strong reference, and its key was lopsided: 12 of 26 answers were B, so a model that only ever answered "B" scored 0.462. Does a revised 40-row fixture leave the rails and balance the key? - Hardware and stack: An Apple M1 with 32 GB on MPS, checkpoint-native dtype, greedy decoding, a 256-token cap. The strong reference is Qwen2.5-1.5B-Instruct and the weak one SmolLM2-135M-Instruct.
- Result: The suite grows from 26 to 40 rows and no letter answers more than 30% of it (the key is now A 11, B 12, C 11, D 6). Qwen2.5-1.5B scored 0.850 (34/40) and SmolLM2 0.350 (14/40). The bundled scorer revision is 7 at v0.77.0, and a baseline written by v0.76 is not compared silently on the new scale: write it again.
- Caveats: "These numbers show the fixture leaves the rails for a 1.5B model. They do not show that a 7B model is off the ceiling", in the record's words.
soup ab: three records, two superseded
soup ab changed its statistic twice in this window, and the three records are kept as a chain: each carries a "Superseded by" banner naming the record that replaces it. All three are CPU simulations on Gaussian rows (binary metrics should be pre-aggregated per prompt) on Windows with Python 3.12 and numpy 2.5. None uses a model, and this page quotes none of their simulation figures. soup ab is documented with the trace commands.
soup ab: the burn-in (#1227)
- File:
gate-1227-ab-burn-in.md. - Question:
soup abran a two-sided mSPRT and withheld every verdict until each arm had a minimum number of rows, because an operator who re-runs the command after every new pair would otherwise reject a true H0 far more often than--alpha. How many rows are enough? - Status: "Superseded by #1265", in the banner's words:
soup abnow uses a variance-robust statistic with no burn-in, andBURN_IN_ROWS_BY_ALPHAandCALIBRATED_HORIZON_ROWSare gone. - Caveats: The record states that its alpha-0.01 tier passes only on its own seed set. It stays as the baseline the next record compares against, and is not a description of current behaviour.
soup ab: the mixture (#1265)
- File:
gate-1265-ab-nig.md. - Question: Can a normal-inverse-gamma Bayes factor in its right-Haar limit replace the burn-in, keeping the Type-I rate under peeking without withholding verdicts?
- Status: "Accept rule superseded by #1418", in the banner's words. The statistic,
PRIOR_SCALE_ROWSand the reject rule recorded here are unchanged. - Caveats: The record says its sweeps predate the accept hold (
ACCEPT_HOLD_SPREAD_RATIO, #1419), so every number in it comes from a run with the hold off.
soup ab: the accept rule (#1418)
- File:
gate-1418-ab-cs-accept.md. Shipped in v0.76.0. - Question: Can a confidence-sequence accept rule replace the Bayes-factor threshold without wrongly accepting a true difference of exactly the effect size and without moving the reject rule's Type-I rate or power?
- Result:
soup abaccepts H0 once a confidence sequence from the same mixture lies inside plus or minus the effect size, instead of when the Bayes factor falls to a threshold.--betais retired: v0.76.0 listedsoup ab --betaas deprecated and ignored it, and since v0.77.0 it is refused with exit 2. - Caveats: A simulation on Gaussian rows only, compared against designs that were not released; this page states no speed-up or rows-needed figure. The reject rule is #1265's.
QuEST: a failed gate and three sequels
Four records follow one contributor's research experiment. The first failed its own preregistered criterion and is published as written. The second passed an evaluation-only gate on a different, mixed route frozen afterwards, the third remained above its target, and the fourth, added in v0.77.0, still fails its parity gate. Soup gained an experimental training.quantization_aware: quest path in v0.76.0 (#1105) whose retained result is evaluation-only; none of these records is an efficiency claim.
QuEST W4A4: the gate failed
- File:
gate-674-quest-w4a4-sft.md. A research record for issue #674, not a release and not shipped code, measured by @Shutaru (the main experiment on 2026-09-09, the last bounded diagnostic on 2026-09-11), and in a tagged release since v0.75.0. It leaves #674 open. - Question: Can an experimental QuEST-style W4A4 supervised fine-tune, with 4-bit weights and 4-bit activations simulated by fake quantization, come within a preregistered 0.1 nat of the strongest full-precision run on the same held-out panel?
- Hardware and stack: NVIDIA GeForce RTX 5090 Laptop GPU (24,463 MiB reported), Windows, Python 3.12.10, PyTorch 2.11.0+cu128, Transformers 5.16.1. The model is
ahxt/LiteLlama-460M-1Tand the data isdatabricks/databricks-dolly-15k, with revisions pinned in the record. The NVIDIA driver was not captured at the time, and the record refuses to backfill it from a later query of the installed driver. - Result: The gate failed. The group128 candidate scored 2.349768 against 2.235367 for the strongest retained full-precision endpoint, a gap of 0.114 nat with a paired 95% interval of [0.099, 0.130], where the preregistered criterion required both the gap and its upper bound to be at most 0.1.
- Status: The record's opening sentence is "Gate failed: 0.114 nat [0.099, 0.130], one training seed, one model, fake quantization. Not parity, not an integration, not an efficiency claim." The separate requirement of a demonstrated improvement over the non-group128 control also fails, since that interval crosses zero. No alternate arm was promoted and the reserved confirmation panel was never evaluated.
- Caveats: The scripts and full records remain local, so the record publishes descriptions and SHA-256 bindings rather than a runnable integration, and the result cannot be re-run from the repository. In its descriptive desktop timings the reference implementation was slower and used more memory than the full-precision run; those are not benchmarks. There was no QuEST option in Soup when this record was written and none was implied. (Since changed in v0.76.0:
training.quantization_aware: questis an experimental mixed-route SFT path, #1105, whose retained result is evaluation-only. The strict W4A4 route this record gated is not what it offers.) - Read to the end: The record retains its failed directions, its cost table and a history of every evaluation panel it spent, and it says it can stand as a negative result without another training run.
- On this site: What was measured in v0.75.0.
QuEST mixed route: an evaluation-only gate that passed
- File:
gate-674-quest-mixed-route.md. A research record for #674 that follows the failed strict gate above and does not revise or replace it. - Question: After strict W4A4 failed, does a mixed-precision route frozen before it was scored on the confirmation panels pass an evaluation-only quality gate against fixed full precision? The route keeps all 168 transformer linear weights at W4 and uses A16 activations for the seven linears of block 23, the last of 24, and A4 for the other 161.
- Hardware and stack: 2 x NVIDIA GeForce RTX 3090 (24,576 MiB each), on the same 460M-parameter model as the failed gate.
- Result: The seven-A16 route passed the record's evaluation-only gate against fixed full precision on one model with fake quantization; the numbers are in the record.
- Status: The record's opening sentence: the predeclared seven-A16 rescue "passed its evaluation-only quality gate". It adds that this is "one model, fake quantization and no training in this confirmation; it is not strict W4A4, upstream QuEST parity, a packed-INT4 result or an efficiency claim."
- Caveats: The work performed was 2,496 forwards, zero backwards and zero optimizer updates, with dense fake quantization and no packed INT4 kernels. It makes no latency, throughput, VRAM or deployment-efficiency claim and does not validate mixed-route training quality. The strict route and the post-fit continuation below did not pass.
QuEST post-fit: still above the parity target
- File:
quest-674-postfit-development-results.md, with its evidence fileevidence/quest-674-postfit-v1.json. Dated 4 October 2026. - Question: How does the best retained post-fit QuEST checkpoint compare with the strongest retained full-precision checkpoint on the reused development panel?
- Status: In the record's words, the best retained integrated checkpoint "remains above the zero-margin parity target". The record preserves its selection, its last-stage recipe, its numeric evidence and nine negative or inconclusive follow-ups, and says it "does not authorize final confirmation".
training_quality_validatedstays false and the three-seed confirmation remains pending. - Caveats: The panel was already used for candidate selection, so the interval is descriptive after adaptive selection. This page quotes no ratio from it.
QuEST development follow-up
- File:
quest-674-development-followup.md, with its evidence fileevidence/quest-674-development-followup-v1.json. A research record for #674; the release body credits @Shutaru for it (#2042). - Question: After the strict and post-fit records above, does the closest retained reconstruction reach zero-margin parity with full precision on the development panels?
- Hardware and stack: RTX 3090 GPUs, on the same 460M-parameter model as the records above.
- Status: In the record's words, "Neither replica passes the frozen zero-margin parity gate", and the strict native W4A4 and public mixed routes "remain separate measurements, with larger gaps". It leaves #674 open and asks for a decision on future acceptance criteria.
- Caveats: Development panels only, already used for selection. It changes no runtime default and validates no production readiness, and this page quotes no ratio from it.
Other records (v0.76.0)
LoRA-FA: an operating point
- File:
gate-725-lorafa-operating-point.md. A contributor's micro-benchmark for #725, dated 2026-09-14. - Question: What is the measured operating point of optional PEFT LoRA-FA, which freezes the A matrix and trains B only?
- Hardware and stack: A two-layer LLaMA with 1,856,768 parameters at rank 16, PyTorch 2.7.0 (2.14.0+cu130 for its GPU check).
- Result: Trainable adapter parameters fell from 32,768 to 16,384 and AdamW state elements from 65,544 to 32,768, 50.0% fewer in each case. The activation-retention ratio is analytic, r divided by d_in (0.0625 at d_in 256).
- Status: A micro-benchmark with no verdict.
- Caveats: In the record's words, these are "NOT total or peak LLM VRAM savings, an end-to-end throughput result, or a downstream model quality claim". Step times in milliseconds on a toy are noise.
MoE expert coverage
- File:
probe-moe-expert-coverage.md. A probe under a rule committed before the run (the rule was edited once, before any model was downloaded, and the record says so). - Question: If a training step's tokens route to nearly every expert anyway, expert-granularity streaming would save memory and not reads. How much of a layer's experts does one step touch?
- Result: Union coverage, the fraction of a layer's experts that at least one token in the step routes to, was 0.967 to 1.000 over 18 combinations of corpus and batch shape on two models: granite-3.0-1b-a400m-base (32 experts, top-8, bf16) and OLMoE-1B-7B-0924 (64 experts, top-8, NF4). The worst single layer was 0.909.
- Status: The record's
C >= 0.85branch, "memory only": expert-granularity streaming "saves memory and NOT reads", and the record concludes it would be "a capacity tier for MoE" shipped with an explicit promise of no throughput gain. - Caveats: Expert-granularity streaming was not built and is not a Soup feature. (Since changed in v0.77.0: it ships as
training.stream_expert_layout: per_expertfor DeepSeek-V3-family bases (#2008, #2029), as the capacity tier this record predicted, with no throughput gain claimed. See Streaming a Mixture-of-Experts base.) The largest expert count measured is 64, and a 128-expert model (Qwen3-30B-A3B) was not measured.
GPU test runs
- File:
gpu-test-runs.md. Not a gate: a log of every run of the CUDA-dependent tests (pytest -m gpu) on a real card, since CI has none (#833). - Status: A log, one row per card and date. The first two rows predate the
gpumarker. - Caveats: This page quotes none of its counts. Its finding that matters is that a regression can be visible only on a real card, and that a green run on the RTX 5070 includes nine tests that pass by asserting a streamed-versus-resident NF4 difference (#776) is still non-zero, so green there does not mean #776 is resolved. Its rows do not include a run after the v0.76.0 NF4 dispatch change. (The one post-change row is in
nf4-on-blackwell.md, above; the log itself gained one packed-LoRA row in v0.78.0.)
The scripts
The harness/ directory holds measurement scripts that can be run against a released Soup, so a claim in a record can be re-measured instead of taken on trust. At v0.75.2 it held six scripts, and most of the original session's harnesses lived only in a scratchpad on a machine that no longer exists, which upstream tracks as issue #379. At v0.76.0 it held 61 files and at v0.77.0 it held 82; at v0.77.1 it held 88 (85 Python scripts, two PowerShell scripts and one shell script); at v0.77.2 it held 95 (89 Python scripts, two PowerShell scripts, one shell script, two Markdown notes and one notebook); and at v0.78.0 it holds 185 (149 Python scripts, 11 shell scripts, two PowerShell scripts, eight Markdown notes, one notebook and 14 data and settings files). The six original scripts and two of the v0.76.0 additions are in the table below, with the v0.77.0 additions after it. The other new files came with the records of this window (the disk tier and drive probes, the batch and layer-major probes, the CUDA graph gate, the soup ab sweeps, the QuEST evidence, the MoE probe and the LoRA-FA micro-benchmark); each record's header names the scripts it used, and upstream's benchmarks/README.md has the full list under its Harnesses heading.
| Script | Record | What it asks | What it needs |
|---|---|---|---|
issue331_qlora_scope.py | 8x H100 | Does the wrong-gradient defect reach ordinary QLoRA? Three arms in one process with a positive control. The answer on one Linear4bit is no: 0.0 against a control that diverges by 3.77e-01 | About 15 s on a 4 GB card, no downloads |
bnb_repro.py | 8x H100, bitsandbytes issue 2034 | A minimal standalone reproducer of the NF4 stale-gradient mechanism. Private buffers are the reference and bf16 is the control | A CUDA GPU, no downloads, about a minute. Historical environment: torch 2.13.0+cu130, bitsandbytes 0.50.0 |
bitexact.py | 8x H100, its Reproducing section | Shard, stream, then compare logits, LoRA gradients and the loss curve against a resident reference of matching numerics. NF4 uses a resident NF4 reference | A CUDA GPU, model weights, and room for the resident reference as well |
fsdp_sharding_probe.py | 8x H100, step 20 and issue #373 | Is the base actually sharded, or was the flag merely accepted? Each rank's local count must equal the total divided by the world size, with a single-GPU control expecting the full count | The accounting is pure Python and CPU-tested, but a real reading needs the multi-GPU box |
issue395_second_stack_vram.py | A10G | Does the long-sequence under-prediction reproduce on a second stack? Three modes: the sweep, --measure and --sdpa | A 24 GB card, downloads of two small models, and jinja2 3.1.0 or newer |
mlx_sft_smoke.py | 8 GB M1, issue #23 | Does the MLX wrapper's training loop run on a real MLX runtime, and what does it peak at? It asserts MLX dispatch before measuring, so a transformers-path number cannot be published as an MLX one | Apple Silicon and the [mlx] extra, and it downloads the chosen model |
issue361_nf4_throughput.py | NF4 streaming, issue #361 | The protocol behind the post-repair 8B row: batch 1, sequence 512, 50 steps after 10 warm-up, PagedAdamW8bit, and the GEMM ceiling measured in the same session at the stated SM clock. It prints the commit it was run from | A CUDA card (the row was taken on a 4 GB RTX 3050 Laptop) and the Llama-3.1-8B weights |
stream_probe.py | the RTX 5070 records | Builds the model through the shipped path, replicates the two shipped copy paths with CUDA events added (a source guard refuses to run if the shipped body drifts), and runs the ceiling, step, sweep and interleaved-ablation modes; --tier disk measures the disk tier and writes JSON after every point | A CUDA GPU and the model or shard cache it is pointed at; the cold 70B-shaped fixture is a 36 GB store on disk |
One more script sits beside the records rather than in harness/: bench_validator.py. It is a timing script, not a record. It compares a preserved copy of the previous validate_and_stats against the current one over 20,000 replicated fixture rows, takes the median of 15 interleaved runs, and asserts the two return identical output. The upstream README does not list it and no result is recorded in the directory, so this site quotes no figure from it.
Added in v0.77.0 (21 files), each named in the header of the record it belongs to:
expert_stream_gate.pyfor the expert-streaming gate, andmoe_stream_step.pyfor Moonlight-16B, the K2-shaped step and its longer-sequence probe, withsynth_moe_store.pywriting the random-weight K2-shaped store.fp8_block_quantize.pyandfp8_shard_gate.pyfor the FP8 shard gate, and the sevenfp8_loss_check_*files for the FP8 loss check.kimi_k2_gate.pyfor the Kimi K2 admission gate.stream_recompute_exactness.pyandstream_recompute_ab.pyforstream_recompute, andstream_ab_pairs.pyfor the NF4 scale cache.stream_loss_chunk_vram.py, the peak-VRAM harness fortraining.stream_loss_chunk(#1693). When it was added it had no record: upstream's index said it had not yet been run on a GPU. (Since closed in v0.77.2, in part: one card has run it, see the v0.77.2 section. The table is in upstream's performance documentation and on #1693, still not in abenchmarks/record, so this site prints no figure from it.)pincost.py,repeat_backward.py,depth_vs_bytes.pyandlayercount.py, which rebuild measurements from the H100 session, whose harnesses mostly lived only in a scratchpad on a machine that is gone (#379). The 6.56x pinning figure thatpincost.pyis built to re-measure was measured on the H100 at 32B NF4 before the #331 repair; upstream's benchmarks index records that the pinned arm's gradients were wrong on all three repetitions (only 8 of 256 gradient tensors matched the reference), so it is a historical timing, not a standalone throughput claim. Since v0.77.2 the H100 record says the same under its finding 9, as a dated clarification (the H100 entry).
Three of the earlier scripts changed at v0.77.0 (issue361_nf4_throughput.py, stream_probe.py and synth_checkpoint.py); the table keeps their v0.76.0 links.
Added in v0.77.1 (six files), each named in the header of the record it belongs to:
vram_estimate_calibration.pyfor the shared VRAM estimate: peakmax_memory_allocated()over two residentsoup trainsteps per row, each row in a process of its own, and a row whose collated length is not the requested one is void.issue2050_selective_checkpoint.py(one arm of the unchanged #361 recipe with--policy full,qkvorqkvo, plus an untimed--auditand a--profile),stream_selective_checkpoint.py(the installer those arms use, with no Soup config field),issue2050_campaign.py(the whole campaign in the protocol's order),issue2050_audit.py(re-derives the verdicts from the committed evidence) andissue2050_report.py(renders the record), all for the selective-checkpointing trial.
Two earlier scripts changed at v0.77.1. stream_ab_pairs.py gains --harness, so the pair runner can drive a script other than the #361 throughput one (the trial above uses it), and stream_loss_chunk_vram.py now runs each arm in a process of its own, because only the first arm per process got the RAM tier. That is step 1 of #1693's order. (Since closed in v0.77.2, in part: the harness has run on one card and its rows are published outside benchmarks/; it still has no record file, see the v0.77.2 section.)
Added in v0.77.2 (seven files), each named in the header of the record or index row it belongs to:
issue999_t4_bracket_check.ipynb, the harness behind the T4 record's third session. It installssoup-cli[train]==0.77.0and asserts that version, so it is not a second proof notebook and this site links no "open in Colab" button for it.record_fast_lora_cpu_bf16.py, the recorder behind the CPU bf16 Fast-LoRA record.fast_lora_qkv_bench.py(#838): forward and backward time and peak allocated memory of one attention block's Q/K/V projections with and withoutpatch_fast_lora_qkv, three fresh processes per cell. (Since v0.78.0 the upstream index records it as run once, on one card, with the results in two issue comments and no record file: Fast-LoRA, two CUDA readings that disagree.training.fast_lorakeeps its "no speedup is claimed".)prep_convergence.pyand its.mdnote: the data-preparation slice of #379 for the convergence runs of the H100 session (five disjoint 3,000-row subsets ofdair-ai/emotionand a 300-row held-out sample). Upstream calls it a reconstruction, not recovered historical data: its hashes "cannot establish the original dataset bytes or reproduce the recorded accuracy", and it claims no training or CUDA result.grpo_nan_numeric_probes.pyand its.mdnote: CPU-only arithmetic and autograd controls for #342. Upstream says it is not an H100 reproduction, not an SDPA backend attribution and not a root-cause determination, and its operand bounds are assumed, not measured.
One earlier harness file changed at v0.77.2: stream_loss_chunk_vram.py (a docstring and the self-test vocabulary).
Added in v0.78.0 (90 files, no earlier harness file changed), each named in the header of the record it belongs to:
ds_compare/, the pre-registered DeepSpeed protocol and its arms (PROTOCOL.md,DEVIATIONS.md, the job scripts and one JSON file per arm) for the re-measurement, andkbit_fp32_ab/for the fp32-cast measurement.stream_infer_throughput.py(forward-only inference),nf4_stream_bench.py,nf4_stream_model.pyandbnb_nf4_path_divergence.py(NF4),flash_liger_record.pywith theissue378_*files (Flash-Attention and Liger),issue722_distill.pyandissue722_compare.py(chunked distillation),vram_estimate_distill.py(the distillation estimate),kimi_k2_resident_parity_record.py(resident K2),issue2677_smoke.py,onnx_export_compat.pyandfast_lora_mlp_bench.pywithfast_lora_mlp_report.py.- The #2162 files,
issue2162_*,stream_mlp_*andstream_projection_*, plustwo_read_compare.py, for selective projection retention. grpo_crash_capsule.pyandgrpo_nonfinite_workflow_cpu.pywith their notes (#342), andhf_hub_cpu_fixture.pywith its note.
See also
- The layer streaming paper, the preprint the layer streaming records are the evidence behind.
- Validation on hardware we do not own, the H100 record in full with its caveats.
- Layer streaming, scaling a streaming run and preference losses over streaming, the pages whose numbers come from these records.
- The free GPU tier and the v0.73.2 release gate repairs.
- The disk tier, measured and
soup bench, the pages for the v0.76.0 records on the second laptop. - Streaming a Mixture-of-Experts base, the page for the v0.77.0 MoE records.
- The hardware-fit calculator, the page for the v0.77.1 resident-VRAM record.
- v0.77.2: the token rule and explicit refusals, the release page, which lists what v0.77.2 added to
benchmarks/. - v0.78.0: a continual-learning loop, a per-row ship audit, native MLX chat and merge, the release page.
- The benchmarks directory at v0.78.0, where every file named above lives.
Soup is free and Apache-2.0. If it saved you a training run, starring the repo costs nothing and helps most. You can also fund the GPU time behind the work a 4 GB laptop cannot reach.