Validation on hardware we do not own

Until August 2026, every number Soup had published about layer streaming came from one machine: a 4 GB RTX 3050 Laptop running Windows. That is the machine the feature exists for, and it is also the machine that cannot check the feature's central claim.

The claim is that a streamed run computes the same thing a normal run computes. Checking it needs a resident reference: the same checkpoint, same quantisation, loaded the ordinary way, so the two can be compared tensor by tensor. On a 4 GB card that reference cannot exist for any model worth streaming. So the published checks ran on 3-layer models built from a config, with a hidden size of 32 and an NF4 layer of 0.01 MiB. The paper says so in its own limitations, and names the missing experiment as the most useful one absent from it.

In August 2026 someone lent us an 8x H100 80 GB box for three days. This page is what the experiment returned, including the defect it found.

The full record is published as written, in the order things happened, at benchmarks/gate-h100-validation.md. It is a working record, not a report assembled afterwards, so it contains rejected hypotheses, withdrawn findings and measurement errors that later controls caught. Read it from the top.

This page is about that session, so "the box" below always means the H100 machine, and every figure in the sections about it was measured in August 2026 on code from just after the v0.72.4 tag, with the repair applied only where a row says so. The free Colab T4 section near the end is a separate, later set of runs. Records from other machines have been added since, and the measurement records page indexes them.

The box, and why the stack matters

• 8x H100 80 GB HBM3, 503 GB host RAM, 32 cores, Ubuntu 24.04, driver 590.48.01
• PCIe between all pairs, no NVLink. This matters for the DeepSpeed comparison below and it favours nobody by accident: all-gather traffic is exactly what NVLink exists for.
• Page-locked ceiling is 62.96 GB, not 503 GB. Pinned memory is a separate budget from RAM.
• No NVMe. The only device is a virtual disk the kernel reports as rotational, which is why this session could not answer the RAM-versus-disk question. v0.76.0 measured it later, on a different machine; see What this does not say.

The software stack is a different one end to end: torch 2.13, bitsandbytes 0.50.0, trl 0.26.2, peft 0.20.0. Of the five components that matter, only transformers matches the laptop. That is a feature of the experiment rather than a nuisance: a result that survives a stack change is a result about the method, not about one pinned set of wheels.

Bit-exactness is two claims, and they are not the same claim

This is the most important sentence on the page, and it is the one two expert readers got wrong before it was written down.

Every exactness result below is two independent measurements: the forward (logits, compared with torch.equal) and the backward (every LoRA gradient tensor). Above roughly 165 MiB per NF4 layer they disagreed. "Bit-exact at 72B" is a forward statement and only a forward statement.

Each row was compared against a resident reference of matching numerics: resident NF4 for a streamed NF4 run, never resident bf16, because quantisation error is wide enough to hide a real defect inside it.

In the backward columns the fraction counts the gradient tensors that matched the resident reference: WRONG 8/256 means only 8 of 256 matched, and WRONG 8/320 only 8 of 320.

The 8B NF4 and 14B NF4 rows each survived a 50-consecutive-backward soak with a worst absolute difference of 0.0. A cell that was not measured says "not tested" rather than being left blank, because a blank reads as a pass.

Scope of this ledger, as of v0.76.0 and unchanged at v0.77.0. v0.77.2 did not change the streamed NF4 forward dispatch described here, and none of its records adds an exactness row for it. Every cell above was measured before the streamed NF4 forward changed. Since v0.76.0 (#842, #1143) that forward asks bitsandbytes' own dispatch which kernel a resident NF4 layer would use for the shape and runs the same one, where the repaired path measured here always dequantised; the backward still dequantises. No benchmark record re-states exactness for the new dispatch, so the table describes the earlier path and is not extended to the new one. The Blackwell streamed-versus-resident NF4 difference tracked as #776 is the case the change was aimed at: upstream's contributor notes report the CUDA exactness tests passing at exactly 0.0 on an RTX 5070 afterwards, but no benchmark record has a post-change row, so this page does not call it fixed.

The defect this found

In NF4, above roughly 165 MiB per decoder layer, a streamed run produced silently wrong gradients. Filed as issue #331.

It is worth being precise about how silent. The forward stayed bit-exact on every run. The loss matched a resident reference to every digit. The run exited 0, printed no warning, and the loss curve fell normally. Only a direct gradient comparison could see that 62 of 64 layers were wrong.

• Threshold, bracketed by sweeping one shape parameter: exact at 163.8 MiB per layer, broken at 171.5. Monotone on both sides over seven points, with no exceptions. It is a bracket, not a number.
• Which models. 8B sits at 105 MiB per layer and 14B at 132, both below with margin. 32B (234) and 72B (432) are above. Nothing Soup has ever published is above the boundary.
• Why NF4 only. bf16 is exact at 935 MiB per layer, moving 3.9 times more bytes than the failing NF4 run, so it is not about transfer size.
• The cause is aliasing, not a race. A full cuda.synchronize() does not fix it. Giving each pooled tensor a private copy fixes it completely. bitsandbytes.MatMul4Bit stashes the packed weight and its quantisation state on the autograd context as ordinary Python attributes instead of going through save_for_backward, so gradient checkpointing cannot see them: the reference is captured in the forward, aliases the streaming buffer, and is read in the backward after that slot has been refilled with a different layer. bf16 goes through a native op that does use save_for_backward, which is why it was never affected. The upstream half is filed as bitsandbytes-foundation/bitsandbytes#2034.
• The obvious fix was rejected on measurement. Cloning every pooled tensor is correct and costs only about 6% of throughput, but it takes peak VRAM from 4,220 MiB to 19,720 MiB on a real 32B, which is approximately the whole model. A repair that is O(model) instead of O(one layer) deletes the reason the feature exists.
• The shipped repair keeps streamed NF4 weights out of that code path entirely, so the saved reference goes through the ordinary mechanism that checkpointing knows how to recompute. Gated on two real models against a control that reproduced the defect in the same process: 32B at 256/256 and 72B at 320/320, five repetitions each. It costs +2.9% peak VRAM and 4.8% throughput at 32B, and +2.6% and 3.7% at 72B.

Are you affected?

Almost certainly not, and here is the exact test. You need all three of: quantization: 4bit, layer streaming, and a decoder layer above roughly 165 MiB. On the 4 GB hardware this feature targets, that layer size is out of reach: a 32B NF4 store is 14.99 GB of pinned host RAM and a 72B store is 33.74 GB. If you have streamed an 8B or smaller, you are below the boundary with margin and there is nothing to re-run.

If you did stream a 32B or larger in NF4, the adapter is not trustworthy and re-running is the only fix, exactly as with the v0.72.1 adapter-key defect. The two have the same shape: an exit-0 run that looked healthy the whole time.

The repair shipped in v0.73.0. Every release from v0.72.0 to v0.72.4 carries the defect at those sizes, so upgrade before streaming anything that large in NF4. The library behaviour it works around is upstream and unchanged, filed as bitsandbytes-foundation/bitsandbytes#2034.

Peak VRAM is flat in model size

This is the feature's whole claim, and it is the first time it has been measured across a real size range on one card. Five runs per row, streamed, NF4, through the shipped CLI.

The model grows fourfold and peak VRAM moves from 3,397 to 4,845 MiB. Qwen2.5-72B extends the shape at n=2: 37.9 tok/s in 7,411 MiB, against a 33.74 GB pinned store.

Two honest readings of that table. 14B being faster than 8B is the opposite of the naive expectation, it was not investigated, and no mechanism is asserted for it; the two rows are different architectures with different vocabularies, so the clean comparison is the two Qwen2.5 rows, which do fall. And every row here was taken before the #331 repair, which costs a few percent of throughput at the sizes it applies to.

The 72B row deserves its caveat spelled out rather than buried: under the configuration that shipped, 72B's gradients were wrong, so that line describes a memory and throughput result, not a valid training result. After the repair the gradients are exact. And the 33.74 GB host store is the real gate on reproducing it: this is not a claim about a laptop.

The laptop number reproduces on completely different hardware

The published headline is Llama-3.1-8B NF4 at 119.6 tok/s at 952 MHz in a 3.32 GB peak, on an RTX 3050 Laptop under Windows, measured on v0.72.2 before the repair. The same configuration on an H100 gives a median of 113.00 tok/s in a 3.32 GB peak, also measured before the repair.

An H100 is not slightly faster here. It is slightly slower, and mean GPU utilisation on those runs was 54.1%.

The record originally read that as proof the method is bound by host-to-device transfer rather than by the GPU. That reading was retracted on 11 August 2026, by measurement. Four ablation arms interleaved in one process on the original laptop at a pinned clock: removing all host-to-device traffic, 6.864 GB per step, buys 1.44%; removing the NF4 dequantisation buys 9.80%; removing both leaves 88.7% of the step standing. The compute stream is blocked on a copy for 8.4 ms of a 4190 ms step, and the step runs at 71.3% of that card's same-session, shape-matched GEMM ceiling. The claim is true below roughly 128 tokens per step, where the fixed transfer volume dominates, and the published configuration is not there.

What survives is the measurement itself, and it is untouched: a card with roughly two orders of magnitude more compute came back no faster than the laptop. The constraint is common to both machines and is not the compute the datacenter card adds. Beyond that we no longer offer an explanation for why: the one we offered was measured and refuted, and a second guess would be the same mistake in a new sentence. The H100's own bottleneck was never instrumented and no claim is made about it. Every occurrence upstream is annotated in place rather than deleted, including in the record that first made the claim.

What the retraction does not touch is what a streamed run needs from its host. The record's STEP 8 compares a page-locked arm with a pageable one on 32B NF4 and reports a ratio of 6.56x, but the page-locked arm was timed while its gradients were wrong on all three repetitions (only 8 of 256 gradient tensors matched the reference) and the pageable arm was exact (256 of 256). Upstream's benchmarks index says so, and since v0.77.2 the record says so too under its finding 9, in a dated clarification of 5 November 2026 (#2271, credit @xiemocun, who raised #2224): the 6.56x is, in upstream's words, "a defective fast arm over a correct slow one", and no number in the record changes. It was measured on the H100 at 32B NF4 before the #331 repair, so it is a historical timing, not a standalone throughput claim, and upstream adds that the pinning cost scales with bytes per layer. That is why a failed page-lock is never silent: by default Soup falls back to a pageable store and prints the cost, and training.stream_pin: true turns that fallback into a refusal.

Against DeepSpeed ZeRO-3

The first comparison against another tool, run through Soup's own --deepspeed entry point so it is one tool against itself. One H100, Llama-3.1-8B, same data, same LoRA, 256 optimizer steps.

At matched numerics, bf16 against bf16, layer streaming is 2.93x the throughput in 9.7x less peak VRAM. ZeRO-3 was then given a second, memory-tuned attempt: tightening every knob bought 5.6% less VRAM for 12% less throughput.

Do not read that as "streaming beats DeepSpeed." The record is emphatic about this and so is this page. ZeRO-3 shards across ranks, and with one GPU there is no partner to shard to, so it is being used outside the regime it was built for. That regime happens to be layer streaming's home ground, which is the point, but the honest claim is the narrow one: not faster than DeepSpeed, built for a different situation, namely one card that is too small.

The eight-card picture makes the same point from the other side, on the same model and data:

Eight cards of ZeRO-3 are slower than one card training resident, for a model that fits on one card. Against streaming on a single GPU they buy 1.70x, for 34 GB on each of eight cards against 2.9 GB on one. If the model fits, the right answer is the top row and every clever technique here is a loss. This is a PCIe box with no NVLink, and ZeRO-3 would improve on an NVLink node while the streaming row would not.

Does streaming change the model you get?

Bit-exactness answers this for one step. Convergence over a real fine-tune is a separate question, and it had never been measured.

Protocol: Llama-3.1-8B, NF4 in both arms, five paired runs where streamed run i and resident run i see byte-identical training data from disjoint subsets, scored by Soup's own soup ship against a fixed 300-row held-out set that no run trained on.

The mean paired difference is +0.0053, which is 1.6 items out of 300, against a within-arm spread several times larger. The general-capability leg points against the convenient direction: 3 of 5 resident runs came back DON'T SHIP against 1 of 5 streamed, and the single worst regression in the matrix belongs to a resident run.

Read this as "no difference is detectable at this resolution", not as "there is no difference." The experiment resolves roughly one percentage point on a 300-item set. It does not license a claim of quality-neutrality to arbitrary precision.

Preference losses, on a real 8B

The v0.72.4 claim that DPO's reference model costs no extra weights was measured on a 365M synthetic fixture. It now holds on a real 8B through the shipped CLI, streamed, timed for the first time:

Peak VRAM is flat across all five, within 64 MiB or 1.7%. DPO costs 44 MiB more than SFT, against roughly 5.6 GB for a second resident copy of an NF4 8B. That is the claim, measured at a size where a second copy would be impossible to miss.

Two cautions. These are samples per second, not tokens per second, and they come from 64-row single-epoch runs where setup is a large share of an 11 to 14 second step, so treat the ratios as directional. And kto runs a different dataset, so only its VRAM compares directly.

The same session also measured the streaming trade through the shipped CLI on a real model for the first time: streaming holds an 8B in 3,681 MiB against the resident path's 12,087 MiB, 3.28x less, for 1.13x the wall time.

The other machine nobody here owns, and it is free

An 8x H100 box is external hardware that almost nobody reading this can repeat on. A free-tier Colab Tesla T4 is external hardware anyone can repeat on in about twenty minutes, and it is the other end of the same question. A streamed 8B NF4 run now completes there, inside a 4.00 GB process cap, at a measured peak of 2.91 GB.

It is a far weaker result than anything else on this page and is filed that way upstream: not a gate. One run, one seed, no repeats, on a session that cannot be returned to, and with its correctness comparison empty on that first session, so no gradient exactness on Turing. No throughput is quoted from those first two sessions, because a card held under an artificial cap is not a benchmark (a third, uncapped session added in v0.77.2 did measure one, below). What it establishes is that the path executes end to end on a pre-Ampere card at 8B, which had never been run before.

The notebook was run again later on a free T4 and its outputs were committed (recorded by the commit of 12 September 2026). That run printed library versions (torch 2.11.0, transformers 5.16.1, peft 0.20.0, bitsandbytes 0.50.2) and filled in the comparison the first session left empty: the streamed forward matched the resident model exactly (torch.equal true, maximum difference 0.0) on SmolLM2-135M in fp16, unquantised, forward only. That is forward exactness on Turing for one small model and nothing more: not 8B, not NF4, not the backward. It also re-measured the 8B NF4 section, on a different memory plan from the run above (large-layer streaming had since moved the embedding and output head into the pinned store, a 5.70 GB store). Its peak is in the record and is not set beside the first run's. Both runs are kept in the record, and neither quotes a throughput.

A third session, added to the record in v0.77.2, ran without a cap (2026-10-26, #999, credit @VanshSharmaPES, #2230 and #2216) and is the only one that measured throughput. On a free Colab T4 with soup-cli 0.77.0 (plus two repairs from #2216 that the harness applied at launch, each only after checking that 0.77.0 lacked it, because 0.77.0's soup bench train could not run a 4-bit config), Llama-3.1-8B NF4 streamed in fp16 at batch 1 with gradient accumulation 4 measured 233 tok/s when every sequence is 256 tokens (232.93 and 233.09, every position counted, padding included; 215 supervised) and 45 tok/s on the proof notebook's own 37-token rows (45.49 and 44.87; 21.2 to 21.5 supervised). Both belong together. The bracket published for an earlier T4 session (288 to 423 tok/s) over-predicted on this card: 233 is 0.81x of its low end, and about 0.65x of the low ends (327 to 357) that the panels of this session printed, so read the panel's range there as an upper bound, not a forecast. The peak at 256 tokens was 1.93 GB against the roughly 1.97 GB the panel printed, on the later memory plan (a 5.70 GB pinned store), not the plan behind the 2.91 GB figure above. The record treats the clock as a likely part of the gap and not as a shown cause, and its own list of what it does not establish names one session on one Colab host, clock and power observed and not controlled, and no measured link between the bracket's startup probe and a sustained rate. It is not a 4 GB result and is not set beside any RTX 3050 row, and backward, NF4 and 8B exactness on Turing remain unshown. The figures and every rider.

Run it yourself, with the full frame. The precision fix it needs shipped in v0.73.1. Since v0.76.0 the notebook installs soup-cli[train]==0.75.0 from PyPI rather than from git, caps huggingface-hub below 1.32.0, falls back to --ignore-requires-python on Colab's default Python 3.13 (the pinned 0.75.0 declares <3.13), and its recorded outputs are from 0.75.0, not from v0.76.x, v0.77.0, v0.77.1 or v0.77.2. At the v0.77.0 tag its install cell is unchanged, only its text and comments changed, and the notebook is byte-identical at v0.77.1 and v0.77.2.

What this does not say

The record's own limitations section is long, and it is the reason to trust the rest.

• The RAM-versus-disk gap was not measured here. There was no NVMe on the box, and a number from a virtual disk published under a heading people would read as "NVMe" is worse than no number. That was true of this machine. v0.76.0 later measured the disk tier on a different one: an RTX 5070 Laptop 8 GB with two NVMe drives, a synthetic 70B-shaped NF4 store of 36 GB with random weights, batch 1, sequence 512. A cold step took 110.5 s on the previous synchronous reader and 16.2 to 17.3 s on the direct-I/O reader that ships, and across two NVMe drives a 17.4 s step fell to 9.97 s; a sustained-load probe measured 1.28x and 1.41x in two rounds and reached no formal verdict. Timing only, Windows only, one machine, no quality claim. A store that fits in RAM is still faster on the RAM tier, which stream_source: auto picks. The disk tier page has the rest.
• Why the defect is NF4-only, and why its boundary is so sharp, is not explained. Seven hypotheses were tested and rejected. None replaced them.
• The repair is gated at two sizes, 32B and 72B. No size between them was tested, and each gate is one sequence length and one buffer count.
• The quality result resolves about one percentage point, on 300 items, with the adapter initialisation seed uncontrolled.
• Every number is one machine, one session. No claim here is a multi-machine result.
• The published 119.6 tok/s laptop figure was measured before the #331 repair. At real training shapes bitsandbytes already took the same code path the repair forces, so the arithmetic is unchanged, and the measured cost at 32B is 4.8%. An 80 GB server card cannot stand in for a 4 GB laptop, which the H100 replication showed by coming back no faster rather than faster. Treat 119.6 tok/s as a pre-repair figure. (Since changed in v0.76.0: the post-#331 row was recorded by @umran666 (#1103), Llama-3.1-8B NF4 on a different RTX 3050 Laptop 4 GB at 1935 to 1957 MHz, 208.6 tok/s, 2.40 GB peak allocated. The 119.6 tok/s row was taken at 952 MHz, and the two are not directly comparable: upstream reads 59% against 68% of the same-session GEMM ceiling as about 14% slower per unit clock, so the raw 1.74x gain is the clock, not a speedup, and the record attributes the shortfall to none of the repair, the memory change or the stack. Reserved memory was 2.66 GB and the store 5.70 GB, from one 50-step run with the slowest step kept. The peaks are not comparable either: between the two rows large-layer streaming moved the embedding and output head into the pinned store. The paper page has the two rows side by side.)

The result that is not a number

Twelve defects surfaced in three days. Seven were repaired in the same window, five were filed with reproducers. The record's own summary of why is worth carrying:

Every one of them was found by running something that had never been run, not by reading code.

Four of the twelve were features the project ships and documents that had never executed once on any machine. Two produced successful, exit-0 runs the entire time: an adapter that reloaded as all zeros, and a data.max_length above 1024 that was silently ignored on every supervised run. A green test suite measured neither the features that had never run nor the paths that cannot run on one card.

That is the argument for borrowing hardware, and it is a better argument than any throughput number in the table above.

See also

• Layer streaming — the mechanism as it ships, the measured numbers and the full refusal table.
• The paper — the preprint, the correctness protocol and the limitations this page answers.
• Scaling a streaming run — architectures, batches, the VRAM pre-flight and the disk tier.
• The disk tier, measured — the v0.76.0 reader and the figures from the second laptop.
• Measurement records — an index of every record in the benchmarks directory, this one included.
• Preference losses over streaming — DPO, ORPO, SimPO and KTO against a streamed base.