v0.73.1: the free GPU tier
A patch release carrying everything that landed after v0.73.0. Its highest-value item is not the issue the slot was opened for.
The slot was opened to make the streaming VRAM pre-flight measure instead of predict. What sat beside it in the same pile was larger: bf16 was assumed on every CUDA card, in fourteen places, so every pre-Ampere card failed on every task, not only on layer streaming. That is a T4 on Colab's free tier, a T4 or a P100 on Kaggle, and every V100, GTX 16xx and RTX 20xx. The entire free notebook tier ran in a dtype its GPU has no units for, and said nothing about it.
Neither this release nor the one after it has a codename upstream. Borrowed Hardware is carried here as the name of the line.
Re-run this
| If you | What happened | What to do |
|---|---|---|
| Trained on a pre-Ampere card (T4, P100, V100, GTX 16xx, RTX 20xx) | The run asked for bf16 on a card with no bf16 units. Every task, not only streaming | Re-run on v0.73.1. A streamed run on these cards needs v0.74.0, which fixed a second, separate defect |
Used backend: mlx | It never dispatched to the MLX trainer at all, silently falling through to the transformers path. And without a freeze first, the saved "adapter" was a full fine-tune: 172 tensors where 24 were expected | Re-run |
| Loaded an MLX adapter you had trained | adapter_config.json shipped target_modules unresolved, as {"keys": ["auto"]}, so loading dropped every LoRA tensor without a word. Generation with the adapter was bit-identical to the base model. auto is the schema default, so this was every MLX run that did not name its modules by hand | Re-train. The file cannot be repaired after the fact |
Resumed from a periodic checkpoint-* under use_fsdp2_compile | v0.73.0 repaired the final save only. Every intermediate checkpoint still loaded as a dead adapter, so --resume and load_best_model_at_end continued from a re-zeroed lora_B in total silence. Measured at 70B: 320 canonical keys in the output root against 320 prefixed ones in checkpoint-100 | Re-run from the start |
Set training.seed on anything other than supervised fine-tuning | It was parsed and ignored. A seeded GRPO run trained at the framework default of 42, with no error and no warning, so replicates that differed only in their seed were the same run | Re-run if you drew a conclusion from a seed sweep |
| Streamed at a sequence length above roughly 4,300 tokens | The VRAM pre-flight can under-predict, against a contract that says it never does | Check the table below, or turn on the probe |
That MLX adapter is the third time in this project's history that a healthy loss curve shipped a dead artifact, after v0.72.0's adapter keys and the full-fine-tune save above. The loss curve structurally cannot see it. Only the artifact can.
The headline: bf16 was assumed on every CUDA card
Fourteen places. Twelve trainer wrappers (bco, classifier, distill, dpo, embedding, ipo, kto, online_dpo, orpo, pretrain, reward_model, simpo) plus the streaming setup and the supervised wrapper's own precision resolver, each carrying some form of "if the device is CUDA, use bf16" with no capability check at all.
The trap recorded beside the fix is worth knowing, because the first attempt was a no-op on exactly the hardware it was written for. The capability check defaults to counting emulation, so a T4 answers yes when asked whether it supports bf16. The predicate now asks the question that distinguishes hardware support from emulation.
Two things make this the same shape as the four backends the borrowed H100 box found had never run: it cannot fail on an Ampere development box, and the knowledge already existed in the repository. One wrapper, the ASR trainer, already carried a comment naming pre-Ampere cards and had already fixed it, in that one file, never propagated. The repair replaced the hand-written list of wrappers with a scan of the trainer package, because a hand-written list is what hid them.
The correctness cross-check: streamed-versus-resident logits are bit-exact at 0.000000e+00 in float16 as well as bfloat16, in both quantisations, against resident references of matching numerics. That was measured using fp16 on an Ampere card, so it establishes the plumbing, not the Turing and Pascal kernels. What a genuinely pre-Ampere card does with bitsandbytes NF4 is not yet measured. (Since partly closed in v0.76.0: a forward-only comparison in fp16 on a free Colab T4 was recorded, on a 135M model without quantisation; NF4 exactness on Turing is still unshown. See the next section.)
An 8B on a free Colab T4
With the repair in, the notebook that had been unrunnable on the published release runs.
Llama-3.1-8B in NF4, streamed, on a free-tier Tesla T4, batch 1, max_length: 256, LoRA r=8, fp16, with the process capped at 4.00 GB. The cap was shown to bite rather than assumed: a deliberate 4.29 GiB allocation was refused.
- Measured peak 2.91 GB against a predicted 3.02, an over-prediction of 3.8%, which is the safe direction. This is the first of three recorded sessions; the second ran a different memory plan, the third (v0.77.2) had no cap, and both are described with the notebook on Layer streaming
- 7 steps, exit 0, adapter written with 128 of 128 tensors non-zero
What that run does not establish is stated as plainly as what it does. No throughput figure is quoted from it, because a capped card is not a benchmark. (Since v0.77.2 the record also holds a separate, uncapped session that did measure throughput; see the next section. It does not turn this capped run into a benchmark.) Gradient exactness at 8B on a Turing card is not shown: non-zero tensors prove gradients flowed, not that they were right. The notebook's own streamed-versus-resident comparison produced no captured output on that run and is recorded as unrun, not as a pass. (Since closed in v0.76.0: a later run of the notebook recorded that comparison, as a forward pass only, on SmolLM2-135M in fp16 without quantisation, with a maximum difference of 0.0. It says nothing about the backward, about NF4, or about 8B.) And one run, one seed, no repeats.
One thing it surfaced: the pre-flight reported 15.10 GB free, the whole device, because the cap is enforced by the allocator and not by the driver. The fit decision was taken against a number 3.8 times larger than the budget actually in force. That is the open issue training.stream_vram_override exists to work around.
At the time of this release the notebook installed Soup from git, not from PyPI, and its own text still said the fix was unreleased. (Since closed in v0.76.0: at the v0.76.0 tag its install cell is
soup-cli[train]==0.75.0from PyPI besidehuggingface-hub<1.32.0, so it runs v0.75.0 and not v0.76.0, and on Colab's Python 3.13 it passes--ignore-requires-pythonbecause Soup declares<3.13. At v0.77.0 the notebook still installssoup-cli[train]==0.75.0(it is byte-identical at v0.77.1 and v0.77.2), so the fallback is for that pin; Soup itself declares>=3.11,<3.14from v0.77.0, as the notebook's own text at the tag now says. Layer streaming has the current frame.)
The uncapped session added in v0.77.2
A third session on a free Colab T4 (2026-10-26, #999; credit @VanshSharmaPES, #2230 and #2216) ran with no process cap and measured throughput, which the two capped runs deliberately did not. It used soup-cli 0.77.0 plus the two repairs from #2216, which the harness applied at launch after checking that the installed release lacked them (0.77.0's soup bench train could not run a 4-bit config; v0.77.2 ships both repairs), and measured Llama-3.1-8B NF4, streamed, fp16, batch 1, gradient accumulation 4 (the schema default), with soup bench train: 20 counted optimizer steps per run, runs A, B, A, B in fresh processes after an unmeasured priming run. Both of its data shapes are the result:
- Every sequence 256 tokens: 233 tok/s (232.93 and 233.09, every position counted, padding included; about 215 supervised), peak VRAM allocated 1.93 GB against the roughly 1.97 GB the pre-flight panel printed.
- The proof notebook's own 37-token rows: 45 tok/s (45.49 and 44.87; about 21.5 supervised), peak 1.51 GB, not comparable with the estimate because the sequences are shorter than the 256 it is for.
The pre-flight's bracket for that kind of run (288 to 423 tok/s) over-predicted on this card: 233 is 0.81x of its low end, and about two thirds of the low end (0.65x to 0.66x, as the record states it) of the higher brackets this session's own pre-flight printed at 675 to 840 MHz, so read the panel's range as an upper bound there, not a forecast. The record reads the clock as a likely part of the gap and does not show it as a cause. Riders that travel with the numbers: one session on one Colab host, clock and power observed and not controlled, throttle reasons not logged. It is not a 4 GB result, it is not set beside any RTX 3050 row, and this page quotes no other soup bench train rate (upstream's own record for that command found a 2.43x spread over thirteen identical runs on a power-limited laptop). Backward, NF4 and 8B exactness on Turing remain unshown. The figures with every rider, and the record, which is linked at v0.77.2 because that section exists only from that tag.
More T4 records in v0.78.0
Five more records were measured on a Tesla T4 (Kaggle or Colab, sm_75) and arrived with v0.78.0. Four of the five use a synthetic fixture or a single block (the aliasing probe, BCO, IPO and the Fast-LoRA block); the GRPO record ran real Qwen2.5-0.5B-Instruct and Qwen2.5-7B-Instruct checkpoints on synthetic prompts whose reward did not score correctness. None adds a streamed-training throughput, so the 8B-on-T4 statements on this page and on Layer streaming are unchanged. None uses the Colab proof notebook.
- NF4 aliasing on a T4 (
gate-842-nf4-streaming.md). A four-layer synthetic probe whose unsafe arm overwrites the pooled weight on reuse produced finite wrong gradients on a T4 (the same aliasing mechanism as the #331 defect); bypassing the pool, and the shipped repair, each restored exact gradients. A full-model arm on a four-layer Llama fixture (181.60 MiB per NF4 shard) was bit-exact against resident NF4 on the forward and on 16 of 16 adapter gradients per backward both with the repair and with it disabled: in the record's words "the full-model negative control did not fire", so that arm does not distinguish the repair. The aliasing probe and the full-model arm are backward observations on a Turing card, on synthetic fixtures with four layers of about 171 to 182 MiB each; backward exactness at 8B, and NF4 on a real checkpoint, are still unshown. - BCO and IPO (
bco-streaming-t4.md,ipo-streaming-t4.md). One optimizer step per cell, on a two-layer vocabulary-64 fixture, to qualify the two losses for streaming. IPO on a T4: the fp16 gradient scaler skips its first updates. The four default-setting cells failed (loss 25.0, NaN gradient norm); the diagnostic controls recorded the skipped first update and an unchanged adapter. With the scaler's initial scale lowered in the test harness only, all eight cells applied an update, and the default setting, on the directipospelling with no quantization only, skipped four updates and applied the fifth at scale 4096, streamed and resident alike (a run of up to 12 attempts, not one step). No throughput, peak VRAM or capacity figure. The preference page. - Fast-LoRA on one block (
fast-lora-mlp-cuda-t4-2026-11-08.md): one synthetic Llama-3.1-8B-shaped SwiGLU MLP block on a Colab T4, no model loaded, with bf16 emulated, not native. No measurable time difference, and a higher peak allocated memory for Fast-LoRA whose cause the record says is unknown. The measurement records. - GRPO on T4s (
run-342-kaggle-t4-grpo.md, #342): four fp16 GRPO arms on real Qwen2.5-0.5B-Instruct and Qwen2.5-7B-Instruct checkpoints with synthetic prompts whose reward did not score correctness, one of them a 7B NF4 run that ran out of CUDA memory before the backward, and a longer 7B run with one watchdog trip. In the record's words the results "provide no evidence for or against the original H100 BF16 failure".
The pre-flight was breaking its own contract
The streaming VRAM pre-flight's contract, as it stood at v0.73.1 and through v0.77.x, is that it never under-predicts, because on Windows an over-budget allocation does not raise. (At v0.78.0 upstream's own statement is narrower: the budget is fitted on named grids and "does not promise a bound at other sequence lengths or devices"; on the RTX 3050 ten-run grid the total reads 101.04 to 106.57% of the recorded peak since the cuBLAS workspaces were added to the charge, #2749.) It spills silently into host memory, and the run slows by multiples with no error at all.
Measured through the real soup train on the 4 GB laptop, SmolLM2-135M streamed in bf16 at batch 1:
| Sequence | Predicted | Real peak | Ratio |
|---|---|---|---|
| 4352 | 3.282 GB | 3.036 GB | 1.081x, over-predicts, safe |
| 5120 | 3.844 GB | 4.118 GB | 0.934x, under-predicts |
| 6144 | 4.590 GB | 5.830 GB | 0.787x, under by 21% |
The existing guard could not have caught it. All ten rows of the grid that validates the estimator sit at sequence 256 or 512, so it varies batch and says nothing whatever about sequence length. A control only covers the variable it varies, and that is this project's own test applied to itself.
The mechanism is deliberately not claimed. The obvious candidate, the quadratic term from the attention score matrix, does not settle the numbers, and that is precisely the argument for measuring rather than adding another coefficient: a formula cannot model a term nobody has identified.
A second box has since said this does not reproduce there. A contributor ran the same protocol on an NVIDIA A10G with a much newer stack (torch 2.13, transformers 5.16.1) and the ratio is flat at about 1.16x over-prediction from sequence 2048 through 6144, on two models. The record publishes its leading hypothesis as a negative result with a positive control, and explicitly does not refute the reading for this 3050. Taken together the honest conclusion is that the term varies per-stack and per-sequence-length, which is why the answer is a measurement rather than a constant. See the v0.74.0 record.
Two new config keys
Both are config keys, not CLI flags. Both are refused at config load if stream_layers is false, the same footgun gate the other streaming keys carry.
training:
stream_layers: true
batch_size: 1 # 'auto' is refused on a streaming run
stream_vram_probe: true # decide the fit by MEASURING one real step
# stream_vram_override: 4000000000 # or: assert what is free, in bytestraining.stream_vram_probe runs one real forward and backward at your configured shape and decides on that, instead of on the fitted formula.
- Off by default, because it costs a step and it can refuse a run the formula would have accepted
task: sftonly. A preference loss is not the step the probe runs, and its agreement with one is not established. The four streaming preference losses keep the fitted prediction (since v0.78.0bcoandipostream too and keep it as well), which is exactly where it was measured least accurate- Costs 1 to 5 seconds at ordinary shapes, and 52 seconds at a shape that does not fit, which is the honest worst case paid once instead of every step
- Runs 12 to 14% conservative against the real training step, the correct direction for a gate
- Cannot overrule a prediction more than four times over budget. The largest disagreement ever measured is 21%, so past a small multiple the config is simply too big and is refused by arithmetic without touching the GPU
- Gates on allocated memory, not reserved. Reserved runs 1.08 to 1.41 times allocated, and gating on it would refuse this feature's own flagship configuration, which runs
training.stream_vram_override replaces the free-VRAM figure the pre-flight checks against, in either direction. It changes the yardstick, not the check: an override below real free VRAM still refuses a config that would otherwise fit, which is what makes it an escape hatch rather than a bypass flag. It is an assertion you are making, not a measurement.
The rest
training.seednow reaches every trainer wrapper, and is applied before the model loads, because the adapter's own initialisation is drawn before the trainer's seeding used to happen. An unset seed still resolves to 42 anddata_seedstill stays unset, so the values a run trains at are unchanged; what changed is when they arrive. Full detail on seeds and reproducibility.- The MLX backend reads neither seed field, and now says so in its existing "MLX backend ignores" line. A warning rather than a rejection, because a config valid on transformers should not become unloadable by switching backend. (Since closed in v0.77.0, #1855: MLX SFT applies
training.seedandtraining.data_seed, andtraining.seedno longer appears on itsMLX backend ignoresline, so an MLX run started before v0.77.0 cannot be reproduced.) training.batch_sizegained a lower bound.batch_size: -4used to load, and then meant whatever each trainer's arithmetic did with it, including the streaming pre-flight, which multiplies by it.- The dead calibration hook was wired up. The estimator's calibration entry point had no caller anywhere in the source; it now feeds both the budget and the printed panel. Every streamed run pays one transient 96 MiB allocation and two device synchronisations for it. On today's stacks that is a no-op in effect, which is the point of a guard against a stack that has not shipped yet.
mlx-lmfloor raised to the version the MLX path is built against.--deepspeedon a card without bf16 hardware (v0.78.0, #2266). Every DeepSpeed preset declaresbf16, and accelerate refuses that against an fp16 trainer, so by upstream's account--deepspeedstopped before the first step on a T4 or V100 and under GRPO'sgrpo_fp16through v0.77.2. Since v0.78.0 the config'sbf16andfp16are matched to the trainer's precision before the file is written, and the change is printed. The evidence is one run by the pull request's author on a T4 (68 optimizer steps, exit 0) that upstream did not repeat on the final code, so this page claims no throughput and no multi-GPU result. Multi-GPU and DeepSpeed.
Known limitations
- The probe is supervised-fine-tuning only, which is where the formula was measured least accurate.
- The mechanism behind the long-sequence divergence is unidentified, so the formula is not fixed, only overrulable.
- The probe measures a plain causal-language-model step, not the run's own step.
- Windows still does not raise on an over-budget allocation, so the refusal direction cannot be verified by OOM on that platform.
- fp16 streaming is not yet measured on a genuinely pre-Ampere card. (Since partly closed in v0.76.0: one forward-only comparison on a Colab T4, as above. The backward, NF4 and 8B on Turing remain unshown. (Since v0.78.0: a four-layer synthetic fixture was observed on a T4, see the T4 records above; at 8B and on a real checkpoint they remain unshown.) Also, since v0.77.2: an uncapped Colab T4 throughput session is recorded, which says nothing about exactness.)
- Layer streaming remains BETA.
The working record carries three readings that were withdrawn during this work, two of which looked like the headline result. They are left standing in the benchmarks directory with their refutations beside them, which is why that directory is worth reading and why you should link a file's top rather than deep-link a passage the same page later corrects.
See also
- v0.73.2: the release gate — the release after this one.
- v0.73.0 Borrowed Hardware — the release before it, and its own re-run list.
- Seeds and reproducibility — what a seed does and does not buy you.
- Layer streaming — the mechanism, the measured numbers and the refusal table.
Soup is free and Apache-2.0. If it saved you a training run, starring the repo costs nothing and helps most. You can also fund the GPU time behind the work a 4 GB laptop cannot reach.