Layer streaming: fine-tune a model that does not fit in your card (BETA, v0.72.0; NF4 v0.72.2; breadth v0.72.3; preference losses v0.72.4; external validation and the NF4 gradient repair v0.73.0; disk tier rewritten v0.76.0; MoE bases, FP8 sources and three opt-in keys v0.77.0)

The usual answer to "this model does not fit" is quantize harder, rent a bigger GPU, or give up. Layer streaming is a fourth answer: never load the frozen base at all. Soup keeps it in CPU RAM (or on NVMe) and copies it into VRAM one decoder layer at a time, so peak VRAM is bounded by the size of one layer instead of the whole model.

Quantizing that streamed base to NF4 shrinks it about fourfold, and that is what turns the idea into a useful one:

Llama-3.1-8B fine-tunes on a 4 GB laptop GPU at 119.6 tok/s at 952 MHz in 3.32 GB — on a card that cannot hold even a quarter of the model. (That row was measured on v0.72.2, before the NF4 gradient repair. The post-#331 row, on another RTX 3050 Laptop 4 GB (@umran666, #1103), reads 208.6 tok/s at 1935 to 1957 MHz in a 2.40 GB peak, and the two are not directly comparable: see Measured numbers.)

This ships BETA, and both the scope and the claim below are narrow on purpose.

If you would rather watch it than read it: the 90-second run shows the pre-flight panel (a 3.60 GB base store pinned in RAM across 32 layers, two 113 MB VRAM buffers) and then the measurement card settling at 3.32 GB and 119.6 tok/s (the pre-repair v0.72.2 figure at 952 MHz, from the table below), stopping short of the 4 GB line. The video was recorded on v0.72.2: since v0.74.0 the panel also shows a large-layer buffer line, and the store is larger because the embedding and output head moved into it. The mechanism is written up in a preprint, summarised on the paper page.

v0.72.3 widened it. The first three releases were deliberately narrow: Llama and Qwen only, batch size 1, no gradient accumulation, no resume, RAM only. All of that is lifted. Eleven architectures are supported as of v0.77.0 (ten from v0.74.0), batches and accumulation work, --resume works, a pre-flight predicts peak VRAM and refuses a run that will not fit, and an NVMe overflow tier handles a base too big even for RAM. See Scaling a streaming run for the sizing rules, the batch-versus-accumulation measurement and the disk tier.

v0.72.4 added alignment. Streaming used to mean supervised fine-tuning only. dpo, orpo, simpo and kto now run against a streamed base, and DPO's reference model is the same streamed base with its adapters switched off, so it needs no second copy of the model. (On an untied checkpoint, dpo and kto do hold one copy of the output head per step, since v0.76.0 a documented and pre-flight-charged cost.) See Preference losses over streaming.

v0.76.0 rewrote the disk tier. It used to read each layer from the compute thread through a memory-mapped file. It now reads on a background thread through direct I/O, stages layers in host memory, can stripe the cache across several NVMe drives, and has one new key (training.stream_read_ahead) and one new environment variable (SOUP_LAYER_STREAM_STRIPE_DIRS). It was also measured for the first time, on a different machine from the 4 GB laptop rows on this page. See the disk tier for the knobs, the figures and what they do not show.

v0.77.0 reached Mixture-of-Experts bases. deepseek_v3 joins the allowlist, DeepSeek-V3's FP8 block-quantised checkpoints are dequantised at shard time, Kimi-K2-Base and -Instruct are admitted through a field-by-field config check (verified on CPU with a tiny K2-shaped checkpoint only; nothing has been trained on K2), and training.stream_expert_layout: per_expert streams one routed expert at a time. Three opt-in keys arrived beside it: stream_recompute, stream_loss_chunk and stream_nf4_scale_cache. See Streaming a Mixture-of-Experts base and Scaling a streaming run.

Read this first if you already trained with streaming

Four defects in this feature's history produced runs that looked completely healthy and were not. Each is narrow, each comes with a way to tell whether your run was affected, and none affects a run outside its stated scope.

If you streamed a 32B or larger model in NF4

Above roughly 165 MiB per decoder layer, a streamed NF4 run produced silently wrong gradients. In practice that means a 32B or a 72B: 8B sits at 105 MiB per layer and 14B at 132, both below the boundary with margin, and both verified exact over a 50-backward soak. Filed as issue #331.

The forward stayed bit-exact throughout, and the loss matched a resident reference to every digit, so nothing in a training log could show it. The cause is buffer aliasing rather than a race: bitsandbytes keeps the packed 4-bit weight outside PyTorch's save_for_backward mechanism, so gradient checkpointing cannot recompute it, and the backward reads a streaming slot that has already been refilled with a different layer. bf16 was never affected, at any size.

It was present in v0.72.0 through v0.72.4, and it is repaired in v0.73.0. The repair keeps streamed NF4 weights out of that code path entirely, and it is gated on two real models against a control that reproduced the defect in the same process: 256 of 256 gradient tensors exact at 32B, and 320 of 320 at 72B, the size where the defect was worst. It costs about 3% of peak VRAM and about 5% of throughput at 32B. If you streamed a 32B or larger model in NF4 on any earlier version, re-run it on v0.73.0; if you streamed 8B or smaller, there is nothing to do. The full diagnosis, the threshold measurement and the repair's cost.

If you trained with streaming on v0.72.0

That adapter is inert. Re-run it on v0.72.1.

The streaming wrapper held the real decoder layer as a child named inner, so every adapter tensor was written to disk under a key carrying an extra .inner. segment:

base_model.model.model.layers.0.inner.self_attn.q_proj.lora_A.weight
                               ^^^^^^

Loading that file into an ordinary model matches nothing. PEFT reports missing keys as a UserWarning rather than an error, so soup merge, soup serve, soup chat and PeftModel.from_pretrained all completed normally and handed back the untuned base model. Nothing failed, and nothing looked wrong.

Check a file you already have:

python -c "from safetensors.torch import load_file; \
print([k for k in load_file('adapter_model.safetensors') if '.inner.' in k][:3])"

A non-empty list means the adapter is affected. Since v0.76.0 there is a command for the same check: soup adapters check ./out/adapter ends in inactive: leaked .inner. keys detected (n tensors) and exits 2 on such a file (it needs numpy and safetensors, which [train] installs). More on soup adapters check.

What was not affected matters just as much. The training itself was correct: the streamed run's numerics are unchanged and v0.72.0's bit-exactness results still stand. Only the saved file was wrong. There is no way to restore a broken file's association with its base model beyond renaming keys, so re-running is the reliable path. On v0.72.1 a streamed adapter saves byte-for-byte in the same layout as an ordinary LoRA run, portable to any tool that has never heard of layer streaming.

If you trained with streaming on peft 0.21 or newer

That adapter is empty. Re-train it on v0.75.1 or newer.

This is the same class as the defect above, found again at a different seam, and this time the file has nothing in it at all rather than the wrong key names.

peft 0.21 changed how it selects an adapter's tensors: it reads prefixes off model.named_modules() and keeps the matching state_dict() entries, where 0.20 filtered by the lora_ substring. The streaming wrapper's module and parameter names still carried .inner. while its state_dict() keys were canonical, so the two no longer met in the middle. trainer.save_model(), every save_steps checkpoint and get_peft_model_state_dict() all returned nothing, and nothing raised.

What lands on disk is a 40-byte adapter_model.safetensors holding zero tensors. Check a file you already have:

python -c "from safetensors.torch import load_file; \
print(len(load_file('adapter_model.safetensors')))"

0 means the run produced nothing; soup adapters check ./out/adapter reports the same file as inactive: no lora weights found and exits 2. It cannot be recovered: there is no key to rename and no association to restore, so the only remedy is to train it again, either on v0.75.1 or newer, or on a Soup older than this with pip install "peft<0.21" to hold peft below the change.

Soup does not pin peft below 0.21, and that is worth saying plainly rather than leaving to be discovered: the declared bound is peft>=0.20.0,<1.0.0 at this tag, so a fresh install resolves 0.21 by default. The fix is the rename, not a pin. The wrapper now enumerates the inner layer's tree at its own prefix, so names and keys are one spelling.

Since v0.76.0 a streamed run checks this itself. After the final save and after every save_steps checkpoint, the streamed trainers (sft, dpo, orpo, simpo, kto) read the adapter file's keys back and stop with a RuntimeError naming the file (and the tensor counts where they differ) if the file is missing, holds no LoRA tensors, holds a different number than the model trained, or carries any key with the wrapper's .inner. segment. Resident runs are unaffected. The manual checks above are still the way to inspect an adapter saved by an earlier version.

One more thing rode along with that rename: transformers builds its weight-decay grouping from named_children(), and with names and keys now one spelling that grouping cannot drop every LoRA parameter into the no-decay group. Upstream does not say whether the old spelling did.

v0.72.1 also closed a second hole in the same area: the guard refusing --resume for a streaming run only tested that one flag, so --hf-resume reached resume_from around it. That combination used to appear to work by accident, and once adapters save canonically it would instead have matched nothing and continued with a freshly initialised adapter, silently. Both flags were refused until v0.72.3 fixed the load side and unblocked them; see Scaling a streaming run.

If you streamed an MoE base with moe_lora: true before v0.77.0

Some of the modules you asked for got no adapter. In upstream's words, the streaming setup replaced the resolved target_modules list with the moe_lora list, so "on Qwen3.5-MoE the linear-attention in_proj_qkv / out_proj were dropped; and an explicit lora.target_modules list was dropped entirely" (#2072). A string target_modules under moe_lora (a regex, all-linear, or auto on qwen4_exp) was replaced by the MoE list as well, resident or streamed, and a regex "was dropped outright" (#2095). Both were verified in the code at v0.76.0 and v0.76.1. Check a file you already have by listing the module names its tensors belong to:

python -c "from safetensors.torch import load_file; \
print(sorted({k.split('.lora_')[0].rsplit('.', 1)[-1] for k in load_file('adapter_model.safetensors')}))"

If a module you named is missing and it mattered, re-train on v0.77.0, which adapts the union of your targets and the MoE targets, in the same order as a resident run of the same config. A third case, target_modules: auto on a DeepSeek-V3 base with q_lora_rank: null (#2071), is on Streaming a Mixture-of-Experts base.

All four defects above share a shape, and it is worth naming rather than smoothing over. v0.72.0 passed every correctness gate it set itself and shipped a clean throughput table, and no test in that release saved an adapter and loaded it back. #331 passed every gate too, on hardware where the reference it needed could not exist. In each case the run exited 0 and the loss curve looked healthy. A green suite measures what it was pointed at.

What is actually new here

Layer streaming itself is not new, and fine-tuning on a small card is not new. An early draft of this page claimed no published system does this; that claim was fact-checked against the sources and cut, because it is false. What is true is narrower:

• For inference, streaming a small card is well established. AirLLM runs 70B-class models through a 4 GB GPU, and its own documentation states that layer sharding is not compatible with gradient propagation. Hugging Face's disk_offload is documented as big-model inference and runs under no_grad.
• For training, the published small-card result already exists: LSP-Offload (arXiv:2406.10181) fine-tunes 1.3B on a 4 GB laptop GPU. It gets there by compressing the optimisation into a learned sparse subspace, not by streaming weights.
• For training, streaming layers through a small device memory is published too. Layer streaming is Soup's implementation of layer-to-layer streaming of a frozen base (L2L, Pudipeddi et al. 2020, "Training Large Neural Networks with Constant Memory using a New Execution Algorithm", arXiv:2002.05645), in the words of upstream's v0.77.0 release notes, and upstream's records name L2L as the prior art for streaming layers through a small device memory. Soup does not implement L2L's layer-major micro-batching: nothing in src/ at v0.77.0 does. The preprint does not cite L2L; upstream says its next version will.

So the honest headline is not "nobody has done this", and the method is not new. It is up to 8B on a 4 GB card, by streaming the base rather than approximating the optimisation, with the forward pass verified bit-exact against a resident run at the same precision — and packaged as two config keys in a general-purpose CLI rather than a research prototype.

Turn it on

It is a config key, not a CLI flag:

training:
  stream_layers: true          # enable layer streaming
  quantization: 4bit           # NF4: ~4x smaller store, so 8B fits a 4 GB card
  stream_source: auto          # RAM when the base fits, NVMe disk when it does not
  stream_buffers: 2            # double-buffering; range [2, 8], default 2
  # stream_pin: false          # v0.74.0: force the pageable store, and print what it costs
  # stream_read_ahead: 2       # v0.76.0, disk tier only: layers the reader stages (1-8)

  batch_size: 4                # any concrete value; the pre-flight sizes it for you

quantization accepts none (bf16) or 4bit (NF4) and nothing else. NF4 is what you want unless you have a specific reason to stream bf16: it shrinks the RAM store about fourfold, and a smaller store is far more likely to page-lock, which is worth more than the arithmetic (see below). Every streaming key, with its default and the release that added it, is in the knob reference.

# There is no --stream-layers flag. You train exactly as before:
soup train --config soup.yaml

The one value batch_size cannot take is auto. The auto-batch probe sizes a resident model, which a streaming run never loads, so it is refused by name. Set a concrete number instead: as of v0.72.3 a pre-flight predicts peak VRAM for that exact batch and sequence length and refuses the run before any GPU work if it will not fit. Reach for batch_size before gradient_accumulation_steps, because at the same effective batch a real batch measured about 2.52x faster than accumulating. Why, and how to size it.

How it works

The base model is built on PyTorch's meta device, so its weights never occupy VRAM. Only the embeddings, the final norm, an untied LM head and the LoRA adapters with their gradients and optimizer state get real storage on the card, and those are small. The checkpoint is rewritten once into one safetensors shard per decoder layer, and that store is held in CPU RAM, page-locked when the machine allows it, or read from an NVMe disk tier when the base is too large for RAM.

During the step, each decoder layer is copied into one of N pre-allocated VRAM buffers on a dedicated CUDA stream, so the load of layer i+1 overlaps the compute of layer i. The pooled buffer is substituted into the unmodified decoder layer, which means the same kernels run on the same weight bytes.

Each layer is read twice per step — once in the forward pass and once when the backward pass recomputes it — because dL/dx = Wᵀ · dL/dy needs the weights to push gradient down to the layers below. That is physics, not an implementation detail, and it is why streaming costs time rather than being free. (Since v0.77.0, training.stream_recompute: off keeps a layer's activations instead of running its forward a second time, but the backward still reads the weights again: details.)

Model code is never the repository's. Under stream_layers the model config, the meta skeleton used by the setup probes and the streamed model are transformers' own classes, whatever --trust-remote-code says. Since v0.77.0 that flag reaches only the tokenizer (#1933), which is where a base such as Moonlight-16B-A3B needs it. A base that has only repository model code cannot be streamed.

Correctness is not part of the trade. A streamed forward pass was verified bit-exact against a resident one, and a 100-step streamed loss curve matched resident exactly.

Scope of the NF4 comparison. The NF4 comparison is against a resident NF4 model that is a plain quantised load, without PEFT's prepare_model_for_kbit_training. The default resident 4-bit or 8-bit SFT path does call that function, and it casts every non-quantised parameter (the embedding, the norms and an untied head) to float32, whereas streaming keeps those tensors in the store dtype. So a default resident run and a streamed run start from a different numerical setup even with identical quantised blocks. Upstream's performance doc reports one contributor case, marked as not part of its benchmark record: on one forward pass of one text, a default resident run differed from streaming by a maximum logit difference of 0.52 while the control without the upcast matched streaming bit for bit. That says nothing about the backward or a training trajectory, and the LoRA initialisation differs by construction (streaming materialises the adapters from a seeded CPU generator).

Bit-exact is two claims, not one

Treat "bit-exact" as two claims, not one, because they can come apart: the forward (the logits) and the backward (every LoRA gradient tensor) are measured independently. On the development box both hold. On borrowed H100 hardware, where a resident reference for a real model can actually exist, the forward is exact up to 72B while the backward turned out to be wrong above roughly 165 MiB per NF4 layer, which is the defect described above. Whenever this page or any other says a run is bit-exact, it means both halves at the size stated, and nowhere else. The full ledger.

There was a third qualifier in v0.75.0, and it was the card. On an RTX 5070 (sm_120, Blackwell) the two CUDA-only streamed-NF4 exactness tests failed at 4.9e-4 in fp16 and 3.9e-3 in bf16 against resident NF4, while the quantization: none variants of the same tests passed (tracked as #776). That split was the useful part of the report: layer streaming itself was still exact there, and the divergence was in the bitsandbytes NF4 path on that architecture. It was not the defect v0.73.0 repaired, which had a megabytes-per-layer boundary rather than a card-architecture one. Upstream's later investigation attributes the gap to which of two bitsandbytes 4-bit paths each model ran, not to streaming: bitsandbytes uses a fused GEMM for a small number of rows and dequantise-plus-matmul above that, resident NF4 took the fused path where it applied, and Soup's streamed path always dequantised. Upstream reports that two resident NF4 models with byte-identical weights differ by about the same amount purely on which path they take. Upstream's test pins the boundary at 32 and 33 leading rows, found by bisection on that card with bitsandbytes 0.50.2: a gap above zero at 32 rows and exactly 0.0 at 33.

v0.76.0 changes the code. This page does not upgrade the evidence. #1143 routes the streamed NF4 forward through bitsandbytes' own fused-versus-dequantized dispatch, so it runs the same kernels as resident NF4. The backward still dequantises, because bitsandbytes exposes no transposed 4-bit GEMM. The only evidence that this closes the Blackwell inexactness is contributor-credit text in upstream's CONTRIBUTORS.md (the CUDA exactness tests passing at exactly 0.0 on that RTX 5070) and the tests themselves. benchmarks/ has no row recorded after #1143, and #776 is still open in the records. So the supportable statement is that the dispatch changed; it is not that "verified bit-exact" now holds for the new dispatch. CI has no GPU runner, and these tests skip on every hosted platform, so only a card shows it.

So the full form of the claim has four parts: which half (forward or backward), which quantisation, which size, and which card (and, for NF4 at small row counts, which bitsandbytes path). Every CUDA exactness result published on this site was measured on Ampere or Hopper, apart from the Blackwell failures above, one forward check on a Turing T4 (see the Colab section), and two v0.77.0 checks on an RTX 5070 (Blackwell): the expert-streaming gate on OLMoE layers, which compares one streamed experts block with the resident one, and the stream_recompute gate, which compares two streamed modes with each other, not with a resident run. The CPU-fixture and Apple MPS toy-model controls are separate and say nothing about a CUDA card. The disk-tier timings under the disk tier were taken on a Blackwell card and are timings only.

Measured numbers

Measured on the development box: RTX 3050 Laptop 4 GB, Windows 11, 16.9 GB RAM, LoRA, batch 1, gradient checkpointing on, PagedAdamW8bit, 50 steps after 10 warm-up.

Read the 8B rate as a v0.72.2 figure. Every row above was measured on v0.72.0 or v0.72.2, before the NF4 gradient repair that shipped in v0.73.0. That repair cost 4.8% throughput at 32B. The post-#331 row, measured by @umran666 on another RTX 3050 Laptop 4 GB (#1103), reads 208.6 tok/s at 1935 to 1957 MHz in a 2.40 GB peak; the 119.6 tok/s row was taken at 952 MHz, before the repair. The two are not directly comparable. In upstream's words, that is 59% of its same-session GEMM ceiling against 68%: about 14% slower per unit clock, so the raw 1.74x gain is the clock, not a speedup. Large-layer streaming also changed the memory pattern between the two rows, so even that fraction is not like-for-like, and the record attributes the shortfall to none of the repair, the memory change or the stack.

The two 8B rows side by side, each with the conditions it was taken under:

The post-repair row is one 50-step run after 10 warm-up steps, batch 1, sequence 512, with the slowest step (3479.0 ms against a median of 2431.5 ms) kept in the record. It was measured by @umran666 on their own laptop (#1103). It was run from main before the v0.76.0 tag existed, not from the tag. The two rows are different memory plans as well as different clocks: large-layer streaming moved embed_tokens and lm_head into the pinned store between them, so the store grew while the peak fell. The two also ran on different software stacks (torch 2.5.1, transformers 4.57.6, peft 0.18.1 and trl 0.19.1 on the first; torch 2.6.0, transformers 5.16.1, peft 0.21.0, trl 0.29.1 and bitsandbytes 0.50.2 on the second), which the record lists without attributing any difference to them. The record keeps the original rows verbatim and adds the new one beneath them.

A server card cannot stand in for either measurement, which the H100 replication showed by coming back no faster rather than faster, so throughput here does not carry across machines. The H100 cross-check is a median 113.00 tok/s in the same 3.32 GB peak, five runs, which is a cross-hardware reproduction rather than a re-measurement, and it adds no post-repair claim.

To fit the 8B row on a 4 GB card, set data.max_length: 512; the default is 2048. The measured table is at sequence 512. Upstream's performance doc reports, from a probe on the same card on v0.73.0 code (LoRA r=8 on q, k, v and o), Llama-3.1-8B NF4 streamed at sequence 512 peaking at 3.448 GB with a microbatch of 1, and about 1.06 GB more for each extra microbatch of 512 tokens, so a microbatch of 2 (1024 tokens per step) peaked at 4.512 GB, already above the card's 4.294 GB. Batch 1 at sequence 1024 was not measured. That is the probe's reading under its own LoRA configuration, not a restatement of the 3.32 GB row.

For scale, 1M training tokens is about 2.3 hours at the pre-repair 119.6 tok/s on that card. That is arithmetic from the measured rate, not a separate measurement.

The honest cost: 1.43x slower than resident training, measured at 0.5B. That is the only apples-to-apples comparison available on this box, because 1.5B and above cannot run resident here at all.

Why NF4 also made the 3B row faster

The same model went from 143.1 to 264.2 tok/s, and the reason is not that 4-bit arithmetic is faster. A 1.43 GB store fits under this machine's page-locked memory ceiling where a 5.55 GB one did not. Pinned memory is what lets the host-to-device copy run asynchronously, so the layer loads hide behind compute again, and utilisation goes from 79.3% to 100%.

Treat the mechanism as the claim, not the multiplier. Those two rows come from different sessions and this card's boost clock varies by about 13% between sessions, so the exact factor is indicative. What is solid is the cause: pinning restores overlap, and NF4 is what makes pinning possible at this size.

The same reasoning explains the bf16 3B row's 79.3%. It is not a model-size effect: that box could not page-lock a 5.55 GB base, so the run fell back to a pageable store. Soup performs that fallback automatically and prints what it costs instead of absorbing it silently. The bf16 3B throughput is therefore a lower bound.

A later finding about page-locking. For a 14B NF4 store, upstream found that torch's host allocator rounds every pinned request up to the next power of two, and the store used to be pinned one tensor at a time, so a store cost well over its own size in page-locked memory (the 14B record measured 1.73x). Since v0.76.0 the store is pinned in a few power-of-two arenas and, when that page-locks fewer bytes, packed largest tensor first. The measured mechanism above (pinned copies overlap, pageable ones are synchronous) is unchanged, the 3B bf16 row is still a lower bound for the reason it was measured, and it has not been re-run. The page-lock ceiling figures that older records print are not repeated here. The arena change is a memory change with no speed claim, and it has no setting. The 14B record.

One number worth knowing before you size a run: at the time this table was measured, the untied embed_tokens and lm_head stayed resident and unquantised, and they account for 2.10 GB of the 8B row's 3.32 GB. Roughly two thirds of that peak was not streamed at all, which is exactly why 8B sat close to this card's ceiling.

v0.74.0 changed that, and the first table above is deliberately not restated. An untied embedding and LM head are now sharded separately and reuse one vocabulary-sized device buffer instead of both staying resident, so the 8B peak moved. A peak taken after that change is in the post-repair row above, on a different laptop: 2.40 GB allocated (2.66 GB reserved) with a 5.70 GB pinned store, against 3.32 GB before; the store grew by exactly the two 0.525B bf16 tensors (2.10 GB) that moved into it. Upstream's docs say the historical figure is not relabelled as a new measurement, and it is not relabelled here. Tied embeddings keep the existing resident path and numerics. The rows in the first table stand as what they were: v0.72.0 and v0.72.2, before both the NF4 gradient repair and this change. Since v0.76.0, dpo and kto on an untied checkpoint also take a private copy of the output head's weight for the step, which the VRAM pre-flight charges as a second large-layer slot; sft, orpo, simpo and tied checkpoints do not (details).

Be careful how utilisation is argued in general: GPU utilisation on its own only says a kernel was resident, which is necessary for overlap but not sufficient to prove it. The evidence is the step arithmetic, not the utilisation column.

Run it yourself on a free Colab T4

Open the proof notebook in Colab and the argument runs on a card you do not have to own: a free-tier Google Colab Tesla T4 (sm_75, Turing, 15.6 GB). The notebook holds the process to 4.00 GB with torch.cuda.set_per_process_memory_fraction before anything loads, then proves the cap bites rather than assuming it, by asking for 4.29 GiB and being refused.

The notebook installs a released Soup, pinned, and that release is v0.75.0, not v0.76.x or v0.77.0. At the v0.77.0 tag its install cell is still soup-cli[train]==0.75.0 from PyPI, beside huggingface-hub<1.32.0 (v0.75.0 refuses the shared blob store that huggingface-hub 1.32 introduced, #1509); only its text and comments changed in that release. Its install section says why: 0.75.0 is the exact release its committed outputs were produced against, and the pin is to be bumped only after re-running the notebook end to end on a real T4. So what it prints is a v0.75.0 reading, and no run on v0.76.x or v0.77.0 is recorded. It also sets CUDA_VISIBLE_DEVICES to a single card before torch is imported, because layer streaming cannot run under nn.DataParallel. On Colab's default Python 3.13 the install cell passes --ignore-requires-python, because the pinned 0.75.0 declares <3.13, which trades that guarantee away. Soup itself supports Python 3.11 to 3.13 since v0.77.0, and the notebook says the fallback goes away when its pin moves to v0.77 or later. Through v0.75.2 the notebook installed from git+, which resolves to the default branch whichever revision you opened.

Two runs of this kind are recorded, under different memory plans, and each is reported on its own terms.

• The first run printed Meta-Llama-3.1-8B-Instruct, NF4, stream_layers: true, stream_buffers: 2, batch 1, max_length: 256, LoRA r=8, fp16 (a T4 has no bf16 hardware). Seven steps, exit 0, an adapter written with 128 of 128 tensors non-zero, and a measured peak of 2.91 GB against the pre-flight's predicted 3.02, an over-prediction of 3.8% in the direction the estimator was fitted to err. Its library versions were not recorded.
• The second run is the notebook as committed with its outputs: soup-cli 0.75.0, torch 2.11.0, transformers 5.16.1, peft 0.20.0, bitsandbytes 0.50.2. Its section 4 compares a streamed forward pass with a resident one on SmolLM2-135M-Instruct in fp16, unquantized, and printed a maximum difference of 0.0 (torch.equal = True). Its section 5 trained the 8B NF4 configuration again under the 4.00 GB cap with 128 of 128 non-zero tensors, on a different memory plan from the first run (the embedding and output head are streamed, and the pinned store is larger), so its peak is in the record and is not set against the first run's.

Read the frame before you read either number:

• No throughput is claimed, here or in the notebook. A card held under an artificial memory cap is not a benchmark. The pre-flight panel's tok/s line is a compute bound derived from a GEMM probe on that card, not a measurement of what the run achieved. The first run's bracket is marked stale upstream (#714): that probe measured the GEMM ceiling in bf16 through Turing's software emulation, and a corrected probe prints a different bracket that upstream also leaves unvalidated against a measured rate. No bracket is repeated here as a rate.
• Gradient exactness on Turing is not shown. An adapter with 128 non-zero tensors proves gradients flowed, not that they were right, and that distinction is exactly the shape of the NF4 defect v0.73.0 repaired. The streamed-versus-resident comparison was empty on the first session. The second run's comparison is a forward pass, on a 135M model, in fp16, with no quantisation. It establishes forward exactness on Turing for that case and nothing at 8B, nothing in NF4 and nothing about the backward.
• On a capped card the pre-flight reads the whole device. torch.cuda.mem_get_info() cannot see a per-process cap, so the panel reported 15.10 GB free while the run was held to 4.00. Nothing was harmed here, since the runs genuinely fit, but on Colab, Kaggle or a MIG slice the pre-flight is not what enforces your real budget. The second run also set the cap as training.stream_vram_override, so the pre-flight read 4.00 GB free instead of the whole card.
• One run each, one seed, one configuration, no repeats, on sessions that cannot be returned to.

What it does establish is worth having on its own: the streaming path executes end to end on a pre-Ampere card, at 8B, inside a 4 GB process budget. Before this it had never been run there. The full record, including the longer list of what it does not establish.

Honest scope

This ships BETA, and the claims stop where the measurements stop.

• Models measured on the 4 GB card: Llama-3.1-8B under NF4, plus Qwen2.5-0.5B, 1.5B and 3B, and a live soup train on SmolLM2-135M. Nothing above 8B has been measured on this hardware, so no number on this page speaks for a larger model. Sizes above that were later measured on a borrowed 8x H100 box, up to 72B, and what those runs do and do not establish is a page of its own. Note the host side is the real gate on reproducing them: the 72B run needed a 33.74 GB pinned RAM store, which a 16.9 GB laptop cannot hold at all. The 14B shape was later trained end to end on an 8 GB card in v0.76.0, a memory result with synthetic weights and no throughput or quality claim: 14B on an 8 GB card.
• Eleven architectures (llama, qwen2, qwen3, mistral, gemma, gemma2, gemma3_text, phi, phi3, qwen4_exp since v0.74.0 and deepseek_v3 since v0.77.0, plus the qwen3_5 dense and MoE aliases; a Kimi-K2-Base or -Instruct config streams as deepseek_v3 behind a config check). For the original nine, forward logits were compared with the same model loaded resident under both bf16 and NF4 and came out equal (tiny from-config CPU float32 fixtures, not real checkpoints; layer-0 gradients were checked non-zero, not equal; Llama and Qwen were also compared on real checkpoints, see Validation on hardware we do not own); the v0.74.0 and v0.77.0 additions carry weaker evidence and say so, and Qwen4-Exp streams only with task: sft and quantization: none. No throughput is claimed on this page for any family beyond Llama and Qwen, and every tok/s figure on it is Llama or Qwen. Upstream's v0.77.0 records time two other families on an RTX 5070 Laptop 8 GB, a different card from the 4 GB rows: Mistral-7B NF4, as relative step-time changes from stream_recompute, and Moonlight-16B (deepseek_v3), as GPU memory and step time per expert layout. Those are quoted with their scope where the keys are described, never as a tok/s for the family. Multimodal gemma3 is refused; only gemma3_text is accepted.
• Mixture-of-Experts bases (v0.77.0). The DeepSeek-V3 family streams, with FP8 block-quantised sources dequantised at shard time and a per-expert layout that streams one routed expert at a time. Kimi-K2-Base and -Instruct are admitted through a config check verified on CPU with a tiny K2-shaped checkpoint only; nothing has been trained on K2, and resident Kimi K2 is refused at load. No preference-loss parity was measured on a DeepSeek-V3-family base. Streaming a Mixture-of-Experts base.
• task must be one of sft, dpo, orpo, simpo, kto; backend: transformers; modality: text. The four preference losses landed in v0.72.4, and no throughput figure is claimed for any of them, because none was measured. ipo, bco and the unified task: preference dispatcher are still refused. See Preference losses over streaming.
• quantization must be none or 4bit. Other quantisations store weights in formats that cannot be streamed into a pooled buffer. The same holds for the base checkpoint itself: since v0.77.0 a checkpoint saved already quantised (GPTQ, AWQ, bitsandbytes, compressed-tensors and the like) is refused by name before a byte of the cache is written (#2103), and the exceptions are DeepSeek-V3's FP8 block format, which is dequantised at shard time (#2061), and Qwen4-Exp's oQ bundles.
• batch_size cannot be auto — the auto-batch probe sizes a resident model. Any concrete value is fine, subject to the VRAM pre-flight.
• Loading an adapter into a streamed run works as of v0.72.3. In memory, named_parameters() and state_dict() disagreed until v0.75.1, the deliberate cost of a serialisation-only fix, and peft 0.21 turned that into an adapter saved with nothing in it. Since closed in v0.75.1: names and keys are one spelling. End-to-end soup train --resume could not be demonstrated on the development box, for a reason that has nothing to do with streaming: transformers refuses torch.load below torch 2.6 under CVE-2025-32434, which blocked every resume there. (Since unreachable in v0.75.0, which raised the declared floor to torch>=2.6.0.)
• The disk tier was first measured in v0.76.0, on a different machine from the 4 GB laptop rows on this page: an RTX 5070 Laptop 8 GB with 31.7 GB of RAM and two NVMe drives, on a synthetic 70B-shaped NF4 store of 36 GB with random weights. A cold step took 110.5 s on the previous synchronous reader and 16.2 to 17.3 s on the direct-I/O reader; across two NVMe drives a 17.4 s step fell to 9.97 s. A sustained-load probe measured 1.28x and 1.41x in two rounds and reached no formal verdict. A store that fits RAM is still faster on the RAM tier, which stream_source: auto picks. No disk-tier throughput exists for a 4 GB card, and the figures are timings with no quality claim. The synchronous disk tier's correctness was verified bit-exact against the RAM tier in v0.72.3; the v0.76.0 reader is gated by byte-identity against that source. The figures and what they do not show.
• Plain LoRA only, on the decoder layers. DoRA, VeRA and the PiSSA / OLoRA / LoftQ initialisations all read a real base weight, which streaming keeps on the meta device at adapter-build time. use_rslora is fine. A LoRA target on the streamed embed_tokens or lm_head of an untied checkpoint is refused, whatever target_modules spelling matched it (a name, a regex, a dotted path): the forward pass is correct, but saving or resuming that adapter is not supported. Since v0.77.0 the check runs when the model is built, after the adapter is attached and before any streaming buffer is allocated, not at config load, so soup train --dry-run passes such a config (#2082). A tied checkpoint streams neither module, so a target there trains and saves with the defaults; v0.76.x refused the literal names there too. training.fast_lora (v0.77.0) is admitted on a streamed sft run and needs lora.dropout: 0 (the default 0.05 is refused); upstream verified it on CPU (float32 and float64 against peft) only, the bf16/fp16, NF4 and streamed-CUDA checks have not been run, and no speedup is claimed.
• Pre-Ampere cards (T4, P100, V100, GTX 16xx, RTX 20xx) stream in fp16 rather than bf16. Until this fix the store dtype was the literal "bf16 on any CUDA device", so the whole free notebook tier ran in a dtype its GPU has no units for, and it could not fail on the Ampere card every number above was measured on. fp16 is bit-exact against a resident reference of matching numerics, 0.000000e+00, in both quantisations. Two honest edges. The capability question has to be asked as is_bf16_supported(including_emulation=False), because the bare call counts software emulation and a T4 answers True to it, which made the first version of this fix a no-op on exactly the hardware it targeted. And that exactness was measured using fp16 on an Ampere card, so it establishes the plumbing and not the Turing or Pascal kernels. It shipped in v0.73.1, where it turned out not to be a streaming fix at all: the same assumption sat in fourteen places, so every task died on that hardware, not only a streamed one.
• Apple Silicon is experimental. backend: transformers on MPS streams from a pageable CPU source into MPS layer buffers; host pinning does not apply there, so stream_pin: true is announced and the run proceeds. bf16 is used only after a live MPS capability probe and float32 otherwise. Upstream's docs report an untied float32 toy decoder bit-exact against its resident MPS control, and make no claim that streaming fits a larger model or runs faster than resident MPS training. backend: mlx is refused with stream_layers.
• Numbers are Windows/WDDM and therefore systematically pessimistic versus Linux. expandable_segments:True is silently ignored on Windows, and Soup detects that rather than claiming it is active. On Linux (WSL2 included), since v0.77.0 a streamed soup train sets PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True before CUDA starts unless you exported a value of your own (an empty one counts as unset, #2156). Before that, soup train had already started CUDA when it detected the device, so every streamed CUDA run printed expandable_segments allocator hint not enabled: CUDA was already initialised before Soup could set it.
• If the model already fits resident on your card, do not enable it. Streaming trades time for memory, and there is nothing to buy when memory is not the constraint.

Rejected at config load

Every refusal comes before training starts, and the config-parse ones before a single GPU byte is touched. Where a later release lifts a limit, the refusal message names that release; the rows marked n/a are refused on a structural ground and carry no roadmap promise either way.

None of this constrains an ordinary resident run. These are the refusals a run with stream_layers: true can hit, and several of the rows below name things that are perfectly legal, and common, with streaming off.

() One honest footnote: not every row here is a config-parse validator. The rows marked () need the model's own config or checkpoint, so they run a moment later, at trainer setup or when the streamed model is built (since v0.77.0 that includes the Kimi K2 check, the quantised-source and FP8-format refusals, per_expert on another architecture, a LoRA target on an untied head or embedding, and the stream_loss_chunk head checks), and so do the checks that read the machine: the RAM fit and the physical-host ceiling, the disk-tier host-staging check, the NVMe rule, the stream_pin: true page-lock refusal, the meta-device adapter guard and the predicted-peak VRAM refusal. They all still come before training starts, but soup train --dry-run validates the config and the data and stops there, so if you are counting on it to catch everything, those are the ones that arrive late.

The generic pre-flight hardware-fit gate is skipped for streaming runs, because it models a resident run and would otherwise refuse exactly the runs streaming exists to enable. Since v0.72.3 a streaming-specific budget replaces it, and that one does refuse: it predicts peak VRAM from your batch, sequence length and vocabulary, which are the terms streaming does not bound. See Scaling a streaming run.

Full config

base: meta-llama/Llama-3.1-8B-Instruct
task: sft
backend: transformers

data:
  train: ./data.jsonl
  format: alpaca
  max_length: 512
  val_split: 0.1

training:
  epochs: 3
  lr: 2e-5
  batch_size: 4                    # v0.72.3: bigger batches amortise the weight read
  gradient_accumulation_steps: 2   # v0.72.3: values above 1 are allowed
  quantization: 4bit               # NF4; 'none' streams bf16 instead
  gradient_checkpointing: true     # handled per-layer by the streamer
  stream_layers: true
  stream_source: auto              # RAM when it fits, NVMe disk when it does not
  stream_buffers: 2
  lora:
    r: 64
    alpha: 16

output: ./output

Gradient checkpointing is handled per layer by the streamer, and the Hugging Face Trainer's own is left off so layers are not recomputed twice.

That batch_size: 4 is illustrative, not a recommendation: whether it fits depends on your card, the model's vocabulary and max_length, and the pre-flight will tell you before the run starts. Sizing rules.

Troubleshooting

• "a streaming step is predicted to need X GB of VRAM but only Y GB is free" — the v0.72.3 pre-flight. Streaming bounds the weights, not the activations or the logits, and both of those scale with batch_size times data.max_length. Lower either one. On a large-vocabulary model the logits tensor is usually the whole story.
• "but only Y GB of RAM is free" — you asked for stream_source: ram and the base does not fit in it. ram means insist, so it refuses rather than falling back. Set stream_source: auto to use the NVMe disk tier instead, free RAM, or pick a smaller base. If you are streaming bf16, quantization: 4bit shrinks the store about fourfold and is often enough on its own.
• "layer streaming needs NVMe or more RAM" — the base does not fit in RAM and the detected disk is not NVMe. soup doctor --disk reports what Soup detected on your machine. On a cloud box, check this one before you believe it: until v0.73.3 a paravirtual disk was read as a spinning one on the strength of a flag it defaults to. Since v0.73.3 the media type is measured when that flag is unreliable, and training.stream_disk_kind: nvme overrides it if detection is still wrong.
• "could not page-lock the base ... falling back to a PAGEABLE RAM store" — expected on a busy machine, and it costs real throughput: pageable memory makes the host-to-device copy synchronous. Training continues, more slowly. Close other applications, or switch to quantization: 4bit so the store is small enough to pin. Since v0.76.0 the store is pinned in a few power-of-two arenas, which asks the host for far less than pinning one tensor at a time did, so a store that fell back before may pin now.
• "layer streaming's disk tier would hold X GB of host staging ... at training.stream_read_ahead=N" — the v0.76.0 host-RAM check on the disk tier. The async reader keeps whole layers in host memory (the embedding and lm_head take one slot each at any depth), and staging plus the resident extras must come in under 70% of free RAM. It is a refusal, never a silent clamp. Lower training.stream_read_ahead, free RAM, or use a smaller base. The disk tier.
• AcceleratorError: CUDA error: out of memory inside SFTTrainer.__init__ with gigabytes of VRAM free — in the case reported as #901 (a 14B on an 8 GB card) this was not about VRAM. A failed page-lock of a large store left a stale CUDA error, the fallback to a pageable store did not clear it, and the first kernel launch reported it as an out-of-memory error. Fixed in v0.76.0: the stale error is cleared before the pageable fallback, and an out-of-memory AcceleratorError during the VRAM probe now counts as an out-of-memory verdict. The 14B record.
• Access violation (exit 139) while sharding a very large checkpoint on Windows — fixed in v0.76.0 (#926 in #934). The sharder used to hold every source safetensors mapping open, which charged the whole checkpoint to Windows commit; it now keeps at most two source handles alive. On the measuring box it appeared at roughly 96 to 106 GB of live mappings; the point depends on the machine's commit limit.
• "layer streaming's disk tier was handed N distinct layer shapes ...", "layer N's shard changed on disk since its header was read ..." and "layer-stream reader has been reading layer N ... past the 300 s limit" — the three disk-tier guards. The first means an index with more than eight different layer shapes (a real model has one to three). The second means the shard cache was re-sharded while a run was reading it; re-shard, restart, and do not re-shard a base that a run is reading. The third means a single read was in flight for more than 300 seconds, which upstream treats as a wedged read, not a slow one; the run stops instead of blocking silently.
• "layer streaming's staging is not sector-aligned on this box; the disk tier reads through the page cache instead of direct I/O" — a yellow notice, not an error. The staging memory was not aligned for direct I/O, so the disk tier reads through the page cache, which gives up what the direct-I/O reader is for.
• "Layer streaming reads <root> through the page cache: direct I/O is unavailable there (<reason>)." — a yellow notice since v0.77.0 (#1962): direct I/O is decided per drive, and that drive's filesystem refused it, so it is read through the page cache. Before v0.77.0 this fallback was logged rather than printed. With a striped cache the ready line also reports direct I/O k/N across the N drives.
• "training.stream_layers does not yet support a LoRA target on the streamed lm_head of an untied checkpoint" (or embed_tokens) — since v0.77.0 this arrives when the model is built, not at config load. Change training.lora.target_modules so it matches neither module, or train without stream_layers (above).
• A message naming a quantised checkpoint and saying layer streaming cannot dequantise it — the base checkpoint is itself quantised (#2103). Point base at a bf16 copy, or for DeepSeek-V3 at its block-FP8 release (MoE streaming).
• "Kimi K2 (model_type 'kimi_k2') ..." followed by a list of fields — the K2 config check refused this config, naming each field that differs (Kimi K2). Only Kimi-K2-Base and -Instruct as published pass it.
• "training.stream_expert_layout='per_expert' is gated on resident parity per architecture" — only deepseek_v3 takes per_expert; use fused on anything else.
• A yellow "training.stream_expert_layout='per_expert' reads every routed expert from the cache volume ... not NVMe" — a warning, not a refusal: the experts are read from the cache volume on every tier, so a slow volume bounds the step.
• "layer streaming does not support model_type=..." — the message prints the full allowlist. Note that a real google/gemma-3-* checkpoint reports gemma3, the vision-capable wrapper, which is refused; the text-only gemma3_text is what streams.
• "Qwen4-Exp layer streaming is initially validated for task='sft' only" or "Qwen4-Exp layer streaming currently requires quantization='none'" — Qwen4-Exp streams only with task: sft and quantization: none; the message says preference-loss and streamed NF4 parity are pending. It comes at trainer setup, before sharding, so soup train --dry-run passes the same config.
• "training.stream_layers supports quantization='none' or '4bit'" — those are the only two. Other quantisations store weights in a form that cannot be streamed into a pooled buffer.
• "layer streaming needs X GB of additional disk space ... Refusing before copying or sharding" — the disk pre-flight, which checks free space on the volume each cache write lands on before it starts. Disk space and the second cache covers where the two caches live and how to move them.
• "base exceeds the physical RAM safety ceiling" — stream_source: auto fell back to the NVMe tier because the RAM tier would have kept the store plus the resident extras above Soup's physical-host ceiling. Set stream_source: ram only if you want that case to refuse instead of falling back.
• "trainable LoRA parameters remain on the meta device after adapter ..." — PEFT attached an adapter without real storage, and Soup refused the run before installing the streaming runtime. The guard looks at the final parameter state rather than at how many tensors Soup materialised, because some PEFT versions create real adapters themselves. If you report it, include the PEFT version and the parameter the message names.
• A streamed adapter asks for --base when you open it — read adapter_config.json. A healthy adapter records the exact configured model reference in base_model_name_or_path. An empty value means it was written by a streaming path older than v0.74.0, which lost the origin of the placeholder model it built; passing --base is a valid workaround for that existing adapter.
• Slower than you expected — layer streaming trades time for memory. If the model already fits resident on your card, do not enable it.

Roadmap

Each refusal above names its release, and this is the same list from the other side:

• v0.72.1 (shipped) — the adapter-key correctness fix above, plus the --hf-resume refusal. It took the .1 slot out of turn, which is why every later slot below moved up by one and every refusal message in the CLI was corrected to match.
• v0.72.2 (shipped) — 4-bit (NF4) streaming, the capability jump that made 8B reachable. It also fixed a display bug where a streamed 4-bit run reported its parameter count about 6.5x too high (training was unaffected), and a startup regression that had the CLI importing PyTorch on every invocation, taking soup --help from 6.0 s back down to 1.15 s.
• v0.72.3 (shipped) — breadth: six more architectures, batches above 1, gradient accumulation, --resume and --hf-resume, a batch- and vocabulary-aware VRAM pre-flight, and the NVMe disk overflow tier. The full page.
• v0.72.4 (shipped) — preference losses: DPO, ORPO, SimPO and KTO against a streamed base, with DPO's reference model taken from that same stream with its adapters switched off, so it needs no second copy of the model. The full page.
• v0.73.0 (shipped) — the release that came out of three days on borrowed hardware. It repaired the NF4 gradient defect that only a resident reference could have found, refused streaming under nn.DataParallel rather than quietly using one card of eight, and fixed the four preference losses whose gradient-checkpointing flag was never set. What the whole exercise measured.
• v0.73.1 (shipped) — not a streaming release at heart, though it repaired streaming too: bf16 was assumed on every CUDA card in fourteen places, so every pre-Ampere card failed on every task. It also caught this feature's own VRAM pre-flight under-predicting at long sequence and shipped training.stream_vram_probe, which measures one real step instead of predicting it.
• v0.73.2 (shipped) — nothing streaming-specific: it repaired the release gate that decides whether a tuned model ships at all.
• v0.73.3 (shipped) — two streaming repairs, both contributed. A paravirtual disk reports itself as rotational with no media hint, so a genuinely NVMe-backed cloud disk measured at 1.5 GB/s was refused the disk tier, which is exactly the audience that tier exists for; when the rotational flag is unreliable the media type is now settled by a bounded direct sequential read, with a 1 GB/s floor, and training.stream_disk_kind as the manual override. And the streamed build now enforces its own postcondition: PEFT 0.18 creates streamed adapters on the meta device for Soup to materialise while PEFT 0.19 may create them as real tensors immediately, so "materialised zero" could not tell a healthy no-op from a missed adapter. The run is refused, naming the stranded parameter, rather than proceeding into a silent no-training run.
• v0.74.0 (shipped) — a tenth architecture (qwen4_exp, float32 tiny-model parity only) with the Qwen3.5 aliases, an untied embedding and LM head that share one device buffer, training.stream_pin, and a second pre-Ampere repair. What changed.
• v0.75.0 and v0.75.2 (shipped) — v0.75.2 changed nothing in streaming. v0.75.0 raised the declared floor to torch>=2.6.0 and made the pre-flight's throughput forecast measure the GEMM ceiling in the card's resolved stream dtype (#648) instead of always in bf16.
• v0.75.1 (shipped) — the empty-adapter repair above.
• v0.76.0 (shipped) — the disk tier rewritten as a background direct-I/O reader that can stripe across NVMe drives, and measured; streamed NF4 following bitsandbytes' own dispatch; pinned-arena packing; the memory half of a 14B-on-8-GB run; a post-repair 4 GB row; a streamed-adapter save guard; an untied-head fix for DPO and KTO; and new config-load refusals (use_liger, use_flash_attn, a LoRA target on the head, quantization_aware, fp8_attention, nvfp4 and others). The disk tier.
• v0.76.1 (shipped) — no streaming module changed. Two things reach this page anyway: streamed dpo and kto runs load their rows through the same converters as a resident run, so v0.76.1's one-layout rule for preference rows applies to them too, and upstream's README and performance docs now print the 8B row with its clock beside the post-#331 row. v0.76.1.
• v0.77.0 (shipped) — Mixture-of-Experts bases: the DeepSeek-V3 family (#1914), FP8 block-quantised sources dequantised at shard time (#2061), Kimi-K2-Base and -Instruct admitted through a config check (#2107) with resident Kimi K2 refused at load (#2130), and stream_expert_layout: per_expert (#2008, #2029). Three opt-in keys: stream_recompute (#2152), stream_loss_chunk (#1880, not measured on a GPU) and stream_nf4_scale_cache (#1892). Header-sized disk and RAM pre-flights (#2039, #2068, #1703), base-revision cache slots (#2094), a LoRA target on a streamed head or embedding checked when the model is built (#2082), quantised sources refused by name (#2103), use_cut_ce, moe_aux_loss_coeff and use_mod refused under streaming (#2023, #1749, #1752), moe_lora targets kept (#2072), the allocator hint set in time on Linux (#2156), and per-drive direct-I/O reporting (#1962). Two internal changes, quoted here without a figure because no benchmarks/ record backs one: decoder layer 0 is no longer re-copied at a step boundary when it is still in its slot (#1750), and a streamed layer skips the tie walk in functional_call when none of its substituted weights is tied (#1890). Streaming a Mixture-of-Experts base.
• GRPO and PPO are explicitly not planned. Rollouts need generation, and generation re-reads the model per token.

What is next is a measurement, not a feature. The 14B reference benchmark that was going to follow v0.72.4 did not become v0.73.1: that slot went to the pre-Ampere repair and the measured VRAM probe instead, and v0.73.2 went to the release gate. The benchmark first intended for that slot, a real 14B checkpoint with its throughput measured, has not been published. Parts of the question have been answered since. The borrowed H100 box measured 14B, 32B and 72B against resident references that a 4 GB machine cannot hold, though it was a validation campaign and not that benchmark. v0.76.0 then added the memory half of a 14B-on-8-GB run, with synthetic weights and no throughput claim (14B on an 8 GB card), and the first RAM-versus-disk measurements (the disk tier), both on an RTX 5070 Laptop 8 GB. What is still open is a throughput benchmark on a real checkpoint.

Measurement records and citation

The gate records behind every number on this page are published in full, including the checks that failed, the diagnoses that turned out wrong, and the figures that were measured and then discarded: the benchmarks directory. They are working records rather than a report assembled afterwards, so read them front to back; a passage quoted out of order may be one the same page later refutes.

The records added in v0.76.0 are the disk tier's, measured on an RTX 5070 Laptop 8 GB (the disk tier page lists them), and a post-repair row for the 8B laptop configuration. The records added in v0.77.0 are the MoE, FP8-source and new-key records, measured on the same RTX 5070 Laptop 8 GB and on CPU (Streaming a Mixture-of-Experts base and the measurement records list them). The one from hardware nobody here owns is three days on an 8x H100 box, which is the first machine able to hold a resident reference for a model worth streaming. That record reproduces the headline row on a completely different card and stack, extends the forward check to 72B, compares the method against DeepSpeed ZeRO-3, shows a streamed model converging indistinguishably from a resident one, and contains the gradient defect it found on the way. Read the summary.

The work also has a preprint, cited by its concept DOI so a revision never strands a reference: 10.5281/zenodo.21771064. The paper page is the short version: what it measures, the correctness protocol it defends hardest, the three findings that have nothing to do with streaming, what it explicitly does not claim, and the BibTeX entry.

Shard cache

The first streaming run rewrites the checkpoint into one safetensors shard per decoder layer under ~/.soup/layer-stream/ (override with SOUP_LAYER_STREAM_CACHE_DIR), quantising as it goes when you asked for 4bit. The rewrite works one tensor at a time, so sharding a model that does not fit never requires it to fit, and the quantisation happens once, offline, not on every run.

The cache is keyed to the quantisation, the dtype, the quantisation device and a fingerprint of the source checkpoint. Switching between none and 4bit, or retraining a base in place, therefore re-shards instead of silently streaming the wrong bytes. The index also records each source file's name, size and modification time, so a re-shard that has to happen reports which component changed instead of rebuilding for minutes without saying why. Since v0.76.0 that report prints as Re-sharding layer cache: <reason>., and a layer-cache file that the committed index does not name prints as Unindexed layer-cache file <path> (<n> bytes) is present; it was not named by the committed cache index. Changing SOUP_LAYER_STREAM_STRIPE_DIRS re-shards too, because the cache is laid out per drive; the disk tier page has the rules.

Since v0.77.0 a Hub base is bound to one commit (#2094). On the transformers backend, a streamed run on a Hub id resolved online resolves the base's commit once and passes it to every base download, and the layer cache and the regular-file copy get a slot per commit: ~/.soup/layer-stream/<org>__<name>@<commit>. So the first streamed run on a Hub base after upgrading re-shards and re-copies the base, which costs the disk space and the time of a first run. The old slot (the folder without @<commit>) stays on disk, and nothing names or removes it (tracked as #2196). Offline runs, local paths and cache-path hints are not bound and keep the slot they had.

An interrupted re-shard. Before v0.76.0, a re-shard that changed the cache's content (a dtype, quantisation or source-checkpoint change) and was interrupted could leave the old index pointing at layer files that had been partly rewritten. v0.76.0 moves the old index aside before rewriting, reports unfinished re-shards and unindexed files, and treats a cleanup failure after the commit as non-fatal. Upstream does not say whether any earlier run ever reused such a cache. As a precaution, if you interrupted a re-shard of that kind on v0.75.2 or earlier, delete that model's folder under the layer-stream cache root (~/.soup/layer-stream/ by default, or SOUP_LAYER_STREAM_CACHE_DIR) and any matching folder under a stripe root before the next run.

Disk space and the second cache

Streaming has two on-disk costs, and since v0.74.0 Soup prints both before it writes either, in a Layer streaming disk pre-flight panel: the source checkpoint, any regular-file copy Soup still has to make, the layer-shard cache, the projected total, and the free space on the volume each write lands on. If a target volume lacks the space the run is refused before anything is copied or sharded, with layer streaming needs X GB of additional disk space ... Refusing before copying or sharding. Free some space, choose a smaller base, or move the cache.

Since v0.77.0 the disk pre-flight sizes the cache from the source safetensors headers, tensor by tensor, by the rules the sharder writes with, instead of charging the NF4 rate to every parameter (#2039). A dense base under quantization: 4bit is charged more, because its embeddings, head and norms stay at the stream dtype: upstream gives Llama-3.1-8B as 4.14 GB before and 5.70 GB now. A run that passed the old check by less than that margin is refused now, before sharding; the old figure was low, and the sharder would have written the larger amount. A fused MoE base is charged what it really writes. A cache Soup will reuse is reported at the size of its own files rather than as if it were rewritten (#1703). With stream_source: ram, the RAM check before sharding now charges the same estimate and refuses by the rule of the check after sharding (70% of free RAM, or 55% of physical RAM), so a base that would be refused after minutes of sharding is refused first (#2068). When the headers cannot be listed or read, that check charges the old rate and its message says approximately.

Since v0.77.0 a checkpoint is the files its model.safetensors.index.json names (#1906). Mistral's repositories also ship a consolidated.safetensors copy beside the shards, which layer streaming used to download (29.0 GB for a 14.5 GB model, in upstream's figure) and keep among the resident extras. It is neither fetched nor sharded now.

The extra copy exists because a Hugging Face snapshot normally exposes symlinks into the Hub blob cache, and the sharder deliberately does not follow symlinks. Soup therefore copies those weights as regular files under ~/.soup/spectrum/weights/ first (override the directory with SOUP_SPECTRUM_CACHE_DIR, the same one the Spectrum scan cache uses). If the Hub cache already holds real files rather than links, or the base is a local checkpoint directory, the files are read in place and no second copy is made. The layer shards then go under ~/.soup/layer-stream/. Both overrides have to resolve under your home directory, the current directory or the temp directory: one that points anywhere else is ignored without a message and the default is used, so redirecting the cache to a bigger disk works only for a disk mounted inside one of those.

base: may also be a local path to a Hugging Face cache snapshot (.../models--org--name/snapshots/<commit>). Soup copies it to regular files and reuses the copy while the commit and blob ids match. When the path is the folder the Hub id resolves to in the active Hugging Face cache, the copy lives in that Hub id's slot, weights/org__name, so both ways of naming the model share one copy; a snapshot folder anywhere else gets a slot of its own, weights/<folder name>@<digest of its resolved location>, and costs one more copy on disk. A directory of symlinks that is not such a snapshot is refused with a message naming the accepted layouts: the Hub id, the snapshot directory, or a directory of regular files. With huggingface_hub 1.32 or later, links into its marked cache-wide blob store are followed; any other link target is refused, the repo's own blobs directory has to be a real directory rather than a link, and a junction inside a snapshot directory is refused like a directory symlink.

With HF_HUB_OFFLINE=1 the copy still works when the snapshot is complete in the Hub cache. Soup copies only verified files from the commit its first cache lookup resolved, makes no second metadata request, and a missing blob aborts the copy before anything is published, so the sharder never sees a half-written checkpoint.

See also

• Scaling a streaming run — v0.72.3: which architectures stream, how to size batch and sequence length against the VRAM pre-flight, batch versus gradient accumulation, resume, and the NVMe disk tier.
• Preference losses over streaming — v0.72.4: DPO, ORPO, SimPO and KTO against a streamed base, why the reference model needs no second copy of the weights, and what it costs in time instead.
• Layer streaming disk tier — v0.76.0: the background direct-I/O reader, stream_read_ahead, striping across NVMe drives, every streaming knob in one table, and the first RAM-versus-disk measurements with their caveats.
• Validation on hardware we do not own — the same mechanism on 8x H100: a resident reference at 8B through 72B, the DeepSpeed comparison, the convergence result, and the defect the exercise found.
• Training — the training guide the streamed run otherwise follows unchanged.
• Spectrum targeted training — the other way to train a large model small: pick the high-signal layers instead of streaming all of them.
• LISA — full fine-tune quality without a full fine-tune, for models that do fit resident. Measured at 3B and 8B it beats full fine-tuning on held-out loss but costs more memory than LoRA, not less.