Scaling a streaming run (v0.72.3; disk tier rewritten in v0.76.0; DeepSeek-V3 and three opt-in keys in v0.77.0)

Layer streaming proved a mechanism: the frozen base never loads resident, so peak VRAM is bounded by one decoder layer instead of the whole model, and NF4 shrank that store far enough to put Llama-3.1-8B on a 4 GB laptop GPU.

It was also, on purpose, barely usable for planning a real run. Llama and Qwen only. Batch size 1. No gradient accumulation. No resume. RAM or nothing.

This page keeps its v0.72.3 shape and carries the later changes in place. The disk tier was rewritten in v0.76.0 and has a page of its own; the section below is the short version. v0.77.0 added a Mixture-of-Experts family, which also has a page of its own, and three opt-in keys for step time and long contexts (below).

v0.72.3 lifts all of it, and each capability was gated against a streamed-versus-resident bit-exactness reference before it was written rather than after.

Which models stream

Eleven model_type values are on the allowlist. Six of them arrived in v0.72.3, the tenth in v0.74.0 and the eleventh in v0.77.0:

Five aliases route onto families already on the list rather than counting as new ones: qwen3_5, qwen3_5_text, qwen3_5_moe and qwen3_5_moe_text all reach the qwen3 streamer, and qwen4_exp_text reaches qwen4_exp. Upstream states why each alias needs its own control: mapping a model type by name alone is not enough to establish that its decoder graph is safe.

The v0.74.0 additions carry weaker evidence than the original nine, and the difference is stated rather than smoothed over. Only the original nine are verified bit-exact in both bf16 and NF4. Qwen3.5's dense and MoE decoder paths are verified against resident controls on CPU; the MoE path also has live streamed training on Qwen3.5-35B-A3B with NF4 and MoE LoRA on a 3072-token dataset, and that run has no resident control at all, because no available machine can hold 35B resident. Upstream's #2072 records that until v0.77.0 the streaming setup dropped Qwen3.5-MoE's linear-attention in_proj_qkv / out_proj adapters under moe_lora; it does not say whether that run's targets were affected. Qwen4-Exp has an exact float32 tiny-model parity gate, including its external N-gram table, and real-checkpoint plus NF4 validation are still pending, so Qwen4-Exp streams only with task: sft and quantization: none: anything else is refused at trainer setup, before sharding, and soup train --dry-run does not catch it. Its N-gram access is governed by training.stream_ngram_source (auto, ram, disk).

Each of the six added in v0.72.3 had its forward logits compared with the same model loaded resident (a resident NF4 load for the NF4 arm) and came out equal, on tiny from-config CPU float32 fixtures rather than real checkpoints; the layer-0 LoRA gradient was checked to be non-zero, not equal. Phi-3 is the interesting one: streaming has to substitute weights into a layer whose attention projection is fused, and it comes out bit-identical anyway.

deepseek_v3, added in v0.77.0, was gated differently: on a tiny three-layer model built from a config, on CPU in float32, the streamed forward and the streamed LoRA gradients equal a resident run's, which checks the backward as the original nine's gate did not; but no NF4 model parity was run for it, so like the v0.74.0 additions it is not verified in both bf16 and NF4. Nothing at real size has a resident reference, its FP8 checkpoints are dequantised at shard time, and no preference-loss parity was measured on it. Streaming a Mixture-of-Experts base has the evidence and its limits.

gemma3 is not gemma3_text. A real google/gemma-3-* checkpoint reports model_type: gemma3, which is the vision-capable wrapper, and it is refused. Streaming a multimodal wrapper as if it were a causal LM is exactly the failure an allowlist exists to prevent, so the refusal is deliberate rather than an oversight.

No throughput is claimed on these pages for any family beyond Llama and Qwen — not for the six added in v0.72.3, not for Qwen3.5 or Qwen4-Exp, and not for DeepSeek-V3. Every tok/s figure published here for layer streaming is Llama or Qwen. Upstream's v0.77.0 records time two other families on an RTX 5070 Laptop 8 GB, a different card from the 4 GB rows: Mistral-7B NF4, as relative step-time changes from stream_recompute, and Moonlight-16B (deepseek_v3), as GPU memory and step time per expert layout. Those are quoted with their scope where the keys are described (below and on the MoE page), never as a tok/s for the family. What was established for the newer families is correctness, not speed.

Batch size beats gradient accumulation

Both were refused before v0.72.3 and both work now. They are not interchangeable, and the release measured the difference instead of reasoning about it.

Measured on Qwen2.5-0.5B in bf16, sequence length 256, pinned RAM store, 50 steps after 10 warm-up:

At the same effective batch of 4, raising batch_size measured about 2.52x faster than accumulating.

The reason is structural, not incidental. A real batch amortises each layer's load over more tokens. Accumulation does not: every micro-batch re-reads the entire base, so the layer reads per 1000 tokens stayed flat across accumulation 1, 2 and 4. Accumulation is not useless, though. It holds peak VRAM almost exactly flat (0.842 GB to 0.846 GB) where batch 4 cost 2.28 GB, which is the whole point of it.

The rule: raise batch_size while the VRAM budget allows, then accumulate for the rest. Soup prints that advice at run start when it sees you accumulating.

The VRAM pre-flight

Once batches scale, a new way to fail appears, and it is not obvious from the outside.

Streaming bounds the weights. It does nothing about activations or the logits tensor, and both scale with batch_size times data.max_length. On a model with a 151,936-token vocabulary at batch 8, the logits tensor alone measured 8.71 GB, which is 146 times the entire layer-buffer pool of 0.060 GB. Streaming the weights perfectly and then allocating an 8.71 GB logits tensor is not a win.

So soup train now predicts peak VRAM before the run and refuses one that will not fit. The panel it prints carries the budget and a forecast, here for a small model at batch 2 on the 4 GB development card:

  peak VRAM    ~0.48 GB at batch 2 x seq 256 (logits 0.35 GB)
  free VRAM    3.46 GB
  forecast     5685-8361 tok/s — a compute-bound bound, not a promise
               (from a GEMM ceiling measured on this card now @ 862 MHz)

Above those lines the panel also names the architecture and tier, the size of the layer store, the buffer pool and what stays resident. Those figures scale with the model, so the four lines above are the ones worth reading before a run: they are the only part that changes when you change batch_size or max_length.

And when it does not fit:

a streaming step is predicted to need X GB of VRAM but only Y GB is free.
Streaming bounds the WEIGHTS, not the activations or the logits — lower
training.batch_size or data.max_length, both of which scale this linearly.

The estimator had a real bug in it

The first version of the budget charged 6 bytes per logit element from first principles. The measured peak is 14. The old constant under-predicted that term by 2.33x, which on the largest term in the whole budget is the difference between a prediction and a guess.

A later measurement on different hardware took that 14 apart stage by stage, with three repeats and no spread at all, and the total held while the explanation did not. The number is 12 plus 2: the loss arithmetic costs 12, being the bf16 logits alongside three fp32 buffers of the same shape, and the remaining 2 is a retention, the cost of holding the model's output object alive across backward(). An earlier telling of this said "bf16 logits, fp32 upcast, log-softmax output and gradient, all live at once", which is right in total and wrong in detail: the upcast is freed when the loss function returns. The 12 is also stack-independent, measured identically on two very different torch and TRL versions.

The corrected estimate was fitted against ten real runs across two models with a 3.1x vocabulary contrast, at batches 1 to 8 and two sequence lengths. Worst error: 0.85%, and it never under-predicted any of them.

And then it did under-predict, at long sequence (v0.73.1)

That "never under-predicts" was the estimator's contract, and it is the reason the whole gate is trustworthy: on Windows an over-budget allocation does not raise, it spills silently into host memory. v0.73.1 measured the contract failing.

Through the real soup train on the same 4 GB laptop, with SmolLM2-135M streamed in bf16 at batch 1:

The ten-run fit above could not have caught this, and that is the lesson. Every one of its rows sits at sequence 256 or 512. It varies batch, so it says nothing whatever about sequence length. A control only covers the variable it varies.

The mechanism is deliberately not claimed. The obvious candidate, the quadratic term from the attention score matrix, does not settle the numbers, and a formula cannot model a term nobody has identified. So the answer shipped is not another coefficient, it is a measurement, and the fitted estimate is left in place as the default. See v0.73.1 for the probe that replaces it on request.

A second box has since said this does not reproduce there. A contributor ran the same protocol on an NVIDIA A10G with a much newer stack (torch 2.13, transformers 5.16.1) and the ratio is flat at about 1.16x over-prediction from sequence 2048 through 6144, on two models. The record publishes its leading hypothesis as a negative result with a positive control, and explicitly does not refute the reading for this 3050. Taken together the honest conclusion is that the term varies per-stack and per-sequence-length, which is why the answer is a measurement rather than a constant. See the v0.74.0 record.

Why refusing beats trusting the driver

On Windows, WDDM does not raise on overcommit. It pages to host memory instead. During this work a run that should have failed outright instead reached a 9.27 GB peak on a card with 4.29 GB, silently, at a throughput nobody would want. That is the case the pre-flight exists to catch: not a crash, but a run that appears to work.

What else the pre-flight charges (v0.76.0 and v0.77.0)

• Host RAM on the disk tier. The disk tier's staging plus the resident extras must fit 70% of free RAM, or the run is refused naming training.stream_read_ahead. See the disk tier.
• A second large-layer slot for untied dpo and kto. Those two losses run a reference forward before the backward, which refills the slot the output head shares with the embedding, so the head takes a private copy for the step and the pre-flight charges it as a second large slot. It is a charge the pre-flight makes, not a measured peak. Details.
• Each stripe drive's share. With SOUP_LAYER_STREAM_STRIPE_DIRS set, the disk pre-flight charges each volume the bytes it will receive.
• An out-of-memory error during stream_vram_probe is now classified as an out-of-memory verdict (the shape does not fit), where it used to read as an instrument failure.

Since v0.77.0:

• The disk pre-flight sizes the cache from the source safetensors headers (#2039), so a dense base under quantization: 4bit is charged more than before (its embeddings, head and norms stay at the stream dtype) and a run that passed by a small margin can be refused now. With stream_source: ram the RAM check before sharding charges the same estimate (#2068). Details.
• stream_nf4_scale_cache is charged as a scale cache line, and one routed expert in flight under per_expert as an experts line.
• Under stream_loss_chunk, the logits are charged on min(seq, chunk) tokens plus two hidden-sized tensors.
• Under moe_lora, the union of the targets and one adapter per routed expert for each fused expert tensor (#2072), where the configured value used to be charged.
• Not charged: the activations stream_recompute: off keeps. stream_vram_probe: true measures them; auto sizes what it keeps from a step it measured at setup.

The probe's own contract is unchanged: task: sft only.

Step time and long contexts (v0.77.0)

Three opt-in keys. With their defaults a run behaves as it did before v0.77.0, and each one is refused while stream_layers is false. Where they were measured at all, it was on one RTX 5070 Laptop 8 GB, on development code before the v0.77.0 tag: a different card from the 4 GB rows above, and upstream calls it a new baseline.

Keep activations instead of recomputing them: stream_recompute

A streamed layer normally runs its forward twice: once in the forward pass and again as the checkpoint recompute in the backward. training.stream_recompute: off keeps every layer's activations instead. The backward still re-reads each layer's weights before it uses them, so the buffer slots are never read stale (the #331 lesson), and the gradients are the recompute's. auto runs one synthetic forward and backward at your batch_size x data.max_length before step 1, keeps activations for as many of the last layers as fit, and prints what it chose; it works on the RAM tier on CUDA only and keeps the recompute anywhere else. on is the default and the shipped behaviour. An unquoted YAML on or off reads as a boolean, and Soup maps it back to the mode; an integer is refused.

training:
  stream_layers: true
  batch_size: 1
  quantization: 4bit
  stream_recompute: auto       # on (the default) | off | auto
  stream_vram_probe: true      # measures the step; the formula pre-flight does not count kept activations

Measured on one laptop (the record): on the RTX 5070 Laptop's RAM tier, a Mistral-7B NF4 step took 22% / 17% / 8.5% off the step time at 512 / 1024 / 2048 tokens, with no gain on a read-bound disk tier and slower at a high VRAM fill. Under deterministic algorithms the loss and the LoRA gradients were bit-identical to the recompute; that compares two streamed modes with each other, not a streamed run with a resident one. The cost is VRAM, every kept layer's activations.

• off and auto are task: sft only, and refused beside activation_offloading.
• The formula pre-flight does not count the activations off keeps. Turn on stream_vram_probe: true, which measures the step after the switch. Without it nothing refuses a run that keeps too much, and on Windows an over-budget allocation spills into host memory rather than raising (why that matters).
• On the disk tier auto keeps the recompute and off prints a warning: the step is read-bound there, and skipping the recompute does not skip the backward read.

The head and the loss in chunks: stream_loss_chunk

The logits term is vocab x seq x batch, which is what caps the context length on a small card. training.stream_loss_chunk: 128 runs the output head and the loss 128 tokens at a time and drops each chunk's logits, so the pre-flight charges the logits on min(seq, 128) tokens plus two hidden-sized tensors. The value is a token count from 16 to 8192; there is no default and a boolean is refused.

training:
  stream_layers: true
  batch_size: 1
  quantization: 4bit
  stream_loss_chunk: 128       # tokens per chunk, 16 to 8192

• Not measured on a GPU. The peak-VRAM table it was built for is still open (#1693); upstream ships a harness for it and no record.
• The loss and the gradients match the unchunked step within 1e-5 of the largest value, not bit for bit, so the bit-exactness the streamed path is published with does not extend to a run with this option.
• A labelled forward returns no logits. The trainer's entropy, mean_token_accuracy and num_tokens columns and rewind.jsonl are written as before.
• task: sft only; refused with data.format: raft and beside use_cut_ce at config load, and, when the model is built, with a trainable output head or a model that changes its logits after the head (Gemma 2's soft-capping).

Expanded NF4 scales kept on the card: stream_nf4_scale_cache

Under double quantisation an NF4 weight stores its block scales in compressed form, and bitsandbytes' dequantise-plus-linear path expands them on every visit, forward and recompute alike. training.stream_nf4_scale_cache: true keeps each weight's expanded fp32 block scales on the card after its first use. It costs 4 bytes of VRAM per 64-element block for the whole run, which the pre-flight charges as a scale cache line. Results are bit-identical, and bitsandbytes' fused 4-bit path is left unchanged. It needs quantization: 4bit with double quantisation, which is the default.

training:
  stream_layers: true
  batch_size: 1
  quantization: 4bit           # double quantisation is on unless you turn it off
  stream_nf4_scale_cache: true

No step-time figure is quoted for it here.

The throughput forecast

The panel quotes a throughput range, and where that range comes from is a deliberate design choice: a GEMM ceiling measured on your own card in that session, printed next to the SM clock. Never a compiled-in per-card constant.

That is a response to a measurement rather than a preference. This one card produced 3.5 and 7.6 TFLOPS in two different sessions at the same reported clock. A baked-in table would have been wrong about half the time, on the same hardware. Streamed runs measured in v0.72.3 landed at 68 to 100% of their measured ceiling; the post-#331 8B row (@umran666, #1103) reads 59% of its own same-session ceiling against the pre-repair row's 68%, which the record says is not like-for-like on efficiency. That spread is why the line calls itself a bound and not a promise.

Resume works now, and why it could not before

--resume and --hf-resume were refused for streaming runs. The refusal was not caution; it was covering a bug.

Adapters could not be loaded into a streamed model at all. PyTorch's load_state_dict narrows keys by child name, and the streaming wrapper holds the real decoder layer as a child, so a canonically saved checkpoint matched 0 of N tensors. PEFT reports missing keys as a warning rather than an error, so nothing failed. A resumed run simply reproduced the from-scratch loss curve exactly, which is the kind of bug that looks like a successful run.

The fix redirects canonical keys at load time, mirroring the save-side fix from v0.72.1. It is load-side only, which is worth stating precisely: v0.72.0's forward-path bit-exactness results stand without being re-run, because nothing in the forward path changed.

What was verified: the adapter round-trip now lands every tensor, the device map is preserved, decoder parameters stay on the meta device (so the run is still streaming, not quietly materialised), and loss continuity holds on the production CUDA path.

What could not be demonstrated, and why it is not streaming's fault: end-to-end soup train --resume on the development box. transformers refuses torch.load below torch 2.6 under CVE-2025-32434, and that blocked every resume on that machine, streaming or not. (Since unreachable in v0.75.0, which raised the declared floor to torch>=2.6.0 and proved it in CI, so no supported install sits below it. The streaming half was verified on the production CUDA path at the time; what was missing was a box that could run the control.)

That asymmetry was a known limitation here for three minor versions (v0.72, v0.73 and v0.74), and it is closed in v0.75.1. Loading an adapter into a streamed model worked, but in memory named_parameters() and state_dict() still disagreed, which was the deliberate cost of fixing serialisation rather than rewriting the wrapper. It stopped being a cosmetic debt the moment peft 0.21 began selecting tensors by the names rather than the keys: at that point the run saved an adapter with nothing in it. v0.75.1 makes names and keys one spelling, and ten peft-version-independent tests pin them together. The four tests that documented the old asymmetry were inverted rather than deleted, which is the honest way to retire a limitation you once relied on.

The disk overflow tier

If the base does not fit in RAM either, v0.72.3 streams it from NVMe. Since v0.76.0 a background reader stages stream_read_ahead layers at a time in host memory (page-locked when the box allows it), so the base is not resident but the host still needs a few GB of free RAM, and the pre-flight refuses a run it will not fit.

training:
  stream_layers: true
  batch_size: 1          # streaming refuses batch_size 'auto', which is the default
  stream_source: auto    # RAM when the base fits, NVMe when it does not

Either tier writes the layer shards to disk first, and a Hugging Face base may need a second regular-file copy beside them. Soup prints that footprint and refuses before writing if the volume is too small: see Disk space and the second cache.

NVMe-class only. SATA SSD, spinning disk and undetectable media are all refused. That is not fussiness: the streamer reads every shard twice per step, so an 80-layer model is 160 seeks per step on a spinning disk and the run thrashes for hours. Unknown media is refused rather than guessed, because guessing wrong costs more than stopping.

What "NVMe-class" means changed in v0.73.3, and it matters on a cloud box. Detection used to trust the kernel's rotational flag, which a paravirtual (virtio) device reports as 1 with no media hint — so a genuinely NVMe-backed cloud disk, measured at 1.5 GB/s, was classified as a spinning one and denied the tier it was built for. rotational=0 still settles it as solid state. When the flag is unreliable, the media type is now decided by a bounded direct sequential read rather than by the flag: at or above 1 GB/s the disk earns the tier, and a genuinely slow one still does not. Deliberately above SATA's practical ceiling, and an unmeasurable disk is still treated as the slow case, which is the safe direction.

training.stream_disk_kind (nvme / ssd / hdd) is the escape hatch for when even that is wrong. It prints what it overrode beside what was detected, and it carries no measured rate on purpose, so a later refusal can never cite a reading you overrode.

Check what Soup detects on your machine:

soup doctor --disk

It reports NVMe, SATA SSD, HDD or Unknown, and it is opt-in because the probe costs about 9 seconds cold (about 2.4 seconds warm) on Windows. On Windows, where a machine can present several physical disks with no way to attribute the volume, Soup reports the worst one it found. Since v0.77.0, on Linux, detection follows the probed path's own device through /sys/dev/block, partitions and device-mapper or RAID members included, and a mixed volume counts as its slowest member, so an unrelated NVMe disk in the machine no longer qualifies the cache volume (#1847). During a run that path is the layer cache's; soup doctor --disk still probes the current directory (and each SOUP_LAYER_STREAM_STRIPE_DIRS root when that is set).

On Linux the probe writes, and that is a side effect rather than just a time cost, so it is worth stating plainly: where the kernel's rotational flag is unreliable, Soup writes a small scratch file (about 64 MiB) into the directory it is probing, which is the current directory for soup doctor --disk and the shard directory during a run, reopens it with O_DIRECT to bypass the page cache, and times a few bounded page-aligned sequential reads, keeping the fastest one so a single cold sample cannot under-measure a fast device and wrongly refuse it. The scratch file is always removed, any failure (no O_DIRECT on that filesystem, no write permission) returns nothing so the caller stays conservative, and each read is bounded so the probe cannot grow into the 9-second cost the Windows path already carries. Media it cannot identify stays unknown, and the disk tier refuses unknown rather than guessing.

The disk tier since v0.76.0

The v0.72.3 synchronous disk reader's correctness was verified bit-exact against the RAM tier; the v0.76.0 reader is gated by byte-identity against that source. The tier's speed was left unmeasured then, with no figure claimed. v0.76.0 replaced that reader, and then measured the tier.

The disk tier now reads on a background thread through direct I/O and can stripe across NVMe drives (SOUP_LAYER_STREAM_STRIPE_DIRS). On one RTX 5070 Laptop 8 GB with a synthetic 70B-shaped NF4 store of 36 GB (random weights), a cold step took 110.5 s on the previous synchronous reader and 16.2 to 17.3 s on the direct-I/O reader; across two NVMe drives a 17.4 s step fell to 9.97 s. A sustained-load probe measured 1.28x and 1.41x in two rounds and reached no formal verdict. A store that fits RAM is still faster on the RAM tier, which stream_source: auto picks.

That is one box, a synthetic store, timings with no quality claim, and an 8 GB card, not a 4 GB one. No warm RAM-versus-disk ratio is quoted here, and no disk-tier throughput exists for a 4 GB card. The knobs, the figures and everything they do not show.

14B on an 8 GB card

v0.76.0 added the memory half of a 14B-on-8-GB benchmark. A Qwen2.5-14B-shaped NF4 model with random weights (14.77 B parameters, 48 layers, a 9.93 GB store) trained end to end through soup train for 18 optimizer steps at batch 1 and sequence 384 on an RTX 5070 Laptop 8 GB with 31.7 GB of RAM. Peak memory was 2.944 GB allocated (3.244 GB reserved) against the pre-flight's 3.39 GB prediction, with the store page-locked as exactly 10 GiB (about 1.08x its size). The record makes no throughput and no quality claim: peak memory and pinned-memory cost depend on tensor sizes, not on values, and that is all it measures.

The report that led to it (#901) was AcceleratorError: CUDA error: out of memory inside SFTTrainer.__init__ on an 8 GB card with gigabytes free, and the record's finding is that it was never about VRAM. torch's host allocator rounds every pinned request up to the next power of two, so pinning the store one tensor at a time cost 1.73x its size and the page-lock failed. The shipped fallback to a pageable store then did not clear the stale CUDA error the failed page-lock had left on the runtime, so the first kernel launch reported it as an out-of-memory error. Both are fixed in v0.76.0: the store is pinned in a few power-of-two arenas, and the stale error is cleared before the fallback. A store that is pinned in arenas is also packed largest tensor first when that page-locks fewer bytes (a Qwen3-8B NF4 store page-locks 6 GiB where it took 8 GiB); that is a memory change, not a speed claim, and there is no setting for it. The record.

A config that uses all of it

base: meta-llama/Llama-3.1-8B-Instruct
task: sft
backend: transformers

data:
  train: ./data.jsonl
  format: alpaca
  max_length: 512

training:
  epochs: 3
  lr: 2e-5
  batch_size: 4                    # raise this first; the pre-flight sizes it
  gradient_accumulation_steps: 2   # then accumulate for the rest
  quantization: 4bit               # NF4: ~4x smaller store
  gradient_checkpointing: true
  stream_layers: true
  stream_source: auto              # RAM, falling back to NVMe
  stream_buffers: 2
  lora:
    r: 64
    alpha: 16

output: ./output

Those batch and accumulation values are illustrative, not a recommendation. Whether they fit depends on your card, the model's vocabulary and max_length, and the pre-flight will tell you before the run starts rather than after.

Also in v0.72.3

Three fixes and one dependency cap that are not about capability:

• The guard meant to refuse non-NVMe media was wired to a hardcoded constant and could never fire. It fires now.
• Streaming weight sources are released when training ends or raises. This matters more on the disk tier, which holds one open shard handle per decoder layer.
• Subprocess helpers resolve tools to absolute paths, because on Windows CreateProcess searches the current directory before PATH.
• The [mcp] extra is capped below 2.0. The MCP SDK's 2.0.0 release removed an API soup mcp serve round-trips through, which broke its round-trip tests for anyone installing fresh. Support for the 2.x API is tracked separately. (Since lifted: v0.74.0 requires mcp>=1.10.0,<3 and picks the API by probing the server constructor, so both majors work.)

What is still out of scope

Layer streaming remains BETA.

• task must be one of sft, dpo, orpo, simpo, kto. The four preference losses landed in v0.72.4; ipo, bco and the unified task: preference dispatcher are still refused.
• GRPO and PPO are explicitly not planned. Rollouts need generation, and generation re-reads the model per token, which destroys the amortisation streaming depends on.
• backend: transformers, modality: text, plain LoRA, and quantization of none or 4bit. A base checkpoint saved already quantised is refused, except DeepSeek-V3's FP8 block format (MoE streaming) and Qwen4-Exp's oQ bundles.
• stream_recompute: off / auto and stream_loss_chunk are task: sft only, and stream_expert_layout: per_expert is DeepSeek-V3-family only.
• batch_size: auto is refused: the auto-batch probe sizes a resident model, which a streaming run never loads.
• Nothing above 8B has been measured on the 4 GB card, so nothing on this page speaks for a larger model there. 14B, 32B and 72B were measured later on a borrowed 8x H100 box, which also turned up a gradient defect in NF4 above roughly 165 MiB per decoder layer. A 14B-shaped run was trained end to end on an 8 GB card in v0.76.0 (above), a memory result with no throughput.

Measurement records and citation

Two of this page's results are written up in the preprint rather than only logged: the peak-VRAM predictor, including the corrected 14-bytes-per-logit-element constant, and the finding that gradient accumulation is per-token I/O-neutral with a 2.52x opportunity cost against batch size. The paper page summarises both. The raw gate record for this release, failures and discarded numbers included, is in the benchmarks directory. The disk-tier and page-lock records added in v0.76.0 are listed on the disk tier page, and the MoE, FP8-source and new-key records added in v0.77.0 on the measurement records page. The preprint covers neither: upstream says its next version covers the v0.77.0 streaming work.

See also

• Layer streaming — the mechanism, the measured numbers and the full refusal table.
• Layer streaming disk tier — v0.76.0: the background direct-I/O reader, stream_read_ahead, NVMe striping, every streaming knob, and the measurements with their caveats.
• Preference losses over streaming — DPO, ORPO, SimPO and KTO against a streamed base, and why the reference model needs no second copy of the weights.
• The layer streaming paper — the preprint behind these numbers, what it measured and what it explicitly does not claim.
• Training — everything a streamed run otherwise follows unchanged.
• Speed and memory — the knobs that matter when the model does fit resident.