Layer streaming: fine-tune a model that does not fit in your card (v0.72.0, BETA)

The usual answer to "this model does not fit" is quantize harder, rent a bigger GPU, or give up. Layer streaming is a fourth answer: never load the frozen base at all. Soup keeps it in CPU RAM and copies it into VRAM one decoder layer at a time, so peak VRAM is bounded by the size of one layer instead of the whole model.

Qwen2.5-3B trains in 2.15 GB on a 4 GB card, where a resident run runs out of memory. That OOM was confirmed on the box, not assumed.

This ships BETA, and both the scope and the claim below are narrow on purpose.

What is actually new here

Layer streaming itself is not new, and fine-tuning on a small card is not new. An early draft of this page claimed no published system does this; that claim was fact-checked against the sources and cut, because it is false. What is true is narrower:

• For inference, streaming a small card is well established. AirLLM runs 70B-class models through a 4 GB GPU, and its own documentation states that layer sharding is not compatible with gradient propagation. Hugging Face's disk_offload is documented as big-model inference and runs under no_grad.
• For training, the published small-card result already exists: LSP-Offload (arXiv:2406.10181) fine-tunes 1.3B on a 4 GB laptop GPU at a 31% slowdown. It gets there by compressing the optimisation into a learned sparse subspace, not by streaming weights.

So the honest headline is not "nobody has done this". It is 1.3B to 3B on a 4 GB card, by streaming the base rather than approximating the optimisation, with the forward pass verified bit-exact — and packaged as one config key in a general-purpose CLI rather than a research prototype.

Turn it on

It is a config key, not a CLI flag:

training:
  stream_layers: true          # enable layer streaming
  stream_source: auto          # 'auto' resolves to 'ram'; 'disk' lands in v0.72.2
  stream_buffers: 2            # double-buffering; range [2, 8], default 2

  batch_size: 1                # required in v0.72.0
  gradient_accumulation_steps: 1   # required, and NOT the default (see below)

# There is no --stream-layers flag. You train exactly as before:
soup train --config soup.yaml

The one thing that trips people up: gradient_accumulation_steps defaults to 4, so adding stream_layers: true to an otherwise normal config is refused until you also pin it to 1. That is deliberate rather than an oversight, because every micro-batch re-reads the entire base, so accumulation multiplies the streaming IO linearly. batch_size must be 1 for the same reason. Both limits lift in v0.72.2.

How it works

The base model is built on PyTorch's meta device, so its weights never occupy VRAM. Only the embeddings, the final norm, an untied LM head and the LoRA adapters with their gradients and optimizer state get real storage on the card, and those are small. The checkpoint is rewritten once into one safetensors shard per decoder layer, and that store is held in CPU RAM, page-locked when the machine allows it.

During the step, each decoder layer is copied into one of N pre-allocated VRAM buffers on a dedicated CUDA stream, so the load of layer i+1 overlaps the compute of layer i. The pooled buffer is substituted into the unmodified decoder layer, which means the same kernels run on the same weight bytes.

Each layer is read twice per step — once in the forward pass and once when the backward pass recomputes it — because dL/dx = Wᵀ · dL/dy needs the weights to push gradient down to the layers below. That is physics, not an implementation detail, and it is why streaming costs time rather than being free.

Correctness is not part of the trade. A streamed forward pass was verified bit-exact against a resident one, and a 100-step streamed loss curve matched resident exactly.

Measured numbers

Measured on the development box: RTX 3050 Laptop 4 GB, Windows 11, 16.9 GB RAM, bf16 LoRA, batch 1, gradient checkpointing on, 50 steps after 10 warm-up.

The honest cost: 1.43x slower than resident training, measured at 0.5B. That is the only apples-to-apples comparison available on this box, because 1.5B and above cannot run resident here at all.

At 1.5B the transfer is effectively hidden, but be careful how that is argued: GPU utilisation on its own only says a kernel was resident, which is necessary for overlap but not sufficient to prove it. The evidence is the step arithmetic, not the utilisation column. The 3B row's 79.3% is not a model-size effect: that box could not page-lock a 5.55 GB base, so the run fell back to a pageable store, which makes the host-to-device copy synchronous and costs overlap. Soup performs that fallback automatically and prints what it costs instead of absorbing it silently. The 3B throughput is therefore a lower bound.

Honest scope

v0.72.0 is proof-of-mechanism, and the claims stop where the measurements stop.

• Models measured: Qwen2.5-0.5B, 1.5B and 3B, plus a live soup train on SmolLM2-135M. Nothing above 3B was measured, and no 8B / 14B / 70B claim is supported.
• RAM tier only. stream_source: auto resolves to ram; the disk overflow tier is v0.72.2.
• Llama / Qwen only, task: sft, backend: transformers, modality: text.
• `batch_size: 1`, no gradient accumulation, no --resume.
• bf16 base, `quantization: none` required. 4-bit weights carry a quantisation state and cannot be byte-copied into a buffer.
• Numbers are Windows/WDDM and therefore systematically pessimistic versus Linux. expandable_segments:True is silently ignored on Windows, and Soup detects that rather than claiming it is active.

Rejected at config load

Every refusal is deterministic and happens before a single GPU byte is touched. Where a later release lifts a limit, the refusal message names that release; the rows marked n/a are refused on a structural ground and carry no roadmap promise either way.

The pre-flight hardware-fit gate is skipped for streaming runs, because it models a resident run and would otherwise refuse exactly the runs streaming exists to enable.

Full config

base: Qwen/Qwen2.5-3B
task: sft
backend: transformers

data:
  train: ./data.jsonl
  format: alpaca
  max_length: 512
  val_split: 0.1

training:
  epochs: 3
  lr: 2e-5
  batch_size: 1                    # required; larger batches in v0.72.2
  gradient_accumulation_steps: 1   # required; accumulation in v0.72.2
  quantization: none               # required; NF4 streaming is v0.72.1
  gradient_checkpointing: true     # handled per-layer by the streamer
  stream_layers: true
  stream_source: auto
  stream_buffers: 2
  lora:
    r: 64
    alpha: 16

output: ./output

Gradient checkpointing is handled per layer by the streamer, and the Hugging Face Trainer's own is left off so layers are not recomputed twice.

Troubleshooting

• "layer streaming needs the base to fit in RAM" — the base is larger than free RAM. Free RAM or pick a smaller base; the disk overflow tier is v0.72.2.
• "could not page-lock the base ... falling back to a PAGEABLE RAM store" — expected on a busy machine. Training continues, more slowly. Close other applications to keep the pinned store.
• "layer streaming does not support model_type=..." — v0.72.0 covers Llama and Qwen only.
• Slower than you expected — layer streaming trades time for memory. If the model already fits resident on your card, do not enable it.

Roadmap

Each refusal above names its release, and this is the same list from the other side:

• v0.72.1 — 4-bit (NF4) streaming, the first genuinely useful capability jump.
• v0.72.2 — disk overflow tier, batch size above 1, gradient accumulation, checkpoint/resume, more architectures (Mistral / Gemma / Phi).
• v0.72.3 — preference losses (DPO / ORPO / SimPO / KTO). DPO gets it nearly free, because the reference model is the same streamed base with the adapters disabled, so there is no second pass over the weights.
• GRPO and PPO are explicitly not planned. Rollouts need generation, and generation re-reads the model per token.
• A published 14B-on-8 GB reference benchmark is hardware-blocked: it needs an 8 GB card and 32 GB of RAM the development box does not have, and it will not be published until it is actually run.

Shard cache

The first streaming run rewrites the checkpoint into one safetensors shard per decoder layer under ~/.soup/layer-stream/ (override with SOUP_LAYER_STREAM_CACHE_DIR). That costs disk space roughly equal to the base. The rewrite works one tensor at a time, so sharding a model that does not fit never requires it to fit.

The cache is keyed to a fingerprint of the source checkpoint, so a base retrained in place re-shards instead of silently training against stale weights.

See also

• Training — the training guide the streamed run otherwise follows unchanged.
• Spectrum targeted training — the other way to train a large model small: pick the high-signal layers instead of streaming all of them.
• LISA — full fine-tune quality at LoRA-like memory, for models that do fit resident.