About Soup

Fine-tuning has a hardware floor that has little to do with the work itself: a model whose weights do not fit on your card cannot be trained on it, so the answer has always been to rent a bigger one. Layer streaming moves that floor.

The frozen base is never loaded into VRAM as a whole. It is rewritten into one shard per decoder layer, held in CPU RAM or on NVMe, and copied one layer at a time into a small pool of pre-allocated VRAM buffers on a dedicated CUDA stream, so the load of the next layer overlaps the compute of the current one. Only the LoRA adapters stay resident, which bounds peak VRAM by a single layer instead of by the whole model. Quantizing the streamed base to NF4 shrinks the store about fourfold, and that is what brings an 8B model within reach of a card that cannot hold a quarter of it.

Eleven architectures stream, covering Llama, Qwen, Qwen4-Exp, Mistral, Gemma, Phi and, since v0.77.0, the DeepSeek-V3 MoE family. The original nine were verified bit-exact against the same checkpoint loaded resident, under both bf16 and NF4, in the v0.72 gates; the newest additions carry weaker evidence and the docs say which. Since v0.76.0 the streamed NF4 forward follows the dispatch bitsandbytes itself uses for resident NF4, and no new bit-exact record exists for it. That check is the load-bearing one, because a streamed run that has quietly severed its autograd graph still produces a falling loss curve and looks healthy.

For a long time that check could only run on toy models, because comparing a streamed run against a resident one needs a card big enough to hold the resident one, and a 4 GB card never is. In August 2026 three borrowed days on an 8x H100 box closed that gap. The streamed forward pass proved exact against a real resident reference from 8B up to 72B, and the same exercise found a gradient defect above a certain layer size that three releases of passing tests had missed.

Layer streaming ships BETA. Every number behind it is published as a measurement record, including the ones that were measured and then discarded, and including the defects the measuring turned up.