Training Speed & Memory

Soup v0.28.0 shipped six throughput and memory features. They were SFT-only in v0.28, a gate lifted in v0.33 and finished in v0.35; tasks outside the 13-task allowlist (see Which tasks below) are still refused at config load, so a task that would ignore these keys is refused rather than allowed to ignore them. At v0.76.0 two of the features below changed: FP8 training now stops a run that cannot do FP8 instead of training on in bf16, and kernel_auto_compose: true is refused at config load. Gradient checkpointing tiers now differ on task: sft, and cross-document attention masking is a refused key, not a working mask.

If the model does not fit at all, none of these six is the answer. They shave a resident run; they cannot make a model smaller than your card. For that, see layer streaming: the frozen base never loads resident, so peak VRAM is bounded by one decoder layer instead of the whole model, and Llama-3.1-8B fine-tunes on a 4 GB card. Since v0.72.4 that covers preference losses too, not just supervised fine-tuning. It is BETA, and it is slower than resident training, so reach for it only when resident training is impossible.

Cut Cross-Entropy

training:
  use_cut_ce: true

Cut CE avoids materialising the full [seq_len × vocab_size] logits tensor during the cross-entropy backward pass. Biggest expected win on models with vocab ≥ 128k (Llama 3, Qwen 3), when it engages (see the log note below). Architecture detection reads config.model_type first, so a local checkpoint directory resolves as correctly as a hub id; a substring match on the last path component (model_name.rsplit("/", 1)[-1]) is only the fallback when no config can be read. That order was the v0.74.0 repair: before it, name matching came first, so a soup merge output called checkpoint-2000 matched nothing and cut-cross-entropy stayed off on a flag you had explicitly set. It also means org prefixes like deepseek-ai/...-phi-... can't trigger the wrong patcher.

Install the optional dep:

pip install "soup-cli[cce]"

Check the log: it may not engage. use_cut_ce patches the model only if the third-party cut_cross_entropy.transformers.cce_patch imports. When it cannot, or no patcher matches the architecture, the run continues without it and prints a yellow Cut Cross-Entropy: no matching architecture or cut_cross_entropy not installed. That line is also what you see when the package is installed and an import inside it fails, and nothing checks beforehand that the package is present. A success prints Cut Cross-Entropy enabled. The v0.73.0 probe record (see Measurement records) could not make the flag engage on any pip-installed stack it tried: cut-cross-entropy 25.1.1 ships no transformers submodule, and 25.9.3 failed importing symbols that transformers 4.57.6 no longer exports. That was measured on transformers 4.57.6, not on the 5.x that v0.76.0 requires, and it has not been re-measured, so whether the current cut-cross-entropy release engages on transformers 5.x is unverified. Treat the memory saving as unproven until your own log says it is enabled.

FP8 training (Ada and newer)

training:
  quantization_aware: fp8
  fp8_recipe: tensorwise   # tensorwise | rowwise | rowwise_with_gw_hp

fp8_recipe picks the torchao scaling recipe and defaults to tensorwise (one scale per tensor). rowwise scales per row, and rowwise_with_gw_hp is rowwise with the weight gradient kept in high precision. The source describes tensorwise as the fastest and the two rowwise recipes as more accurate; none of that has been measured here. The key is refused unless quantization_aware is fp8, and a value outside those three is refused at config load.

Configures float8 linear layers via torchao. The key's literal accepts false, "fp8" and "quest" (the experimental route covered under Backends); true is the refused int8 QAT, and unknown strings like "fp16" are rejected at config load.

FP8 needs CUDA and a card of compute capability 8.9 or higher: RTX 40/50-series, L4, L40S, RTX 6000 Ada, H100/H200 and B100/B200. Through v0.75.2 this page said Hopper and said Soup did not check at launch. v0.76.0 checks, at setup before the model loads, and stops the run with the reason when this card, OS or torch build cannot run it (FP8HardwareUnsupportedError); a missing torchao stops it the same way (FP8DependencyMissingError, naming pip install "soup-cli[qat]"), and so does a float8 conversion that fails, which quotes torchao's error. Before v0.76.0 the trainers printed a yellow advisory and trained in bf16 under a config that said FP8 (#835), so if you ran quantization_aware: fp8 or fp8_attention: true on a card or torch build that could not do it, the run was a bf16 run. quantization_aware: fp8 is refused at config load on backend: mlx and backend: unsloth and with stream_layers: true, as are fp8_attention and nvfp4. soup train --dry-run prints this machine's card as a note and exits 0, because FP8 configs are often written for another machine; a missing torchao still fails it. The [qat] extra declares torchao>=0.18.0 since v0.76.0 (it was >=0.4.0), so an older torchao is no longer accepted by the extra: run pip install -U "soup-cli[qat]".

tensorwise has no further limits. rowwise and rowwise_with_gw_hp run a separate torch kernel, so they also need a torch that dispatches it on your card, and they are never built on Windows. Before torch 2.11 that kernel is only built against CUDA 12 or newer. By compute capability: Ada (8.9) needs torch 2.7 or newer; Hopper (9.x) and datacenter Blackwell (10.x) work on any supported torch; the RTX 50-series (12.x) needs torch 2.8 or newer; 11.x needs torch 2.10 or newer. No torch release checked through 2.14 runs it on 13.x. This is a table transcribed from torch's source, not queried from torch.

Tiered gradient checkpointing

training:
  gradient_checkpointing: auto   # or: selective | medium | full

The schema accepts false, true, selective, medium, full and auto. On task: sft (and tts, which trains through it) the tiers differ since v0.76.0: full checkpoints every block; medium uses Transformers' native every_n_layers=2 path, which checkpoints every other block; selective wraps each decoder block's direct attention module and leaves Hugging Face's full-model checkpointing off, so nothing is recomputed twice. If an architecture exposes no direct attention child, Soup prints that selective is unavailable and applies full. auto picks full below 24 GB of VRAM, medium from 24 to 80 GB and selective above 80 GB. The other task wrappers read the key as on or off: any enabled tier is full checkpointing there, and the tier name changes nothing. Through v0.75.2 that was true of every task. The per-tier slowdowns in the source comments (about 30% for full, about 10% for selective) are upstream's estimates, not measurements, so no per-tier memory or speed figure is published here.

On a 4bit, 8bit or mxfp4 run, gradient_checkpointing: false was silently overridden to on through v0.75.2, because the k-bit preparation step used peft's default of checkpointing on (#797). Since v0.76.0 the setting is honoured there (fifteen call sites now pass it through), so a config that relied on the override can use more VRAM than it did. This covers that preparation step only.

When stream_layers is on, Soup leaves Hugging Face's checkpointing off, because the streamed layers already recompute; see layer streaming.

Kernel auto-composition (refused since v0.76.0)

training.kernel_auto_compose: true is refused at config load (#801): kernel_auto_compose is not supported: it benchmarks the same already-loaded model for every candidate and does not apply the selected flags. false, the default, still loads. Through v0.75.2 the key was accepted: it enumerated the installed combinations of Liger Kernel, FlashAttention and Cut Cross-Entropy, timed a few forward passes per candidate and printed Kernel auto-compose picked: <name>, but it never applied the pick (nothing set use_liger, use_flash_attn or use_cut_ce from the result, and the SFT trainer never called it at all), and the timing loop did not switch kernels between candidates. Set the kernels explicitly instead:

training:
  use_liger: true       # task sft or tts only
  use_flash_attn: true  # task sft or tts only

Both keys are refused at config load on any other task.

Cross-document attention masking

training:
  packing: true      # this is the whole answer

When sample packing is on, documents in one packed sequence must not attend across each other. That isolation is TRL's default bfd strategy, and you get it by setting packing: true with a FlashAttention attn_implementation. There is no second flag to set.

packing_cross_doc_attn_mask is refused at config load since v0.75.0, and it never worked before that. Soup mapped it to TRL packing_strategy="attention_free", which has never been in TRL's allowlist on any released version, so the flag was a TypeError at trainer setup rather than a working mask. It is now rejected by a @model_validator that names bfd / bfd-requeue / wrapped as the real allowlist and points at packing: true. This is a false-capability correction, not a regression: no release ever built that mask. Remove the key from any config that carries it.

Activation offloading

training:
  activation_offloading: cpu    # or: disk

Saved-tensor hooks offload activations to pinned RAM (cpu) or an on-disk scratch dir (disk). The disk variant keeps the mkstemp file descriptor open until torch.save flushes, closing the TOCTOU window between os.close(fd) and torch.save(path). torch.load(weights_only=True) prevents arbitrary Python deserialization on reload. Scratch dirs are containment-checked against cwd. Unsloth and MLX backends are rejected.

Which tasks

use_cut_ce, quantization_aware: fp8 and activation_offloading were SFT-only in v0.28. Multi-trainer wiring landed in v0.33.0 and every transformer-backend trainer was wired in v0.35.0. fp8_attention and nvfp4 (the Quant Menu II keys) arrived later and use the same task allowlist. At v0.76.0 all five are accepted for 13 tasks (sft, dpo, pretrain, grpo, kto, orpo, simpo, ipo, bco, preference, ppo, reward_model, embedding) and refused at config load for the other ten (prm, tts, classifier, reranker, cross_encoder, distill, unlearn, moe_lora_routing, online_dpo, asr), and on backend: mlx; quantization_aware: fp8, fp8_attention and nvfp4 are also refused on backend: unsloth and with stream_layers: true. See multi-trainer speed and memory for the per-trainer wiring.

See also

• Multi-GPU mastery — ZeRO++ / FSDP2 / pipeline
• Backends — transformers / unsloth / MLX
• soup bench — soup bench train runs a fixed number of steps and fails when the model was not training; before you compare two throughput numbers from it, note that thirteen runs of one byte-identical config spanned 2.43x in upstream's reference run