Backends & Performance

Soup has three training backends: transformers (default), unsloth (2-5x faster on CUDA), and mlx (native Apple Silicon, new in v0.25.0).

Apple Silicon MLX Backend (v0.25.0)

Train on M1–M4 Macs without CUDA or emulation:

pip install "soup-cli[mlx]"

base: mlx-community/Llama-3.1-8B-Instruct-4bit
backend: mlx
task: sft       # sft ONLY: the MLX backend refuses every other task

See the MLX backend guide for supported tasks, diagnostics, and MLX-native recipes.

Unsloth Backend (2-5x Faster Training)

Use the Unsloth backend for significantly faster training and up to 80% less VRAM:

pip install "soup-cli[fast]"

Add one line to your config:

base: meta-llama/Llama-3.1-8B-Instruct
task: sft
backend: unsloth    # 2-5x faster, -80% VRAM

data:
  train: ./data/train.jsonl
  format: alpaca

training:
  epochs: 3
  lr: 2e-5
  quantization: 4bit
  lora:
    r: 64
    alpha: 16

Works with the tasks that have an unsloth setup: SFT (text; SFT with vision or audio still runs the transformers setup), DPO, GRPO, PPO, KTO, ORPO, SimPO, IPO, BCO, preference, Pretrain, Embedding and TTS. The other ten (reward_model, prm, classifier, reranker, cross_encoder, distill, unlearn, moe_lora_routing, online_dpo and asr) have no unsloth setup, so backend: unsloth is refused at config load for them (#1357). Before v0.76.0 seven of them (reward_model, prm, classifier, reranker, cross_encoder, unlearn and moe_lora_routing) loaded and trained on plain transformers while soup train printed unsloth (fast mode); distill was refused by #1186 in the same release, and asr and online_dpo already had their own refusals.

On the unsloth backend lora.use_dora, use_vera and a non-random init_strategy, moe_lora, freeze_layers, freeze_ratio, rope_scaling_type, quantization_aware: fp8, fp8_attention and nvfp4 are refused at load: unsloth builds its own adapter and kernels and never reads them.

Tip: Soup auto-detects unsloth. When it is installed, soup train prints a hint if you haven't enabled it yet and your config would load with it: there is no hint for a task with no unsloth setup, or for a config using freeze_layers, rope_scaling_type or moe_lora.

Performance Optimizations

training:
  use_liger: true             # Liger Kernel fused ops (measured once: 12.9% less VRAM, 1.051x throughput)
  use_flash_attn: true        # FlashAttention v2/v3 auto-detection
  gradient_checkpointing: true  # Required for long sequences

Both keys are read by the SFT trainer only (and by tts, which trains through it); on any other task they are refused at config load (v0.76.0, #806), and with stream_layers: true too, because streaming builds the model itself and never reaches either kernel. The Liger figure is a single measurement from v0.73.0 and replaced an earlier "20-60% memory savings" claim that upstream corrected in the same release, so treat it as a data point on one setup, not a promise.

use_flash_attn needs a CUDA device and asks transformers which implementation is installed: FlashAttention 3 if it reports one (that is the separate flash_attn_3 package, not flash-attn), otherwise FlashAttention 2, otherwise nothing. Before v0.73.0 the FlashAttention 3 branch tested flash_attn.__version__ for a 3.x number, which no real install reports, so on Hopper hardware the setting silently fell back to FlashAttention 2 or SDPA while the docs advertised 3. v0.73.0 fixed the detection (#334). Upstream is explicit that this makes detection honest and does not make FlashAttention 3 measurable on its test hardware, so this site publishes no FlashAttention 3 speed or memory figure.

Install optional packages:

pip install "soup-cli[liger]"       # Liger Kernel
pip install flash-attn --no-build-isolation  # FlashAttention
pip install "soup-cli[ring-attn]"   # Ring FlashAttention

Long-Context Training (128k+)

training:
  rope_scaling_type: yarn       # linear, dynamic, yarn, llama3
  gradient_checkpointing: true

data:
  max_length: 131072            # Up to 1M tokens supported

rope_scaling_type is read by task: sft, tts and pretrain on backend: transformers (for sft, the plain text path without stream_layers), and it only does something when data.max_length is larger than the model's native context: the scaling factor is that ratio, worked out for you. Since v0.76.0 the RoPE configuration is built before the model is constructed, because transformers 5 derives the rotary frequencies in the model constructor. Through v0.75.2 the trainer wrote the factor into model.config after the model had loaded, which in a check on a small randomly initialised Llama changed neither the rotary frequencies nor the outputs while the run printed Long-context enabled (#793). On any other task, on backend: mlx or unsloth, on vision or audio SFT, or with stream_layers: true, the key is now refused at config load (#1697), and longrope is refused everywhere (#1239). yarn_factor, yarn_attn_factor, yarn_beta_fast and yarn_beta_slow are read, with rope_scaling_type: yarn only. See Long context for the details and the Llama 3.1 composition rule.

use_ring_attention is different: it is accepted and checked before training (CUDA, at least two GPUs, ring-flash-attn installed, max_length of 8192 or more), and nothing in v0.76.0 applies Ring FlashAttention to the model afterwards, so sequence parallelism does not happen.

Multi-GPU Training

DeepSpeed

soup train --config soup.yaml --deepspeed zero2          # ZeRO Stage 2
soup train --config soup.yaml --deepspeed zero3          # ZeRO Stage 3
soup train --config soup.yaml --deepspeed zero2_offload  # Stage 2 + optimizer offload
soup train --config soup.yaml --deepspeed zero3_offload  # Stage 3 + CPU parameter offload (v0.73.0)

FSDP2 (PyTorch Native)

soup train --config soup.yaml --fsdp full_shard    # Like ZeRO-3
soup train --config soup.yaml --fsdp shard_grad    # Like ZeRO-2
soup train --config soup.yaml --fsdp full_offload  # With CPU offload

To span several machines, soup train takes --nodes, --node-rank, --master-addr and --master-port (v0.76.0); see Multi-GPU Mastery.

No GPU at all: train on a serverless cloud one

The rest of this site is about making a small card work. If you have no card, --cloud modal renders a self-contained Modal app from your soup.yaml and runs it on a per-second-billed GPU.

pip install "soup-cli[modal]"    # only needed to submit live

# Plan-only, the DEFAULT: write the stub and print the modal run command.
soup train --config soup.yaml --cloud modal --gpu a100

# Submit live (authenticate once with `modal setup`, or set
# MODAL_TOKEN_ID and MODAL_TOKEN_SECRET).
soup train --config soup.yaml --cloud modal --gpu a100 --cloud-submit

--gpu accepts t4, l4, a10g, a100 (the default), a100-80gb, l40s and h100.

Plan-only is the default and that is deliberate: nothing is submitted, and nothing is billed, until you add --cloud-submit. You get the generated soup_modal_app.py and the exact command, so you can read what would run before it does.

Your config is base64-embedded as data, not interpolated as code, so nothing in the YAML can execute in the generated stub and no secret is written into it. The image pins soup-cli[train] to the version you are running locally, writes the embedded config inside the container, and runs soup train on the chosen GPU.

Where the checkpoints end up is the part that was missing until v0.75.1, and the answer used to be nowhere. The container is ephemeral, so when the run finished it took the outputs with it, while the entrypoint printed a line telling you to download checkpoints from a directory that had none. Outputs now go to a Modal volume named soup-outputs, created on first use, in a directory named after the run, committed in a finally so a crashed run keeps what it produced, and downloaded into your local output directory at the end including after a failure. If that download is the thing that fails, the command prints the modal volume get line to retry it by hand.

Both cloud paths now refuse an output that leaves the working directory, at plan time. An absolute path, or one reaching upward with .., is rejected before anything is provisioned rather than after. That matters most for the case it was written for: a soup.yaml shared between people, where the output path was written on somebody else's machine.

Lambda Cloud (v0.74.0)

soup train --config soup.yaml --cloud lambda

--gpu takes one of exactly four Lambda instance types: a10, a6000, a100 and h100. That list is shorter than Modal's, so a value that works there (t4, l4, a10g, l40s) is refused here. You need LAMBDA_API_KEY, LAMBDA_SSH_KEY_NAME and LAMBDA_SSH_PRIVATE_KEY in the environment, with the public key already registered in your Lambda account; LAMBDA_REGION is optional and defaults to us-tx-1. Output paths must be relative. Keep the local controller running until it reports termination: shutting down the guest does not stop the billing, the controller does. Since v0.76.0 the controller polls training status over short SSH connections with keepalive probes instead of holding one session open. It tolerates network drops for up to 15 minutes, with exponential backoff capped at 60 seconds, before it terminates the instance to prevent runaway billing, and the checkpoint copy back is retried for the same window.

Plan-only by default, like Modal. The design decision worth knowing is where the credential lives: the API key never enters the instance. Termination belongs to a local controller, and its finally both terminates the instance and polls to confirm that the termination actually happened, rather than firing the call and assuming.

That guarantee had a hole until v0.75.1, and it cost money rather than data. Pressing Ctrl+C during --cloud-submit killed the controller a quarter of a second later, while it was still inside the finally that terminates the paid instance, so the instance could keep running and keep billing. Two narrower windows survived the first fix and were closed after it: a signal arriving between the process being spawned and the guarded region being entered, and one landing on the "still waiting" notice, which is written from the interrupt handler itself. soup now waits for the controller across the interrupt. Check your Lambda console for a still-running instance if you ever interrupted a submit on an earlier version. Two scopes upstream states rather than implies: the fix is Linux and macOS, because the POSIX signal test needs a real process group and is skipped on Windows, and the one way left to skip the cleanup is to kill the controller process itself from another terminal, which can still leave the instance running. Upstream is explicit that the Lambda submission path still needs a paid live-validation pass before it can be described as provider-validated; plan-only rendering and the lifecycle boundaries are covered by offline tests.

--cloud takes modal or lambda, and nothing else. --cloud runpod is a recognised name with no implementation behind it: it prints RunPod cloud training is not yet live; use --cloud modal or lambda and exits 2, and any other value is refused as an invalid --cloud, also with exit 2.

Quantization-Aware Training (QAT)

training.quantization_aware: true (int8 QAT) is refused at config load on every task and backend as of v0.76.0 (#1222). It never ran quantization-aware training. Where it was applied, it ran torchao's post-training int8 weight-only quantization over every Linear layer after LoRA, the adapter's own layers included, and froze them all, so a LoRA run had zero trainable parameters; elsewhere (backend: mlx, backend: unsloth, the other tasks' trainers, layer streaming) it was not applied at all. Remove the key. The "5-10% slower, better quality" claim that stood on this page described a feature that did not train.

Two other values of the same key are separate features with their own code paths:

• quantization_aware: fp8 is FP8 training, covered on Speed and memory. Install pip install "soup-cli[qat]" for it.
• quantization_aware: quest is an experimental mixed W4/A4 plus A16 route retained from an evaluation-only result. It requires task: sft, backend: transformers, modality: text, quantization: none, lora.r: 0, an explicit integer batch_size and auto_mixed_precision: false, and it refuses layer streaming, nvfp4, activation_offloading, freeze_layers, freeze_ratio, unfrozen_parameters, lisa_enabled and expand_layers. Upstream's own record of the strict W4A4 route is a failed gate (see Measurement records); the mixed route makes no packed-INT4, efficiency or training-quality claim.

To ship a smaller model, train normally and quantize at export: soup export --format gguf --quant q4_k_m.

Advanced LoRA Variants

DoRA (Weight-Decomposed LoRA)

training:
  lora:
    r: 64
    alpha: 16
    use_dora: true

use_dora is refused at load on backend: unsloth and backend: mlx (neither builds DoRA, and the run used to train plain LoRA), and with quantization gptq, awq, aqlm or eetq (peft cannot apply DoRA to those layers and raised after the model had loaded). 4bit, 8bit, hqq:Nbit and an unquantized base are allowed.

LoRA+ (Differentiated Learning Rates)

training:
  lr: 2e-5
  loraplus_lr_ratio: 16.0    # lr_B = lr x 16
  lora:
    r: 64
    alpha: 16

The ratio is applied on every LoRA task since v0.76.0 (through v0.75.2 only sft, pretrain and embedding read it, and the others trained B at the base rate). It is refused on prm, moe_lora_routing and unlearn, on classifier, reranker, cross_encoder and asr unless their LoRA flag is on with lora.r > 0, on backend: mlx, with lora.use_vera, with use_lorafa, with use_galore, and with optimizer apollo_adamw, lomo or adalomo. See Configuration. To freeze the A matrices instead, see LoRA-FA.

GaLore (Memory-Efficient Full-Parameter Training)

training:
  quantization: none          # Required: incompatible with quantization
  use_galore: true
  galore_rank: 128
  galore_update_proj_gap: 200
  galore_scale: 0.25

Note: GaLore requires quantization: none and backend: transformers (not unsloth).

NEFTune (Noisy Embeddings)

Add noise to embedding vectors during training for better generalization:

training:
  neftune_alpha: 5.0    # Noise magnitude (0.0–50.0, typical: 5–15)

NEFTune has been shown to improve instruction-following quality without extra data or compute. It is read by the SFT, DPO, KTO, ORPO, SimPO, IPO and BCO trainers; on any other task the key is accepted and ignored.

rsLoRA (Rank-Stabilized LoRA)

Scale LoRA outputs by 1/sqrt(r) instead of 1/r for better training at high ranks:

training:
  lora:
    r: 128              # Higher ranks benefit most from rsLoRA
    alpha: 64
    use_rslora: true

Tip: Combine NEFTune + rsLoRA + Unsloth backend for the best training quality and speed.

MoE Model Support

Fine-tune Mixture of Experts models (Mixtral, Qwen3-30B-A3B, DeepSeek V3):

base: Qwen/Qwen3-30B-A3B
task: sft

training:
  moe_lora: true               # Target expert + attention layers
  moe_aux_loss_coeff: 0.01     # Router load-balancing loss
  quantization: 4bit
  lora:
    dropout: 0.0               # required on a fused-expert MoE such as this one

Soup auto-detects MoE architectures. moe_lora is read by sft, tts, pretrain, dpo, kto, orpo, simpo, ipo, bco, grpo, ppo, online_dpo, reward_model, embedding, distill and unlearn, by the classifier family with classifier_lora: true, and by the shared layer-streaming setup. It is refused at load on asr, prm, moe_lora_routing, SFT with vision or audio, and backend: unsloth. Through v0.75.2 only SFT and pretrain read it (#798), so the DPO and GRPO MoE recipes that set it (eight in the v0.75.2 catalog, fifteen at v0.76.0; upstream's changelog says "8 DPO and 7 GRPO") trained attention-only LoRA, and on transformers 5 they could not attach LoRA at all (peft: No target_modules passed but also no target_parameters found). If you ran one of them, re-run. Expert adapters attach through peft's fused-parameter path (target_parameters) on qwen3_moe, deepseek_v3, glm4_moe, olmoe, qwen2_moe, qwen3_5_moe_text, minimax and mixtral (#1421; measured on tiny stand-ins with transformers 5.16.1 and peft 0.20.0, never on a real 30B+ checkpoint). minimax-m3-sft still trains attention-only LoRA, but because it sets no moe_lora: it loads through SFT's vision path, which never runs the MoE step. The seven kimi-k2.x recipes that set moe_lora are untested rather than known-good: no tiny stand-in for them builds in the installed transformers. moe_aux_loss_coeff is read by sft (text), tts and pretrain, and a non-default value is refused on every other task.

On a transformers 5 fused-expert MoE (every in-library MoE family on the supported stack, Qwen3-30B-A3B included) moe_lora needs lora.dropout: 0.0, because peft adapts the fused expert tensors through a wrapper that refuses dropout; with the default 0.05 the attach stops with a message naming the flag. moe_lora with lora.use_vera is refused on such a model, and moe_expert_quant is refused on one at setup, naming the architecture.

Adapting routed expert tensors directly (v0.74.0)

moe_lora targets expert and attention modules. Some architectures keep their experts as raw 2-D or 3-D nn.Parameter tensors instead, which a module-level target cannot reach, so lora.target_parameters adapts those tensors through PEFT:

training:
  lora:
    r: 16
    alpha: 32
    dropout: 0                 # required: PEFT cannot wrap a raw parameter with dropout
    target_modules: auto
    target_parameters: auto    # routed gate_up_proj + down_proj tensors
    rank_pattern:
      experts.gate_up_proj: 2
      experts.down_proj: 2

auto resolves the architecture's registered parameter mapping, currently Qwen4-Exp routed experts, and fails closed when an architecture has none; an explicit list of parameter-name suffixes is also accepted, capped at 256 entries. The scope is narrow and every bound is refused when the config is read, not minutes into a run: task: sft or pretrain, backend: transformers, modality: text, plain LoRA or rsLoRA with init_strategy: random and use_dora: false, and not with training.stream_layers: true, which stays resident-only for now.

Two operational notes upstream states rather than leaves to be discovered: torch.compile may recompile or graph-break around parameter wrappers, and a parameter-targeted MoE adapter materializes a contribution for every expert at inference, so merge it into the base for deployment unless you need to hot-swap adapters.

Curriculum Learning (v0.23.0+)

Sort the training rows by length, shortest first, before they reach the trainer (the note under the example says what the trainer then does with that order):

training:
  curriculum: true
  curriculum_metric: length       # Sort by sequence length
  curriculum_buckets: 5           # 1-20 difficulty buckets

This sort is applied by the SFT trainer only, and it always orders rows by length, shortest first. curriculum_metric: perplexity or loss is accepted but prints a notice and falls back to the same length sort, and curriculum_buckets only appears in that printout. Only the curriculum_dynamic callback reads the other two metrics. The sort reorders the list handed to the trainer, but Soup does not change the trainer's default sampler, which is random (checked on transformers 5.17, the stack v0.75.2 installs), so the model does not see the rows in that order. Do not expect a short-to-long schedule from this key.

Freeze Training (v0.24.0+)

Freeze bottom layers to reduce compute while preserving base knowledge. Only task: sft (and tts, which trains through it) applies this, on the text path with backend: transformers and no layer streaming; since v0.76.0 any other task, mlx, unsloth, vision, audio or stream_layers refuses the keys at config load instead of training every layer (#1497, #1581):

training:
  freeze_layers: 16               # Freeze bottom 16 layers
  # freeze_ratio: 0.5             # Or freeze 50% of layers

Loss Watchdog (v0.24.0+)

Automatically stop training if loss spikes or diverges:

training:
  loss_watchdog: true
  loss_watchdog_threshold: 5.0    # Stop if loss exceeds threshold
  loss_watchdog_patience: 3       # Wait N steps before stopping

The watchdog is attached by every trainer that attaches the live training callback, which includes task: tts because it runs SFT's training loop. task: prm, task: unlearn, task: moe_lora_routing and backend: mlx refuse loss_watchdog, loss_spike_recovery and grad_accum_auto_tune at config load (v0.76.0), naming the reason. Training stability lists the trainers that do.

Sample Packing (v0.23.0+)

Pack multiple short samples into one sequence for efficient training:

training:
  packing: true                   # Enable sample packing (SFT and Pretrain)

Packed SFT keeps the assistant-only loss mask since v0.76.0 (#1236). Through v0.75.2 a packed SFT run trained on every non-first token, prompt and system tokens included; see Re-run this.