Training Methods

Soup supports 23 training tasks via the task config key. That number is the exact cardinality of the task field in the config schema, not a marketing round-up: the full list is in All Training Tasks below. Switching method means changing one line of YAML, not changing tools.

soup edit set (ROME / MEMIT / AlphaEdit) and soup steer train (CAA / ITI / RepE) are not task: values. They are weight-surgery and inference-time commands, documented under Unlearning & knowledge editing and RAG & steering.

If the model does not fit in your card, add training.stream_layers: true and the frozen base never enters VRAM at all. That works for sft and, since v0.72.4, for dpo, orpo, simpo and kto, with DPO's reference model taken from the same stream with its adapters switched off, so it needs no second copy of the model (on an untied checkpoint, dpo and kto still hold one copy of the output head per step, which the pre-flight charges). See Layer streaming and Preference losses over streaming.

Variables that move Soup's own files, and the ones that carry credentials, are listed on Environment variables and state files.

Moving a soup.yaml from v0.75? Since v0.76.0, dozens of combinations that loaded on v0.75 are refused with a message that names the field and what to change, and a further set of settings that nothing reads only warns (upstream says v0.77 will refuse those). Run soup doctor --config soup.yaml before you train: it exits 2 on a refusal and 0 on warnings alone, and it stops at the first cross-field refusal, so fix that setting and run it again until it prints no error. The list, grouped by what each setting touches, with the start of each message, is Settings Soup refuses or warns about at load; the runs that may need repeating are in the v0.76.0 release page.

Supervised Fine-Tuning (SFT)

The most common method. Train on instruction-response pairs.

base: meta-llama/Llama-3.1-8B-Instruct
task: sft

data:
  train: ./data/train.jsonl
  format: alpaca

training:
  epochs: 3
  lr: 2e-5
  batch_size: auto
  quantization: 4bit
  lora:
    r: 64
    alpha: 16

Two things SFT refuses since v0.74.0 (#535). A row that leaves nothing to learn is refused rather than trained as an all-masked sequence. On the text path with assistant-only masking on (the default, data.train_on_responses_only: true), a row whose assistant reply is missing, or is cut off entirely by data.max_length, stops the run before training with a message like train row 3: no causal-loss target remains after tokenization/truncation at data.max_length=2048; the assistant response is absent or fully truncated. Increase data.max_length or shorten the prompt. The row is named by split and number, and the same check covers pre-tokenized datasets. Long answers under a small max_length are how most people meet it. Second, the final save is refused (refusing to save the final model) when the last logged loss, grad_norm, entropy, train_loss or eval_loss is NaN or infinite, or when any trainable parameter is, so a run that diverged does not write an adapter that merely looks finished. A metric that was non-finite earlier in the run and then recovered does not trigger it.

Packing keeps the assistant-only mask (v0.76.0, #1236). With training.packing: true the loss mask built by data.train_on_responses_only (the default), train_on_messages_with_train_field, mask_history or a pre-tokenized labels column is carried through packing. Through v0.75.2, TRL 0.29.1 dropped the labels column when packing and fell back to training on every input token but the first, prompt and system tokens included. See Re-run this.

data.mask_history: true is read since v0.76.0 (#761). With the default data.train_on_responses_only: true it keeps only the last assistant turn of each conversation in the loss; every earlier assistant turn is masked along with the user and system turns. It matters only for multi-turn chat shapes (chatml, sharegpt and the other message lists): a single-turn row trains the same either way, and a flat format such as alpaca has no turns. With train_on_responses_only: false it is refused at config load, because every token would train, including the history it asks to exclude. The transformers and unsloth text SFT paths and task: distill honour it; backend: mlx and vision or audio SFT do not, and soup doctor --config reports it as ignored there.

SFT also writes a per-row loss record, <output>/rewind.jsonl (training.rewind_log, default true), which soup rewind reads to show which dataset rows sat in a step whose loss spiked: see soup rewind.

Direct Preference Optimization (DPO)

Train with preference pairs (chosen vs rejected).

base: meta-llama/Llama-3.1-8B-Instruct
task: dpo

data:
  train: ./data/preferences.jsonl
  format: dpo

training:
  dpo_beta: 0.1
  quantization: 4bit
  lora:
    r: 64
    alpha: 16

Group Relative Policy Optimization (GRPO)

Reasoning training (DeepSeek-R1 style) with reward functions instead of a reward model.

base: meta-llama/Llama-3.1-8B-Instruct
task: grpo

data:
  train: ./data/reasoning_train.jsonl
  format: sharegpt
  max_length: 4096

training:
  grpo_beta: 0.1
  num_generations: 4
  reward_fn: accuracy     # or 'format', or path to custom .py
  quantization: 4bit
  lora:
    r: 64
    alpha: 16

Built-in reward functions:

• accuracy — 1.0 when the completion's final answer matches the gold's, else 0.0 (no partial credit); see How the final answer is read
• format — checks for structured <think>...</think> reasoning blocks

Custom reward functions — point to a Python file:

# my_reward.py
def reward_fn(completions, **kwargs):
    # kwargs carries your dataset's own columns, row-aligned with completions.
    golds = kwargs["answer"]
    return [1.0 if g in c[-1]["content"] else 0.0 for c, g in zip(completions, golds)]

What is in kwargs. Source columns are preserved and passed through, and the trainer additionally exposes the target as answer: an alpaca output, or the final assistant turn of a chat row. Required columns are validated before generation starts, so a missing one refuses the run rather than producing a silently useless reward: accuracy and the verifiable math domain need answer, verifiable code needs expected or answer, and json_schema needs schema.

What a reward function must return. Exactly one finite number per completion, in the order of completions. The GRPO trainer wraps every reward function (built-in, .py file or ensemble member) in a check, and stops with a message naming the function when it returns the wrong count, a non-number, nan or infinity, or a boolean: return 1.0 and 0.0, not True and False. A comma-separated reward_fn: "accuracy,format" trains an ensemble and is refused on every task except grpo; reward_fn: verifiable is refused unless verifiable_domain (math, code or json_schema) is set.

How the final answer is read

accuracy and the verifiable math domain read the completion and the gold with the same parser. The final answer is, in this order: the rest of the line after the last ####; the content of the last \boxed{} (nested braces are fine); what follows the last The answer is or Answer:, up to the end of its clause. A completion with none of these is read by its last line and its last number. A numeric gold is compared by value, so #### 1,000, \boxed{1000} and The answer is $1000. all match a gold of 1000; any other gold is compared as text, ignoring case, whitespace and LaTeX wrappers. Units are not stripped (42 apples does not match 42), and nothing is evaluated (\frac{1}{2} does not equal 0.5).

An answer phrase that names more than one distinct value is a hedge and states no answer: The answer is either 41 or 42. scores 0.0 against every gold. A gold that states no final answer, or hedges, is refused before generation, naming its split, row and the number of other rows with the same problem, because it would score every completion 0.0 and give GRPO no signal. math scores a numeric gold 1.0 within 1e-4 and 0.6 within 1e-2.

KL penalty, grpo_beta: 0, and the gradient watchdog

training.grpo_beta is the KL coefficient (default 0.1). Since v0.76.0 every grpo_variant applies it: gspo, dapo, dr_grpo, bnpo, two_sided and rft used to validate grpo_beta and drop it, so a variant run trained with no KL term (#1232). A recipe that relies on being KL-free should now say so with grpo_beta: 0, which is valid, skips the reference-model forward pass, and is refused together with reward_hack_mitigation kl_control or pid_lagrangian (they need a positive beta to steer).

Every task: grpo run also attaches a gradient watchdog (#342): if non-finite gradients (NaN or Inf) appear, the optimizer step does not update the weights, and the skipped-step count is logged at the end of the run (as a console warning when the fraction is 5% or more) and saved in trainer_state.json, where soup adapters audit reads it.

PPO / Full RLHF Pipeline

Three-step pipeline: SFT warmup -> Reward Model -> PPO alignment.

# Step 3: PPO alignment
base: meta-llama/Llama-3.1-8B-Instruct
task: ppo

data:
  train: ./data/prompts.jsonl
  format: chatml

training:
  reward_model: ./output_rm   # From step 2
  epochs: 1                   # passes over the prompt dataset
  ppo_epochs: 4               # optimization passes inside each PPO update
  ppo_clip_ratio: 0.2
  ppo_kl_penalty: 0.05
  quantization: 4bit
  lora:
    r: 64
    alpha: 16

On task: ppo, reward_hack_detector, reward_hack_mitigation and echo_trap_enabled are refused when the trainer is built, before any model loads (#1441, #1514): the trl PPO trainer this build supports takes no reward functions, so the signal those callbacks read is never captured. They are wired on GRPO only. task: ppo itself is not refused and still runs; it is those flags that it refuses. soup train --help still describes --reward-hack-detector, --reward-hack-mitigation and --echo-trap-tokenizer-aware as applying to GRPO and PPO, which no longer matches. soup train --dry-run does not see the refusal. PPO also needs a train set of at least batch_size x gradient_accumulation_steps x processes rows, or soup train refuses before loading anything.

The two epoch keys are different things. epochs (default 3) is complete passes over the training data; ppo_epochs is the optimization passes within each PPO update. Soup forwards both, plus ppo_kl_penalty, to the names the installed TRL PPOConfig uses.

All Training Tasks

All 23, exactly as the schema accepts them.

Validation split and evaluation

data.val_split (default 0.1) holds that share of the rows out of training, and since v0.76.0 the run evaluates them. Until then no trainer on the transformers backend scheduled an evaluation: the held-out rows were withheld from training and used for nothing, and val_loss stayed empty (#1223).

data:
  val_split: 0.1    # the default
training:
  eval_steps: 50    # optional; unset = evaluate at the end of every epoch

• Default: every epoch. At the end of each epoch the trainer runs one pass over the validation split and records eval_loss. It appears as the Val loss row on the live panel, as val_loss in the experiment tracker, and as TrainEvent.val_loss in the Web UI. Upstream states that runs take a few percent longer.
• training.eval_steps: N evaluates every N optimizer steps instead, counted after gradient accumulation like save_steps. The last step is evaluated too when it is not a multiple of N.
• Batch size. The evaluation runs at the resolved per-device train batch (after batch_size: auto), not Hugging Face's default of 8. Layer-streamed runs evaluate the same way.
• val_split: 0 trains on every row and evaluates nothing, unless the data brings its own validation split (a Hub dataset's validation split, or a pre-tokenized cache's val or validation split).

Generation-based tasks (grpo, ppo, online_dpo) do not hold rows out. Evaluating them means generating completions, which costs about as much per row as training on it. So data.val_split is ignored on them, every row trains, and the run prints a one-line note saying so. Their default config therefore trains on more rows than it did on v0.75.2. On grpo, set training.eval_steps to hold the split out: the validation number is then the held-out reward, eval_reward, shown as Val reward on the panel and stored as val_reward in the tracker (val_loss stays empty, because TRL's GRPO eval_loss sits near zero and does not measure held-out quality, #1389). ppo and online_dpo refuse eval_steps.

Refused at config load: training.eval_steps on backend: mlx (mlx-lm evaluates the split on its own cadence), on task: unlearn, on ppo and online_dpo, and with val_split: 0 on local files or remote URIs, where there would be nothing to evaluate. prm and moe_lora_routing evaluate, but attach no live callback, so their eval_loss reaches the trainer's log history and not the tracker.

Which tasks quantize the base

training.quantization is 4bit unless you set it, except on eight tasks whose trainers never quantize: distill, classifier, reranker, cross_encoder, prm, moe_lora_routing, unlearn and asr. On them an unset value resolves to none, an explicit 4bit or 8bit loads with a warning and is treated as none (Soup v0.77 will refuse it, per its own warning), and a Quant Menu value such as gptq is refused. Details are under Keys a task ignores.

Running Training

# Start training
soup train --config soup.yaml

# Resume from checkpoint
soup train --config soup.yaml --resume auto
soup train --config soup.yaml --resume ./output/checkpoint-500

# With W&B logging
soup train --config soup.yaml --wandb

# With TensorBoard
soup train --config soup.yaml --tensorboard

# With DeepSpeed (multi-GPU)
soup train --config soup.yaml --deepspeed zero2

# With FSDP2
soup train --config soup.yaml --fsdp full_shard

# Skip confirmation
soup train --config soup.yaml --yes

# Continual learning: rehearse an old dataset so the new task
# does not erase it (v0.71.36, sft/pretrain only)
soup train --config new_task.yaml --replay old_task.jsonl --replay-ratio 0.1

--replay interleaves a seeded sample of old_task.jsonl into training; --replay-ratio is the fraction of the final mixed set. Validation stays pure new-task. See Data Moat II for the full mechanism and its measured limits.

The same three settings exist as config keys, for a soup.yaml you commit: data.replay (the path), data.replay_ratio (default 0.1; above 0 and at most 0.5, so 0.6 is refused at load) and data.replay_seed (0 to 2147483647; unset means seed 0). The ratio and the seed are refused without data.replay, data.replay is refused on any task other than sft and pretrain, and it is refused alongside training.packing or training.multipack, whose bin-packing breaks the ratio. A run that uses it prints Replay: +N old rows interleaved (P% of M).

Read the console before trusting --find-lr

soup train --config soup.yaml --find-lr sweeps the learning rate and writes a report (how it works), but the sweep is not a dry run of your training run. It loads the base model without quantization and without LoRA, puts the whole model on the device and trains every parameter with plain AdamW, one row at a time, on at most 256 tokens of each of the first rows of your data. So it needs the full-precision model to fit, and the rate it recommends is one for full fine-tuning, so it does not tell you the rate for the LoRA run you will launch. It reads each row's text, prompt or messages field only; a row in any other shape, an alpaca instruction and output row for instance, is tokenized as a blank string.

If the sweep cannot run at all (the config fails to load, the base or dataset cannot be loaded, the base does not fit unquantized, torch is missing), soup train --find-lr names the cause, exits 1 and writes no report (v0.76.0, #1203). Through v0.75.2 it printed a yellow line ending writing synthetic curve, wrote a placeholder report and exited 0, so a recommended learning rate from such a run described no model and no data.

The sweep takes one training row per step. With fewer rows than --find-lr-steps it runs one step per row over the same --find-lr-start to --find-lr-end range and says so; the recommendation needs at least 4 points, so --find-lr-steps must be at least 4 and a training set of fewer than 4 rows is refused before the model loads. If the loss turns non-finite partway through, the report covers the steps before it; if that leaves fewer than 4, the command says where it diverged instead of writing a report.

Re-run this (v0.76.0)

Each of these changed what a finished run trained. In every case the run looked normal: a loss curve and exit 0. The wording of each cause is upstream's; the PR numbers are the CHANGELOG's.