Training Stability & Auto-Tuning (v0.32.0)

Pre-flight tuning + in-training stability nets. All flags are opt-in. Not all of them do something at v0.76.0: warmup_auto and convergence_detection are accepted by the schema, but nothing in v0.76.0 reads warmup_auto or acts on convergence_detection, and the loss watchdog, loss-spike hint and VRAM advisory are refused at config load on the tasks and backend that cannot run them. Each section below says which.

LR Range Finder

Run a fast.ai-style geometric LR sweep before the real training run. Soup writes a JSON report with the recommended LR, the loss curve, and the divergence point.

soup train --config soup.yaml \
  --find-lr \
  --find-lr-start 1e-7 \
  --find-lr-end 1e-1 \
  --find-lr-steps 100 \
  --find-lr-output ./lr_finder.json

The report contains the geometric lrs[], raw + EMA-smoothed losses[], the recommended LR (steepest negative gradient before divergence), the LR with min loss, and the divergence point if any.

v0.33.0: --find-lr now runs an in-process LR-sweep training loop. NaN/Inf loss terminates the sweep early so diverged_at is honest.

Since v0.76.0 (#1203) it refuses instead of inventing a curve. If the sweep cannot run at all (the config does not load, the model or dataset cannot be loaded, torch is missing), --find-lr names the cause, exits 1 and writes no report; through v0.75.2 it wrote a placeholder curve and exited 0. The sweep takes one training row per step: with fewer rows than --find-lr-steps it runs one step per row over the same range and says so, and the recommendation needs at least 4 points, so --find-lr-steps must be at least 4 and a training set of fewer than 4 rows is refused before the model loads. If the loss turns non-finite partway through, the report covers the steps before it; if that leaves fewer than 4, the command says where it diverged instead of writing a report. See Training methods.

Warmup (warmup_auto is accepted, not read)

training:
  warmup_ratio: 0.03   # 3% of total update steps (default)

warmup_ratio (0.0 to 0.5) sets warmup as a fraction of the optimizer steps the run really executes. Since v0.76.0 (#1431) Soup passes the fraction to the trainer and transformers resolves it as ceil(real_optimizer_steps x warmup_ratio), with no clamp, so a run with fewer than 34 optimizer steps at the default 0.03 gets exactly one warmup step, and that first step trains at learning rate 0. Through v0.75.2 the ratio was turned into a fixed step count from the raw dataset rows before the trainer existed, so a run whose real step count differed (GRPO, where prompts repeat num_generations times; SFT with packing or multipack; pretrain with packing) got a warmup that was too short or covered the whole run; if the learning curve looked flat or never took off on one of those, re-run. warmup_auto: true is accepted by the schema, but nothing in v0.76.0 reads it, so setting it changes nothing about a training run. A helper that clamps warmup to [10, 1000] steps exists as library code, but no training path and no command calls it, and soup autopilot does not write a warmup into the config it generates.

Auto mixed-precision

training:
  auto_mixed_precision: true

Picks bf16 on Ampere+, fp16 on Turing or known fp16-stable models (Qwen2 / Qwen2.5 / Phi-3 / Phi-3.5), no on pre-Pascal. Multi-version pairs match the longest substring deterministically.

Only task: sft reads it, and task: tts, which trains through the SFT trainer. Setting auto_mixed_precision: true on any other task is refused at config load (v0.76.0, #1618): through v0.75.2 a DPO, GRPO or KTO run carrying the key trained at its trainer's default bf16 or fp16 choice with nothing said. It is also refused together with quantization_aware: quest.

Loss spike recovery hint

Builds on the loss watchdog. The watchdog itself stops the run when the loss stays above loss_watchdog_threshold (default 3.0) for loss_watchdog_patience (default 5) consecutive steps. loss_spike_recovery does not change that: it makes the watchdog also write a spike_recovery.json hint next to the run output, so that you (or a wrapper script) can relaunch with a lower learning rate.

training:
  loss_watchdog: true                   # required
  loss_spike_recovery: true
  loss_spike_recovery_max_attempts: 3
  loss_spike_recovery_lr_decay: 0.5

The hint records the loss at the spike, the previous learning rate, a recommended one (the previous rate times loss_spike_recovery_lr_decay), and an attempt counter against loss_spike_recovery_max_attempts. Soup does not decay the learning rate or resume the run by itself, and nothing in Soup reads spike_recovery.json back: relaunch with the recommended lr and --resume yourself. Because the run stops on the first firing, the attempt counter does not get past 1 within one run. Since v0.76.0 every trainer that attaches the live training callback reads loss_spike_recovery (#802): the SFT trainer (and task: tts, which trains through it), pretrain, DPO, IPO, ORPO, SimPO, KTO, BCO, online DPO, GRPO, PPO, reward-model, embedding, classifier (which also serves reranker and cross_encoder), distill and ASR trainers, and so task: preference, which hands off to the paired-loss trainers. Through v0.75.2 only the SFT trainer read it. task: prm, task: unlearn and task: moe_lora_routing attach no live callback, and backend: mlx has no stop control, so those four refuse loss_watchdog, loss_spike_recovery and grad_accum_auto_tune at config load, naming the reason (they used to be accepted and never read). task: tts is not in that refused list: it trains through the SFT trainer, so it reads all three exactly as sft does. On an SFT run, soup rewind can show which dataset rows sat in a step whose loss spiked: see soup rewind.

Convergence detector (accepted, not enforced; two tunables warn at load)

training:
  convergence_detection: true
  convergence_window: 50
  convergence_rel_tol: 0.005

These keys are accepted by the schema, but nothing in v0.76.0 acts on them: the only reader is a startup check in soup train that prints a note when convergence_detection is set ("set but not enforced during training in this build"), and no advice is surfaced during a run. The plateau-detection logic that would produce continue / early_stop / lower_lr advice exists as library code that no training path calls. Since v0.76.0 a convergence_window or convergence_rel_tol that differs from its default also prints Warning: training.convergence_window is accepted but read by nothing; v0.77 will refuse it at load (#808), and upstream says v0.77 will refuse the config, so delete the two tunables. convergence_detection itself carries no load warning and no announced deadline.

VRAM pressure advisory

training:
  grad_accum_auto_tune: true
  grad_accum_pressure_threshold: 0.92

Records peak memory each step. When pressure crosses the threshold, recommends a new (batch, accum) pair preserving effective batch (capped at accum=1024). It is an advisory printed once per run on CUDA, not an automatic change (restart with the new pair to take effect). Since v0.76.0 every trainer that attaches the live callback reads grad_accum_auto_tune (task: tts too, through the SFT trainer), and prm, unlearn, moe_lora_routing and backend: mlx refuse it at config load (MLX has no VRAM total to measure pressure against on unified memory). Pressure is now the peak allocated memory divided by the total memory of the CUDA device the run uses, read on the first advisory call, and the advisory prints the measured numbers (peak 7.8 of 8.0 GB (97%)). Through v0.75.2 the total was a fixed 24 GB (#1620), so on every card smaller than about 24 GB the advisory could never fire, and on a 40 to 80 GB card it fired at about 22 GB of peak use.