Skip to content

Seeds and reproducibility

Until v0.73.0 there was no way to set a seed at all. Every run trained at 42, the framework default, and the only variation between two runs of one config was row ordering plus whatever the GPU did differently that day.

That matters more than it sounds, because a project that reports "arm A beat arm B by 0.006" needs to know what one arm does against itself. Without a seed knob, the natural spread was missing its largest source, and every measurement quoted against it was weaker than it looked.

The two keys

yaml
training:
  seed: 1234        # weight init of new parameters, data order, dropout
  data_seed: 99     # optional: vary data order while holding init fixed

Both are config keys, not CLI flags. There is no --seed, and soup train takes no --output either, so for replicates write one config per seed, each with its own output:, and run soup train --config once per file:

yaml
# soup-seed-1.yaml (repeat for seeds 2 and 3; only seed and output differ)
base: HuggingFaceTB/SmolLM2-135M
task: sft
data:
  train: ./train.jsonl
training:
  seed: 1
output: runs/seed-1
  • Both default to unset, not to 42. That is deliberate rather than tidy: an unset seed has to reproduce two different historical defaults, 42 for the trainer and 0 for the multipack sampler, and defaulting the field to 42 would have silently re-ordered every existing multipack run.
  • An unset seed resolves to 42; an unset data_seed stays unset. The fields are optional precisely so the trainer can tell "unset" from "explicitly 42".
  • Bounds are 0 to 2^32 - 1. Zero is legitimate. A YAML true is refused by name rather than coerced to 1. Since v0.78.0 (#2462) a boolean in the other scalar numeric fields of training and training.lora is refused too (<field> must be a number, not a bool, or an int; YAML's yes, on and no are booleans and are refused), where v0.77.2 read true as 1 and false as 0 in most of them; fields with their own check, such as seed, data_seed, eval_steps and grpo_beta, refused a boolean before and keep their own messages; see Configuration.
  • data_seed is forwarded to Accelerate's dataloader configuration, which takes it from accelerate 1.1.0. Every supported install is already above that: the transformers 5.17 and trl 0.29 releases this was checked against require accelerate 1.1.0 and 1.4.0 respectively, and [train] requires trl>=0.29, so on a working install it applies. Soup's own declared floor, accelerate>=0.27.0, is lower than what the rest of the stack allows.

What changed in v0.73.1, and what did not

In v0.73.0 the seed reached the supervised trainer and nothing else. A seeded GRPO or DPO run was accepted, trained at 42, and said nothing. Seventeen of eighteen task wrappers ignored it. Replicates that differed only in their seed were the same run.

It now reaches every wrapper, and it is applied before the model loads, because the adapter's own initialisation is drawn before the trainer's seeding used to happen.

The values a run trains at are unchanged. What changed is when they arrive. An unseeded run's adapter and classification-head initialisation used to vary from process to process; it is now deterministic at 42.

What a seed does not buy you

This is the part worth reading before you design an experiment on top of it.

  • It is not a determinism guarantee. For bit-exact reruns you also need PyTorch's deterministic-algorithms switch, and Soup does not set that for you.
  • A resident 4-bit run is still not bit-reproducible from a seed. This is diagnosed and not fixed in the source. Never treat a resident quantised arm as reproducible.
  • A streamed run is bit-reproducible: five runs, one adapter hash, zero movement on every behavioural suite. That asymmetry is real and is why the streamed arm is the control.
  • The mlx backend applies both since v0.77.0 (#1855). MLX SFT seeds mx.random (LoRA init and dropout) with seed before the adapter is built, and numpy's global RNG (mlx-lm's batch shuffle) with data_seed, or seed when that is unset, right before training. The native DPO, ORPO and SimPO loops seed mx.random and their batch order the same way (since v0.77.2 task: preference with preference_loss dpo, orpo or simpo selects those same loops), and KTO keeps dataset order on both backends. Since v0.77.1 MLX GRPO seeds mx.random and shuffles its prompts each epoch from seed too, with one exception: its data_seed is recorded but moves nothing, because the prompt order follows seed alone, as TRL's sampler does. Upstream's check is that two runs with the same seed reproduce each other's losses; this site claims nothing beyond that. Through v0.76.1 MLX read neither field and said so on its MLX backend ignores line, so no MLX run reproduced, and one started then still cannot be. See the MLX page.
  • The multipack sampler still takes 0.
  • Since v0.76.0 the rows a default grpo, ppo or online_dpo run trains on changed: those three ignore data.val_split and train on every row, so a seeded replicate of one of them draws from a larger set than it did on v0.75.2. See Training methods.
  • task: unlearn builds no Trainer, so Soup's own seeding call is the only place the seed lands there. Its RMU control direction follows seed, so two RMU replicates at different seeds really are different runs. An unset seed leaves that one draw at 0 rather than 42, for the same reason as the multipack sampler: it has been seeded 0 since RMU landed.

Why the ship gate cares

The consequence is concrete, and it is the reason this page sits next to the release notes rather than in a corner.

Three runs of one unchanged resident configuration moved two behavioural suites by 0.375 and 0.269, against a regression threshold of 0.05. Five of seven suites could cross the line on a re-run that changed nothing at all.

Two separate things follow, and they compose:

  1. Set a seed, so a replicate is a replicate.
  2. Measure what your instrument can resolve before you call a delta real. That is soup ship --noise-floor, added in v0.73.2, and it exists because greedy decoding is not deterministic on a GPU either.

Neither one alone is enough. A seed removes a source of variation you control; the noise floor sizes the one you do not.

See also

Soup is free and Apache-2.0. If it saved you a training run, starring the repo costs nothing and helps most. You can also fund the GPU time behind the work a 4 GB laptop cannot reach.