Seeds and reproducibility
Until v0.73.0 there was no way to set a seed at all. Every run trained at 42, the framework default, and the only variation between two runs of one config was row ordering plus whatever the GPU did differently that day.
That matters more than it sounds, because a project that reports "arm A beat arm B by 0.006" needs to know what one arm does against itself. Without a seed knob, the natural spread was missing its largest source, and every measurement quoted against it was weaker than it looked.
The two keys
training:
seed: 1234 # weight init of new parameters, data order, dropout
data_seed: 99 # optional: vary data order while holding init fixedBoth are config keys, not CLI flags. There is no --seed, and soup train takes no --output either, so for replicates write one config per seed, each with its own output:, and run soup train --config once per file:
# soup-seed-1.yaml (repeat for seeds 2 and 3; only seed and output differ)
base: HuggingFaceTB/SmolLM2-135M
task: sft
data:
train: ./train.jsonl
training:
seed: 1
output: runs/seed-1- Both default to unset, not to 42. That is deliberate rather than tidy: an unset seed has to reproduce two different historical defaults, 42 for the trainer and 0 for the multipack sampler, and defaulting the field to 42 would have silently re-ordered every existing multipack run.
- An unset
seedresolves to 42; an unsetdata_seedstays unset. The fields are optional precisely so the trainer can tell "unset" from "explicitly 42". - Bounds are 0 to 2^32 - 1. Zero is legitimate. A YAML
trueis refused by name rather than coerced to 1. Since v0.78.0 (#2462) a boolean in the other scalar numeric fields oftrainingandtraining.lorais refused too (<field> must be a number, not a bool, oran int; YAML'syes,onandnoare booleans and are refused), where v0.77.2 readtrueas 1 andfalseas 0 in most of them; fields with their own check, such asseed,data_seed,eval_stepsandgrpo_beta, refused a boolean before and keep their own messages; see Configuration. data_seedis forwarded to Accelerate's dataloader configuration, which takes it fromaccelerate1.1.0. Every supported install is already above that: the transformers 5.17 and trl 0.29 releases this was checked against requireaccelerate1.1.0 and 1.4.0 respectively, and[train]requirestrl>=0.29, so on a working install it applies. Soup's own declared floor,accelerate>=0.27.0, is lower than what the rest of the stack allows.
What changed in v0.73.1, and what did not
In v0.73.0 the seed reached the supervised trainer and nothing else. A seeded GRPO or DPO run was accepted, trained at 42, and said nothing. Seventeen of eighteen task wrappers ignored it. Replicates that differed only in their seed were the same run.
It now reaches every wrapper, and it is applied before the model loads, because the adapter's own initialisation is drawn before the trainer's seeding used to happen.
The values a run trains at are unchanged. What changed is when they arrive. An unseeded run's adapter and classification-head initialisation used to vary from process to process; it is now deterministic at 42.
What a seed does not buy you
This is the part worth reading before you design an experiment on top of it.
- It is not a determinism guarantee. For bit-exact reruns you also need PyTorch's deterministic-algorithms switch, and Soup does not set that for you.
- A resident 4-bit run is still not bit-reproducible from a seed. This is diagnosed and not fixed in the source. Never treat a resident quantised arm as reproducible.
- A streamed run is bit-reproducible: five runs, one adapter hash, zero movement on every behavioural suite. That asymmetry is real and is why the streamed arm is the control.
- The
mlxbackend applies both since v0.77.0 (#1855). MLX SFT seedsmx.random(LoRA init and dropout) withseedbefore the adapter is built, and numpy's global RNG (mlx-lm's batch shuffle) withdata_seed, orseedwhen that is unset, right before training. The native DPO, ORPO and SimPO loops seedmx.randomand their batch order the same way (since v0.77.2task: preferencewithpreference_lossdpo,orpoorsimposelects those same loops), and KTO keeps dataset order on both backends. Since v0.77.1 MLX GRPO seedsmx.randomand shuffles its prompts each epoch fromseedtoo, with one exception: itsdata_seedis recorded but moves nothing, because the prompt order followsseedalone, as TRL's sampler does. Upstream's check is that two runs with the same seed reproduce each other's losses; this site claims nothing beyond that. Through v0.76.1 MLX read neither field and said so on itsMLX backend ignoresline, so no MLX run reproduced, and one started then still cannot be. See the MLX page. - The multipack sampler still takes 0.
- Since v0.76.0 the rows a default
grpo,ppooronline_dporun trains on changed: those three ignoredata.val_splitand train on every row, so a seeded replicate of one of them draws from a larger set than it did on v0.75.2. See Training methods. task: unlearnbuilds no Trainer, so Soup's own seeding call is the only place the seed lands there. Its RMU control direction followsseed, so two RMU replicates at different seeds really are different runs. An unset seed leaves that one draw at 0 rather than 42, for the same reason as the multipack sampler: it has been seeded 0 since RMU landed.
Why the ship gate cares
The consequence is concrete, and it is the reason this page sits next to the release notes rather than in a corner.
Three runs of one unchanged resident configuration moved two behavioural suites by 0.375 and 0.269, against a regression threshold of 0.05. Five of seven suites could cross the line on a re-run that changed nothing at all.
Two separate things follow, and they compose:
- Set a seed, so a replicate is a replicate.
- Measure what your instrument can resolve before you call a delta real. That is
soup ship --noise-floor, added in v0.73.2, and it exists because greedy decoding is not deterministic on a GPU either.
Neither one alone is enough. A seed removes a source of variation you control; the noise floor sizes the one you do not.
See also
- v0.73.1: the free GPU tier — where the seed reached the rest of the trainers.
- v0.73.2: the release gate — where the instrument got measured.
- v0.73.0 Borrowed Hardware — where the seed keys first shipped.
- The ship gate — the verdict these numbers feed.
Soup is free and Apache-2.0. If it saved you a training run, starring the repo costs nothing and helps most. You can also fund the GPU time behind the work a 4 GB laptop cannot reach.