Find the rows behind a loss spike: soup rewind

A loss curve tells you that step 340 spiked. It does not tell you which rows were in the batch. soup rewind does, from a small flight recorder that an SFT run writes while it trains. It is new in v0.76.0, which takes Soup to 78 top-level commands.

Upstream's own account of it, as one run: on an 8 GB M1 it flagged step 58 of a Qwen2.5-0.5B run and put the poisoned row at 100% of that step's loss. That is a single anecdote, not a measured rate of any kind.

The recorder

training:
  rewind_log: true   # the default; false writes nothing

With training.rewind_log on, an SFT run writes rewind.jsonl into the run's output directory (<output>, or <output>/<experiment_name> when experiment_name is set). It holds a header line (format version, backend, task, number of rows, batch size, gradient accumulation, and a fingerprint of the training rows) and then one line per micro-batch: the step, the dataset rows in the batch, each row's mean supervised-token loss and its supervised-token count. A non-finite loss is stored as null. A previous run's log in the same directory is kept as rewind.jsonl.1.

It records an SFT run on the transformers or the MLX backend, in a single process. A recorder failure never raises into training; it prints one Rewind recorder disabled: ..., Rewind log disabled: <path>: <error> (a log write that failed, such as a full disk) or Rewind log off: <reason> line and the run carries on. On the transformers backend the recorder switches itself off, with that one line, when it cannot attribute a loss to a row:

• a distributed launch (each rank samples its own shard);
• a resumed run (a resume skips batches the recorder cannot see);
• packing or padding_free, which flatten each micro-batch into one sequence;
• multipack, RAFT, a vision run, an audio run, or a pre-tokenised dataset;
• any task other than sft.

soup rewind then has nothing to read. Look for the Rewind log off or Rewind recorder disabled line in the training output.

Reading it

soup rewind                              # the latest run: list the spikes, detail the worst
soup rewind <run_id> --step 340          # rank the rows of one step
soup rewind <run_id> --top 25 --no-preview
soup rewind <run_id> --json report.json

It needs no model and no GPU. It finds the run in the local tracker (~/.soup/experiments.db), then looks for the log in the run's recorded output directory, then in <output>/<experiment_name> and <output> from the run's saved config, so a run that crashed before the tracker recorded its output directory can still be read.

What counts as a spike. A step is critical when its token-weighted loss is not finite. It is a warning when its loss is strictly more than 2.0 times the median of the previous 20 measured finite steps, and at least 3 such steps lie behind it. Steps with no supervised tokens are skipped and never enter the baseline. The table lists critical spikes first, then warnings by ratio.

How rows are ranked. Within the chosen step, each row's share is loss x tokens divided by the sum of that quantity over the step. The share follows the row that moved the optimizer, not the row with the largest per-token mean, so one long malformed row stands out even when its per-token loss is ordinary. A row whose loss is not finite is listed first with a share of 0.0%.

 Loss spikes
 Severity   Step     Loss   Baseline     ×
 warning       9   7.8429     0.9550   8.2

 Step 9 rows
 Row     Loss   Tokens   Share
  17   9.0000      300   98.4%
  16   0.9000       50    1.6%
row 17: 98% of step 9's loss, 300 tokens
Inspect or drop this row, then retrain.

That output is from a synthetic twelve-step log. The last line names the worst row. "Inspect or drop this row, then retrain." prints whenever the top row carries 50% or more of its step, or its loss is not finite, including on a step that was not a spike: soup rewind <run_id> --step 3 on a clean run prints it for whichever row happens to be first.

Previews. Soup rebuilds the dataset from the run's saved config and shows each row's text only if the rebuilt rows fingerprint identically to the ones the run trained on. Otherwise it prints Preview withheld: the dataset on disk differs from the one this run trained on (fingerprint mismatch). and shows the ranking alone. The row number is an index into the training list Soup built from your data, after any validation split was held out, so it is not necessarily the line number in your file. Text from the dataset is stripped of control characters and shown literally before it reaches the terminal, so a crafted row cannot restyle your screen or inject markup.

Exit codes

The --json destination must stay under the current directory, must not be a symlink or a directory, and an existing regular file is replaced atomically.

Two things the JSON does

Both are upstream behaviours at v0.76.0, stated as they are.

1. --json writes no file on a run with no spike. When no step qualifies and you did not pass --step, the command prints No loss spikes recorded. and returns before the write, with exit 0. A script that reads the file afterwards finds none. Pass --step N to force a detail table and a file.
2. A non-finite loss is written as NaN or Infinity. Python's json.dumps default emits those bare words, which are not valid JSON, and a strict parser rejects the file. This is tracked upstream as #1818. Parse it with a reader that accepts them, such as Python's json.load.

Related

• Training stability: loss-spike recovery and the watchdog act during the run; soup rewind explains one afterwards.
• Observability: soup why, soup tui and crash bundles.
• Experiment tracking: soup runs, which holds the runs soup rewind reads.
• Configuration reference: training.rewind_log in the training table.
• Exit codes