Fine-tune Doctor (v0.71.27)

Most fine-tunes don't fail loudly. They fail silently: the model never stops generating, the assistant turns were never actually trained, or your DPO run quietly learns "longer = better" instead of the preference you meant. v0.71.27 "Fine-tune Doctor" is a pure-CPU, zero-GPU pre-flight that catches these before a single training step.

We have not found these checks in Unsloth, Axolotl or LLaMA-Factory. Without them you find out after the GPU hour.

soup data doctor — 8 chat-template checks

soup data doctor chats.jsonl --model meta-llama/Llama-3.1-8B

It renders your data through the model's real tokenizer and chat template, then runs eight checks:

The verdict uses the same OK / MINOR / MAJOR taxonomy as soup diagnose: exit 0 on OK/MINOR, exit 2 on MAJOR, so it drops straight into CI. A missing path, an empty dataset or a tokenizer that fails to load is exit 1; a mistyped flag, preference data, or RAFT data without --show-mask is exit 2, the same code as MAJOR, because soup data doctor is not one of the commands that moved usage errors to 3 in v0.76.0. Its one exit 3 is described under --mask-history below. See Exit codes.

--max-length (default 2048) is the length the truncation_risk check compares against, so pass the data.max_length you train with; it also truncates the --show-mask rows. The check is OK when the p95 rendered length is within it, MAJOR when at least 20% of the sampled rows are longer, and MINOR otherwise. --sample (default 200, at most 20000, otherwise exit 1) is how many rows the checks run on, spaced evenly across the file rather than taken from the top.

See the mask, don't guess it

soup data doctor chats.jsonl --model meta-llama/Llama-3.1-8B --show-mask 3

--show-mask N prints N sample rows with every token coloured trained vs masked, rendered through the real collator path (answer-only, per-message train-field, or RAFT span-mask), not a reimplementation. An assistant-mask bug becomes visible instantly. The flags --train-on-responses-only and --train-on-messages-with-train-field pick the exact masking strategy soup train would use, so what you see is what you'll train.

--mask-history (v0.76.0) previews data.mask_history, which trains only the last assistant turn of each conversation and masks the earlier ones (--no-mask-history is the default). It needs response-only masking: combined with --no-train-on-responses-only or with --train-on-messages-with-train-field the command exits 3 before it loads a tokenizer or reads the dataset, with a message naming the flag in conflict. Those are the same two combinations soup.yaml refuses at load. data.mask_history is read by text SFT and distill training; the MLX backend, vision SFT and audio SFT ignore it.

Dataset-derived strings are stripped of control characters (C0, DEL and, since v0.76.0, the C1 range) before hitting your terminal, because Rich's markup escape doesn't neutralise raw ESC bytes; the --output JSON is unaffected.

soup data lint — the preference-data linter

soup data lint prefs.jsonl              # word-count length bias
soup data lint prefs.jsonl --model Qwen/Qwen2.5-7B   # exact token-length bias

For dpo / orpo / simpo / ipo / bco / kto data it flags:

• length_bias — chosen systematically longer than rejected (the #1 silent DPO degradation), reported as a Cohen's d effect size so you know how bad it is, not just that it exists. It is MAJOR at |d| of 0.8 or more and mean lengths at least 10% apart, MINOR at |d| of 0.3 or more and at least 5% apart, so a consistent one-word gap between near-constant lengths is not flagged. For conversational chosen and rejected (message lists) it reads only the assistant turns.
• label_imbalance — KTO desirable:undesirable ratio out of balance.
• near_duplicates — MinHash/LSH near-dupes (reuses the soup data dedup kernel).
• identical_pairs — chosen == rejected, i.e. zero preference signal.
• prompt_leak — the prompt echoed verbatim inside the completion.

--sample sets how many rows the linter checks (default 2000, at most 50000, otherwise exit 3), spaced evenly across the file.

Exit codes: 0 for OK or MINOR, 2 for MAJOR, and 3 for a rejected input: a missing path, an empty dataset, data that is not preference data (the message says to use soup data doctor), --sample above 50,000, an unwritable --output, or a mistyped flag. Before v0.76.0 a missing path was 1 and non-preference data was 2, the same code as MAJOR. The soup data doctor bound on --sample (20,000) is unchanged and still exits 1.

soup doctor --config: what this release refuses or warns about (v0.76.0)

soup doctor --config soup.yaml

Run it on any config that loaded on an older release before you train with v0.76.0. Dozens of combinations that loaded on v0.75 are now refused with a message that names the field and what to change, and 29 staged fields warn (upstream says v0.77 will refuse them): see the settings refused at load. A config that cannot load makes soup doctor --config exit 2, after printing the loader's error and Config could not be loaded (see above).. A config that loads with warnings only exits 0. Unknown keys are reported together, but a cross-field refusal prints only the first one it hits, so fix that setting and run the command again until it prints no error.

With a loadable config it then prints a Config check table: the settings your file switches on that its task and backend never read. For example use_galore: true on backend: mlx is listed as ignored (GaLore has no MLX implementation). That table is small and an empty one is not an all-clear. It knows four declared (task, backend, modality) combinations: SFT on MLX (18 entries), SFT on Unsloth (3), and SFT on transformers with vision (2) or audio (2) data. For any other pair it prints None of the N setting(s) known to be unread on task=... backend=... is switched on in this config., which means "nothing to report", not "every setting is read"; upstream tracks the gap as #1815. See the MLX page for the MLX entries.

soup bench train: a short run that fails if the model was not training (v0.76.0)

soup bench train --config soup.yaml --steps 20 --warmup 3 --output bench-train.json

Trains for a fixed number of optimizer steps in a scratch directory, saves nothing, and writes a JSON report with step timing, supervised tokens per second (tokens whose label is not -100, not padded tokens), peak allocated and reserved memory, and the device and library versions. The part that matters before a GPU hour is the verdict: the report is marked not valid, and the command exits 1 with the report still written, when the model was not actually training: no trainable parameter with storage, a gradient norm that is zero or not finite, trainable parameters bit-identical between the first and last step, or fewer steps than requested. It measures the SFT trainer on the transformers backend only; any other task or backend exits 1 with Cannot bench this config, and a light install without PyTorch exits 1 with the [train] hint. --output must stay under the working directory and is checked first (exit 2 if refused). Details are on the bench page.

Numbers

Validated live against the real HuggingFaceTB/SmolLM2-135M-Instruct tokenizer; the smoke pass found and fixed two genuine bugs. Part of the v0.71 release line, which closed at 16,529 tests across 319 files.

See also

• soup diagnose — the post-training report card whose OK/MINOR/MAJOR taxonomy the doctor mirrors.
• Pre-flight & tooling — soup plan/apply, soup tunability, hardware-fit, the rest of the "catch it before GPU" surface.
• MCP server — data_doctor is one of the tools exposed to your coding agent.