Exit codes
The commands that gate a pipeline report their verdict through the exit code. This page lists what each command returns at v0.78.0. v0.78.0 keeps the eleven-command scheme, moves a code inside it only for an input it now reads as unusable, adds refusals under codes already in use and brings two commands that read the same way (what v0.78.0 adds: a CI step that passed on v0.77.2 can fail on v0.78.0). v0.77.2 kept the eleven-command scheme and changed ten results that its release notes list: six commands (soup distill-prompt, soup data evolve and gen-magpie, soup data filter --perplexity, soup data dedup --field, soup registry push --run-id) now stop with a non-zero code where v0.77.1 exited 0; soup eval quant-check reports an unparseable --tasks file as 3 before any model loads, soup plan and soup apply report malformed YAML or an unreadable config as 2, soup quantize reports a --to value Rich read as markup as 2, soup ship --progress jsonl exits 130 on Ctrl+C, and soup loop status exits 2 on a stored spend that is not a finite number (what v0.77.2 adds). v0.77.1 kept the eleven-command scheme and moved one code inside it, a soup eval gate suite that is not valid YAML (1 to 3); the rest of it is refusals under codes the commands already used, and soup plan and soup apply going from 0 to 1 on a config soup train refuses (what v0.77.1 adds). v0.77.0 kept the eleven-command scheme of v0.76.0 and moved none of its codes: it added new conditions under codes these commands already used, and two new uses of 3 outside the scheme (what v0.77.0 adds). v0.76.1 moved no code either: its YAML read limits refuse under codes the commands already used (what v0.76.1 adds). v0.76.0 moved eleven of them onto one scheme and changed which number means "the gate said no" on several, so a CI step written against v0.75.2 can now pass a failing gate without saying so. Read what moved before anything else.
Wherever it could be done without a model or a network service, each row was checked at v0.76.0 by running the command against a small local fixture: soup ship, soup expect, soup data validate, soup data lint, soup lock check, soup lock show, soup lock write, soup eval gate, soup eval against (its error branches), soup eval quant-check (its error and --allow-stub branches), soup eval behavior, soup eval checklist, soup eval unlearning, soup diagnose (the --evidence path), soup plan, soup apply, soup train (its config refusals), soup doctor --config, soup attest emit --sign sigstore without the extra, soup plugins install, soup ab, soup data clean and soup rewind. Rows for commands that need a model, a GPU, an adapter or a network service (a MAJOR verdict from soup eval quant-check, soup data canary check, soup draft measure, the verdict of soup shrink, soup probe, soup adapters scan, soup adapters check, soup adapters audit, soup recipes verify, soup bench train, soup can run, soup ui, the live paths of soup eval behavior and soup diagnose) were read from the call site and not run, and so were the rarer error branches elsewhere, such as the hardware-fit refusal in soup train. The rows that read a YAML config or suite (soup train, soup doctor --config, soup plan, soup apply, soup ship --config, soup eval gate --suite and the config-read branch of soup adapters audit) and the error branches of soup eval quant-check were run again on the v0.76.1 binary. Most rows that v0.77.0 changed were run on the v0.77.0 binary beside the v0.76.1 tree; the --strategy cmaes row and the judge_rank budget and Sesame CSM refusals of soup train were read from the call site. The rows that v0.77.1 changed or added were run on the v0.77.1 binary, most of them beside the v0.77.0 tree: soup eval gate with a malformed suite, soup train --gate, soup plan, soup apply, soup ship --config and --progress jsonl, soup recipes verify --config, soup data with an --output that names an input (each of the eighteen commands, with the same string or a default output name), soup adapters bisect, soup adapters check, diff, scan, merge and arithmetic with NumPy made unimportable, soup train --cloud lambda, soup profile and soup infer --backend. The one exit that was read from the call site is the --progress jsonl exit 1 on an unexpected exception, and a symbolic link or a Windows spelling of the same path was not run. The rows that v0.77.2 changed were run on the v0.77.2 binary: soup plan and soup apply (malformed YAML, a config that cannot be read, an unknown key, and a distill config with and without --minillm-on-policy), soup eval quant-check (a --tasks file with invalid JSON, with a row missing prompt and with a name that does not end in .jsonl, and a model that does not load), soup distill-prompt, soup data dedup --field, soup data filter --perplexity and --score-only without torch, soup quantize --to "[/]", soup registry push --run-id on a running, a failed and a completed run, soup loop status on a hand-edited state, and soup serve --tool-auth-token. The soup ship --progress jsonl exits 130 and 1 and the soup data evolve and gen-magpie exit 1 (which needs a provider) were read from the call site; the release notes say the 130 was not exercised by hand. The rows that v0.78.0 changed or added were run on the v0.78.0 binary, most of them beside the v0.77.2 tree: soup data mix --apply, soup expect (with a file locked against reading), soup data sample, the six options that refuse nan, soup drift-alarm, soup lock show and soup lock check, soup runs curriculum-curve, soup iterative-dpo, soup tokenizer train, soup ci init, soup airgap-bundle, soup adapters merge, the soup train refusals (--resume on task: ppo, --push-public, --cloud, the config refusals), soup eval gate (suite, --write-baseline), soup ship (aliasing, pickle weights, schema_version, evidence that records failed generations, --task-mode reward, --base-adapter and --gate refusals), soup serve (with the server dependencies stubbed out, because the audit environment had no [serve] extra), soup chat, soup bench infer, soup can run, soup migrate, soup docs search, soup data leakage, the input refusals of soup eval retention, soup prune-layer-cache (help and an empty cache), soup deploy hf-space, soup eval quant-check --performance, soup diagnose --require-all-modes, the input refusals of soup export --validate-onnx and the refusals of soup loop init, set, watch, canary and rollback. Read from the call site and not run: the exit of soup merge --low-memory, the live exit 1 of soup ship, the pass and fail of --validate-onnx, and the loop rows marked as read.
The PR gate that soup ci init writes runs soup data validate (with --min-valid-fraction 1.0 in a workflow generated by v0.77.0, so one row that does not convert fails the job; an existing workflow keeps its old step until you regenerate it with soup ci init --force), soup expect and soup ship --evidence in that order, and a non-zero exit from any step fails the job. Everything below is what those exits, and the rest, mean.
The scheme the gate commands share
soup_cli.utils.exit_codes defines four values. Eleven commands adopt the scheme as a whole, a mistyped flag included: soup ship, soup expect, soup data validate, soup data lint, soup eval gate, soup eval quant-check, soup eval against, soup eval behavior, soup eval checklist, soup lock check and soup recipes verify. Two commands new in v0.78.0, soup eval retention and soup data leakage, read the same way and exit 3 on a mistyped flag; upstream's own list still names the eleven.
- 0 the check passed.
- 1 something broke that is not about your input: a model that would not load, an unwritable output, an unexpected exception, an unreachable judge.
- 2 the gate said no: a failed check, a regression, drift, DON'T SHIP, a MAJOR verdict, a failed expectation.
- 3 the invocation or an input is wrong: a missing or unparseable file, a bad value, an unknown flag, a schema the command rejects.
Every other command keeps its own scheme, listed in the tables below, although a few individual codes moved too (the last rows of the table above), and several of them still use 2 for a rejected argument and 3 for a failed check. Do not carry the scheme over to a sibling: soup diagnose borrowed 3 for three conditions (a NOT_RUN verdict, --attach-to-registry without --output, a bad --tokenizer) and kept 2 for the rest, and soup data doctor, soup eval unlearning and soup probe sit next to commands that moved and did not move with them.
What moved in v0.76.0, and the CI step that now passes silently
| Command | Condition | v0.75.2 | v0.76.0 |
|---|---|---|---|
soup eval gate | a task failed or regressed | 1 | 2 |
soup eval gate | suite missing, outside the working directory or failing the schema; baseline missing or not valid JSON; threshold out of range; --write-baseline without --model | 1 | 3 |
soup eval against | regression | 1 | 2 |
soup eval against | bad --metric, locked suite invalid, no series for a run | 1 | 3 |
soup lock check | drift | 3 | 2 |
soup lock check | lock file missing or unparseable, malformed SHA | 2 | 3 |
soup lock show | lock file missing | 2 | 3 |
soup expect | an expectation failed | 3 | 2 |
soup expect | suite or data rejected | 2 | 3 |
soup eval quant-check | a task is MAJOR | 0 | 2 |
soup eval quant-check | a model fails to load | 0 (a stub scored it) | 1, unless --allow-stub |
soup eval quant-check | missing path, unknown --format, .gguf | 1 or stub | 3 |
soup eval behavior, soup eval checklist | run without --evidence (for behavior, also without --base-model) | 0 (neutral OK) | 3 |
soup data validate | missing file, unknown or undetectable format | 1 | 3 |
soup data lint | missing path, empty dataset, not preference data | 1 or 2 | 3 |
soup ship | evidence missing, not JSON or malformed | 1 | 3 |
| the eleven commands in the scheme | unknown flag, missing option, malformed number | 2 | 3 |
soup diagnose | a requested probe did not run | scored OK | 3 (NOT_RUN) |
soup diagnose, soup eval unlearning | --attach-to-registry without --output | skipped, 0 | 3 for diagnose, 2 for unlearning |
soup diagnose, soup eval unlearning | the registry attach fails | warning, 0 | 1 |
soup train --gpus N on Windows | the launcher's exit code | 0 within seconds | the launcher's code |
soup plugins install | any name | 0 (advisory) | 2 |
soup eval benchmark | nothing measured or saved | 0 | 1 |
soup attest emit --sign sigstore | the Sigstore extra is not installed | 0 (unsigned statement) | 2, nothing written |
soup doctor | an installed package in an extra group beyond its declared ceiling | 0 | 1 |
The step that now passes silently is any test for the old number. A script that treats soup eval gate exit 1 as "the gate failed", or soup lock check exit 3 as "drift", or soup expect exit 3 as "an expectation failed", no longer matches a failing gate: it gets 2. A step written as "fail on any non-zero exit" keeps working and cannot tell the causes apart. Change the test to this:
- A failed gate is 2 on every command in the scheme,
soup shipincluded (which already used it). - 3 means your invocation or input is wrong. Do not treat it as a regression, and do not swallow it: a typo in the CI file used to exit 2, which on
soup shipread as DON'T SHIP, and on all eleven commands it now exits 3. - 1 means something crashed.
The two files Soup generates were updated to match. The workflow from soup ci init wraps each of its three steps: exit 2 prints a step-specific message (data validation failed: unusable data, expectations suite failed: quality regression, gate rejected: model failed ship checks) and exits 2, and any other non-zero prints usage or configuration error (exit N) and exits with N. An older committed workflow still fails the job on any non-zero. The pre-push hook that soup eval gate-install writes now exits 2 on a regression and passes any other non-zero exit of soup eval against through, with [soup] pre-push gate error: configuration or usage error (exit N); before v0.76.0 it turned every non-zero exit into 1. The hook still does nothing unless SOUP_CANDIDATE_RUN_ID is set.
What v0.77.0 adds
v0.77.0 moves no exit code and keeps the eleven commands of v0.76.0 on one scheme. Most rows refuse something that used to pass, under codes these commands already used, so a CI step that passed on v0.76.1 can fail on v0.77.0; three rows go the other way (a non-object line in soup data validate, a bf16 soup adapters check, a non-object row in the data commands), and soup recipes use gains a 3 it did not use before.
| Command | Condition | v0.76.1 | v0.77.0 |
|---|---|---|---|
soup eval gate | a suite with no tasks (a missing, empty or misspelled tasks list) | 0, GATE PASSED | 3 |
soup eval gate | two tasks with the same name | 0 | 3 |
soup eval gate | a baseline score that is not a finite number (NaN, -1e999, 1e999, an integer too large for a float) | 0, 2 or 1, depending on the value | 3 |
soup eval gate | a judge task whose prompts file is empty | 0 at threshold 0 | 2 (an ERROR row) |
soup train --gate | a suite with no tasks or a repeated task name | 0 | 1 |
soup data validate | a line that is not valid JSON, with --min-valid-fraction 1.0 | 0 (the line was not counted) | 2 |
soup data validate | a line that is valid JSON but not an object | 1 (a crash) | counted as invalid: 0, or 2 below --min-valid-fraction |
soup data validate | only unusable lines, with --format given | 1 (a crash) with a non-object line; 0 (0/0 rows valid) with only invalid-JSON lines | 2 |
soup data pii, toxicity, score, langdetect, educational, decontaminate | an input with no extractable text in any row (an empty file included), or a decontaminate --benchmark-file with none | 0 | 1 |
soup drift-alarm | no usable live or reference row (it reads output, response or text) | 0, OK | 1, no verdict panel |
soup sweep | every arm failed | 0 | 1 |
soup sweep | a --param that sets a staged field away from its default (a deprecated value such as data.remove_unused_columns=true is not pre-checked) | 0 | 1 |
soup eval auto | no leg evaluated anything | 0 | 1 |
soup ab | --beta, at any value | 0 (a warning) | 2, before any row is read |
soup train | standard input is a pipe or a redirected file, and neither --yes nor --dry-run is given | the prompt read it | 3 |
soup train | a staged field set away from its default, or a deprecated value, in the config | 0 (a warning) | 1 |
soup train --dry-run | a preference task (dpo and the other pair tasks) on rows with no chosen and rejected | 0 | 1 |
soup doctor --config | a staged field set away from its default, or a deprecated value | 0 (a warning) | 2 |
soup recipes use | kimi-k2-sft, kimi-k2-thinking-grpo | 0 (config written) | 3, nothing written |
soup recipes verify | a missing or unparseable --config on an install without torch | 1 (torch was checked first) | 3 |
soup registry show, soup card and the other commands that take a name:tag reference | the name and tag match several entries | 0 (the newest was used) | refused, listing the candidate ids: 1 for soup registry and soup card, 3 for soup eval quant-check and soup eval gate --baseline registry://..., 2 for soup adapters branch --from-registry |
soup loop watch --detach | an invalid --poll-interval or --max-iterations | 0, and a child that exits at once | 2, nothing spawned |
soup ingest | --output is the --logs file, or a hard link to it | 0 | 1 |
soup export --format awq or gptq | a Deprecated: line, then the usual checks (--calibration-data required) | 1 (the formats were removed) | |
soup export --bits, --group-size, --calibration-samples | accepted | 2 (No such option) | |
soup quantize --to awq or gptq | 0 (it printed a command) | 2 | |
soup adapters merge --strategy cmaes | every evaluation failed | 0 (it wrote the first sample) | 2, nothing written |
soup adapters check | a bfloat16 adapter | 1 | 0 or 2 (a verdict) |
soup data clean, inspect, stats, lint | a row that is not a JSON object | 1 (a crash) | skipped with a warning |
Two of these use 3 outside the eleven commands: soup train on a piped or redirected standard input without --yes and soup recipes use on an unavailable recipe. (An ambiguous registry reference passed to soup eval quant-check or soup eval gate --baseline is also 3, but those commands are in the scheme, where 3 already meant a usage or input error.) Do not read them as the scheme spreading: soup train still uses 1 for a refused config and 2 for a mistyped flag, and soup recipes other than verify still uses 1 for an unknown name. Pass --yes to soup train in scripts and CI jobs. A step that expected soup adapters check to exit 1 on a bf16 adapter now gets a real verdict instead.
What v0.77.1 adds
v0.77.1 keeps the eleven commands of v0.76.0 on one scheme. It moves one code inside it (a soup eval gate suite that is not valid YAML, 1 to 3, which is what 3 already meant on that command), and it adds refusals under codes that were already in use, so a CI step that passed on v0.77.0 can fail on v0.77.1. soup plan and soup apply go from 0 to 1 on a config soup train refuses, which is the gap the v0.77.0 notes said was open.
| Command | Condition | v0.77.0 | v0.77.1 |
|---|---|---|---|
soup eval gate | a suite that is not valid YAML, or whose top level is a non-empty list or scalar | 1, a bare Error: ParserError or Error: TypeError line | 3, Cannot load suite: and the file name |
soup plan | a staged field, a refused deprecated value or any other schema error in the config | 0, a plan written | 1, the Config validation error: lines soup train prints, no plan written, an existing state file unchanged |
soup apply, soup apply --dry-run | the same | 0, the config accepted | 1, the state file unchanged and not marked applied |
soup data pii, toxicity, langdetect, educational, decontaminate, doctor, topics, from-traces, canary insert, sample, dedup, filter, convert, merge, active-sample | an --output (a default output name included) that is the same file as an input | wrote the result over the input; upstream: exit 0 | 1, before the data is read |
soup data lint | the same | ran and wrote its report to that path; the code was the verdict's | 3 |
soup data augment, soup data evolve | the same (evolve --plan-only included) | wrote the result over the input; upstream: exit 0 | 2 |
soup adapters bisect | a fixed --eval-command program that is not on PATH | 2, after one probe, with a BROKEN_AT verdict naming the first checkpoint | 2, before the first probe, with no verdict |
soup train --cloud lambda | an output directory with a path part that starts with - | 0, a plan printed | 2, Cloud plan failed: Lambda output_dir parts must not start with '-' |
soup profile | batch_size, max_length or the model's size outside the range the estimate supports | 0, an estimate printed (checked with an oversized batch_size) | 1, Cannot estimate this run: ... Since v0.78.0 the line names the field that stops the estimate, for example Cannot estimate this run: training.batch_size is outside 1 to <max>, the range the estimate prices.; data.max_length, training.quantization and base have their own sentences, and batch size, sequence length and quantization are checked before the model is read (#2306). |
Four things the table does not say.
- The refusal of an output that is an input is upstream's count of eighteen, and it is not every
soup datacommand. The first five came in #2245 and the other thirteen in #2261. Each compares its outputs with its inputs before the data is read; for some commands a missing input file is still reported first (sample,dedup,filter,convert,topics,doctor,lint,canary insert,from-traces) and for the others the same-file refusal comes first, under the same exit code. Upstream lists the spellings it refuses: the same string,./x,sub/../x, the absolute path, a case variant on a case-insensitive file system, a hard link or a symlink, and on Windows also a trailing space or dot,::$DATAand a junction; the check behind this page used the same string and the default output names, not a link or a Windows spelling.canary insertalso refuses a--manifestthat is the dataset or the--output(exit 1). The commands upstream has not checked aremix,forge,generate,gen-magpie,recipeandcanary check, andsoup data persona-mixstill writes over its--promptsfile when--outputnames it (it exited 0 and replaced the file here). Both are tracked in #2226. (Since closed in v0.77.2:soup data ingestrefuses an--outputthat names its source document, andsoup data persona-mix,generate,canary check,mix --optimize,forgeandrecipe --executerefuse an--outputthat names one of their inputs, all before the data is read and all with exit 1;soup adapters arithmeticandmergerefuse an--outputthat is an input adapter with exit 1 (Output path refused: ..., #2274, #2349, #2379). The shared refusal no longer starts withError:orPlan failed:. Herepersona-mix,ingest,canary check,forgeandadapters mergewere run and each left its input untouched; the other commands are as the release notes list them.) The v0.77.0 advice to give--outputanother path is now the refusal's own message:--output 'pii.jsonl' is the same file as --input 'pii.jsonl'; choose a different --output. soup planandsoup applyprint the same lines assoup train. The first line isConfig validation error:, then one line per setting. A syntax error was 1 at v0.77.1 and is 2 since v0.77.2 (what v0.77.2 adds); a file over a read limit or a config that is not a mapping is still 2, a missing file is still 1.soup doctor --configstays 2 andsoup recipes verify --configandsoup ship --configstay 3 on the same config. One config loads undersoup trainonly because of a flag: a distill config withtraining.minillm_enabled: true,training.minillm_teacher_mix_ratio: 0and notraining.minillm_on_policyloads withsoup train --minillm-on-policy, butsoup planandsoup applyhave no such flag and exit 1, so writetraining.minillm_on_policy: truein the YAML (upstream listed this as a known limitation). (Since closed in v0.77.2: both commands take--minillm-on-policy, #2315.)soup adapters bisectstill exits 2 for two different things. Exit 2 isBROKEN_AT, and it is also an--eval-commandwhose program is not onPATH, so read the first line of the output: the second prints--eval-command program 'x' was not found on PATH; install it, put it on PATH, or give its pathand no verdict panel. A program given with a directory part (./tool, an absolute path) ran as given. When{ckpt}is part of the program name, the name was resolved for each probe and a miss counted as a failed probe, as before. (Since closed in v0.77.2: a program that cannot be started exits 2 with no verdict, whether it is fixed (refused before the first probe) or completed by{ckpt}(resolved for each probe, so the refusal can come after earlier probes ran); a program given with a path that is not a file, or is not executable, exits 2 with--eval-command program './x' could not start: .... A program that starts and exits non-zero, or times out, is still a failed checkpoint. On Windows the template is split without POSIX quoting, so write"{ckpt}"in double quotes, #2314, tracked as #2488 (Since closed in v0.78.0, #2603: on Windows the template is split with Microsoft C runtime rules plus one addition, so--opt="a b"and"{ckpt}"group a word and lose the double quotes, a word that starts with a single quote is grouped and loses them too, and a single quote inside a word groups nothing; read from the code, not run on Windows here). Here a missing fixed program, a name that{ckpt}completes and a./path that does not exist all exited 2 with no verdict.)- Missing packages are named now, under the same code.
soup adapters check,diffandscanwithout NumPy exit 1 withnumpy package required; pip install "soup-cli[train]" (or pip install numpy);soup adapters mergeandsoup adapters arithmeticexit 1 withnumpy and safetensors packages required; pip install "soup-cli[train]" (or pip install safetensors numpy)before they read an adapter (#2141). Through v0.77.0checkreported the missing NumPy as a corrupt weights file anddiffended in aModuleNotFoundErrortraceback.
Two options are new and add little to the codes. soup ship --progress jsonl writes JSONL events to stderr and leaves stdout and the exit codes alone: 0 for SHIP, 2 for DON'T SHIP (a completed verdict ends with an end event, not an error), 3 for a usage or input error (an error event with the fixed text execution_aborted); a value other than jsonl exits 3. In that mode an unexpected exception is exit 1 with start and error events and no traceback (read from the call site; since v0.77.2 it also prints an Error: line on stdout, and Ctrl+C is 130, see what v0.77.2 adds). soup infer --backend takes transformers or mlx; any other value is Click's 2, and --backend mlx with an option MLX generation does not take, or without the [mlx] extra, exits 1 naming it (the options).
What v0.77.2 adds
v0.77.2 keeps the eleven commands of v0.76.0 on one scheme and changes the ten results below, which are the rows of its release notes. Six turn a result that was empty, unfiltered or unusable, or a stored value that is not a number, into a refusal, so a CI step that passed on v0.77.1 can fail on v0.77.2; the other four replace a code that said little with one that says more.
| Command | Condition | v0.77.1 | v0.77.2 |
|---|---|---|---|
soup eval quant-check | an unparseable --tasks file | 1, after loading both models | 3, before any model loads (#2185) |
soup distill-prompt | no trace row has a prompt | 0, an empty output written | 3, the output untouched (#2209) |
soup data evolve, soup data gen-magpie | no row was produced and at least one provider call failed | 0, the output emptied | 1, the output kept (#2348) |
soup data filter --perplexity or --score-only | torch or transformers is not installed | 0, unfiltered or partial | 1 (#2377) |
soup data dedup --field <key> | the key is absent from every row | 0, distinct rows dropped | 1 (#2336) |
soup registry push --run-id | the run is not completed | 0, an entry registered | 1 (#2248) |
soup plan, soup apply | malformed YAML, or a config that cannot be read | 1, the raw parser or OS error | 2, naming the config and the YAML location (#2323) |
soup quantize --to "[/]" | a value Rich read as markup | 1 | 2 (#2217) |
soup ship --progress jsonl | Ctrl+C | 1 | 130 (#2417) |
soup loop status | a stored spend that is NaN or Infinity | 0, $nan printed | 2 (#2424) |
Six things the table does not say.
- The old results of rows 2 to 6 are upstream's words, not a finding of this page. The v0.77.2 notes describe the old result of
soup distill-promptwith no usable prompt as "empty output written" (now exit 3 with the output untouched, #2209) and that ofsoup data evolveandgen-magpiewhen no row was produced and a provider call had failed as "output emptied" (now exit 1 with the output kept, #2348; the release notes put it as "every provider call failed", the code at the tag refuses when no row survives and at least one call failed).soup data filter --perplexityand--score-onlywithout torch or transformers exited 0 with "unfiltered or partial" output (#2377), andsoup data dedup --field <key>with a key absent from every row exited 0 with "distinct rows dropped" (#2336).soup registry push --run-idon a run that was notcompletedexited 0 with "an entry registered" (#2248). If a script ran one of these on v0.77.1 and kept the file, look at the file. On v0.77.2 the reason is printed, among other lines, for example--field 'x' is absent from every row; choose a field present in the dataset.andNo prompts found in traces: 0 of 2 trace rows had a prompt .... soup planandsoup applynow exit 2 on a YAML syntax error, and every other code is as at v0.77.1. A config that is not valid YAML, or that cannot be read, prints one line naming the config (Config 'soup.yaml' is not valid YAML: mapping values are not allowed here (line 2, column 7), orConfig 'soup.yaml' cannot be read: Permission denied; a file that is not valid UTF-8 printsConfig 'soup.yaml' is not valid UTF-8: ...the same way) and exits 2. A missing file is still 1, a configsoup trainrefuses is still 1 with theConfig validation error:lines, and a file over a read limit or a config that is not a mapping is still 2. Both commands take--minillm-on-policy(#2315, by @Nari-Ab), so the distill config that v0.77.1 refused (training.minillm_enabled: true,training.minillm_teacher_mix_ratio: 0and notraining.minillm_on_policy) is validated and hashed like any other withsoup plan --minillm-on-policy: here that plan exited 0 and the same plan without the flag exited 1.soup apply --minillm-on-policykeeps the flag in thesoup traincommand it prints and still only records the plan.soup eval quant-checkreads--tasksbefore it touches a model. Invalid JSON, a row withoutprompt, a name that does not end in.jsonl, an empty file and a path that cannot be opened all printCannot load --tasks: ...and exit 3, so the exit-1 note on the quant-check row (the binary against the upstream changelog) is history now. A model that does not load is still 1 without--allow-stub.soup ship --progress jsonlends in 130 on Ctrl+C. The wrapper maps the interrupt to 130 after theerrorevent with the fixed textexecution_aborted; this was read from the call site, and the release notes list it among the paths not exercised by hand. An unexpected exception in that mode is still exit 1 and now also printsError: <ExceptionType>: <message>(control characters removed) on stdout, while stderr stays JSONL events.soup loop statusexits 2 on a stored spend that is not a finite number. The same is true of a negative spend (at v0.78.0loop state invalid: state file has invalid contents: spent_this_month_usd must be in [0, 1000000000000000]; v0.77.2 printed... must be >= 0). The command shows the spend and the run count that the next iteration would see (the month and day resets applied to the savedlast_run_date) and writes nothing toloop.yaml(the loop).- Other refusals under codes already in use.
soup ci init --pythonwith a version outside 3.11 to 3.13 exits 1 (python_version 3.10 is not supported by soup-cli 0.77.2 (needs Python 3.11 to 3.13);03.11and3.011are still accepted and written as typed, tracked as #2534, #2249 (Since closed in v0.78.0: both exit 1 withpython_version must look like '3.11', #2632; the3.10message namessoup-cli 0.78.0)).soup profile --modelexits 1 with the reason when noconfig.jsoncan be read (#2285).soup eval compareandsoup eval leaderboardexit 1 for a stored score that is not finite (#2477).soup ship --adapterreturns DON'T SHIP (2) with the ruleadapter_noopfor a local adapter directory in which everylora_Btensor is zero and every other saved tensor (a head, a bias) is zero too, before any model loads (#2434). The check reads the adapter weights with NumPy, from a directory under the current directory that is not a symlink; a Hub id, an adapter with no LoRA pairs, or one with a non-zero saved tensor goes on to the live evaluation, and if the weights cannot be read (NumPy missing, a path outside the current directory)soup shipprints a warning and runs the live evaluation as before. (Since v0.78.0 a pickle adapter,adapter_model.binand the like, is not one of those cases: it is refused with exit 3, see what v0.78.0 adds.)soup serveexits 2 for a--tool-auth-tokenorSOUP_TOOL_AUTH_TOKENoutside the token rule (16 to 128 characters fromA-Z,a-z,0-9,_and-, trailing newline included), before the model loads, on every bind (#2329; the rule).
What v0.78.0 adds
v0.78.0 keeps the eleven commands of v0.76.0 on one scheme and moves a code inside it only for an input it now reads as unusable (soup expect, soup data validate, soup eval gate and soup lock check; soup lock show, which is not on the scheme, moved too). Most other rows turn something that used to pass, print nothing or crash into a refusal, so a CI step that passed on v0.77.2 can fail on v0.78.0. Two commands new in v0.78.0, soup eval retention and soup data leakage, read the same way (0 pass, 2 gate failed, 3 usage or input, a mistyped flag included), which makes thirteen commands registered on the shared scheme in the code; upstream's own list in docs/commands.md still names the eleven. Each row below was run on the v0.78.0 binary beside the v0.77.2 tree except where it says it was read from the call site.
| Command | Condition | v0.77.2 | v0.78.0 |
|---|---|---|---|
soup data mix --apply | YAML that does not parse, or a data.interleave.probs value that is not a number | 1 (Error: ParserError: ..., or Error: ValueError: could not convert string to float) | 2, apply failed: ..., with no snippet printed for a bad weight (#2905, #2946) |
soup expect | a suite or dataset that exists but cannot be opened (a permission error) | 1 (Error: PermissionError: ...) | 3 (#2945) |
soup data sample | -n 0 or below; --pct 0, above 100 or nan | 0 for -n 0 (an empty output), --pct 0 (one row) and --pct 150 (every row); -n -1 and --pct nan ended in Error: ValueError: ..., exit 1 | 1, --n must be a positive integer. or --pct must be a finite number in (0, 100]. (#2931) |
soup data validate | --min-valid-fraction nan | 0, the gate was off | 3, Invalid value for '--min-valid-fraction': nan is not in the range 0.0<=x<=1.0., before any input is read (#2917) |
soup data toxicity, decontaminate and score (--threshold), soup data forge (--uncertainty-threshold), soup draft measure (--min-acceptance) | nan | toxicity 0 with an empty output file, draft measure accepted (its floor was off), the other three 1, but only after they had read their options or input | 2, the same Click message, before any input is read (#2917) |
soup drift-alarm | rows that exist but hold no tokens (whitespace only) | 0, OK, KL: 0.0000 | 1, ... has 1 readable row but no tokens. Nothing was compared, so no drift verdict is possible. (#2955) |
soup lock show, soup lock check | a directory where the lock file should be | 1, Error: PermissionError: ... on Windows | 3, lock path is a directory: 'lkdir' (#2903) |
soup eval gate | a suite whose top-level mapping has a non-string key | 1, Error: TypeError: keywords must be strings | 3, Cannot load suite: ... has a non-string key 1; top-level keys must be strings |
soup eval gate | a task threshold that is a boolean, or a key repeated inside a task mapping | loaded and ran (true read as 1; the later value of a repeated key won) | 3, ... threshold must be a number, not a boolean / ... has duplicate key 'name' in a task mapping on line 4 (#2696) |
soup eval gate | --write-baseline naming the suite, a custom task's file or a judge task's prompts | no check | 3, --write-baseline 'suite.yaml' aliases --suite 'suite.yaml'; choose a different --write-baseline, nothing written (#2694). The overwrite itself was not shown on v0.77.2: it rests on the absent check |
soup runs curriculum-curve | a history file with a truncated JSON line or with mixed bucket counts (read from the code: a row that is not an object) | 1, Error: JSONDecodeError: ... or Error: ValueError: ... | 2, history malformed: ... (#2731, #2796) |
soup iterative-dpo | a prompts file with no usable prompt (empty, or only blank lines) | went on to load the sampling model | 1, Iterative DPO failed: prompts file 'x' has no usable prompts, before any model code runs (#2958) |
soup tokenizer train | --output names an existing file | not compared (upstream: it trained, then failed) | 1, Output path is not a directory: <path>, before the tokenizers import (#2947) |
soup ci init --python | 03.11 or 3.011 | 0, written into the workflow as typed | 1, python_version must look like '3.11' (#2632). 3.10 is still 1, and its message now names soup-cli 0.78.0 |
soup airgap-bundle | a missing --model, --dataset, --wheel or --kernel folder, or a file where a folder is expected | 0, a bundle whose manifest listed no files | 1, model directory not found: ... or dataset path is not a directory: ..., nothing written (#2568; the gap was #2453). A bundle over the size cap, or --bundle-size-cap 0, is 2 |
soup adapters merge | inputs it cannot merge: a LoRA rank that differs on a shared module, no tensor name in common, or a shared tensor with two shapes (--strategy cmaes included) | 0, an adapter with no weights written | 2, cannot merge: ..., nothing written (#2926, #2907) |
soup merge --low-memory | an adapter whose init_lora_weights can rewrite the base (pissa, olora and the like) | accepted | refused, exit 1 (the refusal text was run on the loader; the exit code is read from commands/merge.py; #2732) |
soup train | --resume or --hf-resume on task: ppo | accepted: --push-as <repo> --hf-resume --dry-run exited 0 (without --push-as the older --hf-resume requires --push-as exit 1 came first; --resume auto exited 1 only because no checkpoint existed; upstream: --resume warned, restarted from step 0 and wrote over the output directory) | 1, before the data is read and under --dry-run too, soup train --resume is not supported for task 'ppo': ... (#2772) |
soup train | --push-public without --push-as | 2, No such option '--push-public' (the option is new) | 1, --push-public requires --push-as <repo>: ..., under --dry-run too |
soup train --cloud | a config that names a local input, or --push-as or --gate beside --cloud | 0, the plan printed | 2, Cloud plan failed: ... / --push-as is not supported with --cloud: ... (see Settings Soup refuses at load) |
soup train, soup doctor --config, soup plan | a boolean in a numeric training or training.lora field, a wrong-typed training value, training.eval_gate.baseline: previous, an enabled gate on backend: mlx or task: prm, unlearn or moe_lora_routing, training.vllm_sleep_mode: true on task: grpo | loaded (0); a wrong-typed value was Error: TypeError: ... from soup train and 2 from soup plan | soup train 1, soup doctor --config 2, soup plan 1 (same page). A training.eval_gate.suite that cannot load stops soup train and soup train --dry-run with 1; soup doctor --config does not read the suite and exits 0 |
soup ship | --output or --emit-evidence that is the same file as --evidence, --task-eval, --config, --corrections, --held-out, --baseline or --reward-fn, or as the other output | 0; for --evidence the input file was replaced by the verdict (run), and for the other inputs the alias check did not exist at v0.77.2 | 3, Error: --output 'evs.json' is the same file as --evidence 'evs.json'; choose a different --output, before a model loads and before anything is written (the config and the files of --gate loop-v2 are read first; --evidence and the task file are not); compared by real path, so a case-insensitive spelling counts (#2894, #2982) |
soup ship | an adapter whose weights are a pickle (adapter_model.bin, .pt, .pth, .pkl, .pickle, .ckpt, or a pickle named .safetensors), on --adapter or --base-adapter | a warning (could not read the adapter weights to check for an all-zero lora_B), then the live eval | 3, ... unsafe weight file (pickle / PyTorch-classic): adapter_model.bin. ..., before any model loads |
soup ship --evidence | a schema_version field | 3, evidence has unsupported field(s): 'schema_version' | 1 and 2 are read; 3 for any other value (evidence.schema_version must be 1 or 2, got 3), and for a version-2 field in a file that says 1 or says nothing |
soup ship --evidence (the MCP ship_evidence tool returns the same message as a tool error) | evidence whose task side records that every generation failed on one side | not readable (a version-2 file) | 3, every task generation failed on the base side (5 of 5), under every task mode; a measured zero and partial failures are replayed as before (#2899) |
soup ship | live --task-mode metric with every task generation failed on one side | 0, the side scored 0.0 (upstream's words) | 1, every task generation failed on the base side (8 of 8); read from upstream's changelog (#2870), not run |
soup serve | --max-tokens outside 1 to 16384 | accepted | 2, --max-tokens: must be between 1 and 16384 (got N)., before any backend loads (#2855, #2898). With no [serve] extra installed the missing-extra message comes first, exit 1 |
soup serve | --trace-log, --record-thumbs or --feedback-log with --backend vllm, sglang or mii | --trace-log and --record-thumbs: accepted and silently ignored (in the run it went on to look for the backend); --feedback-log is new in v0.78.0 (v0.77.2: No such option, exit 2) | 2, --trace-log requires --backend transformers (the vllm backend does not support it, so the flag would be silently ignored). (same server-extra rider) |
soup chat, soup bench infer | --max-tokens below 1; for bench infer also --num-prompts below 1 without --prompts-file | accepted, reached the model load | 2, 0 is not in the range x>=1. for --max-tokens, and --num-prompts must be at least 1 when --prompts-file is not set for --num-prompts, at parse time (#2787, #2788). There is no upper bound on these two |
soup can run | a can that holds <redacted> values (packed by v0.78.0 from a config that carried a credential) | not applicable: v0.77.2 packed the value in clear | 1, this can holds redacted value(s) at: <key paths> ..., before anything is extracted, fetched or trained, but only with --yes: without it the confirmation panel prints and exits 1 first (#2883) |
soup migrate --from axolotl | a custom prompt-format datasets[].type mapping | 1, Error: TypeError: unhashable type: 'dict', nothing written | 0, the config written with data.format: auto and one warning that the template was not carried over |
Notes on the table.
- Two rows of the changelog at the tag are wider than the code (upstream has since narrowed both in the release notes on GitHub).
soup expectmoved from 1 to 3 only for an input that exists and cannot be opened: a missing file, invalid YAML, an invalid UTF-8 file and a directory were already 3 on v0.77.2, so exit 1 still means only an unreachable judge.soup data mix --applymoved from 1 to 2 only for YAML that does not parse and for weights that are not numbers: a negative, zero,NaNor infinite weight is still accepted and prints the snippet with exit 0, and a missing recipe or one without adata:mapping was already 2. soup iterative-dporefuses a prompts file with no usable prompt, not one of blank strings. A row{"prompt": ""}counts as a prompt; only a file with no non-blank line, or none that yields a string or a message list, stops with the new message.soup can packandsoup can forkredact before they write. They replace credential-shaped values and values under credential-named keys with<redacted>, print the key paths they changed (never the values) and exit 0;soup can publishwarns about key paths that still look like credentials. A can that holds a raw credential, packed by v0.77.2, is not refused as redacted: it extracts and runs. Upstream calls the redaction pattern-based and best-effort: read a can before you share it.soup export --verboseand--no-install-depsare still refused with exit 2 on a format other thangguf; v0.78.0 fixes the grammar of the single-flag message (--verbose requires --format gguf., where v0.77.2 printed--verbose require --format gguf.).soup trainand Ctrl+C changed too, and that is not a row: see the rule on Ctrl+C in Three rules that hold for every command.
New commands and options, and the codes they use
| Command | Code | Means |
|---|---|---|
soup eval retention | 0 / 2 / 3 / 1 | No forgetting past the threshold / a task fell further than --threshold below the best score any earlier version had / a usage or input error, a mistyped flag included (no ledger and no --add-task, --add-task NAME=PATH with a missing file or a bad name, --threshold outside 0 to 1, a --ledger outside the working directory) / a model that fails to load or another runtime error. Run with no model: every input refusal is 3, and a valid --add-task with --record reached the model load and exited 1; the regression path and the table were not run |
soup data leakage | 0 / 2 / 3 | No eval record shares its input text with the training file / at least one does (--json prints the report and still exits 2) / an unusable input: a missing file, a line that is not JSON or not an object, an empty file, a path outside the working directory, or a Click error such as --samples 21. Run on the v0.78.0 binary |
soup docs search | 0, 1, 2 | 0 for a table or JSON result and for a query with no match (No matching documentation found for: ...); 1 when the documentation index cannot be read (read from the code, not reached with the bundled index); 2 for --top-k below 1 or a --format other than table or json |
soup prune-layer-cache | 0, 1, 2 | 0 for a dry run, for --yes and for a cache with nothing to prune (Nothing to prune.); 1 when a removal failed; 2 without --model |
soup deploy hf-space --hardware | 1 | A slug that is not lowercase letters, digits and single hyphens of at most 32 characters, or zero-a10g with the streamlit-chat template (ZeroGPU does not support the built-in streamlit-chat template; use --template gradio-chat.) |
soup ship --task-mode reward | 3 | Without --reward-fn (--task-mode reward needs --reward-fn <accuracy, format, verifiable:math, verifiable:json_schema or the path of a .py file>), including through --config; eval.ship.reward_fn beside another task_mode is a config refusal |
soup ship --base-adapter, --gate loop-v2, --corrections, --held-out | 3 | Every refusal comes before a model loads: --base-adapter without --adapter, with --tuned, --evidence, --emit-evidence, --baseline or --require-provenance, with a non-bundled --general-suite, with a path that is not a directory or is the --adapter path; --gate with a value other than loop-v2, or with --evidence, --emit-evidence or --require-provenance; --corrections or --held-out without the gate, naming a file that does not exist, or holding a bad row |
soup eval quant-check --performance | 3 / 1 | 3 for --allow-stub with it, --device or the warm-up and repeat options without it, a device that cannot be resolved, or a Click range error; 1 for a runtime failure. Speed and memory never change the exit-2 verdict |
soup diagnose --require-all-modes | 3 | With --evidence, a mode missing from the file is NOT_RUN and the run exits 3 unless --allow-not-run; with --base-model, or without --evidence, Error: --require-all-modes only applies to --evidence runs (not with --base-model). exits 3 |
soup export --validate-onnx | 3 / 2 / 0 | 3 for a usage problem (no --prompts file, more than 32 prompts, a bad --atol, a prompts file that cannot be read) and for a validation that cannot run; the input refusals were run, the 2 for a numerical failure and the 0 for a pass were read from the code and not run |
soup serve --adapter-dir | 2 | With a backend other than transformers, with --mole, or naming a path that is not a directory or is a filesystem root, before any model loads |
soup loop init | 2 | --train-method outside dpo, kto and sft; --sft-* options on a loop that does not train sft; an out-of-range --sft-lora-r; --canary-traffic of 0, above 100 or not a number; a --canary-on-timeout or --canary-max-wait value outside the allowed ones (--canary-max-wait 31d is refused: must be at most 30d); --paraphraser-endpoint without --paraphraser-model, or an endpoint that is not this machine or a LAN address (a LAN address needs https; plain http is accepted only on loopback); a --trace-dir that is a file; --min-new-edits 0; an invalid --verify; and a second init without --force |
soup loop set | 2 / 1 | 2 for an unknown key, an empty or non-directory trace_dir, an invalid verify, an empty base_model; 1 for a locked loop.yaml (read from the code) |
soup loop watch | 2 | A second --detach, or a foreground watch while a live daemon holds .soup/loop.pid. The daemon exits 1 when it stops itself on a loop.yaml it cannot read (read from the code) |
soup loop canary | 2 | An adapter whose last verdict file says DON'T SHIP: refusing to canary cand2: its verdict file .soup-loops/verdicts/cand2.json says DON'T SHIP. .... --allow-dont-ship exits 0 and records the override in loop.yaml and in the canary's progress.jsonl line; if the canary is promoted it is also in the promotion's note and versions.jsonl line (the promotion half was not run) |
soup loop rollback | 2 / 1 | 2 when there is nothing to roll back to; 1 when the server refuses, with loop.yaml unchanged (read from the code) |
soup loop status | 2 | A hand-edited loop.yaml whose monthly_budget_usd or spent_this_month_usd is not finite or is above 1e15, naming the field (upstream's release notes; the non-finite spend was already 2 at v0.77.2) |
What v0.76.1 adds: new refusals under codes that already existed
v0.76.1 moves no exit code. It puts one loader with nesting, expansion and size limits in front of every YAML file Soup reads (the limits), so a file that loaded on v0.76.0 can now be refused, with these codes:
| Command | A YAML file over a limit | A YAML syntax error |
|---|---|---|
soup train --config | 1, one line naming the file | 1, now one line naming the file (v0.76.0 printed the parser's message) |
soup doctor --config | 2, the same line, then Config could not be loaded (see above). | 2, the same |
soup plan, soup apply | 2 (through v0.76.0 the file was read, and plan wrote a plan for it) | 1 through v0.77.1, 2 since v0.77.2 (#2323) |
soup eval gate --suite | 3 | 3 (through v0.77.0: 1) |
soup train --gate | 1, Invalid --gate suite: ... | 1 |
soup ship --config | 3 (its size cap is 4 MiB) | 3 |
soup adapters audit --config | 1 | 1 |
Three rules that hold for every command
- 0 is success.
--helpexits 0. - Click rejects a bad invocation with exit 2 before the command's own code runs, except on the eleven commands in the scheme, and
soup eval retentionandsoup data leakagesince v0.78.0, where it is 3. That covers an unknown command or flag, a missing required option, and a value that fails Click's own type check.soup ship --bogusandsoup ship --forgetting-threshold abcexit 3 at v0.76.0;soup diagnose --bogus,soup data doctor --bogus,soup eval unlearning --bogus,soup adapters ...,soup lock write,soup lock showandsoup train --bogusstill exit 2. - An unexpected exception prints a short error and exits 1, and Ctrl+C exits 130.
soup --verbose <command>shows the traceback. (A malformed YAML suite passed tosoup eval gatewas the example through v0.77.0, when it stayed 1 rather than 3; since v0.77.1 it exits 3 with a message naming the file.) Through v0.77.2soup traininstalled no Ctrl+C handler of its own, so interrupting a local run was an ordinary 130. Since v0.78.0 a Hugging Face Trainer run handles Ctrl+C itself (#2414): the first press saves a checkpoint after the next completed optimizer step and keeps training, the second saves and stops after that checkpoint, and the third restores the previous handler and raises the interrupt at once, which is the old behaviour and its 130 (read from the code; upstream says it needs a real console and was not exercised by hand, so no exit status is stated for the second press). Native MLX training and loops that do not use the Hugging Face Trainer,task: unlearnincluded, have no such handler, so Ctrl+C there is still the ordinary 130. A script that sent one SIGINT to stop a run now needs two to stop it, or three to interrupt at once. A--cloud runpod --cloud-submitrun instead keeps waiting for the controller to delete the Pod and exits with its code. See Observability.
Where one code means two things
soup diagnose: exit 2 is a MAJOR verdict, and it is also a missing--evidencefile, an invalid--citation-styleand a mistyped flag. Exit 3 is the newNOT_RUNverdict, and it is also--attach-to-registrywithout--outputand a bad--tokenizer.soup data doctor,soup eval unlearningandsoup probe: exit 2 is a MAJOR verdict, and each of them also uses 2 for rejected inputs and for a mistyped flag (preference data handed tosoup data doctor, a missing--evidencefile).soup shrink,soup reward stress,soup draft measure,soup adapters check,soup adapters audit: exit 2 is the verdict (DON'T SHIP, gameable, below the floor, inactive, diverged) and also what Click returns for a mistyped flag.soup adapters scan: exit 1 is a WARN verdict and also an adapter that is missing or unreadable.soup adapters verifyandsoup adapters check-safetensors: a failed check is 1 by default and 3 with--strict. Both also exit 3, with or without--strict, when the directory is outside the working directory or is a symlink, andsoup adapters verifyexits 3 on a.soup-signature.jsonthat cannot be parsed.soup adapters bisect: exit 2 isBROKEN_ATand also a rejected argument, and, since v0.77.1 for a fixed program and since v0.77.2 for any program (one that{ckpt}completes included), an--eval-commandprogram that is not onPATHor cannot be started; the first line of the output says which.soup adapters merge: exit 2 is a rejected argument, since v0.78.0 inputs it cannot merge (a rank that differs on a shared module, no tensor in common, a shared tensor with two shapes) and, with--strict-verdict, a MAJOR canary verdict; exit 3 is a refused gate, not a verdict.soup attest verify: exit 3 is an invalid signature or a policy mismatch, 2 is an input that cannot be read, and 1 is a Sigstore verifier that is not available.soup env check: exit 3 is ABI drift against the lock and also an installed package outside the bounds Soup declares for itself.soup train: exit 2 can arrive after a finished run, and so can exit 3 (--diagnose-gate); a run that a gate stopped early exits 0. See the train rows below.
Release and quality gates
Ship and the eval commands
| Command | Code | Means |
|---|---|---|
soup ship | 0 | SHIP. |
soup ship | 2 | DON'T SHIP: the task leg or the forgetting leg said no, or no general benchmark was measured at all (evidence with no benchmarks and no baseline: "cannot verify the model didn't regress"). Since v0.77.2 also a local --adapter directory in which every lora_B tensor is zero and every other saved tensor (a head, a bias) is zero too, with the rule adapter_noop, decided before either model loads (#2434; the conditions). |
soup ship | 3 | A usage or input error: no mode chosen, --base without --tuned or --adapter, --noise-floor outside 2 to 10 or together with --evidence, --forgetting-threshold outside 0 to 1, an unreadable --config (one over 4 MiB or, since v0.76.1, nested deeper than 100 levels or expanding past its alias allowance), a refused --judge-model, any Click error (an unknown flag, a malformed number), and offline evidence that is missing, is not JSON, has a malformed score block or a malformed numerics stamp, carries a field this release does not recognise, or whose provenance or numerics family does not match the committed --config. Since v0.77.0 also --require-provenance without --evidence, --config and a local --adapter, or against evidence with no binding or with one that no longer matches (stale provenance: artifact, task, training source, suites or policy differs); a --data-sha256 that is not 64 lowercase hex characters; and a --chat-template-kwargs value that is not a JSON object, names a key Soup sets itself, or is given with --evidence. Since v0.77.1 also a --progress value other than jsonl (Error: --progress must be jsonl, no events). Since v0.78.0 also an --output or --emit-evidence that is the same file as an input of the run or as the other output, an adapter whose weights are a pickle (on --adapter or --base-adapter), an evidence schema_version other than 1 or 2 or a version-2 field in a file that does not say 2, offline evidence that records every task generation failed on one side, --task-mode reward without --reward-fn, and the refusals of --base-adapter and --gate loop-v2: see what v0.78.0 adds. |
soup ship | 1 | A runtime error: a model that fails to load, an unwritable --output, or a judge that cannot be reached (a runtime error, not a verdict). With --progress jsonl (v0.77.1) also an unexpected exception, which prints start and error events on stderr and no traceback and, since v0.77.2, an Error: <ExceptionType>: <message> line on stdout. Ctrl+C in that mode was 1 at v0.77.1 and is 130 since v0.77.2 (#2417, read from the call site; upstream did not exercise it by hand). 0, 2 and 3 are unchanged, and a completed DON'T SHIP still ends with an end event. Since v0.78.0 also, in a live --task-mode metric run, every task generation failing on one side (every task generation failed on the base side (8 of 8); from upstream's changelog, not run here). |
soup eval gate | 0 | Every task passed. Without --model the run uses a stub generator that returns empty text, so it checks the wiring and not a model. Since v0.77.0 a suite with no tasks is refused with exit 3 (see the 3 row); through v0.76.1 an empty suite, or one whose tasks key was misspelled, passed with exit 0. |
soup eval gate | 2 | A task scored below its threshold, a task that could not be scored (its row reads ERROR, for example an unreadable task file), or a score that fell further than --regression-threshold below the --baseline. Since v0.77.0 that includes a judge task whose prompts file is empty: it fails even at threshold 0. |
soup eval gate | 3 | Setup you can fix: a --suite that is missing, outside the working directory, fails the schema (it needs a top-level suite: name) or, since v0.76.1, is over 1 MiB, nested deeper than 100 levels or expanding past its alias allowance; a --baseline that is missing or not valid JSON, --regression-threshold outside 0.0 to 1.0, --write-baseline without --model, a judge URL the load refuses, or a Click error. Since v0.77.0 also a suite with no tasks (field 'tasks' must contain at least one task) or with a repeated task name, and a baseline score that is not a finite number (NaN, Infinity, an overflow such as 1e999, an integer too large for a float), each refused before any task runs. Through v0.76.1 a NaN or -1e999 baseline printed GATE PASSED. Since v0.77.1 also a suite that is not valid YAML or whose top level is not a mapping (Cannot load suite: eval-gate suite 'x.yaml' is not valid YAML: ..., or ... must be a mapping with 'suite' and 'tasks', got a list); through v0.77.0 those exited 1 with a bare Error: ParserError or Error: TypeError line (#1895). Since v0.78.0 also a suite whose top-level mapping has a non-string key, a boolean task threshold, a key repeated inside a task mapping, and a --write-baseline that names the suite, a custom task's file or a judge task's prompts; and a suite missing suite: or a task threshold: prints one line instead of a multi-line dump, still 3. A judge task whose prompts file has a bad row is not a suite refusal: it is an ERROR row inside a failed gate, exit 2 (run). |
soup eval gate | 1 | --model fails to load, an unwritable --write-baseline, or an unhandled error. |
soup eval against | 0 | No regression on the metric. |
soup eval against | 2 | A regression. |
soup eval against | 3 | Input you can fix: a --metric outside the allowed list, a locked --suite that is missing, outside the working directory or invalid, no recorded series for one or both runs, or a Click error. |
soup eval against | 1 | The experiment tracker cannot return a per-row series. |
soup eval quant-check | 0 | OK or MINOR on every task, or a run with --allow-stub: the table is then titled as a stub and --format json carries "stub": true, and it measures nothing. |
soup eval quant-check | 2 | At least one task is MAJOR. |
soup eval quant-check | 3 | An unknown --format; a --before, --after or --tasks path that is missing, outside the working directory or an unresolvable registry reference, or (since v0.77.0) a registry reference that matches more than one entry; a standalone .gguf file; or a Click error. Since v0.77.2 also a --tasks file that cannot be loaded (invalid JSON, a row without prompt, a name that does not end in .jsonl, no tasks, a path that cannot be opened), reported as Cannot load --tasks: ... before either model loads (#2185). |
soup eval quant-check | 1 | A model that fails to load (without --allow-stub), naming the side, or an unexpected error. (A --tasks file that is not valid JSONL exited 1 after both models had loaded at v0.76.0 to v0.77.1, although the upstream changelog said 3; since v0.77.2 the binary says 3 too, see the 3 row.) |
soup eval unlearning | 0 | Overall OK or MINOR. Without --evidence every metric is scored a neutral OK and the command exits 0. |
soup eval unlearning | 2 | Overall MAJOR, or a rejected input, including --attach-to-registry without --output. |
soup eval unlearning | 1 | --attach-to-registry failed after the report was written. |
soup eval behavior | 0 | Overall OK or MINOR. |
soup eval behavior | 2 | Overall MAJOR. |
soup eval behavior | 3 | No --evidence and no --base-model (--evidence is required when --base-model is not supplied), an invalid battery or run id, unreadable or malformed evidence, elephant or syceval together with --base-model, or a Click error. |
soup eval behavior | 1 | The live pre or post generation run failed. |
soup eval checklist | 0 | Overall OK or MINOR. |
soup eval checklist | 2 | Overall MAJOR. |
soup eval checklist | 3 | --evidence omitted (it is required), a spec that is missing, empty or invalid, evidence that cannot be read, or a Click error. |
soup diagnose | 0 | Overall OK or MINOR. The worst mode decides: a score of 0.85 or above is OK, 0.60 or above is MINOR, anything lower is MAJOR. Without --base-model or --evidence every mode is scored a neutral OK (probe not run (no evidence)) and the command exits 0, so a pipeline has to pass one of them. Also an overall NOT_RUN when --allow-not-run is given. |
soup diagnose | 2 | Overall MAJOR (it wins over NOT_RUN). Also an --evidence file that does not exist, an invalid --citation-style and a Click error. |
soup diagnose | 3 | Overall NOT_RUN: a probe you asked for produced no measurement, and --allow-not-run was not given. Also --attach-to-registry without --output and a bad --tokenizer. |
soup diagnose | 1 | An --evidence file that is not valid JSON, a live run that failed, an unwritable --output, or an --attach-to-registry that failed after the report was written. |
soup eval capability and soup eval citation return no verdict code. A failed --push PR comment on soup ship prints a warning and leaves the verdict's exit code alone.
Several of these commands score only what you give them, and a missing input is a pass. soup diagnose, soup eval unlearning, soup probe sleeper, soup probe truth and soup probe harm exit 0 when run with no evidence (and, for soup diagnose, no --base-model): they print a neutral OK report and score nothing. The three probes also write that neutral report, with 0 tokens and verdict OK, if you pass -o. Give --weights without --evidence and they exit 2 instead. A CI step that drops the flag stays green. soup eval behavior and soup eval checklist stopped scoring a missing --evidence as a pass in v0.76.0: with no evidence flag they exit 3. soup eval checklist still scores any test whose name is missing from the evidence file as a neutral OK (0 of 0), so an evidence file that omits tests, an empty {} included, exits 0 with Overall: OK.
Data
| Command | Code | Means |
|---|---|---|
soup data validate | 0 | At least one row is usable and any --min-valid-fraction is met. Issues on some rows do not fail the run when no minimum is given, and since v0.76.1 that includes the preference rows the loader now drops because they cannot be written in one layout. An empty file also exits 0 when --format is given explicitly (it reports 0/0 rows valid); without --format it exits 3. Since v0.77.0 a line that is not valid JSON or not a JSON object counts as an invalid row; before, it was skipped uncounted or, for a non-object, crashed the command with exit 1. |
soup data validate | 2 | No usable rows (since v0.77.0 that includes a file of only non-object or invalid-JSON lines with --format given), or the valid fraction is below --min-valid-fraction. |
soup data validate | 3 | A missing file, an unknown --format, a format that cannot be detected (an empty file without --format among them, and since v0.77.0 a file whose lines are all unusable: no object rows remain: N row(s) skipped), --min-valid-fraction outside 0 to 1, or a Click error. |
soup data validate | 1 | A file with an unsupported extension, or a directory. |
soup data doctor | 0 | Overall OK or MINOR. |
soup data doctor | 2 | Overall MAJOR, or data of the wrong kind: preference data (use soup data lint), or RAFT data without --show-mask. Also a Click error. |
soup data doctor | 3 | --mask-history together with --no-train-on-responses-only or --train-on-messages-with-train-field. |
soup data doctor | 1 | A missing path, an empty dataset, rows whose format cannot be detected, a tokenizer that fails to load, an unwritable --output, or (since v0.77.1) an --output that is the same file as the dataset. |
soup data lint | 0 | Overall OK or MINOR. |
soup data lint | 2 | Overall MAJOR. |
soup data lint | 1 | A format that cannot be auto-detected (soup data validate gives 3 for the same file). |
soup data lint | 3 | A missing path, an empty dataset, data that is not preference data (use soup data doctor), --sample above 50,000, an unwritable --output, since v0.77.1 an --output that is the same file as the input (--output 'x.jsonl' is the same file as the input 'x.jsonl'; choose a different --output, which v0.77.1 printed after an Error: prefix), or a Click error. |
soup data canary check | 0 | OK or MINOR. |
soup data canary check | 2 | MAJOR: a canary was memorized. |
soup data canary check | 1 | A runtime error: manifest unreadable, model load failed, or the [train] extra is missing. Since v0.77.2 also an --output that is the same file as --manifest (--output 'x' is the same file as --manifest 'x'; choose a different --output), refused before anything is read. |
soup data brain-rot | 0 | Always, unless --strict is set. |
soup data brain-rot | 3 | With --strict, the fraction of MAJOR rows exceeds --max-major-fraction. |
soup data brain-rot | 2 | A rejected input. |
soup expect | 0 | Every expectation passed. |
soup expect | 2 | At least one expectation failed. That includes expect_chosen_preferred_over_rejected_by_judge with no judge configured, unless the expectation carries args: {advisory: true}. |
soup expect | 3 | The suite or the data was rejected: a missing or invalid suite (an unknown top-level key, an unknown argument, a key beside name), a missing dataset, a path outside the working directory, a line that is not valid JSON or not an object, a file with no checkable rows, or a Click error. Since v0.78.0 also a suite or dataset that exists but cannot be opened (a permission error), which exited 1 on v0.77.2; a missing file, invalid YAML, invalid UTF-8 and a directory were already 3 (#2945). |
soup expect | 1 | The --judge endpoint could not be reached. |
soup data decontaminate, soup data toxicity, soup data langdetect, soup data pii, soup data educational and soup data score filter or score a file and write the result. They exit 0 whatever they find, with three exceptions: soup data score -b <label> without --benchmark-file exits 1 (since v0.76.0); since v0.77.0 an input with no extractable text in any row, an empty input included, or a --benchmark-file with none, exits 1 with no extractable text in N input rows (or benchmark rows) before anything is reported or written; and since v0.77.1 an --output that is the same file as --input (decontaminate: or as --benchmark-file), the default output names pii.jsonl, toxicity.jsonl, langdetect.jsonl, educational.jsonl and clean.jsonl included, exits 1 with --output 'x' is the same file as --input 'x'; choose a different --output before the data is read. Through v0.77.0 an input with one of those names was replaced. Since v0.77.0 these commands also read alpaca, preference, KTO and ShareGPT rows, which they used to scan as empty. soup data clean exits 0 whenever it ran, even when it dropped rows.
Models, adapters and supply chain
| Command | Code | Means |
|---|---|---|
soup adapters scan | 0 | OK. |
soup adapters scan | 1 | WARN, or the adapter is missing or unreadable. |
soup adapters scan | 3 | FAIL: the spectral scan found a likely backdoor pattern (rank-1 dominance, energy concentration, NaN or Inf, norm outliers). |
soup adapters scan | 2 | An unknown --format, or an adapter path outside the working directory. |
soup adapters verify | 0 | A signature record is present and the files match it. An unsigned record is a hash manifest with no signer; an ed25519 record is checked against its own embedded key unless --public-key is given; a Sigstore record needs --cert-identity and --cert-oidc-issuer. |
soup adapters verify | 1 | The signature is absent or does not match, the directory is missing, a Sigstore record is given without both --cert-identity and --cert-oidc-issuer, or the Sigstore verifier is unavailable (it is reported as a failed verification here, unlike soup attest verify). |
soup adapters verify | 3 | The same failure as 1, with --strict. Also without --strict: a directory outside the working directory or a symlink, or a signature file that cannot be parsed. |
soup adapters check-safetensors | 0 | Every weight file is safetensors. |
soup adapters check-safetensors | 1 | A pickle or PyTorch-classic weight file was found, or the directory is missing. |
soup adapters check-safetensors | 3 | The same finding as 1, with --strict (a missing directory stays 1). Also without --strict: a directory outside the working directory or a symlink. |
soup adapters sign | 0 | Signed. |
soup adapters sign | 1 | The adapter directory does not exist, or a Sigstore signing request failed at run time (the trust root, the OIDC provider, Fulcio or Rekor could not be reached, for example). |
soup adapters sign | 2 | --backend sigstore without the [sigstore] extra (nothing is signed), or with no ambient OIDC credential and without --interactive-oidc when the Sigstore trust root can be loaded (if it cannot, for example offline, the run exits 1 first: see the 1 row); an unknown --backend; or another rejected argument. |
soup adapters check | 0 | alive: the adapter moved from its initial state. |
soup adapters check | 2 | inactive, with the reason. Also a Click error. |
soup adapters check | 1 | The path is refused or missing, or the adapter's weights cannot be read. Since v0.77.0 the safetensors package is not needed, but NumPy is, and without it the error is numpy package required; pip install "soup-cli[train]" (or pip install numpy) since v0.77.1 (at v0.77.0 it read as a corrupt file: Corrupt or unreadable safetensors weights ... No module named 'numpy'); a tensor dtype NumPy cannot hold, such as FP8, is refused. A bfloat16 adapter, which exited 1 through v0.76.1, gets a verdict. |
soup adapters audit | 0 | The recorded settings agree with the config. Settings the record cannot speak to are reported unknown and still exit 0. |
soup adapters audit | 2 | At least one setting DIVERGED. A forgotten --config is a Click error, which is also 2. |
soup adapters audit | 1 | The adapter path, the config or the recorded settings cannot be read. Since v0.76.1 that includes a --config over 1 MiB, nested deeper than 100 levels or expanding past its alias allowance, which v0.76.0 read and compared. |
soup adapters merge | 0 | Merged. |
soup adapters merge | 3 | Refused by a gate: the spectral scan failed or could not run on an input (--allow-unscanned skips it), or the licenses conflict (--license-override with a reason of at least 8 characters proceeds and is recorded to the audit log). |
soup adapters merge | 2 | Rejected arguments: an unknown --strategy, fewer than two adapters, a --license count that does not match the adapters, --strategy cmaes without --eval (or with --canary or --strict-verdict), or a --canary file that cannot be read or scored. Also a MAJOR canary verdict when --strict-verdict is given: the merged adapter is already written when it exits. And, since v0.77.0, a --strategy cmaes search in which every evaluation failed: nothing is written (through v0.76.1 it wrote a merge of the first sampled weighting and reported it converged). Since v0.78.0 also inputs it cannot merge: a LoRA rank that differs on a shared module, no tensor name in common, or a shared tensor with two shapes, --strategy cmaes included; nothing is written (through v0.77.2 it wrote an adapter with no weights and exited 0, #2926). |
soup adapters merge | 1 | Since v0.77.1, NumPy or the safetensors package is missing: numpy and safetensors packages required; pip install "soup-cli[train]" (or pip install safetensors numpy), checked before any adapter is read. Since v0.77.2 also an --output that is one of the input adapters (Output path refused: --output 'x' is the same file as adapter 'x'; choose a different --output, #2349, #2379). |
soup adapters arithmetic | 1 | The same two packages are missing (v0.77.1); the message is the same as for merge. Since v0.77.2 also an --output that is a referenced adapter, refused before any adapter is read (#2274, #2349, #2379). |
soup adapters bisect | 0 | ALL_OK: the endpoints both pass. |
soup adapters bisect | 2 | BROKEN_AT, or rejected arguments. The --eval-command is judged by its own exit code: 0 passes a checkpoint and anything else fails it. A mistake in that command's own arguments (a usage error is 3 for soup eval gate) reads as a failed checkpoint, so run the command once by hand first. Since v0.77.1 it is also a fixed --eval-command program that is not in an absolute PATH directory: that is refused here, before the first probe, with --eval-command program 'x' was not found on PATH; install it, put it on PATH, or give its path and no verdict, so exit 2 alone does not tell BROKEN_AT from a rejected command: read the first line. Through v0.77.0 that case ran one probe and reported BROKEN_AT on the first checkpoint. Since v0.77.2 the refusal, with no verdict, also covers a program that {ckpt} completes (resolved for each probe, so it can come after earlier probes ran) and a program given with a path that is not an executable file (--eval-command program './x' could not start: ...); through v0.77.1 a completed name that was not found counted as a failed probe. |
soup probe sleeper | 2 | A MAJOR verdict, or a rejected input. The same holds for soup probe truth and soup probe harm. |
soup probe interference | 2 | The worst pair scores 0.20 or more in absolute value, or a rejected input. |
soup shrink | 0 | SHIP: perplexity regression within --tolerance. Also --plan-only. |
soup shrink | 2 | DON'T SHIP: perplexity regression above --tolerance. |
soup shrink | 1 | Anything else that fails once the command is running: an unreadable --calib, a model that fails to load, a bad value that Click does not check. A mistyped flag or a missing required option is Click's 2, the same number as DON'T SHIP. |
soup draft measure | 0 | The report was produced. Without --min-acceptance no acceptance rate fails the run. |
soup draft measure | 2 | --min-acceptance was given and the measured acceptance rate is below it. Also a Click error. |
soup draft measure | 1 | An error: unreadable prompts, a bad --sweep-k, a model pair that fails to load, or a target that generated no tokens. |
soup reward synth | 0 | A verifier was emitted, or --plan-only printed the plan. |
soup reward synth | 2 | Refused: the verifier could not tell the references from perturbed negatives. Nothing is left at -o. |
soup reward synth | 1 | Usage or runtime error: no -o, an existing output without --force, an output that is not a .py file. |
soup reward stress | 0 | Robust: the junk completions stayed under --max-gameable. |
soup reward stress | 2 | Gameable. Also a Click error. |
soup reward stress | 1 | Usage or runtime error: the target was not found or could not be loaded, a bad --attacks list. |
soup recipes verify | 0 | Every config that could be checked attaches. That can mean everything was unverified, so read the report. |
soup recipes verify | 2 | A config cannot attach. |
soup recipes verify | 3 | A --config that is missing, unparseable or refused by the loader (Config does not parse: ..., a staged field included), or a Click error. |
soup recipes verify | 1 | torch, transformers or peft is missing or fails to import, or a crash. Since v0.77.0 the --config is read first, so a missing or unparseable config exits 3 even on an install without torch (through v0.76.1 this 1 hid it). |
Provenance, locks and plans
| Command | Code | Means |
|---|---|---|
soup attest emit | 0 | Statement printed or written. --sign sigstore signs it with a Sigstore bundle; without the extra it does not fall back to an unsigned statement, see the 2 row. |
soup attest emit | 2 | Rejected input: an invalid stage or SHA, a signing failure (ed25519 without a usable key), --sign sigstore without the [sigstore] extra, without --output, with --key, or with no ambient credential and no --interactive-oidc when the Sigstore trust root can be loaded (nothing is written; if the trust root cannot be loaded, for example offline, the run exits 1 instead), a write failure, or --attach-to-registry without --output. |
soup attest emit | 1 | --attach-to-registry failed after the statement was written, or a Sigstore signing request failed at run time, which includes a trust root that cannot be loaded (for example offline) even when no credential is present either. An explicit signing request never degrades to an unsigned statement. |
soup attest verify | 0 | The signature is valid: an ed25519 signature against the key embedded in the sidecar, or a Sigstore bundle against the --cert-identity and --cert-oidc-issuer you supplied. That shows the statement matches its sidecar, not who signed it: for ed25519 pass --public-key trusted.pem to require a specific signer. The subject digest is asserted by the signer and is not re-checked against an artifact. |
soup attest verify | 3 | Invalid: a tampered statement or wrong key, an unsigned sidecar, no public key to verify with, an embedded key that does not match --public-key, a Sigstore sidecar with no bundle or without both --cert-identity and --cert-oidc-issuer. |
soup attest verify | 2 | An input that cannot be read, a path outside the working directory, one of --cert-identity and --cert-oidc-issuer without the other, or --public-key with a Sigstore sidecar. |
soup attest verify | 1 | The Sigstore verifier is unavailable (for example its trust root cannot be fetched), which is reported as an outage and not as tampering. |
soup bom emit | 0 | BOM printed or written. |
soup bom emit | 2 | Rejected input: an unsupported --format, a SHA that is not 64 hex characters, a bad --energy file, or --attach-to-registry without --output. |
soup bom emit | 1 | --attach-to-registry failed after the BOM was written. |
soup lock check | 0 | The closure matches. |
soup lock check | 2 | Drift. |
soup lock check | 3 | The lock file is missing or unreadable, a SHA is malformed, or a Click error. Since v0.78.0 also a directory in place of the file (lock path is a directory: ...; it was a PermissionError crash, exit 1, on Windows). |
soup lock show | 3 | The lock file is missing. A mistyped flag is Click's 2. Since v0.78.0 a directory in place of the file is 3 as well (lock path is a directory: ...; exit 1 through v0.77.2). |
soup lock write | 2 | Unchanged from v0.75.2: no --env-hash and no soup-env.lock, a SHA that is not 64 hex characters, or a Click error. |
soup env check | 0 | No drift and every installed package is inside the bounds Soup declares. |
soup env check | 3 | ABI drift against the lock, or a package outside the declared bounds. The bounds check needs no lock file and comes first. |
soup env check | 1 | There is no lock file. |
soup env check | 2 | The lock file is unreadable. |
soup plan | 0 | The plan was written to soup.tfstate. Since v0.77.1 soup plan runs the same load checks as soup train, so a config that soup train refuses is exit 1 and no plan (at v0.77.0 it wrote one and exited 0; see the 1 row). |
soup plan | 1 | The config file was not found, or is refused the way soup train refuses it (since v0.77.1, #2242): an unknown key (unknown config key 'x' - refused.), a staged field set away from its default, a refused deprecated value or any other schema error, printed as Config validation error: and one line per setting. No plan is written and an existing state file is not changed. |
soup plan | 2 | The config or the state was rejected: for example a config that is not a mapping or, since v0.76.1, one over 1 MiB, nested deeper than 100 levels or expanding past its alias allowance (config 'soup.yaml' is larger than 1048576 bytes; refusing; through v0.76.0 such a file was read and the plan written). Since v0.77.2 also a YAML syntax error or a config that cannot be read, each reported in one line that names the config (Config 'soup.yaml' is not valid YAML: ... (line 2, column 7), Config 'soup.yaml' cannot be read: ...); through v0.77.1 those exited 1 with the parser's or the operating system's own message. |
soup apply | 0 | No drift. --dry-run stops there. Without it the state is marked applied and the soup train command is printed: soup apply does not start training. |
soup apply | 3 | The config has drifted from the plan. |
soup apply | 1 | The config or the state file does not exist, or the config is refused the way soup train refuses it (since v0.77.1: an unknown key, a staged field, a refused deprecated value or any other schema error, --dry-run included). The state file is left as it was and is not marked applied. |
soup apply | 2 | The config or the state was rejected: for example a config that is not a mapping or, since v0.76.1, one over a YAML read limit. Since v0.77.2 also a YAML syntax error or a config that cannot be read, reported in one line that names the config (through v0.77.1 those exited 1). --minillm-on-policy is accepted by both commands since v0.77.2. |
soup license-advisor | 0 | Advice printed. A warn-level risk does not fail the run. |
soup license-advisor | 3 | --license was given and its risk for the --target is at block severity. |
soup license-advisor | 2 | An unknown --target. |
soup drift-alarm | 0 | The KL divergence is within --threshold. A row without a string output, response or text field is skipped, so check n_reference and n_live in the output. |
soup drift-alarm | 3 | Drift. Any Slack or Discord webhook is posted first. |
soup drift-alarm | 1 | A missing file, a path outside the working directory, or, since v0.77.0, a --reference or --live file with no readable row (has 0 readable rows: ... Nothing was compared, so no drift verdict is possible.). Through v0.76.1 that last case printed an OK panel with KL 0 and exited 0. Since v0.78.0 also a side whose rows exist but hold no tokens (... has 1 readable row but no tokens. Nothing was compared, so no drift verdict is possible.), which printed OK with KL 0 and exited 0 through v0.77.2 (#2955). |
soup drift-alarm | 2 | A --threshold that is not above zero or is above 100, or a webhook URL that is refused. |
soup apple-adapter | 0 | Converted, or --plan-only. |
soup apple-adapter | 2 | Rejected input or a missing source. |
soup apple-adapter | 3 | The conversion is gated upstream and was refused. |
soup apple-adapter | 1 | A dependency is missing. |
Commands that start something
| Command | Code | Means |
|---|---|---|
soup train | 0 | Training finished and the adapter is saved. Also --dry-run on a valid config, and --cloud without --cloud-submit (it prints a plan and starts nothing). A run stopped early by --gate or training.eval_gate with on_regression: stop, by the loss watchdog, or by a reward-hack or echo-trap halt also exits 0: soup train does not turn the stop into a failure, so check the run's output and metrics. |
soup train | 1 | The config is missing, unreadable, not valid YAML, over a read limit (since v0.76.1: larger than 1 MiB, nested deeper than 100 levels or expanding past its alias allowance, each reported in one line naming the file) or invalid (an unknown key is refused, and since v0.77.0 so is a staged field set away from its default, data.remove_unused_columns: true or a quantization the task never applies: Config validation error: with one line per setting), the hardware-fit gate refuses the run (--allow-oom-attempt overrides it), a --gate suite cannot be loaded, a --replay file does not exist, a data.chat_template that is not registered or a dataset that loads zero training rows (both also under --dry-run; since v0.76.1 the second includes a preference dataset none of whose rows can be written in one layout, and since v0.77.0 a preference task whose rows lack prompt, chosen or rejected, which v0.76.1 passed under --dry-run), a --gate suite with no tasks or a repeated task name (since v0.77.0), a reward_fn: judge_rank run that could make more judge calls than training.grpo_judge_max_calls and a Sesame CSM dataset row that is not one clip with one transcript message (both since v0.77.0, checked after the data loads and before the model does, --dry-run included), --find-lr that cannot run its sweep (no report is written), --resume or --hf-resume together with relora_steps or on task: unlearn or, since v0.78.0, task: ppo (refused before the data is read, --dry-run included, #2772), --push-public without --push-as (since v0.78.0), a --diagnose-gate evidence file that cannot be read, --gpus above 1 given with --no-reexec and without --dry-run (it prints the launch command and stops), or training raised an error. The SFT trainer also refuses to save the final model, and exits 1, when the last logged loss or the trainable weights are non-finite. Since v0.78.0 a config is also invalid for a boolean in a numeric training or training.lora field, a wrong-typed training value, training.eval_gate.baseline: previous, an enabled gate on backend: mlx or task: prm, unlearn or moe_lora_routing, training.vllm_sleep_mode: true on task: grpo and a training.eval_gate.suite that cannot be loaded (see Settings Soup refuses at load); a training.eval_gate.baseline that cannot be resolved stops training once the model has loaded. |
soup train | 2 | A flag override that fails validation (the --replay options), an invalid tracker option, an invalid --cloud or --gpu (--cloud takes modal, lambda or, since v0.77.0, runpod), a refused cloud plan (Cloud plan failed: ..., for example, since v0.78.0, a config that names a local input, or a RunPod plan whose output directory already exists, or, since v0.77.1, a Lambda plan whose output directory has a path part that starts with -), since v0.78.0 --push-as or --gate beside --cloud (--push-as is not supported with --cloud: ..., without the Cloud plan failed: prefix), or --diagnose-gate finding a MAJOR mode. The last one fires after training: the adapter is already saved and the run is recorded as finished. A mistyped flag is also 2. |
soup train | 3 | --diagnose-gate finding a mode that did not run (NOT_RUN). Like the MAJOR exit it fires after training, with the adapter already saved. Also, since v0.77.0, a standard input that is a pipe or a redirected file when neither --yes nor --dry-run is given (stdin is not a TTY, so the start-training confirmation cannot be answered. Pass --yes to start without the confirmation.), after the config is read and before the data or the model loads (a config refused at load still exits 1 first): piping y into the prompt no longer answers it, so pass --yes in scripts. |
soup train | the submit call's code | With --cloud modal, --cloud lambda or --cloud runpod plus --cloud-submit, the return code of the submit call (1 when the backend's submit is unavailable, for example RUNPOD_API_KEY or RUNPOD_SSH_PRIVATE_KEY unset). The RunPod controller has not been run against a real RunPod account yet (tracked as #2098). Since v0.78.0 the RunPod controller uses RunPod's REST API v2 (upstream says RunPod announced the retirement of REST API v1 for 2026-11-15, #2877), which rests on the published API reference and not on a real account. |
soup train | the launcher's code | With --gpus above 1 the run is handed to accelerate launch. On Linux and macOS os.execvp replaces Soup's own process, so the code is the launcher's rather than Soup's. On Windows, since v0.76.0, Soup runs the launcher as a child process, waits for it and exits with its code; Ctrl+C waits for the launcher's own shutdown. Before v0.76.0 the Windows call returned at once with exit 0 while the launcher kept running, and the launcher's code never reached the caller. A launcher that cannot be started exits 1. |
soup quickstart | the child's code | Exits with the code of the soup train subprocess it starts, or 0 with --dry-run. |
soup llama | the child's code | Exits with the code of the llama.cpp program it runs, 2 when that program is not found on PATH, 1 when it cannot be launched. |
soup serve | 2 | A refused option: a non-loopback --host without --tool-auth-token, since v0.77.2 a --tool-auth-token or SOUP_TOOL_AUTH_TOKEN value outside the token rule on any bind, loopback included (16 to 128 characters from A-Z, a-z, 0-9, _ and -; a trailing newline is refused; the message names the rule and never the value; v0.77.1 accepted any non-empty value), --bank, --mole or --steer with a backend other than transformers or a path outside the working directory, --kv-cache-type fp8, any --kv-cache-type with --backend vllm, sglang or mii (Use --backend transformers.), --kv-cache-type q8_0 with no quantized-cache backend installed, --auto-quant (it is unavailable), an unknown --device, or an unusable --hub. Since v0.78.0 also --max-tokens outside 1 to 16384 and --trace-log, --record-thumbs or --feedback-log with --backend vllm, sglang or mii (before any model loads, but after the check that the [serve] extra is installed, which a core install fails first with exit 1), and --adapter-dir with a backend other than transformers, with --mole, or naming a path that is not a directory or is a filesystem root (before that check too, so a core install answers 2 as well): see what v0.78.0 adds. |
soup serve | 1 | Other startup failures, for example a model path that does not exist. |
soup mcp serve | 2 | A refused option combination: an unknown --transport, --host, --port or --auth-token with stdio, --allow-execute with sse or http, or an invalid --auth-token. |
soup mcp serve | 1 | The mcp SDK is not installed, or is too old for the transport. |
soup mcp runs reconcile | 2 | --expunge-launching was not passed, and no run record changed. |
soup mcp runs reconcile | 1 | A launching run whose process is still alive was refused. |
soup ui | 2 | Reserved: a non-loopback host (--public or --host) without a valid auth token. The command's own options cannot reach it, because a token is generated at startup and an invalid --auth-token exits 1 first, so soup ui --host 0.0.0.0 starts the server. |
soup ui | 1 | An invalid --auth-token, or the [ui] extra is not installed. |
soup can run | 1 | Without --yes it prints the confirmation panel, exits 1 and runs nothing. Also an unreadable can. Since v0.78.0 also a can that holds <redacted> values, refused only once --yes is given and before anything is extracted, fetched or trained (this can holds redacted value(s) at: ...; #2883). |
soup can run | the child's code | With --yes, the exit code of the training subprocess is passed through, then that of the deploy step when --deploy is set. |
soup doctor | 0 | Advisory findings (a missing optional extra, a torchvision mismatch, a core dependency below its floor, which since v0.76.1 includes a Typer below 0.15.4 wherever soup doctor can start: below Typer 0.12.4 no command can, and 0.12.4 to 0.15.3 start only next to Click 8.1 or older) and settings your config writes that its backend never reads do not change the exit code. |
soup doctor | 1 | A required core dependency is missing or incompatible, or an installed package, in an extra group included, is beyond the ceiling Soup declares for it. A missing or outdated [train] member stays advisory. |
soup doctor | 2 | --config named a file that is missing, unparseable, over a YAML read limit (since v0.76.1) or fails the schema (every load refusal, an unknown key among them). It is reported after the system, GPU and dependency tables and any environment issues. Since v0.77.0 a config that cannot be loaded stops the command before All checks passed! is printed (#2106); through v0.76.1 that line could print above the refusal. Since v0.77.0 the refusals include a staged field set away from its default and a deprecated value, which v0.76 only warned about. Unknown keys are reported together, but a cross-field refusal is reported one at a time (the first it hits), so fix it and run the command again until it prints no error. |
Other commands that return a verdict code
| Command | Code | Means |
|---|---|---|
soup ab | 0 | Any decision, reject_h0 with direction worse included. Read direction to gate on a regression. |
soup ab | 1 | An input file that is not found or is outside the working directory, or a column whose values overflow. |
soup ab | 2 | An invalid metric, alpha, effect size or webhook URL, --beta given at all (refused since v0.77.0, before the input is read), or a Click error. |
soup rewind | 0 | The analysis was printed, including a run with no spikes. |
soup rewind | 1 | No such run, no log, an empty or unreadable log, a --step that was not recorded, or a failed JSON write. |
soup rewind | 2 | An invalid --json path, or a Click error. |
soup bench train | 0 | A valid report. |
soup bench train | 1 | The model was not training (no trainable parameters, a zero or non-finite gradient norm, parameters unchanged, fewer steps than asked): the report is written but marked not valid. Also a config that is missing, invalid or not supported by the bench, --steps not above --warmup, or a report that cannot be written. |
soup bench train | 2 | An --output path that is refused (checked before the config is read), or a Click error. |
soup plugins install | 2 | Always: Soup does not install plugins. Install a trusted distribution that exposes the soup_cli.plugins entry-point group, then run soup plugins enable <name>. |
Commands with no verdict code of their own
Besides Click's exit 2 for a bad invocation, these exit only 0 or 1 (four exceptions: since v0.77.0 soup recipes use on a recipe the catalog marks unavailable exits 3, since v0.77.2 soup distill-prompt with no prompt in any trace exits 3, and since v0.78.0 soup runs curriculum-curve on a malformed history file and soup data mix --apply on YAML that does not parse or a weight that is not a number exit 2): the exit-code lines that v0.76.0 added to them are all 1, and none of them adds another code.
soup init,soup cost,soup card,soup diff,soup history,soup why,soup profile(1 also, since v0.77.1, whenbatch_size,max_lengthor the model's size is outside the range the estimate supports:Cannot estimate this run: ...; since v0.77.2 alsosoup profile --modelwhen noconfig.jsoncan be read for the model, with the reason),soup sweep(1 also when every arm failed, or when a--paramsets a staged field away from its default, since v0.77.0),soup autopilot,soup ci init(1 also, since v0.78.0, for--pythonwith a leading zero in either part,03.11or3.011)soup advise(run, explain and compare),soup recipes(other thanverify; since v0.77.0soup recipes useexits 3 for a recipe the catalog marks unavailable,kimi-k2-sftandkimi-k2-thinking-grpo, and 1 for an unknown name),soup registry(since v0.77.0 aname:tagthat matches more than one entry exits 1 and lists the candidate ids;push --dataand--idempotency-keyrefusals are 1 too, and since v0.77.2 so ispush --run-idon a run whose status is notcompleted:run X has status 'failed'; only a completed run can be registered)soup eval benchmark(1 when nothing was measured or saved),soup eval custom,soup eval judge(1 also when the judge stops answering: three requests in a row with their retries spent end the run, print the judge URL and how many items were not judged, and the scores from before the outage are shown but not saved),soup eval auto(1 also when every leg was skipped, since v0.77.0),soup eval compareandsoup eval leaderboard(1 also, since v0.77.2, when a stored score isNaNor infinite, #2477),soup eval human,soup eval aidersoup datainspect, convert, merge, dedup, filter, stats, sample, split, search, preview, from-traces, review, ingest, preprocess, generate, topics and clean. Since v0.76.0soup data from-traces --judgeagainst a judge that is down exits 1 and writes nothing (it used to write an empty file and exit 0), and a provider-backedsoup data forge,augmentordistill-promptrun in which every call failed exits 1 and writes nothing. Since v0.77.0soup data clean,inspectandstatsskip a row that is not a JSON object with a warning instead of exiting 1, andsoup ingestexits 1 when--outputis the--logsfile or a hard link to it. Since v0.77.1convert,merge,dedup,filter,sample,topicsandfrom-tracesalso refuse an--outputthat is the same file as an input, before the data is read, with exit 1; the other eleven of upstream's eighteen are in the Data table and in what v0.77.1 adds. Since v0.77.2soup data evolveandgen-magpieexit 1 and keep the output when no row was produced and a provider call failed,soup data filter --perplexityand--score-onlyexit 1 without torch or transformers,soup data dedup --field <key>exits 1 when the key is absent from every row, andingest,persona-mix,generate,canary check,mix --optimize,forgeandrecipe --executerefuse an--outputthat names one of their inputs, all with exit 1 (what v0.77.2 adds);soup distill-promptexits 3 when no trace row has a prompt and leaves the output as it was. Since v0.78.0soup data sampleexits 1 for-nbelow 1 and for--pctthat is not a number in (0, 100] (--n must be a positive integer.,--pct must be a finite number in (0, 100].), where it exited 0 and wrote an empty file, one row or every row, andsoup data generateasks the provider again for rows it did not deliver until--countis reached and, after three requests in a row that bring no valid row, saves what it has and exits 1 (upstream's changelog, #2418; not run here, it needs a provider).soup runsshow, compare, replay and delete, and, since v0.78.0,curriculum-curve(2 for a malformed--historyfile, as the table above says)soup canpack, inspect, verify, publish and forksoup deploy ollama,soup deploy hf-space(1 also, since v0.78.0, for a--hardwarevalue that is not a lowercase slug of at most 32 characters, and for--hardware zero-a10gwith thestreamlit-chattemplate),soup agent synth,soup agent eval
soup eval custom never exits non-zero on a low score, so it cannot act as a checkpoint test for soup adapters bisect on its own. Several validating commands outside the tables above, among them soup export, soup merge, soup quantize, soup push, soup infer, soup fetch, soup ingest, soup tunability, soup spectrum scan, soup compile, soup data recipe, soup deploy autopilot and soup loop, use 1 or 2 for a rejected argument and have no verdict. For example soup export exits 1 for an unknown or removed --format (--format awq and gptq since v0.77.0) and 2 for a removed flag such as --bits, soup quantize --to awq or gptq exits 2, and soup loop watch --detach checks --poll-interval and --max-iterations in the parent since v0.77.0 and exits 2 without spawning a child; since v0.77.2 soup loop status exits 2 with loop state invalid: when the stored monthly spend is not a finite, non-negative number, and soup quantize --to with a value outside gguf, onnx and tensorrt exits 2 whatever the value looks like. Since v0.77.1 soup infer --backend takes transformers or mlx (anything else is Click's 2), and --backend mlx with a --task other than text, a --batch-size above 1, --cuda-graphs, any --device, a --hub other than hf, a negative or non-finite --temperature, or without the [mlx] extra exits 1 with a message naming that option.
Soup is free and Apache-2.0. If it saved you a training run, starring the repo costs nothing and helps most. You can also fund the GPU time behind the work a 4 GB laptop cannot reach.