soup eval design — derive evals from your training data

Before v0.55 you wrote your eval suite by hand. Now Soup drafts one from your data.

soup eval design

soup eval design data.jsonl --goal "polite customer support chat" --num-dimensions 5

How it works:

1. TF-IDF salience picks up to num_dimensions (default 5) salient terms over the dataset. 10,000-row DoS-capped subsample.
2. Goal-keyword dispatch maps each dimension to a scorer: json / code / math → rlvr (verifiable reward), classify → exact_match, extract → regex, and everything else → judge.
3. Output: a frozen EvalDesign (JSON) with one EvalDimension per row.

Scorer allowlist: {exact_match, regex, judge, rlvr}.

soup eval discover — canaries

soup eval discover data.jsonl --num-clusters 5 --per-cluster 3

Three sets:

• Held-out canaries — prompts from each cluster of your own data, found by greedy farthest-first Jaccard-distance clustering (_CLUSTER_SUBSAMPLE = 10_000).
• Adjacent-skill probes — rows from the rarest clusters of your own data, used as forgetting probes. They are rarer themes inside your data, not neighbors from outside it.
• Memorization probes — 25%-prefix truncation. If the trained model can continue the rest of a training row from its prefix, it memorized.

Per-group cap: _MAX_CANARIES_PER_GROUP = 1024. The canary set is written to evals/canaries.json by default (--output changes it).

Every canary is drawn from the file you pass in. "Held-out" therefore means held out of the evaluation, not out of training: if you want them held out of training too, remove those rows from the training file yourself.

soup eval lock — pin the suite

soup eval lock my-design.json

Locks the design as a SHA-256-checksummed eval_suite artifact via canonicalise_design_bytes (canonical-JSON for stable hashes across runs), written to evals/locked.json unless you pass -o. It is not registered anywhere by default: pass --attach-to-registry <id-or-name> to attach the frozen LockedSuite (path / sha256 / dimension_count) to a v0.26 registry entry. If that attach fails the command prints a warning and still exits 0.

soup eval coverage — gap analysis

soup eval coverage my-design.json --task reasoning

Checks the locked design against the v0.54.0 TASK_CATEGORIES taxonomy and the _RECOMMENDED_SCORERS allowlist (e.g. reasoning → (rlvr, judge), format_conversion → (regex, rlvr)). Returns a CoverageReport with concrete gap recommendations. --task is required and is one of factual_lookup, style_shaping, format_conversion, reasoning, tool_use, summarization or classification; anything else exits 2.

soup eval gate-install — git regression gate

soup eval gate-install --baseline run-id-7f3a

Writes .git/hooks/pre-push (atomic, POSIX 0o755). --suite names the locked suite (default evals/locked.json), --hook-path writes somewhere else, and --force overwrites an existing hook; without it an existing file is refused, and so is a symlink at the hook path. The hook is a thin wrapper around soup eval against, and three things about it are easy to miss:

1. It does nothing unless SOUP_CANDIDATE_RUN_ID is set. With the variable unset the hook prints SOUP_CANDIDATE_RUN_ID not set; skipping regression gate. and exits 0, so the push goes through ungated. Set the variable in the environment that runs git push (a CI job, for instance) before you rely on the hook.
2. It does not run an eval. With the variable set it runs soup eval against <baseline> --candidate $SOUP_CANDIDATE_RUN_ID --suite <suite> --json-only, which compares metric series that are already recorded in the experiment tracker for the two runs. The locked suite is only checked to exist and parse; a missing or unparseable suite, or a run with no recorded series, blocks the push. A regression makes the hook exit 2 and print [soup] pre-push gate blocked: regression vs <baseline>; any other non-zero exit of soup eval against (a usage error, 3, or a runtime error, 1) is passed through and printed as [soup] pre-push gate error: configuration or usage error (exit N). Before v0.76.0 the hook turned every non-zero exit into 1.
3. It checks one metric, task_accuracy. soup eval against takes one --metric per call and the hook passes none, so every other metric is checked only if you call soup eval against --metric ... yourself. task_accuracy reads that metric's own recorded series and, when a run has none, the custom result that soup eval custom --run-id writes. The verdict is a paired-bootstrap interval (paired_bootstrap_ci, n_samples bounded to [100, 100_000]) compared against the built-in GateThresholds defaults, and decide_regression is direction-aware through _METRIC_DIRECTION: higher-is-better for every metric except p95_latency_ms, which is lower-is-better. A run that recorded one aggregate result instead of a per-row series is compared by its point delta: --json-only then carries "ci_degenerate": true and null bounds, and the panel prints confidence interval unavailable (one aggregate result).

The hook script is rendered via render_pre_push_hook with shlex.quote for every interpolated path — no shell injection.

soup eval against — the check the hook runs

The hook above is a wrapper. This is the command inside it, and you can run it by hand any time you want to compare two runs without pushing anything.

soup eval against <baseline-run-id> --candidate <candidate-run-id>

It reads the per-row metric series for both runs from the experiment tracker, runs the same decide_regression on the paired delta, and exits 0 when there is no regression and 2 when there is, so it drops into any CI step, not only a git hook. A bad --metric, a --suite that is missing, outside the working directory or invalid, or a run with no recorded series exits 3, and 1 is left for a tracker that cannot return a per-row series. Before v0.76.0 a regression was exit 1, so a script that tests for 1 now passes a regression: see Exit codes.

The --suite behaviour is the part worth knowing: the generated pre-push hook passes the locked suite through, so a suite that has gone missing fails the gate instead of quietly turning it off. An unset SOUP_CANDIDATE_RUN_ID still skips the whole check, suite included; see the hook section above.

See also

• Quant-check — same idea, but for quant-induced regression
• Eval-gated training — halt training when quality drops
• Registry — where eval_suite and canaries artifacts live
• Eval depth (v0.65) — 4 deeper optional probes (behavior, capability, checklist, irt-subset) that stack on top of v0.55 dimensions
• Post-train x-rays (v0.66) — 4 mechanistic-interpretability probes (sae-diff, sleeper, interference, pack); failure-mode coverage 7 → 11