Experiment Tracking

Every soup train run is automatically tracked in a local SQLite database (~/.soup/experiments.db).

List Runs

soup runs

Shows all training runs with task, model, status, and final loss. The run database at ~/.soup/experiments.db is machine-wide, so this lists every run on the box; add --cwd-only to narrow it to runs whose output_dir sits under the current directory.

Run Details

soup runs show run_20260223_143052_a1b2

Detailed info including config, metrics, and an ASCII loss curve. A failed run also shows an Error: line with the reason (below). When the curve shows a spike, soup rewind names the dataset rows that were in that step's batch.

A run that is not running no longer says it is

Until v0.73.3 a run whose watcher process died was reported running forever, and the listing hardcoded that word for any row it could not otherwise classify. That trained you to ignore the one status field that guards against starting a second concurrent run.

Both soup runs and soup runs show now reconcile on read: a running row whose recorded process is gone is rewritten to terminated with an unknown exit code. Unknown, deliberately — a lost outcome is never recorded as a success — and the richer completed and failed statuses are left exactly as they are. Only runs recorded with a process id can be checked this way, so a run without one is left alone rather than guessed at.

Since v0.76.0 soup train records its own process id, which is what makes that check reach a run you killed yourself. Through v0.75.2 only runs launched by the MCP server recorded one, so a soup train run killed with SIGKILL or Ctrl+C had nothing to check and stayed running; now the next soup runs marks it terminated (#764). Rows written by earlier versions still have no process id and are still left alone.

A failure anywhere after the run is registered (tokenizer or model load, quantization, the LoRA attach, dataset mapping, the rest of the command) now marks the row failed and stores the exception type and message as the run's error_message, which soup runs show prints as Error:. The message is redacted for secrets and capped at 2,000 characters. Separately, a training log that arrives before the run's first measured loss now stores no loss instead of a fabricated 0.0, and the live panel shows Loss: - until one is measured (#1225).

On Windows this depended on a second fix, and it is a good example of a bug that cannot be reasoned around: the API that reports a process's exit code returns 259 for a process that is still alive, but 259 is also a perfectly legal exit code, so a child that genuinely exited with 259 was indistinguishable from one still running. Liveness is now decided by waiting on the process handle with a zero timeout, which is signalled the instant the process exits whatever code it exits with.

Compare Runs

soup runs compare run_1 run_2

Side-by-side comparison of two runs with loss curves and metrics.

Delete Runs

soup runs delete run_1

Reclaim disk without losing the run

Deleting a run is the blunt option. Checkpoints are what actually fill the disk, and most of that weight is optimizer state you will never load again.

soup runs clean run_1 --dry-run        # estimate the saving first, delete nothing
soup runs clean run_1                  # surgical: drop optimizer states, keep the weights
soup runs clean run_1 --no-keep-weights # the one way to delete whole checkpoints
soup runs clean --all                  # every historical run

If you typed --keep-weights on Click 8.1, it did the opposite and deleted whole checkpoints. The option was a bare flag with a True default, and Click 8.1 treats that as a toggle, so passing it explicitly parsed as False. Anyone who read the flag name, typed it to be safe, and had Click 8.1 installed lost their non-best checkpoints. v0.75.1 makes it the paired --keep-weights/--no-keep-weights, so the default and the explicit flag now mean the same thing on every supported Click version, and --no-keep-weights is the one way to delete. Upstream added a repo-wide check that fails any True-default flag without a negative form, so this shape cannot come back.

--keep-weights defaults to on, which is the whole design: it deletes optimizer.pt from the lesser checkpoints and leaves the model weights, so you can still evaluate or export from any of them. --dry-run estimates the saving without touching anything, and --force skips the confirmation for CI.

This matters more than it sounds at scale: a LoRA run training a fraction of a percent of the model can drag a checkpoint hundreds of times the size of the adapter it produced.

Which checkpoint is left whole. clean leaves untouched the checkpoint with the lowest loss recorded at its own step, compared only among the checkpoints. When no checkpoint has a recorded loss (a run shorter than logging_steps records none), it keeps the latest one and says so: no checkpoint has a recorded loss; keeping the latest, checkpoint-N, whole. It only touches runs whose output directory is under the current directory.

Before v0.76.0 the rule required the checkpoint to match the run's overall lowest loss exactly. Whenever that loss was logged between two checkpoints, or the run was shorter than logging_steps, no checkpoint matched, so soup runs clean treated every checkpoint of the run as expendable: it stripped the optimizer state from all of them, and with --no-keep-weights it deleted all of them (#1225). If you ran it on such a run, check what you still have.

Model Evaluation

Soup includes a comprehensive evaluation platform (v0.19.0+):

pip install "soup-cli[eval]"

# Run benchmarks (mmlu, gsm8k, hellaswag, etc.)
soup eval benchmark --model ./output --benchmarks mmlu,gsm8k

# Custom eval tasks from JSONL
soup eval custom --model ./output --tasks ./eval_tasks.jsonl

# LLM-as-a-judge over saved outputs: --target is a JSONL file of
# {"prompt": ..., "response": ...} rows (soup infer writes this shape)
soup eval judge --target ./results.jsonl --model gpt-4o --provider openai

# Run the benchmarks and custom tasks listed in the eval block of soup.yaml
soup eval auto --config soup.yaml

# Compare eval results between runs
soup eval compare run_1 run_2

# Local leaderboard across models
soup eval leaderboard

# Human A/B evaluation with Elo ratings
soup eval human --model-a ./model_v1 --model-b ./model_v2 --input ./prompts.jsonl

On soup eval judge, --model names the judge model, and --target is the file of pairs to score, not the model under test. Every row needs prompt and response, or the command exits 1 naming the line.

soup eval auto takes its work from an eval block in the config, and --benchmarks and --tasks override the two lists:

eval:
  auto_eval: true                 # also run both lists after soup train (v0.76.0)
  benchmarks: [mmlu, gsm8k]       # lm-evaluation-harness task names
  custom_tasks: ./eval_tasks.jsonl

soup eval auto needs the output directory to exist, so run it after soup train, and it exits 1 when neither list is set.

eval.auto_eval: true runs these two lists from soup train itself, since v0.76.0. Once the model is saved, the chief process prints Running auto-eval..., runs eval.benchmarks and then eval.custom_tasks against the saved output, and records the scores under the run. A benchmark or custom-task failure prints a yellow notice and does not fail an otherwise successful run. Through v0.75.2 no trainer handed the eval block to the training command, so auto_eval: true did nothing and a finished run never started an evaluation by itself (#1487); if you relied on it, run soup eval auto against that output now.

soup train --trust-remote-code is passed on to those evaluations. There is no soup.yaml key for it, and without the flag both stay off. soup eval auto --trust-remote-code now reaches its benchmark and custom-eval steps as well (through v0.75.2 it was accepted and nothing read it), and soup eval benchmark and soup eval custom take their own --trust-remote-code, off by default. The block's judge key is accepted by the schema, but nothing in v0.76.0 reads it: judge evaluation is not part of auto-evaluation, so run soup eval judge yourself.

What the evaluation commands do with what they find

soup eval benchmark saves only the requested names that came back with a numeric metric (a real 0.0 counts; a boolean does not). A name missing from the results, or without a numeric result, is skipped and named (Skipped <name>: not found or no numeric result), and the banner reports the saved count: Results saved to experiment tracker: N of M. When nothing was saved it prints Nothing measured; nothing saved to experiment tracker. and exits 1. Through v0.75.2 such a name was saved as 0.0.

soup eval compare, soup eval leaderboard (per model and benchmark) and soup registry diff use the newest eval row for each benchmark, so re-running a benchmark replaces its score. Through v0.75.2 each of them let the oldest row win. A comparison that read a re-measured benchmark could show an improvement where there was a regression; re-run it, nothing stored changed.

soup eval judge skips a row whose judge call keeps failing (a 429 after its retries, a timeout, a reply with no choices) with a warning, counts it, and still scores and saves the rest; it exits 1 only when every row fails. Once three requests in a row have spent their retries unanswered, it stops: one message names the judge URL and how many items were not judged, the scores from before the outage are shown but not saved, and the exit is 1 (#1522). The judge is asked for one JSON object, {"scores": {"<criterion>": <number>}, "reasoning": "..."}, and Soup reads the first object in the reply that has a scores key. Criterion names match case-insensitively and {"score": N} reads as N. A reply with no object, no finite number for a rubric criterion, or one criterion under two spellings is an error that names the criterion, not a score at the scale minimum, and a reply over 65,536 characters is refused unread. This paragraph is read from the code and upstream's documentation, not run against a live judge.

Custom task scorers

Each row of the soup eval custom --tasks file carries a prompt and optionally expected, scoring and category. scoring takes one of exact, contains, answer, regex, semantic, tool_call_match, tool_call_name_match or tool_call_args_subset. The three string scorers read an output like this:

answer refuses at load an expected that states no single answer (a multi-line worked solution with no ####, an empty \boxed{}, a hedge between values) and names the line. It reads the answer a model states: against a gold Paris, Answer: Paris and Paris match but The capital of France is Paris. does not, so prefer it for math and short-answer tasks and use contains for free text. contains cannot read negation (It is not 4, so 5. still contains 4), and its bound is an ASCII alphanumeric run, so 4 still matches inside 3.4. The eval gate's scorer: override accepts answer too.

contains changed in v0.76.0. It used to be a raw substring test and paid 14 for an expected 4. A result recorded with it (a soup eval gate --baseline file, a registry:// row, a soup eval custom --run-id result) can be higher than the same outputs score now. A baseline recorded before v0.76.0 does print the generic scorer-revision (or unknown-provenance) warning, but that warning does not say that contains changed, so re-measure those baselines or compare only runs made on the same version. The scorer functions were run for the table above.

Aider Polyglot (soup eval aider, v0.74.0)

pip install "soup-cli[aider]"
git clone https://github.com/Aider-AI/aider.git
cd aider
./benchmark/docker_build.sh
cd ..
git clone https://github.com/Aider-AI/polyglot-benchmark.git

soup eval aider --model openai/gpt-4.1 --output ./aider-results \
  --exercises-dir ./polyglot-benchmark --run-id run_20260301_143052_a1b2

This scores a model through Aider's own benchmark harness, in Aider's Docker image. --model is an Aider / LiteLLM model identifier such as openai/gpt-4.1, not a path to a local checkpoint. The [aider] extra alone does not make the command runnable, because the aider-chat wheel does not ship the harness: you build the aider-benchmark image from an Aider checkout and clone the exercises once, as above. Docker has to be running, and Soup checks the Docker CLI, the daemon and the image before it starts. The exercises are mounted read-only (a directory outside the current one is allowed, with a warning), and provider credentials reach the container by environment-variable name, never as a value on the command line: OPENAI_API_KEY, OPENAI_API_BASE, ANTHROPIC_API_KEY, GEMINI_API_KEY, OPENROUTER_API_KEY and the three AZURE_API_* variables, whichever are set.

--output is a directory, not a report file, and it must sit under the current directory. It receives Aider's per-exercise results and a soup_result.json with the model, the score, the error counts and aggregate details. --run-id must name an existing run; the aider_polyglot score is stored under it, which is what lets soup eval compare run_before run_after compare two of them. --threads (1 to 64, default 1), --num-tests (default -1, every exercise) and --timeout (60 to 86400 seconds, default 86400) bound the run. --allow-host-services lets the container reach services on the host, for a trusted local OpenAI-compatible endpoint; it also lets the model-written code the benchmark executes reach those services, so leave it off for a remote provider. The benchmark runs model-generated code, and Soup offers no host-execution fallback.

Hyperparameter Sweep

Search for the best hyperparameters:

# Grid search
soup sweep --config soup.yaml --param lr=1e-5,2e-5,5e-5 --param lora_r=8,16,32

# Random search with max runs
soup sweep --config soup.yaml --param lr=1e-5,2e-5,5e-5 --strategy random --max-runs 5

# Preview without running
soup sweep --config soup.yaml --param lr=1e-5,2e-5 --dry-run

# Early stopping: skip remaining runs if loss exceeds 1.5x best
soup sweep --config soup.yaml --param lr=1e-5,2e-5,5e-5 --early-stop 1.5

# A standalone sweep file instead of repeated --param
soup sweep --config soup.yaml --sweep-config sweep.yaml --dry-run

A parameter that is not a field of the config exits 1 (sweep parameter does not match any config field), and --dry-run runs that check, so a typo fails before any training.

A standalone sweep file

--config stays the base soup.yaml. --sweep-config points at a separate file that holds the whole search, in place of repeating --param:

# sweep.yaml
strategy: random      # grid | random
n_runs: 6             # 0 or absent: use --max-runs (a random sweep with neither plans at most 10 runs; a grid plans every combination)
seed: 7
params:
  lr: [1e-5, 2e-5, 5e-5]
  lora_r: [8, 16, 32]

params maps each name to a non-empty list of scalars (at most 32 names, 64 values per name, a 256 KiB file, under the current directory). Giving both --param and --sweep-config is an error (Use either --param or --sweep-config, not both., exit 1), and so is giving neither. The file's strategy and seed win over --strategy, and a non-zero n_runs wins over --max-runs. Two consequences were run: a file that leaves strategy out is a grid sweep even when you pass --strategy random, because the file's default wins; and n_runs only caps the count, so the 3 by 3 grid above with strategy: grid and n_runs: 20 plans 9 runs.

What a sweep arm is, and what it records

Every arm builds the trainer that soup train would build for the config's task and backend (v0.76.0). Through v0.75.2 soup sweep had its own shorter dispatch that knew SFT and twelve other tasks, so ten task values (distill, tts, classifier, reranker, cross_encoder, prm, unlearn, moe_lora_routing, online_dpo, asr) reached the SFT trainer, and backend: mlx was never read. Each of those arms trained plain next-token cross-entropy while the tracker recorded the configured task, and the sweep ranked the numbers as if they were the configured objective. If you swept any of them, discard that sweep (#1213).

A diverged arm (a NaN or infinite final loss) stays in the results table, is labelled diverged, sorts after the finite losses, can never be marked the best run, and counts as worse than any finite early-stop threshold. An arm whose trainer cannot be built or set up is now marked failed in the tracker with the reason, where it used to stay running (#1464).

Model Comparison

Compare outputs of two models side-by-side:

soup diff --model-a ./model_v1 --model-b ./model_v2 --prompt "Explain gravity"
soup diff --model-a ./base --model-b ./finetuned --prompts test_prompts.jsonl
soup diff --model-a ./a --model-b ./b --prompts prompts.txt --output results.jsonl

Both models must be existing local paths (exit 1 otherwise). For an adapter directory, soup diff reads the base model from its adapter_config.json; --base-a and --base-b name the base for model A and model B explicitly, and are required when that file records none (the command exits 1 with Cannot detect base model). A full (non-adapter) model is loaded from its own directory; the base is not used to load it. Custom modeling code is denied by default: a local model whose config.json has an auto_map is refused unless you pass --trust-remote-code.

Batch Inference

Run a model on a list of prompts:

soup infer --model ./output --input prompts.jsonl --output results.jsonl
soup infer --model ./output --input prompts.txt --output results.jsonl \
  --max-tokens 512 --temperature 0.3

Output is JSONL with prompt, response, and tokens_generated fields.

Two opt-in options arrived in v0.76.0, both described on the soup bench page. --batch-size N (default 1) generates N prompts together, left-padded, and writes the results in input order; in fp16 and bf16 a batched run may not produce the same tokens as batch size 1. --cuda-graphs is experimental CUDA graph decoding for a resident, unquantized Qwen2 or Llama model and needs PyTorch 2.14 or newer.

soup infer, soup chat and soup diff take --device (cpu, cuda, cuda:<index> or mps; mlx is also accepted and loads like cpu), and when you leave it out Soup auto-detects one, so what follows applies either way. Since v0.76.0 the device decides how the model loads: cpu pins it to the CPU in float32, an explicit cuda:<index> or mps pins it to that device in float16, and plain cuda lets accelerate place it in float16. Any other value is refused (Unsupported --device ...). Through v0.75.2 these commands loaded with device_map="auto" in float16 whatever you passed, so --device cpu could still put the model on a visible GPU. The cost to know about: on a CPU-only host the model now loads in float32, which doubles its RAM use (a 7B model needs about 28 GB instead of 14 GB).

Training Profiler (v0.23.0+)

Estimate memory, speed, and GPU requirements before training:

soup profile --config soup.yaml            # model, task and quantization come from the config

Shows estimated GPU memory, training speed, and hardware recommendations. Add --json to print the same figures as one JSON object (memory components, total_memory_gb, tokens_per_sec, samples_per_sec, recommended_batch_size, compatible_gpus, gpu_memory_gb, gpu_memory_source) for scripting.

gpu_memory_source is flag, detected or assumed. With no GPU detected and no --gpu, the memory verdict is made against an assumed 24 GB and says so: the panel reads No GPU detected: would fit in an assumed 24 GB (need ~3 GB). Pass --gpu for a real verdict., the speed panel reads Typical A100 throughput, not specific to your GPU., and the JSON ends "gpu_memory_gb": 24.0, "gpu_memory_source": "assumed". Pass --gpu with a name such as RTX5070LAPTOP to get a verdict for a real card ("gpu_memory_source": "flag"); an unknown name lists the accepted ones and exits 1. (soup profile --help still gives only rtx3090, rtx4090, a100, h100 as examples.)

Adapter Management (v0.22.0+)

# Scan directory for LoRA adapters
soup adapters list ./experiments

# Show adapter metadata (base model, rank, size)
soup adapters info ./output

# Compare two adapters side-by-side
soup adapters compare ./adapter_v1 ./adapter_v2

# Is an adapter trained at all? (v0.76.0)
soup adapters check ./output

soup adapters check ends in one verdict line, alive or inactive: <reason>, and soup adapters audit compares the settings a finished run recorded with the soup.yaml that started it: both are on their own page.

Logging Integrations

TensorBoard

soup train --config soup.yaml --tensorboard
tensorboard --logdir ./output/runs/

Weights & Biases

pip install "soup-cli[wandb]"
soup train --config soup.yaml --wandb

The wandb package comes from its own extra: neither [train] nor [all] installs it.

--tensorboard and --wandb cannot be used together.