Experiment Tracking

Every soup train run is automatically tracked in a local SQLite database (~/.soup/experiments.db).

List Runs

soup runs

Shows all training runs with task, model, status, and final loss.

Run Details

soup runs show run_20260223_143052_a1b2

Detailed info including config, metrics, and an ASCII loss curve.

A run that is not running no longer says it is

Until v0.73.3 a run whose watcher process died was reported running forever, and the listing hardcoded that word for any row it could not otherwise classify. That trained you to ignore the one status field that guards against starting a second concurrent run.

Both soup runs and soup runs show now reconcile on read: a running row whose recorded process is gone is rewritten to terminated with an unknown exit code. Unknown, deliberately — a lost outcome is never recorded as a success — and the richer completed and failed statuses are left exactly as they are. Only runs recorded with a process id can be checked this way, so a run without one is left alone rather than guessed at.

On Windows this depended on a second fix, and it is a good example of a bug that cannot be reasoned around: the API that reports a process's exit code returns 259 for a process that is still alive, but 259 is also a perfectly legal exit code, so a child that genuinely exited with 259 was indistinguishable from one still running. Liveness is now decided by waiting on the process handle with a zero timeout, which is signalled the instant the process exits whatever code it exits with.

Compare Runs

soup runs compare run_1 run_2

Side-by-side comparison of two runs with loss curves and metrics.

Delete Runs

soup runs delete run_1

Reclaim disk without losing the run

Deleting a run is the blunt option. Checkpoints are what actually fill the disk, and most of that weight is optimizer state you will never load again.

soup runs clean run_1 --dry-run     # estimate the saving first, delete nothing
soup runs clean run_1               # surgical: drop optimizer states, keep the weights
soup runs clean --all               # every historical run

--keep-weights defaults to on, which is the whole design: it deletes optimizer.pt from the lesser checkpoints and leaves the model weights, so you can still evaluate or export from any of them. --dry-run estimates the saving without touching anything, and --force skips the confirmation for CI.

This matters more than it sounds at scale: a LoRA run training a fraction of a percent of the model can drag a checkpoint hundreds of times the size of the adapter it produced.

Model Evaluation

Soup includes a comprehensive evaluation platform (v0.19.0+):

pip install "soup-cli[eval]"

# Run benchmarks (mmlu, gsm8k, hellaswag, etc.)
soup eval benchmark --model ./output --benchmarks mmlu,gsm8k

# Custom eval tasks from JSONL
soup eval custom --model ./output --tasks ./eval_tasks.jsonl

# LLM-as-a-judge evaluation
soup eval judge --target ./output --model gpt-4o --provider openai

# Auto-eval from soup.yaml config
soup eval auto --config soup.yaml

# Compare eval results between runs
soup eval compare run_1 run_2

# Local leaderboard across models
soup eval leaderboard

# Human A/B evaluation with Elo ratings
soup eval human --model-a ./model_v1 --model-b ./model_v2 --input ./prompts.jsonl

Hyperparameter Sweep

Search for the best hyperparameters:

# Grid search
soup sweep --config soup.yaml --param lr=1e-5,2e-5,5e-5 --param lora_r=8,16,32

# Random search with max runs
soup sweep --config soup.yaml --param lr=1e-5,2e-5,5e-5 --strategy random --max-runs 5

# Preview without running
soup sweep --config soup.yaml --param lr=1e-5,2e-5 --dry-run

# Early stopping: skip remaining runs if loss exceeds 1.5x best
soup sweep --config soup.yaml --param lr=1e-5,2e-5,5e-5 --early-stop 1.5

Model Comparison

Compare outputs of two models side-by-side:

soup diff --model-a ./model_v1 --model-b ./model_v2 --prompt "Explain gravity"
soup diff --model-a ./base --model-b ./finetuned --prompts test_prompts.jsonl
soup diff --model-a ./a --model-b ./b --prompts prompts.txt --output results.jsonl

Batch Inference

Run a model on a list of prompts:

soup infer --model ./output --input prompts.jsonl --output results.jsonl
soup infer --model ./output --input prompts.txt --output results.jsonl \
  --max-tokens 512 --temperature 0.3

Output is JSONL with prompt, response, and tokens_generated fields.

Training Profiler (v0.23.0+)

Estimate memory, speed, and GPU requirements before training:

soup profile --config soup.yaml            # model, task and quantization come from the config
soup profile --config soup.yaml

Shows estimated GPU memory, training speed, and hardware recommendations.

Adapter Management (v0.22.0+)

# Scan directory for LoRA adapters
soup adapters list ./experiments

# Show adapter metadata (base model, rank, size)
soup adapters info ./output

# Compare two adapters side-by-side
soup adapters compare ./adapter_v1 ./adapter_v2

Logging Integrations

TensorBoard

soup train --config soup.yaml --tensorboard
tensorboard --logdir ./output/runs/

Weights & Biases

soup train --config soup.yaml --wandb

--tensorboard and --wandb cannot be used together.