Benchmarks you can trust: the soup bench group

soup bench became a command group in v0.76.0, with two subcommands: soup bench infer, the inference benchmark that was always soup bench, and soup bench train, which is new. The same release added soup infer --batch-size and an experimental --cuda-graphs flag on soup infer and soup bench infer.

A section marked from code, not run describes a path that needs PyTorch and a CUDA or CPU run. The audit that checked this page had no PyTorch, so it ran every refusal and routing case and read the success paths from the source.

soup bench train

soup bench train --config soup.yaml --steps 20 --warmup 3 -o bench-train.json

It trains for exactly --steps optimizer steps in a scratch directory, saves nothing, never writes into the config's output, and writes a JSON report. It exists so that a throughput number comes with evidence that the model was training while it was measured.

Scope. task: sft on the transformers backend, and nothing else. Any other task or backend is refused by name, for example Cannot bench this config: task: dpo -- bench train measures the SFT trainer only or backend: mlx -- bench train measures the transformers trainer only.

The checks, and why the report can say "not valid"

A step's time runs from the previous step's end to its own, with CUDA synchronised at both boundaries. Throughput is counted in supervised tokens (labels not equal to -100), not padded ones, and the report keeps the allocated and reserved memory peaks separate (null off CUDA, never read from nvidia-smi). Besides the timing it records the device, card, driver, the SM clock sampled during the counted steps, library versions, dtype, optimizer, seeds, and the sha256 and text of the resolved config.

The report is written even when it is not valid, with one exception: a model with no trainable parameter is refused before the first step, with the trainable_parameters message and exit 1, and no report. valid is false when any of these checks fails (the first is asked before the first step, which is why that case has no report):

A failed check prints FAILED <check>: <message> and The report is written but NOT valid: this run was not training., and exits 1. A good run ends with Valid: the model trained while it was measured. and exit 0. parameters_changed proves that something moved, not that the model learned anything useful.

Exit codes

Some refusals arrive only after a slow import: --warmup that is not below --steps is reported once PyTorch has loaded. That is a known limitation upstream, tracked as #1817.

What the reference run found

Upstream published a run on a real card as benchmarks/gate-836-bench-train-contract.md. Its verdict is the contract holds; the number does not repeat. Across thirteen runs of a byte-identical config, the config hash, the token counts, both memory peaks and all three checks were identical to the byte, and the reported throughput ranged from 376.4 to 915.3 supervised tokens per second, a 2.43x spread. The spread was additive rather than proportional, roughly 0.17 to 0.28 s per step across shapes whose step time differs 26-fold, which is the size of difference a comparison of two soup bench train numbers has to clear.

Read that as a statement about the command, not about the card. The machine was a laptop RTX 5070 on battery, power-limited to 50 W of a possible 115 W; the record is explicit that it is not a throughput claim and that a run on mains would be a new baseline. Treat a single soup bench train throughput as one sample from a noisy distribution, and use the checks to know whether the sample means anything.

soup bench infer

soup bench infer ./output
soup bench ./output          # the same command: anything that is not a subcommand routes to infer
soup bench --p50 ./output    # so does an option given first

The inference benchmark is unchanged. soup bench with no arguments prints the group help and exits 2.

It imports PyTorch first, so on a core install it exits 1 with the [train] hint. The output is a Benchmarking Configuration panel, one line per prompt, and an Inference Benchmark Results table (backend, average tokens per second, total latency, peak VRAM or N/A). One warm-up run is discarded. From code, not run.

soup infer --batch-size

soup infer --model ./output --input prompts.jsonl --output results.jsonl --batch-size 8

--batch-size N generates N causal-LM prompts together, left-padded, and writes the results in input order. The default is 1, which keeps the old behaviour; batching is opt-in. In fp16 and bf16 a batched run may not produce the same tokens as batch size 1. The flag is ignored with --task asr. --batch-size 0 is rejected (0 is not in the range x>=1, exit 2). Generation itself: from code, not run.

--cuda-graphs (experimental)

soup infer --model ./output --input prompts.jsonl --output results.jsonl --cuda-graphs
soup bench infer ./output --cuda-graphs --backend transformers

--cuda-graphs replays CUDA graphs for decoding instead of launching every kernel from Python. It needs PyTorch 2.14 or newer and applies to a resident, unquantized, eval-mode, stock Qwen2 or Llama model on one CUDA device, optionally under one unmerged plain LoRA adapter. It is not available in soup chat or soup serve.

Before any output it refuses, with a message ending Omit --cuda-graphs:

• --batch-size above 1 (graphs record a batch of 1; this is a usage error, exit 2, and so is --task asr);
• PyTorch older than 2.14, quantization, CPU or disk offload, layer streaming, flash attention, DoRA or any other LoRA variant, a merged adapter, several adapters, beam search, assisted generation, a non-CUDA device or more than one GPU. These arrive after the model loads and exit 1.

On soup bench infer the flag additionally needs the transformers backend (exit 2 otherwise). Soup runs one greedy warm-up on the longest prompt first, so a run over a single prompt can take longer overall; the summary's Duration and Throughput start after the warm-up. The refusals were run; the decoding path is from code, not run.

What upstream measured

On one RTX 5070 Laptop, unpinned, Qwen2.5-1.5B FP16, the slowest run with graphs decoded 2.9x faster than the fastest run without. Under an unmerged LoRA adapter the same bound was 2.8x. The record is benchmarks/gate-v0.76.0-cuda-graph-decode.md, and it matters how that sentence is built:

• The criterion was pre-declared as a per-run bound (the slowest graph run at least 1.20x the fastest baseline run), not a pooled mean, and every criterion passed. The figure is a worst case, not an average.
• Unpinned, the baseline was the unstable arm: 13.5 to 20.7 tokens per second between runs, against 60.1 to 62.7 with graphs. The record says the unpinned absolute numbers must not be quoted as what a user gets, because the gate ran as a background process rather than from a focused window.
• Token IDs were identical on every request, and reserved VRAM rose by 40 to 42 MiB.
• End to end through soup infer over 24 prompts of varied length, with model load and compile included, the runs with the flag took 96 to 102 s against 131 to 282 s without, with identical responses.

It did not measure soup chat or soup serve, architectures other than Qwen2, Linux or macOS, quantized or offloaded models, sampling (--temperature above 0), long --max-tokens (everything is 128 tokens), or a batch above 1. The flag stays opt-in and experimental.

Related

• Serving: soup serve, which has no --cuda-graphs flag.
• Training speed and memory: what to change when a benchmark says the run is slow.
• Measurement records: how Soup publishes its records, including the ones that disagree with themselves.
• Exit codes