Question 1

What's new in the latest Soup CLI release?

Accepted Answer

v0.76.0, tagged 12 October 2026, has no codename: upstream titled it with a sentence, settings Soup accepted and never applied now stop the load. 490 pull requests merged between 12 September 2026, the day v0.75.0 shipped, and this tag, and 418 of them came from 80 people outside the maintainer (412 of 483, from the same 80, if you count only the ones merged after the v0.75.0 release pull request); the window also holds v0.75.1 and v0.75.2, which were cut off v0.75.0. What they found is the release: settings that Soup validated, documented and accepted, and then read nowhere on the path they claimed to configure, so a soup.yaml could ask for one thing and quietly train another. This release refuses the settings that never took effect, warns about the rest and names the release that will refuse them, evaluates the validation split Soup always held out and never used, and applies the grpo_beta KL penalty to every GRPO variant. A config that loaded on v0.75 can stop loading, so run soup doctor --config soup.yaml after upgrading and before you train: it loads the config under the new rules and reports what is refused or warned about, with what to change (refusals that fail on a single field are listed together, but a cross-field refusal is reported one at a time, so a corrected config can need a second run), exiting 2 when the config cannot load and 0 when it loads, warnings included. Its table of settings known to be unread covers only a few task, backend and modality combinations (upstream tracks that as #1815), so a clean report is not proof that every key in the file is read. A config that must load on both versions needs the refused key removed, not renamed, and deleting the key is the fix; but for training.use_longlora, training.ebft_variant and training.quantization_aware: true a run that finished on v0.75 did not do what its config said, so re-run it (the re-run table is on the settings-refused-at-load page). Refused at load: training.quantization_aware: true (it never ran quantization-aware training), training.use_longlora, training.ebft_variant, every non-null training.gdpo_variant, training.rope_scaling_type on a task whose setup does not read it (only sft, tts and pretrain do), training.freeze_layers and freeze_ratio outside the text sft and tts path on the transformers backend, training.auto_mixed_precision outside sft and tts, backend: unsloth on seven further tasks that have no unsloth setup, quantization: bitnet_1.58 for training, a Sesame CSM text-to-speech config, a relora_steps config that does not also set quantization: none, and the optimizer names transformers 5 rejects after the model has loaded (adam_mini, adamw_apex_fused, ao_adamw_fp8, ao_adamw_4bit, ao_adamw_8bit, badam, came_pytorch and dion everywhere; muon and adamw_hf off the MLX backend). Warned about at load now, and refused from v0.77 (upstream's stated deadline): 29 staged fields that nothing reads, among them training.lr_groups, early_stop_patience, citation_recall_threshold, forgetting_eval_steps and the checkpoint_eval tunables (a field left at its default stays silent), plus three deprecated values: data.remove_unused_columns: true, an explicit training.quantization of 4bit or 8bit on a task whose trainer never quantizes, and soup ab --beta. soup export --format awq and --format gptq (and soup quantize --to awq or gptq) are deprecated too and removal is tracked for v0.77, because both upstream projects are archived and neither extra installs next to [train]. Upstream says plainly that the goal is not fully met: its guard test still lists 36 declared fields with no consumer, 29 of them the staged fields and two more, training.warmup_auto and training.forgetting_threshold, plainly unwired and silent at load. Behaviour that changes even when the config still loads: data.val_split is now evaluated at the end of each epoch, or every N steps with the new training.eval_steps, and the validation loss is recorded, so runs take a few percent longer; grpo, ppo and online_dpo evaluate by generating, so by default they ignore val_split with a note and train on every row; gspo, dapo, dr_grpo, bnpo, two_sided and rft now apply the grpo_beta KL penalty they used to validate and drop, so existing variant configs will not reproduce prior runs, and grpo_beta: 0 runs KL-free and skips the reference forward pass; training.loraplus_lr_ratio now takes effect on every transformers-backend trainer that builds a LoRA adapter (it was silently ignored on all but sft, pretrain and embedding) and is refused at load where it cannot apply (unlearn, prm, moe_lora_routing, backend mlx, lora.use_vera); an FP8 request that cannot run stops the run instead of training in bf16; and eleven gate and verdict commands share one exit-code taxonomy, 0 pass, 2 gate failure, regression or drift, 3 usage or input error, a mistyped flag included. soup eval gate and soup eval against now fail with 2 where they used to fail with 1, soup lock check drift and soup expect failure are 2 where they were 3, and soup data validate input errors are 3 where they were 1, so a CI step that treats only exit 1 as 'gate failed' no longer catches a failing gate. New: soup adapters check (alive or inactive: a near-zero weight delta, dead layers, an all-zero lora_B) and soup adapters audit, soup bench train (trains a fixed number of steps in a scratch directory and exits 1 when the model was not training; soup bench is now a group and soup bench infer is the old inference benchmark), soup data clean (sanitizes and repairs a dataset without touching the input file), soup recipes verify (checks on the meta device, with no weights downloaded, that a recipe can attach a LoRA adapter), soup rewind (ranks the dataset rows behind a loss spike), multi-node launch flags on soup train, opt-in batched soup infer --batch-size, an experimental soup infer --cuda-graphs that needs PyTorch 2.14 or newer, Sigstore keyless signing for adapters and attestations behind the [sigstore] extra (verification takes an out-of-band certificate identity and OIDC issuer), and a soup ab that runs a two-sided, variance-robust test, says whether a difference is better or worse, and retires --beta (accept_h0 can need more rows than before). Layer streaming stays opt-in and BETA. Its disk tier now reads on a background thread, through direct I/O, and can stripe across NVMe drives; on a synthetic 70B-shaped NF4 store of 36 GB with random weights, larger than RAM, on one RTX 5070 Laptop 8 GB box, a cold step went from 110.5 s on the synchronous reader to 16.2 to 17.3 s, and from 17.4 s to 9.97 s across two drives, while a sustained-load probe measured 1.28x and 1.41x in two rounds and reached no formal verdict; a store that fits in RAM is still faster on the RAM tier, which stream_source: auto picks. Nothing was measured on a real 70B checkpoint. The streamed CUDA NF4 forward now follows bitsandbytes' own fused-versus-dequantized dispatch, the same kernels as resident NF4 (the backward still dequantises); no benchmark record shows whether that closes the Blackwell inexactness. A post-repair row for the 8B NF4 laptop configuration is now recorded on a different RTX 3050 Laptop at a different clock; throughput figures are comparable only at the same clock, so no ratio is given against the 119.6 tok/s figure, which was measured on v0.72.2. The preprint is unchanged and still scoped to v0.73.0; the release lists three of its statements as no longer describing the project (the repaired-code re-run, the always-dequantising forward and the unmeasured disk tier). No new security advisory is published with this release; the fixes for GHSA-63h4-gvp4-r26g and GHSA-9f49-hqpf-c849 are carried in 0.76.0, so a user still on 0.75.0 or 0.75.1 should upgrade, and 0.76.x is the supported line. Install: Python 3.10 to 3.12; click is now declared and bounded below 8.5, and an environment that already holds typer 0.9.0 to 0.12.3 fails every command with 'Type not yet supported: int | None' (0.12.4 to 0.15.3 start only with click 8.1.x) until you run pip install -U "typer>=0.15.4,<0.21". Before it, v0.75.2, tagged 1 October 2026, was a one-change security patch on top of v0.75.1 and, like it, was written by the maintainer alone; an outside contributor, @abdugafforov-bobur, is credited for reporting the defect. Outbound endpoint checks now refuse a private, link-local or reserved IP literal over https as well as http: soup data generate --api-base, the vLLM provider and the commands that use it, judge URLs in soup eval judge, eval-gate suites, soup ship --judge-model and eval.ship.judge_model, training.online_dpo_judge (now checked when soup.yaml loads, not only when the trainer starts) and the Web UI chat proxy. Loopback still works, so a server on your network has to be addressed by its hostname, over https. http://0.0.0.0 is no longer treated as local by soup data generate --api-base or the chat proxy, 100.64.0.0/10 and fec0::/10 count as non-public wherever private addresses were already refused (webhooks, the OTLP endpoint, telemetry, hub endpoints over HTTP and soup ingest --pull), and an address written with non-ASCII digits or label separators is classified as the address a client folds it to. soup ship now checks --judge-model before it builds the base and tuned models instead of loading both first, and the chat proxy answers 400 rather than 500 for a URL that does not parse. It is the fix for security advisory GHSA-9f49-hqpf-c849, rated medium (CVSS 4.1) and affecting 0.3.0 through 0.75.1. Nothing needs re-running and no credential needs rotating on its account; until you upgrade, upstream's advice is not to run an untrusted soup.yaml, eval-gate suite or .can package that sets judge_model, online_dpo_judge or eval.ship.judge_model, and to keep soup ui bound to loopback with its token private. Its known limits are upstream's own: hostnames are not resolved, so a hostname that resolves to a private address is still accepted; HF_ENDPOINT and the other hub endpoint variables refuse private literals over plain HTTP only; a private TLS endpoint addressed by an IP literal has no opt-in, so use a hostname; and loopback spellings differ between settings. The diff touches the endpoint validators, soup ship's judge check, the online-DPO judge field and the Web UI chat proxy and nothing in the trainer, so layer streaming and every measured number are unchanged. Nothing was measured: there is no gate record, and the preprint is unchanged. Test files go from 500 to 501. Before it, v0.75.1, tagged 21 September 2026, was a security and correctness backport rather than a feature release, and the first release in four that nobody outside the maintainer wrote. Its tree was the v0.75.0 tag plus 25 cherry-picks, one test adaptation and the release commits; the development branch was 186 commits ahead when it was cut, so releasing from there would have shipped an entire minor under a patch number. It is the fix for security advisory GHSA-63h4-gvp4-r26g, rated high, covering eight issues and affecting 0.7.0 through 0.75.0, found by a full-repository review and verified by the maintainer. Three of them mean a credential should be rotated, and the conditions are upstream's own rather than reworded: ran soup push with hub modelscope or modelers on 0.53.10 to 0.75.0 while logged in to Hugging Face or with HF_TOKEN set, rotate the Hugging Face token, because the token chain ran for every hub and the resolved token was handed to the third-party SDK; had a judge URL on 0.33.0 to 0.75.0 pointing anywhere other than api.openai.com while OPENAI_API_KEY was set, rotate that key, because any https host was classified as the OpenAI provider; passed --tool-auth-token, --auth-token or --api-key on 0.71.3 to 0.75.0 and collect the audit log anywhere, rotate those and purge it, because masking keyed on whether a value looked like hf_ or sk- or Bearer rather than on the option name. The other five: soup adapters verify --public-key consulted the trusted key only inside the ed25519 branch, so an unsigned adapter verified with exit 0 even under --strict, which is the one failure a signature check exists to prevent; the Web UI inserted dataset field names, dataset values and stored run configuration into the page as HTML and sent no Content-Security-Policy, so inspecting somebody else's dataset could run script in the origin that holds your bearer token and can start training; soup serve tool routes and the adapter activate and deactivate routes checked neither Host nor Origin, so a page the operator merely visited could reach them through DNS rebinding, and the tool token was compared with an ordinary equality test rather than in constant time; soup mcp serve --allow-execute digested only some of the config paths a plan reads, two of the names in that list not being schema fields at all, so a reward model, PRM, teacher model, reward-function file or unlearning set could be swapped between approval and execution, while an input outside the working directory was silently left unpinned and the one-active-execution check read only the fifty newest runs; and reading an untrusted can archive could exhaust memory through YAML alias expansion counted after the size cap rather than before it. Four fixes mean something already finished was lost, and none of them raised. A layer-streamed LoRA adapter saved on peft 0.21 or newer is a 40-byte file holding zero tensors, because peft 0.21 selects an adapter's tensors by the prefixes it reads off named_modules while 0.20 filtered by the lora_ substring, and the streaming wrapper's module and parameter names carried .inner. while its state_dict keys were canonical, so trainer.save_model, every save_steps checkpoint and get_peft_model_state_dict returned nothing. That asymmetry had been documented as a cosmetic known limitation for three releases; peft 0.21 made it fatal. It cannot be recovered, and Soup does not pin peft below 0.21, so the remedy is to re-train on v0.75.1 or newer, or hold peft back yourself. Typing soup runs clean --keep-weights on Click 8.1 parsed as false, because the option was a bare flag with a true default and Click 8.1 treats that as a toggle, so it deleted whole non-best checkpoints; it is now a paired --keep-weights/--no-keep-weights. soup train --cloud modal ran in an ephemeral container and the checkpoints died with it while the entrypoint printed a line telling you to download them; outputs now live on a named volume, are committed in a finally so a crashed run keeps what it produced, and are downloaded at the end including after a failure. And Ctrl+C during a Lambda submit killed the controller while it was still terminating the paid instance, so the instance could keep billing. Nothing on the streamed forward path changed, so no measured number moves. There is no gate record, which upstream states rather than leaving implicit, and the preprint is unchanged. Test files go from 475 to 500. Before it, v0.75.0, tagged 12 September 2026, had no codename either: upstream titled it with a sentence, MLX honours its config, unknown config keys refuse the load. All 60 pull requests merged into it came from outside the maintainer, by 22 people, and what they found is the release. A config option can be validated by the schema, documented in the reference, accepted by the loader, and then read by nothing on the backend you chose. On backend: mlx, six were. data.train_on_responses_only defaults to true and reached nothing, so every MLX supervised run trained on system and user turns while the adapter metadata recorded mask_prompt false regardless. training.optimizer, scheduler, warmup_ratio and weight_decay were dropped, so every run built a bare AdamW at a constant learning rate whatever the recipe said, and all 32 names on Soup's allowlist became AdamW. training.max_grad_norm reached nothing, so the documented default that clips every transformers run at 1.0 clipped nowhere here, and the backend's own ignores advisory did not even mention it. gradient_accumulation_steps and gradient_checkpointing took mlx-lm's dataclass defaults rather than the configured values, and the written adapter_config.json hardcoded both, so the drift was not detectable from the output afterwards either. The same soup.yaml therefore trained a different recipe on Apple Silicon than on a CUDA box, silently, and this is the fourth release running whose headline is a flag that validated and reached nothing. All six are honoured now, and a value MLX cannot express is refused by name rather than silently becoming AdamW: eight of the 32 optimizer names have an MLX equivalent and the other 24 are refused, four schedulers are supported, and a non-zero weight decay on an optimizer whose MLX constructor takes none is refused rather than dropped. Schedules are composed in optimizer-update units, because that is what MLX drives a callable learning rate from, and measured on Apple Silicon a cosine run with warmup_ratio 0.2 now reports 10 distinct learning rates across 40 iterations where the old path reported one. Response-only masking is measured rather than asserted: over 16 two-turn conversations, 772 supervised tokens unmasked, 71 with mlx-lm's own flag, and 146 with a correct mask, because upstream's flag masks a single prefix and supervises only the last assistant turn. Soup injects its own per-token mask instead, and the two backends are still not comparable, with MLX the stricter of the two: it excludes the assistant header where transformers includes it, and it refuses a chat template whose partial renderings are not prefixes of the full one rather than approximating. Qwen3 is exactly that case, injecting its thinking block only for the last assistant turn, so multi-turn Qwen3 rows are now refused at dataset construction with a message naming the remedy, which affects the shipped qwen3-8b-sft-mlx recipe. The max_grad_norm scope is stated honestly too: one M1 step on an ill-conditioned batch moved SGD weights by 388.03 unclipped against 0.0085 clipped, while AdamW moved 0.126505 against 0.126504, so it rescues sgd and gradient spikes rather than every run. One upgrade note applies to anyone who never touched the field: gradient_accumulation_steps defaults to 4, so an MLX run that used to update the optimizer on every micro-batch now accumulates over four first, which is a 4x larger effective batch size and roughly a quarter as many updates for the same iteration count. That is the schema default finally taking effect, so a differently-converging run after upgrading is expected rather than a regression. The backend also drives the live terminal dashboard, the SQLite experiment tracker and the soup ui stream, all three of which it used to accept and discard, and it reports validation loss. The second change has the widest blast radius and it is breaking. The deadline v0.74.0 named has arrived: an unknown config key refuses the load. soup train exits 1 before the training stack is imported, and the API and Web UI loader raises a ValueError with the same text, naming the field you probably meant. Nothing is defaulted and nothing is substituted, and the message says Refused rather than Not applied, which would have read as if the run went ahead without the key. Two practical consequences: a config that must stay loadable on v0.74 as well needs the key removed rather than renamed, because v0.74 warns and ignores it while v0.75 refuses it and neither applies it, and soup plan and soup apply read the YAML as a plain mapping so they did not run this check at all (since v0.76.0 they refuse an unknown key too). The detector now sees a config the way the schema does, so a root-level lora block, the LLaMA-Factory and Axolotl spelling the schema has accepted and moved under training since v0.40.1, is remapped before the walk rather than refused, which the release review caught in the first draft. Key names are escaped before they reach the terminal, because Rich markup and raw escape bytes in a YAML key could restyle or spoof it, and the scan stops at 100 findings and says so, because 50,000 bogus keys cost about 31 seconds of CPU per Web UI request. All 167 recipes, 21 templates and the shipped example configs scan clean. Third, validation loss existed nowhere. It was computed on every backend and thrown away (v0.76.0 corrects the premise: on the transformers backend no trainer evaluated the held-out rows at all until #1223 and #1313): the callback read the training loss and never the evaluation loss, so an evaluation step left the last training loss in place and re-reported it to every sink, and there was also nowhere to put it, no metrics column and no event field. The live panel, the metrics table and the streaming frame now each carry val_loss as its own series, never folded into loss, and the panel gains a Val loss row. The panel carries the last measured value forward between evaluations while the stored and streamed values do not, so a step where no evaluation ran records NULL and a series read back has one point per measurement rather than one per logged step. Existing experiment databases are migrated in place, and rows written before this ship read NULL rather than a fabricated zero, because an unmeasured value must not read as a measured one. Fourth, the torch floor moved to 2.6.0, which closes the known limitation v0.74.0 published rather than hid: at torch 2.5.1 trl 0.29 could not import at all, because it needs the public FSDP2 API that arrives in 2.6.0, so every preference trainer was dead in a pinned 2.5.x environment. CI now proves the floor. Fifth, and breaking: grpo_variant gspo is now the published Group Sequence Policy Optimization objective rather than a column-centering heuristic, so existing gspo configs will yield different losses and gradients and will not reproduce prior runs. The heuristic it replaced also centred the per-token log-ratio across the batch before the completion mask was applied, so one masked padding token shifted the loss and gradient of every unmasked row sharing its column, and picked up a non-zero gradient of its own. Also here: training.loraplus_lr_ratio crashed every run that set it, because it was forwarded to TrainingArguments, which has no such field, and it now builds a real PEFT LoRA+ optimizer whose save and resume state is proven to survive; the wasserstein and topk_align distillation strategies forwarded the student's token ids to the teacher clamped into range, which does not translate a token between vocabularies, so a mismatched pair trained on the teacher's logits for the wrong text with a finite and plausible loss, and both now refuse a non-interchangeable pair and name wasserstein_aligned instead; the same loss began honouring the response-only label mask and the causal shift it had ignored; inference prompts are now encoded the way Soup trains them, because soup chat, serve, infer, bench, diff, ship, diagnose and advise all rendered the chat template to text and then let the tokenizer add its own special tokens, sending a doubled BOS on Llama-3, Gemma and Mistral templates, with the vLLM, SGLang and MII engines still affected and tracked separately; soup ingest with source langfuse and the new pull flag fetches generations straight from the Langfuse API, with credentials taken only from the environment so they never reach the audit log; soup doctor with a config path reports the settings that config writes which its task and backend do not read, from a hand-written table scoped to sft on mlx, seven entries, whose all-clear message upstream itself says overclaims; soup data validate now runs the real converters so it agrees with the loader, fails when no row is usable, takes a minimum-valid-fraction threshold for CI, and rejects an unknown format instead of reporting every row valid for it; AWQ exports require explicit calibration data rather than silently downloading AutoAWQ's large default set; multipack FFD placement is O(N log N) via a segment tree, with the packing verified unchanged over 6,000 randomized cases; packing no longer raises on TRL 0.29 while packing_cross_doc_attn_mask is refused outright, because it never mapped to a valid TRL strategy on any release; the Web UI requires authentication on read endpoints and streaming as well as mutating ones, stops serving its FastAPI docs routes to the LAN, and cannot hang when no client drains its output; and four recipes take the catalog to 167. Three contributor measurement records landed in this window and no maintainer gate did: a pinned-store accounting probe on an L4, a QuEST W4A4 supervised gate that FAILED and is published as written, and an 8 GB M1 MLX run whose own header says it is not a gate. That last one is worth a line, because its shape is the useful part: the model the shipped llama3.1-8b-sft-mlx recipe names trained on an 8 GB M1 under a smaller config than the recipe's (LoRA r=8, batch 1, max_length 512), 48 iterations in 71 seconds at a 5.154 GB peak, and a model too big for RAM there does not fail, it pages, so an allocation-failure check would not fire. Its own instruction is to treat every throughput figure in it as an order of magnitude rather than a benchmark, proven by a 3.5x spread on the same config from a warm cache. The preprint is untouched: no measured number moves and the scope does not change, because nothing in this window touches layer streaming's mechanism or its architecture list. One new limitation is worth carrying: streamed-NF4 bit-exactness does not hold on Blackwell. Two CUDA-only tests fail on an RTX 5070 at 4.9e-4 and 3.9e-3 while the unquantised variants pass, so the divergence is in the bitsandbytes NF4 path on that card rather than in layer streaming, CI has no GPU runner and cannot see it, and the root cause is not established. (Since v0.76.0 the streamed NF4 forward follows bitsandbytes' own dispatch, #1143; no record shows whether that closes this.) Before it, v0.74.0, tagged 4 September 2026, had no codename either: upstream titled it with a sentence, the base was loaded in fp32 the whole time. 116 of the 120 pull requests merged into it came from somebody other than the maintainer, by 25 people, and the maintainer's own four were a DeepSpeed guard, a dependency-drift job, an MCP compatibility layer, and the CI fix that stopped that same drift alarm firing every Monday. The headline is the one a contributor found. A LoRA run freezes the base: it never receives an optimizer step, so its only job is to hold the checkpoint's own numbers. All three supervised load paths, text, vision and audio, passed no dtype at all, so the frozen base was materialised in fp32, at twice a bf16 checkpoint's precision, for nothing. Measured on an H100 with Llama-3.1-8B and LoRA, peak memory falls from 48,241 MiB to 18,658 MiB, a 2.59x reduction and 28.9 GB, byte-identical across three repeats. Read that as a saving on a configuration that was already paying it, not as a new capability: the same fix landed in twelve more trainers, dpo, kto, orpo, simpo, ipo, bco, online_dpo, grpo, ppo, pretrain, reward_model and embedding, none of which has a full fine-tuning branch at all, and a genuinely trainable base still loads fp32 on purpose, which is a documented numerics choice rather than an oversight. One measurement supports the number, carried over from the original pull request rather than re-measured, and it states its hardware, model and adapter and nothing else, so no sequence length or batch size should be attached to it. It also does not touch the streaming laptop figure, which was never paying an fp32 base. The second large change is the dependency stack. Training now requires Transformers 5.16.1 or newer, TRL 0.29 and PEFT 0.20, reached through capability probes rather than a version table, because a version table was wrong twice in this project's history. The visible consequence is that pip install soup-cli[train,mlx] resolves again: the two extras had declared ranges that could not be satisfied together at all, so anybody wanting Apple Silicon and the training stack in one environment was simply stuck. Qwen3.5-family text decoders train on the Transformers path as a result. One known limitation is published beside it rather than left to be discovered: the declared torch floor of 2.5.0 did not actually work with TRL 0.29, because at torch 2.5.1 a module TRL imports does not exist, so DPO, KTO, GRPO and BCO were unavailable there. v0.75.0 closed it by raising the floor to 2.6.0 and proving it in CI. A fresh install resolves a newer torch and is unaffected; an environment pinned to 2.5.x is not. No floor bump ships, because 2.5.1 was measured to fail and 2.6 was not measured to work. Third, the free notebook tier could not stream at all, and this is a second defect rather than the one v0.73.1 fixed. That release removed a bf16 assumption made in fourteen places, which had broken every task on every pre-Ampere card. Streaming still crashed on a T4, a P100, a V100 or a GTX 16xx, because PEFT creates LoRA adapters in the base checkpoint's dtype while the fp16 gradient scaler requires fp32 gradients. Trainable adapter parameters are now cast to fp32 before the optimizer is built, from every trainer's own entry point through one shared helper, and the full fine-tuning paths are deliberately left alone so trainable memory cannot double after the VRAM pre-flight has already approved the run. No pre-Ampere measurement was added, so nothing about Turing throughput or gradient exactness is claimed here. Layer streaming itself gained scope rather than speed. The architecture allowlist goes from nine to ten with Qwen4-Exp text, and Qwen3.5's dense and MoE text decoders route onto the existing Qwen3 streamer through explicit aliases, which is stated that way on purpose: only the original nine are verified bit-exact in both bf16 and NF4. Qwen3.5's paths are verified against resident controls on CPU, the 35B MoE was validated live with NF4 and MoE LoRA on a 3072-token dataset but has no resident control because no available machine can hold it resident, and Qwen4-Exp has an exact float32 tiny-model parity gate with real-checkpoint and NF4 validation still pending. No throughput figure exists for any of them. Two mechanical improvements ride along: an untied embedding matrix and LM head are now sharded separately and share one vocabulary-sized device buffer instead of both staying resident, which is the 2.10 GB that dominated the 8B row's 3.32 GB peak, and the resulting peak is deliberately not restated, because it had not been re-measured on the reference card (since changed in v0.76.0: a post-repair 8B row now exists, but on a different RTX 3050 Laptop, so the reference card itself has still not been re-measured). And training.stream_pin makes page-locking an explicit choice: false forces the pageable store and prints what that costs, true forces the pinned store and refuses on the RAM tier if the box cannot page-lock it, naming the store size rather than a ceiling nobody probed, while on the disk tier and on CPU it announces and proceeds because there is nothing there to page-lock (since closed in v0.76.0 for the disk tier, which stages layers in host memory through an async reader: true is honoured or refused there too; CPU stays announce-and-proceed). The MCP server grew network transports. Until now it was stdio only, which suits a client that spawns Soup as a subprocess and leaves remote or multi-client setups with nothing. sse and http now serve the same registry, and the listener is gated three ways: every request needs a bearer token compared in constant time with no opt-out, the SDK's DNS-rebinding protection is on so a page the operator merely visits cannot reach the port, and binding off loopback warns. Gated execution is refused with either network transport, in the command and again in the app factory, because behind a listener a leaked token would mean process execution rather than plan disclosure. The tool count is unchanged at 18. The server also runs on both major versions of the MCP SDK now, chosen by probing the server constructor rather than by reading a version string, which lifts the below-2.0 cap that had pinned anyone who wanted 2.x in the same environment. A quieter change has the widest blast radius: unknown configuration keys were silently dropped. None of the config models overrode the library default, so a key the schema did not declare validated clean and was discarded. A misspelled quantization key trained 4-bit when full precision was asked for; a misspelled gradient-checkpointing key did no checkpointing; a misspelled max-length key truncated at the default. soup train --dry-run said the config was valid and the run exited 0. Loading a config now walks the whole model tree and reports every key it cannot place, in one report, naming the field you probably meant. It was a warning rather than a refusal there, so a config written against a newer Soup still ran on an older wheel, and the release named v0.75 as the version that would reject it, with a test that turns red when that deadline arrives. It has arrived. soup sweep is stricter, because the failure there has no partially useful result to preserve: a swept parameter matching no config field produces a grid of identical arms and a meaningless winner, so it now exits 1 before the first arm starts, and --dry-run validates instead of returning before the config is even loaded. Two breaking changes are worth stating plainly. soup serve now exits 2, rather than printing a warning, when it is bound to a non-loopback host without a tool auth token, because the shell endpoint it protects has been re-enabled under real operating-system isolation and now actually executes. And the SGLang serving backend obeys trust-remote-code instead of hardcoding it on at both runtime call sites, so a custom-code model on that backend fails to load without the flag rather than silently running its code; the warning panel had said so for releases, but a notice is not a gate. On security, four spellings of the same bypass are closed: abbreviated, decimal, hexadecimal and octal IPv4 forms reached the telemetry and webhook guard, and, through a path the first fix never touched, the OTLP tracing validator, which mattered because the cloud metadata address has a decimal spelling. Seven copies of that predicate across the codebase became one, with a guard test that fails if an eighth appears. Also here: soup eval aider runs Aider's Polyglot code-editing benchmark through its official Docker harness behind a new extra; soup train --cloud lambda plans a Lambda Cloud run, with the API key never entering the instance; soup monitor reads real Apple Silicon GPU utilisation and power through powermetrics rather than guessing, leaving NVIDIA-only fields unavailable; soup data best-of-n can sample candidates from Ollama or vLLM and gained a durable two-phase offline workflow; data.interleave is finally read at training time, which closes a gap where every multi-dataset mixture had silently trained on one dataset since v0.42.0, and it now covers streaming and Hub datasets too; LISA accepts continued pre-training and gained training.lisa_train_embeddings, whose default matches the published method and whose false setting freezes the always-on group that is about 70% of what LISA trains at 8B, a real quality-versus-memory trade rather than a free win; baseline eval artifacts carry a scorer-revision provenance stamp, replacing a hardcoded suite-name warning; and nineteen new recipes take the catalog to 163. Two new measurement records were published, neither of them a throughput. A contributor ran the streaming VRAM pre-flight on a second box, an A10G on a much newer stack, and the long-sequence under-prediction that v0.73.1 disclosed does not reproduce there: the ratio is flat at about 1.16x over-prediction from sequence 2048 through 6144 on two models, where the 4 GB laptop had gone under by 21% at 6144. The loss term measures 12.0 with zero spread on that third stack, and the retained copy the laptop's arithmetic implies is absent. The record publishes its own leading hypothesis as a negative result with a positive control, and explicitly does not extend it to the laptop, which is the honest shape: the gap is stack-specific rather than drift with sequence length, which is the argument for measuring a real step rather than adding another coefficient to a formula. The second record is an Apple Silicon Qwen4-Exp mapping gate that passed its cache and parity checks and then stopped short of a completed optimizer step, and says so in its own words: it must not be read as a throughput, peak-memory or trainability claim. Underneath everything, the v0.73 line is unchanged. v0.73.3 was the first release the maintainer did not write, and what its 24 outside pull requests found was four flags that the schema validated, the docs described, and nothing read, plus two runs that completed and were wrong anyway: assistant-only loss masking built its label mask from a tokenizer mapping's key strings, so a run trained on zero tokens behind a normal-looking loss curve, and on Apple Silicon quantization 4bit was silently rewritten to none because device detection did not know MLX. Its one new capability was that soup mcp serve --allow-execute stopped being a reserved gate and began running a planned training or export behind a single-use, server-generated confirmation token bound to the plan and to the kind, taking the server from 16 tools to 18. v0.73.2 repaired the release gate itself: two of soup ship's behavioural suites ranked by the wrong thing, so a stub answering every item correctly scored 0.000 and an 8B naming the right tool 40 times out of 40 scored 0.225, a whole failure direction had no detector until mini_over_refusal became the eighth bundled suite, and soup ship --noise-floor now measures what the instrument itself can resolve before any delta is called significant. Any verdict issued before it is suspect and a stored baseline is on a different scale. v0.73.1 repaired the GPU tier most people actually have, caught its own streaming VRAM pre-flight under-predicting at long sequence against a contract that says it never does, and shipped training.stream_vram_probe, which decides the fit by measuring one real step instead of predicting it. And v0.73.0 is the release that came out of three borrowed days on an 8x H100 box, the first machine able to hold a resident reference for a model worth streaming. It confirmed the method, the streamed forward pass is exact against that reference at 8B, 14B, 32B and 72B, peak VRAM stays flat from 3,397 MiB to 4,845 MiB while the model grows fourfold, the laptop headline reproduces at a median 113.00 tok/s in the same 3.32 GB, and a streamed model converges indistinguishably from a resident one across five paired runs, and it broke it: in NF4 above roughly 165 MiB per decoder layer the backward pass produced silently wrong gradients while the forward stayed bit-exact, the loss matched to every digit and the run exited 0. That affects 32B and larger, never 8B or 14B, and it is repaired and gated at both affected sizes. The same three days found four backends that had never executed once and a supervised path that silently capped every run at 1024 tokens. Beneath that the v0.72 line still carries the flagship. Quantizing the streamed base to NF4 shrinks the RAM store about fourfold, so Llama-3.1-8B fine-tunes at 119.6 tok/s with a 3.32 GB peak on a 4 GB RTX 3050 Laptop, a number measured on v0.72.2, before the v0.73.0 repair, which cost 4.8% at 32B (a post-repair row from another RTX 3050 Laptop at a different clock is recorded; no ratio is given); an H100 cannot substitute, because it ran the same configuration no faster than the laptop. DPO, ORPO, SimPO and KTO run against a streamed base too, with DPO's reference taken from the same stream with its adapters disabled so it needs no second copy of the layer weights, at 1.52x the layer reads per step, and no throughput figure is claimed for any preference loss at any size. GRPO and PPO stay permanently excluded, because rollouts re-read every layer per generated token. Layer streaming remains BETA. One honesty note, rewritten because v0.75.0 changed it: Soup's telemetry is now wired, and it is strictly opt-in. Nothing is built or sent unless SOUP_TELEMETRY=1, a global --no-telemetry flag disables it per invocation, and the payload is hardware-only: seven fields covering version, command name, Python major.minor, OS, architecture, duration and a locally generated anonymous UUID. The published policy states that dataset paths and contents, model names, config contents, usernames, local paths, IP addresses and credentials are never collected. As shipped it still sends nothing even when opted in, because the bundled project key is an unprovisioned placeholder the sender refuses by name, printing a notice instead of opening a socket. Everything from the v0.71 line is still current underneath: soup ship's verdict with an emit-evidence loop and a committed ShipConfig, Reward Forge, the Data Moat II semantic data layer, the compliance pack, LISA, soup shrink, soup draft's honest speculative-decoding gate, Whisper ASR fine-tuning and the judge-in-the-loop suite. Totals as of v0.76.0: 875 test files, 170 ready-made recipes, 21 built-in templates, 23 training tasks, 20 data formats (one of them, cross_encoder, only with task: cross_encoder), 17 quantization formats, Apache-2.0, and Python 3.10 to 3.12.

Question 2

Will my soup.yaml from v0.75 still load on Soup v0.76.0?

Accepted Answer

Not necessarily, and you can find out before you spend a GPU hour: after upgrading, run 'soup doctor --config soup.yaml'. It loads your config under the v0.76.0 rules and reports every setting in it that is warned about, plus the refusal that stops the load, with what to change (when several cross-field refusals apply it prints the first one, so fix that setting and run the command again until it prints no error); it exits 2 when the config cannot load (the last line reads 'Config could not be loaded (see above).') and 0 when it loads, warnings included. The release's rule is that a setting Soup validated, documented and accepted, but never applied on the path it claimed to configure, now stops the load instead of quietly training something else (upstream's known limitations say that goal is not fully met yet); unknown keys have refused the load since v0.75.0. Refused at load: 'training.quantization_aware: true' (it never ran quantization-aware training), 'training.use_longlora', 'training.ebft_variant', every non-null 'training.gdpo_variant', 'training.rope_scaling_type' on any task other than sft, tts and pretrain, 'training.freeze_layers' and 'freeze_ratio' outside the text sft and tts path on the transformers backend, 'training.auto_mixed_precision' outside sft and tts, 'backend: unsloth' on seven further tasks that have no unsloth setup, a Sesame CSM text-to-speech config, 'quantization: bitnet_1.58' for training, a 'relora_steps' config that does not also set 'quantization: none', 'task: kto' with 'batch_size: 1', and the optimizer names transformers 5 rejects after the model has loaded (adam_mini, adamw_apex_fused, ao_adamw_fp8, ao_adamw_4bit, ao_adamw_8bit, badam, came_pytorch and dion everywhere, muon and adamw_hf off the MLX backend). Warned about at load, and refused from v0.77 according to upstream: 29 staged fields that nothing reads, among them 'training.lr_groups', 'early_stop_patience', 'citation_recall_threshold' and 'forgetting_eval_steps' (a field left at its schema default stays silent), plus three deprecated values: 'data.remove_unused_columns: true', an explicit 'training.quantization' of 4bit or 8bit on a task whose trainer never quantizes, and 'soup ab --beta'. A config that must load on both v0.75 and v0.76 needs the refused key removed rather than renamed, and deleting the key is the fix; but for 'training.use_longlora', 'training.ebft_variant' and 'training.quantization_aware: true' a run that finished on v0.75 did not do what its config said, so re-run it (the re-run table is on the settings-refused-at-load page). Behaviour that changes even when the config still loads: 'data.val_split' is now evaluated (at the end of each epoch, or every N steps with the new 'training.eval_steps') and the validation loss is recorded, so runs take a few percent longer; grpo, ppo and online_dpo evaluate by generating, so by default they ignore val_split with a note and train on every row; every non-standard 'grpo_variant' now applies the 'grpo_beta' KL penalty it used to validate and drop, so existing variant configs will not reproduce prior runs; 'training.loraplus_lr_ratio' now takes effect on every transformers-backend trainer that builds a LoRA adapter (it was silently ignored on all but sft, pretrain and embedding) and is refused at load where it cannot apply (unlearn, prm, moe_lora_routing, backend mlx, lora.use_vera); an FP8 request that cannot run stops the run instead of training in bf16; and eleven gate and verdict commands exit 0 on pass, 2 on a gate failure, regression or drift, and 3 on a usage or input error ('soup eval gate' and 'soup eval against' used to exit 1 on failure, 'soup lock check' and 'soup expect' used to exit 3), so a CI step that treats only exit 1 as 'gate failed' no longer catches a failing gate. One limit worth knowing: 'soup doctor --config' only knows which settings are unread for a few task, backend and modality combinations, and for the rest it reports nothing rather than guessing, so a clean report is not proof that every key in the file is read (upstream tracks that as #1815).

Question 3

Can Soup fine-tune and align a model that is bigger than my GPU's VRAM (layer streaming, v0.72.4)?

Accepted Answer

Yes. You can fine-tune Llama-3.1-8B on a 4 GB laptop GPU, measured at 119.6 tok/s with peak VRAM of 3.32 GB (measured on v0.72.2, before the v0.73.0 NF4 gradient repair; a post-repair row from another RTX 3050 Laptop, taken at a different clock, is recorded in the benchmarks directory; throughput figures are comparable only at the same clock, so no ratio is given), and it ships BETA within a scope the release states precisely. Set 'training.stream_layers: true' in soup.yaml and the frozen base is never loaded resident. Soup rewrites the checkpoint once into one safetensors shard per decoder layer (cached under ~/.soup/layer-stream/, overridable with SOUP_LAYER_STREAM_CACHE_DIR and keyed to the quantisation as well as a fingerprint of the source checkpoint, so switching precision or retraining a base in place re-shards instead of silently streaming the wrong bytes), holds that store in CPU RAM page-locked when the machine allows it, and copies one layer at a time into a small pool of pre-allocated VRAM buffers on a dedicated CUDA stream, so the load of the next layer overlaps the compute of the current one. The base model is built on PyTorch's meta device, so only the embeddings, the final norm, an untied LM head and the LoRA adapters with their gradients and optimizer state ever occupy VRAM. Peak VRAM is therefore bounded by the size of one layer plus the embeddings, the adapters and the activations, not by the size of the model. Each layer is read twice per step, once in the forward pass and once when the backward pass recomputes it, because pushing gradient down to the layers below needs the weights, which is why streaming costs time rather than being free. Adding 'quantization: 4bit' quantises that streamed base to NF4 and shrinks the RAM store about fourfold, which is what brings 8B within reach of a card that cannot hold even a quarter of it. Measured on a 4 GB RTX 3050 Laptop at batch 1, sequence length 512, gradient checkpointing on, 50 steps after 10 warm-up: Llama-3.1-8B-Instruct at 119.6 tok/s in 3.32 GB with the base page-locked at a 3.60 GB store and the GPU 100% busy, and Qwen2.5-3B at 264.2 tok/s in 1.76 GB. On the bf16 path Qwen2.5-3B trains in 2.15 GB where a resident run runs out of memory. The honest cost is 1.43x slower than resident training, measured at 0.5B, the only apples-to-apples comparison available on that box. Correctness is not part of the trade: a streamed run was measured bit-exact against a resident run at the same precision in the v0.72.x gates (since v0.76.0 the streamed NF4 forward follows bitsandbytes' own fused-versus-dequantized dispatch, the same kernels as resident NF4, and no new bit-exact record exists for it; the backward still dequantises), and logit equality, non-zero gradients at layer 0 and a matching multi-step loss curve are regression tests rather than one-off measurements. v0.72.3 widened it from a demonstration into something you can plan a run on. Nine model types streamed after it, not three: llama, qwen2, qwen3, mistral, gemma, gemma2, gemma3_text, phi and phi3, each with its forward logits compared against the same model loaded resident (NF4 against a resident NF4 load) and found equal, on tiny from-config CPU float32 fixtures rather than real checkpoints, and v0.74.0 later took the allowlist to ten with qwen4_exp text plus the qwen3_5 dense and MoE aliases, on weaker evidence that the docs state rather than smooth over. Phi-3 fuses Q, K and V into a single qkv_proj, so there is no q_proj to substitute, and it is bit-exact anyway. Multimodal gemma3 is deliberately refused; only gemma3_text is accepted, because streaming a vision wrapper as a causal LM is exactly the failure an allowlist exists to prevent. There is no throughput measurement for any of the six newly supported families, so none is claimed. Batches above 1 and gradient accumulation both work now, and the release measured which one to reach for rather than reasoning about it. On Qwen2.5-0.5B in bf16 at sequence length 256 with a pinned store: batch 1 with no accumulation runs at 556.6 tok/s in 0.842 GB, batch 1 with accumulation 4 runs at 540.1 tok/s in 0.846 GB, and batch 4 with no accumulation runs at 1378.0 tok/s in 2.28 GB. At the same effective batch of 4, raising batch_size measured about 2.52x faster than accumulating, because a real batch amortises each layer load over more tokens while accumulation re-reads the entire base for every micro-batch, which is why the layer reads per 1000 tokens stay flat across accumulation 1, 2 and 4. Accumulation still earns its place: it holds peak VRAM almost exactly flat where a bigger batch does not. Raise batch_size while the budget allows, then accumulate. Because batches now scale, there is a pre-flight that accounts for them. Streaming bounds the weights, but activations and the logits tensor are untouched by it and both scale with batch times sequence length. On a 151,936-token vocabulary at batch 8 the logits tensor alone measured 8.71 GB, which is 146 times the entire layer-buffer pool of 0.060 GB. soup train now predicts peak VRAM before the run and refuses one that will not fit, naming training.batch_size and data.max_length as the two knobs that scale it linearly. The estimate was fitted against ten real runs across two models with a 3.1-fold vocabulary contrast, with a worst error of 0.85%, and it never under-predicted any of them. That contract later failed: v0.73.1 measured it under-predicting at long sequence (0.934x the real peak at sequence 5120 and 0.787x at 6144), which the ten-run grid structurally could not have caught because all ten of its rows sit at sequence 256 or 512, so it varies batch and says nothing about sequence length. It also corrected a real bug: the old estimator charged 6 bytes per logit element from first principles, where the measured peak is 14: 12 for the loss arithmetic (the bf16 logits alongside three fp32 buffers of the same shape) plus 2 for the model's output kept alive across the backward. That under-predicted the largest term in the budget by 2.33x. This matters most on Windows, where WDDM does not raise on overcommit: a run that should have failed instead reached a 9.27 GB peak on a card with 4.29 GB, silently, by paging to host memory. The panel also prints a throughput forecast, quoted as a range from a GEMM ceiling measured on your own card in that session and printed next to the SM clock, never a compiled-in per-card constant. That is a response to a measurement: this one card produced 3.5 and 7.6 TFLOPS in two different sessions at the same reported clock, so a baked-in number would have been wrong about half the time. Streamed runs measured in v0.72.3 landed at 68 to 100% of their measured ceiling, while the post-repair 8B row recorded for v0.76.0 reads 59% of its own same-session ceiling (not like-for-like, per the record), which is why the line calls itself a bound and not a promise. The flags --resume and --hf-resume work with streaming now. They had to be refused before because adapters could not be loaded into a streamed model at all: PyTorch's load_state_dict narrows keys by child name, so a canonically saved checkpoint matched 0 of N tensors, and because PEFT reports missing keys as a warning rather than an error, a resumed run silently reproduced the from-scratch loss curve exactly. The streaming layer now redirects canonical keys at load time, mirroring the save-side fix from v0.72.1. The fix is load-side only, so v0.72.0's forward-path bit-exactness results stand without being re-run. One honest limit: end-to-end resume could not be demonstrated on the development box, for a reason that has nothing to do with streaming, since transformers refuses torch.load below torch 2.6 under CVE-2025-32434 and that blocks every resume there (since v0.75.0 the declared torch floor is 2.6.0, so a supported install can no longer hit that). The streaming-specific half, the adapter round-trip and loss continuity, is verified on the production CUDA path. Loading into a streamed model works. Until v0.75.1 named_parameters() and state_dict() disagreed in memory, the deliberate cost of a serialisation-only fix, and on peft 0.21 or newer that mismatch made a streamed adapter save as a 40-byte file holding zero tensors; v0.75.1 made names and keys one spelling, and an adapter saved with stream_layers: true on peft 0.21 or newer before it must be re-trained. For a base too large even for RAM there is now a disk overflow tier. stream_source: auto is the default and uses RAM when the base fits, falling back to an NVMe disk tier when it does not, holding no resident copy of the base (since v0.76.0 a background thread reads the shards through direct I/O into host staging buffers, staging up to 'training.stream_read_ahead' minus one layers ahead of the layer in use). stream_source: ram insists and refuses rather than falling back, so a run that must stay in RAM cannot quietly become a disk run. Non-NVMe media is refused outright, including media Soup cannot identify, because the streamer reads every shard twice per step, which on a spinning disk is 160 seeks per step for an 80-layer model. soup doctor --disk reports what was detected (NVMe, SATA SSD, HDD or Unknown) and is opt-in because the probe costs about 9 seconds cold and about 2.4 seconds warm on Windows. The v0.72.3 synchronous disk reader's correctness was verified bit-exact against the RAM tier; the v0.76.0 reader is gated by byte-identity against that source. How long a disk-tier step takes was first measured in v0.76.0, on a second laptop (an RTX 5070 Laptop 8 GB with two NVMe drives, Windows) and only on a synthetic 70B-shaped NF4 store of 36 GB with random weights, timing only, with no RAM-versus-disk ratio: a cold step went from 110.5 s on the previous synchronous reader to 16.2 to 17.3 s on the direct-I/O reader, and from 17.4 s to 9.97 s across two drives ('SOUP_LAYER_STREAM_STRIPE_DIRS'), while a sustained-load probe measured 1.28x and 1.41x in two rounds and reached no formal verdict. A store that fits in RAM is still faster on the RAM tier, which stream_source auto picks. No real 70B checkpoint was trained. v0.72.4 then opened the whole path to the preference losses. DPO, ORPO, SimPO and KTO run against a streamed base now, with the same config keys, so alignment stops being the step that forces you to rent a bigger card. The hard part was DPO's reference model, because the obvious implementation builds a second copy of the model and doubles the weights, which defeats the entire premise. Soup takes the reference from the same streamed base with its LoRA adapters switched off, so policy and reference are one set of weights, one RAM store and one buffer pool, and the adapter is the only difference between the two forward passes. That is measured rather than asserted, because a loss curve cannot detect its failure: if disabling the adapter were a silent no-op through the streamed path, every log-ratio would be zero and the loss would sit at 0.6931 forever, which reads as slow training rather than as a bug. On the reference box in bf16 with a 365.2M-parameter model, streamed DPO peaked at 81.87 MB against streamed supervised fine-tuning's 89.53 MB, which is 0.914x, with a byte-identical RAM store and buffer pool, while the same harness with a real second instance forced in reached 812.32 MB, moving the peak by 730.44 MB against 730.44 MB of weights, exactly one copy. A zero-adapter control measured a policy-versus-reference gap of exactly zero, proving the gap in the real run comes from the adapter. Read the 0.914x as no second copy rather than as a saving: it sits below 1 because that fixture splits its 64 positions into a 32-token prompt and a 32-token completion, so the logits tensor is smaller, and the model is synthetic rather than a real checkpoint. Upstream's CHANGELOG states the loss is bit-exact against a resident run for all four, and the benchmark record shows it for DPO (difference exactly 0.0, float32 on CPU, a forward-side comparison of the loss), and a live soup train on SmolLM2-135M under NF4 moved every one of them, saving ordinary LoRA adapters that load into a non-streaming model. The reference needs no second copy of the model but is not free in time: DPO walks the layer stack 1.52x as often per step as supervised fine-tuning does, because it adds a reference forward to the policy forward and the gradient-checkpoint recompute. KTO is not reference-free however it is usually described, since it selects its reference exactly the way DPO does; ORPO and SimPO genuinely are. KTO also requires a batch size of 2 or more, because TRL's KL term is degenerate at batch 1, and Soup refuses that when the config is read rather than minutes into a run after the checkpoint has been sharded. No throughput figure is claimed for any preference loss at any size, because none was measured. And the VRAM pre-flight is a sound upper bound for these losses but a loose one, charging the supervised loss's 14 bytes per logit element over twice the rows where TRL reduces preference logits to per-token log-probabilities instead, so on a 4 GB card with a 128k-vocabulary 1B model DPO is allowed at max_length 512 and refused from 768 up even though it would probably fit. That looseness shipped deliberately, because under-predicting is the strictly worse failure on a platform where an overcommit spills silently into host memory instead of raising. What is still out of scope: task must be one of sft, dpo, orpo, simpo and kto, with ipo, bco and the unified task preference dispatcher still refused; backend transformers; text modality; plain LoRA, since DoRA, VeRA and the PiSSA, OLoRA and LoftQ initialisations all read a real base weight that streaming keeps on the meta device; and quantization values other than none and 4bit. A batch_size of auto is refused because the auto-batch probe sizes a resident model, which a streaming run never loads. GRPO and PPO are permanently excluded rather than pending, and their refusal deliberately names no release, because rollouts need generation and generation re-reads the model per token, which destroys the amortisation streaming depends on. Nothing above 8B has been measured on the 4 GB card, so no figure quoted for it speaks for a larger model; sizes from 14B to 72B were later measured on a borrowed 8x H100 box, where the streamed forward is exact against a real resident reference throughout and the exercise found a gradient defect in NF4 above roughly 165 MiB per decoder layer, affecting 32B and larger, which is repaired in v0.73.0 and gated at both affected sizes. The numbers are Windows and WDDM and therefore systematically pessimistic versus Linux. On prior art the claim is deliberately narrow, because a first draft of it was fact-checked and cut. Layer streaming on a small card is well established for inference: AirLLM streams 70B-class models through a 4 GB GPU and its own documentation says layer sharding is not compatible with gradient propagation, and Hugging Face's disk_offload is documented as big-model inference and runs under no_grad. Small-card fine-tuning has been published too: LSP-Offload (arXiv:2406.10181) fine-tunes 1.3B on a 4 GB laptop GPU by compressing the optimisation into a learned sparse subspace rather than by streaming weights. So nobody-has-done-this would be false. The defensible statement is the measured one: up to 8B on a 4 GB card, by streaming the base rather than approximating the optimisation, with the forward pass verified bit-exact against a resident run at the same precision. If the model already fits resident on your card, do not enable streaming. It trades time for memory, and there is nothing to buy when memory is not the constraint.

Question 4

Has layer streaming been verified on hardware other than the maintainer's laptop?

Accepted Answer

Yes, on an 8x H100 80 GB box borrowed for three days in August 2026, and the record is published in full at the repository's benchmarks directory. The reason it mattered is specific. Layer streaming's central claim is that a streamed run computes the same thing an ordinary run computes, and checking that needs a resident reference: the same checkpoint at the same precision, loaded the normal way, so the two can be compared tensor by tensor. On a 4 GB card that reference cannot exist for any model worth streaming, so every published correctness check had run against 3-layer models built from a config, with a hidden size of 32 and an NF4 layer of 0.01 MB. The preprint states this in its own limitations and names it as the most useful experiment missing from the work. Four results came back. First, the streamed forward pass is bit-exact against a matched resident reference, logits compared with torch.equal, at 0.5B, 8B, 14B, 32B and 72B, on real checkpoints rather than fixtures. Read that carefully, because bit-exactness is two independent claims and not one: the forward and the backward are measured separately, and above roughly 165 MiB per NF4 layer they disagreed. Second, peak VRAM is flat in model size, which is the entire point of the feature: 3,397 MiB at 8B, 4,475 MiB at 14B and 4,845 MiB at 32B while the model itself grows fourfold, and 7,411 MiB at 72B. Third, the published laptop headline reproduces on completely different hardware, a different operating system and a much newer software stack: 119.6 tok/s in 3.32 GB on the RTX 3050, a pre-repair figure that predates the v0.73.0 NF4 gradient repair, against a median 113.00 tok/s in the same 3.32 GB peak on an H100. The server card is if anything slightly slower, and mean GPU utilisation was 54.1%. The explanation first published for that was retracted in August 2026, after an ablation on the original laptop measured the entire host-to-device term at 1.44% of the step; the measurement stands and the explanation does not. Fourth, a streamed model converges indistinguishably from a resident one: five paired runs on disjoint training subsets, scored by Soup's own soup ship against a fixed 300-item held-out set, gave a mean paired difference of 0.0053, roughly 1.6 items out of 300, against a within-arm spread several times larger, and the general-capability leg pointed against the convenient direction with 3 of 5 resident runs coming back DON'T SHIP against 1 of 5 streamed. That is 'no difference is detectable at this resolution', not 'there is no difference'. The exercise also found a defect, which is the honest headline. In NF4, above a boundary bracketed between 163.8 and 171.5 MiB per decoder layer, the backward pass produced silently wrong gradients on every layer but the last few, while the forward stayed bit-exact and the loss matched a resident reference to every digit and the run exited 0. That affects 32B and 72B; 8B sits at 105 MiB per layer and 14B at 132, both below the boundary with margin and both verified exact over a 50-consecutive-backward soak, so nothing this project has published is above it. The cause is buffer aliasing rather than a race, which was established by measurement: a full cuda.synchronize() does not fix it and de-aliasing does. bitsandbytes keeps the packed 4-bit weight and its quantisation state on the autograd context as ordinary attributes instead of using save_for_backward, so gradient checkpointing cannot recompute them, and the reference captured during the forward aliases a streaming buffer that has been refilled with another layer by the time the backward reads it. bf16 was never affected, at 3.9 times the bytes per layer, because it goes through a native operation that does use save_for_backward. The obvious repair, giving every pooled tensor a private copy, was measured and rejected: it costs only about 6% of throughput but takes peak VRAM from 4,220 MiB to 19,720 MiB on a real 32B, which is approximately the whole model, and a repair that is proportional to the model rather than to one layer deletes the reason the feature exists. The shipped repair keeps streamed NF4 weights out of that code path entirely and is gated at 32B and 72B against a control that reproduced the defect in the same process, at 256 of 256 and 320 of 320 gradient tensors exact over five repetitions each, costing about 3% of peak VRAM and about 5% of throughput. It shipped in v0.73.0, so v0.72.0 through v0.72.4 carry the defect at those sizes and anyone who streamed a 32B or larger model in NF4 on one of those should re-run it. The upstream half is filed as bitsandbytes issue 2034 and remains open. One comparison came out of the same session: against DeepSpeed ZeRO-3 with CPU parameter offload, at matched numerics on a single card, layer streaming is 2.93x the throughput in 9.7x less peak VRAM. The right reading is not that streaming beats DeepSpeed, and the record says so explicitly. ZeRO-3 shards across ranks and had no partner to shard to, so it was used outside its intended regime; that regime happens to be layer streaming's home ground. On eight cards ZeRO-3 reached 1,831.7 tok/s using 34 GB on each of eight cards, against 1,077.1 tok/s and 2.9 GB for streaming on one, while one card training the model resident beat all of them at 2,645.4 tok/s. If the model fits on your card, use it resident and skip every clever technique here.

Question 5

Can I run DPO on a GPU that cannot hold the model (preference losses over layer streaming, v0.72.4)?

Accepted Answer

Yes, since Soup CLI v0.72.4. Set 'training.stream_layers: true' and 'task: dpo' in the same soup.yaml, and the frozen base never enters VRAM while DPO trains against it. ORPO, SimPO and KTO work the same way. Before v0.72.4 streaming meant supervised fine-tuning only: task had to be sft, and all four preference losses were refused when the config was read. The accepted list is now exactly five values, sft, dpo, orpo, simpo and kto. Note that ipo, bco and the unified 'task: preference' dispatcher are not in it, so name the loss in task directly, and that grpo and ppo are excluded permanently rather than pending a release, because generation rollouts re-read every layer once per generated token and destroy the amortisation streaming exists to buy. The interesting part is DPO's reference model. DPO scores the policy against a frozen reference, and the obvious implementation constructs a second copy of the model, which doubles the weights on a card that could not hold them once. Soup instead takes the reference from the same streamed base with its LoRA adapters switched off, so policy and reference share one set of weights, one CPU RAM store and one pool of VRAM buffers, and the adapter is the only difference between the two forward passes. The reference therefore needs no second copy of the model's weights (on an untied checkpoint, DPO and KTO add one output-head-sized buffer per step). That was measured with a control rather than asserted, because a passing loss curve cannot detect the failure mode: the streamed layer substitutes base weights through functional_call rather than through the module's own forward, so disabling the adapter being a silent no-op is entirely plausible, and if it were, the reference would be the policy, every log-ratio would be zero and the loss would sit at 0.6931 forever, which reads as slow training. Measured on a 4 GB RTX 3050 Laptop in bf16 with a 365.2M-parameter model, 24 decoder layers, vocabulary 260, sequence length 64 and batch 1: streamed supervised fine-tuning peaked at 89.53 MB, streamed DPO at 81.87 MB (0.914x) with a byte-identical 729.91 MB RAM store and 60.83 MB buffer pool, and the control that forced a real second instance at 812.32 MB, which moved the peak by 730.44 MB against 730.44 MB of weights, exactly one copy. A zero-adapter control measured a policy-versus-reference gap of exactly zero, proving the gap in the real run comes from the adapter and not from some other difference between the two passes. Read the 0.914x as evidence of no second copy, not as a saving: DPO is not cheaper than supervised fine-tuning in general, and that row sits below 1.0 only because the fixture splits its 64 positions into a 32-token prompt and a 32-token completion, so its logits tensor is smaller. The numbers come from a synthetic model, not from a real checkpoint and not from Llama-3.1-8B. Correctness: upstream's CHANGELOG states all four losses are bit-exact against a resident run of the same loss, and the benchmark record shows it for DPO, with a difference of exactly 0.0 (a forward-side comparison of the loss; adapter gradients were checked non-zero, not equal), and DPO's arm ran in float32 on CPU precisely so bit-exact means literally zero rather than bf16 noise. End to end through the released CLI, streaming SmolLM2-135M under NF4, dpo moved 0.6931 to 0.6695, orpo 5.3816 to 5.0559, simpo 5.2033 to 4.8749 and kto 0.5000 to 0.4904, and all four saved ordinary LoRA adapters that load into a non-streaming model. Two things to know before you configure it. First, the reference needs no second copy of the model but is not free in time: DPO walks the layer stack 1.52x as often per step as supervised fine-tuning does, because it adds a reference forward on top of the policy forward and the gradient-checkpoint recompute. Second, KTO is not reference-free however it is usually described, since it picks its reference exactly the way DPO does and gets the same treatment, while ORPO and SimPO genuinely never construct one; KTO additionally requires a batch size of 2 or more, because TRL's KL term is degenerate at batch 1, and Soup refuses that when the config is read rather than minutes into a run after the checkpoint has been sharded (since v0.76.0 for every KTO run, streamed or not). Two honest limits. No throughput figure is claimed for any preference loss at any model size, because none was measured. And the VRAM pre-flight is a sound upper bound for these losses but a loose one: it charges the supervised loss's 14 bytes per logit element over twice the rows, while TRL reduces preference logits to per-token log-probabilities with selective_log_softmax instead of holding a full-vocabulary fp32 upcast, so DPO's entire above-resident cost measured 51.76 MB where the charge for the same shape is roughly 458 MB. In practice, on a 4 GB card with a 128k-vocabulary 1B model, DPO is allowed at max_length 512 and refused from 768 up even though it would probably fit; lower data.max_length if the pre-flight refuses a run you believe fits. That conservatism shipped deliberately, because under-predicting is the strictly worse failure: on Windows an overcommit is not an exception you can catch, since WDDM silently spills into shared host memory and the run completes an order of magnitude slower with no error at all, which means a run completing is not evidence that its configuration fits. The measurement record behind all of this is published in full, including the checks that failed and the figures that were measured and then discarded, in the repository's benchmarks directory, and the layer-streaming work has a preprint under DOI 10.5281/zenodo.21771064.

Question 6

What are 'soup reward synth' and 'soup reward stress' in Soup CLI v0.71.40 and v0.71.41?

Accepted Answer

They are a matched pair (v0.71.40 and v0.71.41) that turns Soup's reward-hacking detector on the reward verifier itself, the thing an RL run trusts most and inspects least. 'soup reward synth -o reward.py' auto-generates a deterministic reward verifier from your own gold outputs (it reads each gold from the 'answer' field unless you pass '--field'): it infers one of four families, auto-detected or forced with --kind, and each is grounded in the golds. 'numeric' extracts the last number, a \boxed{} span or a #### answer and compares exactly or within --tolerance; 'json_schema' induces keys, types and required fields; 'regex' builds positional character classes over equal-length golds; 'tool_call' binds each tool's required and allowed arguments. The emitted .py is self-contained, readable and committable, and it rides the existing load_reward_fn .py path so it adds no new trusted-execution surface: you read, edit, diff and commit it. The moat is a mandatory calibration report: the synthesized verifier is loaded back and run against its own references (it must accept at least 90%) and against auto-generated bad answers (it must reject them), and a degenerate always-accept verifier is refused with exit 2, never silently emitted. '--plan-only' reports the induced spec without writing and '--output-report' persists the calibration JSON. 'soup reward stress --references golds.jsonl' (v0.71.41) is the adversarial half: where synth proves a verifier separates references from friendly perturbations, stress asks the question a reward-hacking model asks at train time, does the verifier pay out for degenerate junk? It feeds six families of degenerate completion scored against the real gold (empty, length-padded, repetition and sentinel-spam, plus two structure-preserving ones added in v0.76.0, wrapped_junk and answer_spray; '--attacks' takes a comma list and '--sentinel GOLD' sets the spam token), so a numeric, tool_call or json_schema verifier still gets a valid target and must reject the garbage, and it reports a per-attack accept-rate table plus an overall gameability verdict tuned by '--max-gameable' and '--threshold'. Since v0.76.0 the built-in 'format' reward, which awards 1.0 to 'a', is reported as gameable. It probes a synthesized .py and a builtin (accuracy, format, verifiable) through the same loader, and it is pure and offline: exit 0 for robust, 2 for gameable, 1 for error. A gold-requiring verifier probed with no --references is a hard error naming the flag, never a false 'robust'. v0.71.40 also made a comma-separated 'training.reward_fn: "accuracy,format"' actually train, resolving it to a reward ensemble (a GRPOTrainer with a list of reward_funcs) which also unlocks the rm_ensemble reward-hack detector, and it fixed a recipe that shipped exactly that form and previously crashed. Nothing in TRL, Unsloth, Axolotl or OpenRLHF synthesizes a reward or tests one for gameability.

Question 7

How does 'soup ship' decide SHIP or DON'T SHIP, and can I commit the evidence (v0.71.38, v0.71.39)?

Accepted Answer

'soup ship' answers one question after a fine-tune, did the model get better or did I break it, as a single binary SHIP or DON'T-SHIP with a one-screen reason. It fuses two legs: leg 1 is that the task metric strictly improved (an eval accuracy, an LLM-judge score, or a swap-debiased pairwise judge win-rate against a 0.5 base), and leg 2 is that no general benchmark regressed past a forgetting threshold (default 0.05 absolute points), so a model that wins the task but breaks general knowledge is refused. v0.71.38 gave leg 2 real teeth. It used to be fifteen trivia prompts scored by case-insensitive substring containment, which credited 'B' for Berlin and '3' for 13 and had zero coverage of tool-calling, JSON validity or safety; it is now an answer-extraction plus boundary-aware scorer over a bundled, offline general suite of eight hand-authored benchmarks shipped in the wheel (mini_mmlu, mini_common_sense, mini_instruction, mini_arithmetic, mini_tool_call for function calling, mini_format_json for JSON validity, mini_safety for refusal rate, and since v0.73.2 mini_over_refusal, which scores the fraction of benign-but-alarming-sounding requests NOT refused, because a model that refuses everything used to read as a monotone safety improvement), each roughly forty items or fewer (24 to 40) so a few wrong answers trip the default threshold instead of being rounded away, all scored offline with no lm-eval and no download. Usage errors moved to exit 3 so exit 2 now means only DON'T-SHIP; since v0.76.0 an unreadable or malformed --evidence file exits 3 too (it was 1). v0.76.0 also re-scaled three of the suites so a strong small model no longer sits at 1.000 (mini_common_sense and mini_arithmetic now have 40 items each and 24 of mini_tool_call's 40 rows were replaced), moved the bundled scorer to revision 5, and makes a --baseline stamped at an older revision warn, because it is on a different scale. v0.71.39 closed the evidence loop: 'soup ship --emit-evidence <path>' re-serializes the verdict back into the --evidence input schema so a run's output replays as input; a ShipConfig block under eval.ship in soup.yaml plus 'soup ship --config soup.yaml' lets you commit the gate policy with CLI over config over default precedence; and 'soup ship --push owner/repo#N' posts the verdict as a GitHub PR comment as a best-effort step that never flips the exit code. Emitting evidence stamps a provenance block with an order-insensitive config hash, the base model and a best-effort data hash, and reading evidence back with --config refuses (exit 3) evidence whose config drifted from the committed recipe, while excluding the gate policy from the hash so tuning the threshold never falsely invalidates evidence. 'soup ci init --config soup.yaml' binds the generated PR gate to that committed config, and since v0.73.3 the noise floor is committable there too as 'eval.ship.noise_floor'. That floor, added in v0.73.2 as 'soup ship --noise-floor N', re-runs the base model N times (N from 2 to 10) and refuses to call a delta significant below what the instrument itself can resolve, because greedy decoding is not deterministic on a GPU. v0.73.2 measured it on the leg-1 task axis in '--task-mode metric' only; v0.73.3 measures it in every task mode, scoring the base side N times through the judge in 'judge_score' and judging the base model against itself in 'pairwise', where the expected win rate is 0.5 by construction so the spread is measured rather than inferred. Because those repeats fold in the judge's own sampling noise, that floor is labelled 'decode + judge' on the panel and stamped judge_inclusive in the evidence, so it is never read as a decode-only number. It still sizes an effect rather than calibrating a threshold: n=3, one model, one dataset. That is CI for weights, not prompts: a provenance-bound SHIP gate that renders on every pull request. One honesty note: Soup sends nothing about your runs unless you ask it to. Telemetry was wired in v0.75.0 but is strictly opt-in, off unless SOUP_TELEMETRY=1, with a --no-telemetry option on the root soup command, so it goes before the subcommand (soup --no-telemetry train --config soup.yaml; after the subcommand it is refused), and it is hardware-only: seven fields covering version, command name, Python major.minor, OS, architecture, duration and a locally generated anonymous UUID, never dataset paths or contents, model names, config contents, usernames, local paths, IP addresses or credentials. As shipped it still sends nothing even when opted in, because the bundled project key is a placeholder the sender refuses by name, printing a notice instead of opening a socket.

Question 8

What is 'Data Moat II' in Soup CLI v0.71.36 (semantic dedup, topics, canary, replay)?

Accepted Answer

Data Moat II (v0.71.36) puts a semantic layer over your training data and adds two tools for what a fine-tune forgets and leaks, with zero new dependencies (it reuses transformers from the [train] extra, a mean-pooled all-MiniLM-L6-v2, so no sentence-transformers). 'soup data dedup --semantic -o clean.jsonl' removes near-duplicates over embedding cosine instead of MinHash shingling, catching reworded duplicates MinHash scores as distinct (measured 0.88 to 0.91 cosine on rewordings) while keeping genuinely-distinct-but-similar instructions; flags are --threshold (default 0.8), --field and --embed-model. The honest known limitation ships with it: it is not a paraphrase detector, because measured paraphrase cosines (0.49 to 0.76) overlap with genuinely distinct rows (for example 'Add two numbers' vs 'Multiply two numbers' scores 0.759, higher than the true paraphrase 'reverse a string' / 'invert the order of characters' at 0.491), so no threshold separates them and lowering --threshold to chase paraphrases deletes real training rows; the default 0.8 is deliberately conservative. 'soup data topics <data>' clusters the dataset with k-means over the same embeddings, labels each cluster with c-TF-IDF terms, and prints a coverage table (for example '82% code, 6% math') plus a warning for thin topics; the labels are emergent term clusters, not classification against a fixed taxonomy, and there is no automatic join to an eval-coverage suite. 'soup data canary insert train.jsonl -o canaried.jsonl --count 16 --manifest secrets.json' plants K high-entropy secrets, and after training 'soup data canary check --manifest secrets.json --base ./model --adapter ./lora' ranks each secret's loss against never-inserted controls drawn from the same space (a Secret-Sharer memorization probe); the verdict is a binomial-tail test at alpha 0.05 over the count of memorized canaries, with exit 2 on MAJOR so CI can gate a leak. That count-based verdict replaced an 'any single canary in the bottom 1%' rule that fired MAJOR on a clean model about 15% of the time at K=16 (a CI gate crying wolf one run in seven); the fix dropped the false-MAJOR rate to about 1.2%. 'No exposure' is the sampled-control approximation, not full-space rank enumeration, and the manifest is written 0600 on POSIX. 'soup train --config new_task.yaml --replay old_task.jsonl --replay-ratio 0.1' is continual-learning rehearsal: it interleaves a seeded sample of an old dataset into training so a new task does not erase the old one, where the ratio is the fraction of the final mixed set, rows are interleaved rather than appended, an undersized pool reports the shortfall instead of repeating rows, and validation stays pure new-task; replay v1 is sft/pretrain only and is incompatible with packing/multipack, and on SmolLM2-135M + LoRA it retained the old task 7% better than a no-replay control, the correct direction, but forgetting without it was only about +4%, so the effect size at full fine-tuning or 7B+ is proof-of-mechanism, not yet demonstrated at scale. Data Moat II also fixed the hardware-fit gate, which had refused to train any locally-merged model because the size guesser returned its 7B default for a name with no size marker; local checkpoints are now measured from their safetensors header. The follow-up patch v0.71.37 made every 'pip install soup-cli[extra]' hint use double quotes so it works on Windows cmd.exe, and fixed eval-gate benchmark tasks.

Question 9

How do I fine-tune a model for HIPAA, SOC 2, the EU AI Act or SR 11-7 with Soup CLI?

Accepted Answer

The v0.71.35 compliance pack covers it end to end. Start with 'soup init --template hipaa|soc2|eu-ai-act|sr-11-7'. Be clear on what a template is, because the honest constraint shaped the design: Soup's compliance controls are CLI flags and commands, not config keys (audit_log, bom, attest, repro_receipt, annex_xi, track_energy, pii and decontaminate have zero matches in the config schema), so a template cannot 'pre-wire audit-log on' as YAML. Each one is instead a valid, immediately trainable config on a license-clean Apache-2.0 base, plus header comments naming that regime's exact commands: PII flagging and air-gap for HIPAA, BOM/attest/sign for SOC 2, Annex XI plus energy tracking for the EU AI Act, repro-receipt plus diagnose and ship for SR 11-7. Built-in templates went from 17 to 21. The common path is then: 'soup data pii' (flags rows that match email, phone, SSN or card-number patterns; it does not redact) and 'soup data decontaminate --benchmark-file ' (drops rows that overlap a benchmark corpus you supply; Soup bundles none) to check the data; 'soup train --repro-receipt receipt.json' (adding '--annex-xi annex_xi.md --track-energy --energy-country DEU --energy-out energy.json' for the EU AI Act); 'soup registry push'; 'soup bom emit --format both' for CycloneDX plus SPDX and 'soup attest emit --sign ed25519' for in-toto plus SLSA-3; 'soup adapters scan/sign/verify'; 'soup airgap-bundle' if you need to cross an air gap; then 'soup card -o MODELCARD.md' to publish. The audit log records every command automatically ('soup audit-log tail'). Finally 'soup ci init' writes .github/workflows/soup-gate.yml, a pull-request gate chaining 'soup data validate', then 'soup expect', then 'soup ship --evidence', where exit 2 blocks the merge in each step and any other non-zero exit (a usage or configuration error) also fails the job and is labelled as one.

Question 10

What is 'soup card' and 'soup ci init' in Soup CLI v0.71.35?

Accepted Answer

'soup card -o MODELCARD.md' (v0.71.35) turns a Local Model Registry entry into a publishable, provenance-carrying Hugging Face model card: base model, the real training config, an eval scorecard joined from the tracker, config and data sha256 hashes, lineage (ancestors), and a table of every registered artifact. Adapter versus full model is inferred from the registered artifacts and falls back to the training config's LoRA rank, with Spectrum ('unfrozen_parameters') and LISA ('lisa_enabled') full fine-tunes correctly treated as dense, so the card sets the right 'library_name' and never misreports the model type. That detail was found by a live smoke, not by tests: a real LoRA run with no attached artifacts rendered 'Type | Full model' and 'library_name: transformers', a false claim in a provenance document that 90 green tests had missed. 'soup push --model ./out --repo you/m --card ' renders the same card and uploads it as README.md, overriding the auto-generated one; a bad ref fails fast before any network call and it is Hugging Face only (it warns and ignores on other hubs). 'soup ci init [--data d.jsonl --suite s.yaml --evidence ev.json] [--branch main --python 3.11] [--force]' writes .github/workflows/soup-gate.yml, a fine-tuning CI gate that runs 'soup data validate' for dataset format compliance, then 'soup expect' for PII, token-length and refusal expectations, then 'soup ship --evidence', where exit 2 blocks the merge in each step and any other non-zero exit (a usage or configuration error) also fails the job and is labelled as one. Every interpolated path is validated to stay under the repo root and shell-quoted (a '; rm -rf /' in a path is quoted as one token), the branch and Python version are regex-gated so they cannot break out of their YAML context, and the write is atomic, symlink-rejecting, and refuses to clobber an existing workflow without --force.

Question 11

What is 'soup draft', and does training a speculative-decoding draft actually make serving faster?

Accepted Answer

'soup draft' (v0.71.33) trains and, above all, MEASURES a speculative-decoding draft, and the honest answer to whether it makes serving faster is: measure it, because on the pair we validated it did not. 'soup draft measure --target --draft --prompts p.jsonl' reports a draft's acceptance rate (the fraction of the target's own greedy tokens the draft would have proposed correctly, teacher-forced argmax agreement, the metric the Medusa and EAGLE papers report) plus real plain-versus-assisted throughput measured on your box; there is no single acceptance rate where speculative decoding pays (the one pair upstream measured at scale scored 81% acceptance and ran at 0.48x), so since v0.76.0 it also times the draft on its own and reports the break-even acceptance rate at the --num-assistant-tokens in use and a modelled best draft length ('--sweep-k 1,2,3,5,8' measures assisted throughput at each length; the model assumes independent acceptance and excludes framework overhead), and a draft no faster than its target is reported as no acceptance rate paying; '--min-acceptance 0.6' still exits 2 below the floor so CI can gate on it (exit 0 = ok, 2 = below floor, 1 = error). 'soup draft distill --target --draft-base --data d.jsonl -o draft/' distils your target into a tiny base via logit KD through the existing 'task: distill' trainer and emits a dense model, because a PEFT adapter directory cannot be loaded as an assistant_model; flags include --steps, --device, --force and --plan-only. 'soup draft list' shows local drafts, and a registry at ~/.soup/drafts.json is consulted by 'soup serve --auto-spec' BEFORE the built-in pairing table, so a draft you trained yourself is picked up automatically. Draft and target must share a tokenizer and a mismatch is refused up front, because speculative decoding proposes draft token ids into the target's vocabulary and a mismatched pair silently produces garbage rather than failing. The measured result, which is the feature: on SmolLM2-360M-Instruct with a SmolLM2-135M-Instruct draft, the stock draft already scored 69.3% acceptance and distilling it moved that to 69.7% after 2 epochs and back to 69.3% after 10, no gain beyond noise, because a small same-family draft is already near its capacity ceiling for agreeing with the target and logit KD cannot buy capacity it does not have; assisted decoding measured 0.55 to 0.64x, a net slowdown, since the draft's forward pass costs more than the tokens it saves at that size. The 'faster serving' pitch was therefore withdrawn rather than shipped, and 'soup draft measure' is what tells you so before you ship. Whether distillation pays off on a larger or genuinely diverged pair is unproven on a 4 GB box; cross-tokenizer drafts were deferred at the time and shipped in v0.74.0, routed through Wasserstein-aligned logits.

Question 12

What is 'soup adapters arithmetic' (LoRA task arithmetic) and LISA in Soup CLI v0.71.34?

Accepted Answer

'soup adapters arithmetic "coder + 0.5*math - toxic" --adapter coder=./coder-lora --adapter math=./math-lora --adapter toxic=./toxic-lora -o ./blended' (v0.71.34) applies task arithmetic (arXiv:2212.04089) to LoRA deltas: add to blend two skills, scale to dial one down, and, the differentiator, negate to subtract one. The output is a single loadable adapter (exit 0 = ok, 1 = refusal). The coefficient math is the point: a LoRA does not store its delta directly but two factors whose product is the effective weight change (delta W = B @ A), so scaling both factors by c scales the delta by c squared, which would make '- toxic' square to +1 and become a no-op instead of a removal. Soup splits the coefficient as the square root of its absolute value across the two factors and carries the sign, so the delta scales linearly (verified live on two real rank-8 adapters: the ratio of the norms for 2*a versus a is exactly 2.000). That exactness holds on the self (diagonal) term, so scaling or negating a single adapter is exact, while a multi-adapter sum also carries cross-terms this element-wise combine does not cancel; and the mechanism is what is proven (the delta really negates), not the behavioural outcome, since whether subtracting a toxicity adapter measurably reduces toxicity on a real safety eval is not yet benchmarked. Guards: a different base model is refused unless --allow-cross-base, a FAIL-scanned input is refused by the backdoor-scan gate unless --allow-unscanned, and paths are cwd-contained and symlink-rejecting. The expression parser is hand-written with no eval, so an adapter name can never become code. v0.72.0 removed the one real limitation: mixed ranks used to be refused with a 'harmonize rank' message, and they are now handled exactly. Stacking the A factors and the scaled B factors of the inputs produces a combined adapter whose delta reproduces the sum of the individual deltas by construction, with no approximation, and the new '--rank N' flag then truncates that result with an SVD when you want a fixed output rank rather than the concatenated one. Ranks are no longer a reason to refuse; genuinely incompatible layer shapes still are.

Question 13

What is LISA in Soup CLI (v0.71.34)?

Accepted Answer

LISA (Layerwise Importance Sampled AdamW, arXiv:2403.17919) is the other half of v0.71.34 and reaches for full-fine-tuning quality without a full fine-tune's memory: where Spectrum picks the layers once, LISA re-samples a small random set of decoder layers every N steps and freezes the rest, with the input embeddings, LM head and final norm always trainable. Because only a handful of layers train at any moment and their optimizer state is cleared when re-frozen, peak optimizer memory is roughly embeddings + head + lisa_num_layers, while every layer still gets updated over the run. Read that arithmetic carefully, because it is true and it is not a saving over LoRA: measured on an H100 at 3B and 8B, LISA beats full fine-tuning on held-out loss at both learning rates, and it costs 1.22x LoRA's memory at 3B and 1.51x at 8B, with the gap widening because the embeddings, the LM head and the final norm stay trainable through every interval and are 70.7% of everything LISA trains at 8B. Both LoRA rows in that comparison were measured before v0.74.0 fixed a frozen base being loaded in fp32, so a LoRA run today needs less memory than they show; no repeat of the comparison is recorded. Stated positively: LISA trains an 8B on one 80 GB card where full fine-tuning needs roughly 120 GB and cannot run at all. Enable it with 'training.lisa_enabled: true' plus 'lisa_num_layers', 'lisa_interval_steps' and 'lisa_reset_optimizer' on a task: sft or, since v0.74.0, task: pretrain run, with transformers, text and quantization: none; it is mutually exclusive with LoRA features, freeze_layers/freeze_ratio and Spectrum's unfrozen_parameters, since each independently decides what trains. Live on a 4 GB GPU for small models.

Question 14

Does 'soup export --format gguf' work on Windows?

Accepted Answer

Yes, as of v0.71.35, and it was validated end to end on Windows for the first time: llama.cpp built CPU-only with VS2022 BuildTools and cmake, SmolLM2-135M exported to q4_0 (92 MB), q4_k_m (105 MB), q8_0 (145 MB) and f16 (271 MB), then 'soup deploy ollama', then live inference. Doing that validation surfaced four real bugs, each of which independently broke the path, and which 361 green export and ollama tests had not caught. First, 'soup export --format gguf' cloned llama.cpp into your current directory: SOUP_DIR is the bare name '.soup' but the lookup used it relatively instead of anchoring to the home directory like the rest of the codebase, so the canonical ~/.soup/llama.cpp was never found and a fresh ~200 MB checkout was dropped into whatever directory you ran from. Second, and worst, the first GGUF export downgraded your PyTorch and broke CUDA: the auto-clone ran 'pip install -r <llama.cpp>/requirements.txt' into your interpreter, and llama.cpp pins torch~=2.2.1 against the CPU wheel index, so torch 2.5.1+cu became 2.2.2+cpu and transformers 4.57 became 4.46, meaning a user's first GGUF export silently destroyed their training setup; Soup now installs only the convert script's extra dependencies (gguf, sentencepiece, protobuf), unpinned, never touching torch, and non-fatally. Third, a correctly built llama.cpp was 'not found' on Windows because MSVC, like Xcode, is a multi-config generator and emits build/bin/Release/llama-quantize.exe, while only the flat single-config layout was searched. Fourth, 'soup deploy ollama' failed on a relative GGUF path with 'pull model manifest: file does not exist', because Ollama resolves FROM against the Modelfile's directory and Soup writes the Modelfile to a temp dir; the Modelfile now emits an absolute path. This closes the CPU-validatable half of the long-standing GGUF-on-Windows issue; the CUDA half remained open at the time, and the AWQ/GPTQ half is superseded because 'soup export --format awq' and '--format gptq' are deprecated in v0.76.0 and their removal is tracked for v0.77.

Question 15

Can Soup CLI fine-tune Whisper (ASR / speech-to-text) models?

Accepted Answer

Yes. As of v0.71.32, 'soup train' with 'task: asr' fine-tunes OpenAI Whisper speech-recognition models on your own audio, locally. Data is plain JSONL rows of {"audio": "clip.wav", "text": "the transcript"} with 'data.format: asr' and a containment-checked 'data.audio_dir'; the trainer wraps Hugging Face Seq2SeqTrainer + WhisperProcessor (clips become 16 kHz mono log-mel input features, transcripts become the decoder labels). 'training.asr_lora: true' trains LoRA on the q/v projections instead of a full fine-tune, so whisper-tiny (39M) and whisper-base (74M) train on a 4 GB GPU; 'training.asr_language' and 'training.asr_task' (transcribe or translate) pin the decoder prefix and persist to an 'asr_generation.json' sidecar that inference restores automatically. 'soup infer --task asr --model <whisper-or-adapter> --input eval.jsonl --output preds.jsonl' transcribes a JSONL of clips and, when references are present, reports per-row and corpus WER/CER via a pure-Python metrics module with no new dependency; word accuracy (1 - WER) plugs into 'soup ship' as a higher-is-better task metric. Recipes: 'whisper-tiny-asr' and 'whisper-base-asr' (live-validated on a 4 GB RTX 3050: WER 1.000 to 0.000 on a memorization smoke at 0.4 of 4 GB) plus 'whisper-large-v3-asr' (parse-tested, needs 16 GB+). Audio decodes only through a hardened loader (soundfile pre-probe, symlink rejection, O_NOFOLLOW, byte caps); a non-Whisper base or an mlx/unsloth backend is rejected with a friendly error before any download.

Question 16

What is the judge-in-the-loop suite in Soup CLI v0.71.31 (Online DPO, best-of-N, Evol-Instruct, soup ship pairwise)?

Accepted Answer

The judge-in-the-loop suite (v0.71.31) puts an LLM judge in the loop across the whole post-training workflow as four integrated commands sharing one swap-debiased pairwise-judge core, and no competitor (Unsloth, Axolotl, LLaMA-Factory, OpenPipe) ships it as an integrated CLI suite. 'soup train with task: online_dpo' trains on-policy against a judge: each step samples two completions per prompt and a pairwise judge (or a reward model) names the winner 'chosen' and the loser 'rejected', wrapping TRL's OnlineDPOTrainer (config online_dpo_judge or reward_model, exactly one of the two; online_dpo_loss_type sigmoid or ipo; online_dpo_max_new_tokens; beta reuses dpo_beta; transformers and text only). 'soup data best-of-n --base --prompts p.jsonl --n 8 --judge -o sft.jsonl [--emit-pairs dpo.jsonl]' is best-of-N rejection sampling (BOND-lite): sample N completions locally, a judge scores each one pointwise, the winner becomes an SFT row with provenance and, with --emit-pairs, winner-versus-loser DPO pairs. 'soup data evolve --input seeds.jsonl --provider ollama|vllm --model --strategy depth|breadth --rounds N -o out.jsonl' is Evol-Instruct (WizardLM) instruction evolution that grows instruction diversity, completing the synthetic-data suite (Magpie, Forge, Persona, evolve). And 'soup ship --task-mode pairwise --judge-model ' adds a third task-win mode to the SHIP or DON'T-SHIP verdict: a true pairwise judge win-rate where the judge picks base versus tuned per prompt (swap-debiased, a win counts only when the (A,B) and (B,A) orders agree), the base is a 0.5 coin-flip and the tuned model wins only if its win-rate exceeds 0.5, fused with the catastrophic-forgetting gate and CI exit codes (0 = SHIP, 2 = DON'T SHIP, 1 = error). It is honest about scale: the training claim is proof-of-mechanism only, validated on SmolLM2-135M with a synthetic length-preferring judge on CPU, not a production RLHF claim (community scale ask tracked in issue #286). On the supported trl range (0.29 up to, but not including, 1.0) it uses TRL's pairwise-judge path with Soup's swap-debiased judge; the code also carries a pointwise reward-function adapter for trl 1.x, which is outside the declared range. Judge and provider URLs are SSRF-validated, dataset paths are cwd-contained with symlink rejection and atomic writes, and untrusted model code defaults to trust_remote_code off. It adds the online-dpo-smollm2-135m recipe.

Question 17

What is PRM-guided GRPO in Soup CLI v0.71.30?

Accepted Answer

PRM-guided GRPO (v0.71.30) lets a trained Process Reward Model drive GRPO reinforcement learning, the o1-era process-supervision signal, and no OSS fine-tuning CLI ships this. 'soup train with task: grpo' with training.prm_reward set to a PRM directory or Hugging Face id (from 'soup train with task: prm') and training.prm_aggregate set to min, prod or last scores each reasoning step of a generated answer and folds the per-step scores into one reward that GRPO optimizes, replacing reward_fn; the default aggregation 'min' is weakest-link and the safe choice, and it is gated to grpo, transformers and text. v0.71.30 also bundles three ready-to-run toy rollout environments (soup_cli.envs.calculator, retrieval_qa and guess_number) so the GRPO openenv rollout path works with no external setup, with the recipes grpo-env-calculator, grpo-env-retrieval-qa and grpo-env-guess-number. It is proof-of-mechanism on SmolLM2-135M with a tiny synthetic PRM and synthetic reward, not a production reward-model claim (scale ask tracked in issue #286).

Question 18

What is 'soup shrink' in Soup CLI v0.71.29, and how does depth pruning work?

Accepted Answer

'soup shrink --model [--drop-ratio 0.25 | --drop-layers N] --calib calib.jsonl [--heal heal.jsonl --heal-steps 200] [--tolerance 0.10] [-o shrunk] [--device cpu] [--attach-to-registry ] [--plan-only]' (v0.71.29) makes a model permanently smaller and faster by removing its least-useful decoder layers, optionally healing the damage by distillation, and printing a single SHIP / DON'T-SHIP perplexity verdict. It implements 'The Unreasonable Ineffectiveness of the Deeper Layers' (Gromov et al., arXiv:2403.17887): over a calibration set it ranks every contiguous block of decoder layers by the mean per-token angular distance the residual stream travels from the block's input to its output, drops the least-important block (the first and last decoder layers are always protected), reloads the sliced model so surviving layers re-index cleanly, and with '--heal' distills the full-depth original into the pruned student via an isolated LoRA logit-distillation 'soup train' subprocess whose adapter is then fused back — so the shipped artifact stays a single dense smaller model, not a base plus adapter. The verdict SHIPs only if the perplexity ratio (final / original) stays within '--tolerance' (default 10%), else DON'T SHIP; exit codes are 0 = SHIP, 2 = DON'T SHIP, 1 = error, and '--plan-only' prints the layer-importance table and the chosen block without writing anything. The arch allowlist v1 is Llama / Qwen / SmolLM (checked twice). It is validated live on SmolLM2-135M on a single RTX 3050: drop 25% took it from 30 to 22 layers (~21% of params) at perplexity x2.98 unhealed, while drop-4 plus a CPU heal recovered perplexity to x1.35. No fine-tuning CLI (Unsloth, Axolotl, LLaMA-Factory) ships one-command importance-ranked depth pruning with a distill-heal and a binary perplexity verdict. Security: --calib, --heal, --output-dir and every derived write path stay under cwd with realpath + commonpath + O_NOFOLLOW + symlink rejection re-validated right before each write (TOCTOU defence, including after the heal), the heal subprocess is an argv list with a timeout and a schema-validated config, subprocess output is control-char-stripped, input files are capped at 64 MiB / 10,000 calib rows, and --model defaults trust_remote_code off.

Question 19

Does Soup CLI have an MCP server, and what does 'soup mcp serve' do?

Accepted Answer

Yes. 'soup mcp serve [--allow-mutating]' (v0.71.28) is a Model Context Protocol server that lets you drive Soup from any MCP client (Claude Code, Cursor, Cline, Continue) over stdio by default (or a bearer-gated sse or http listener, described below), so your coding agent can inspect data, search recipes, read runs and give ship verdicts without leaving the chat. Since v0.73.3 it exposes 18 tools: 14 read-only ones that each map to a Soup command and return JSON (advise, data_inspect, data_validate, data_score, data_doctor, recipes_search, recipes_show, runs_list, runs_show, registry_list, registry_show, profile, diagnose_evidence, ship_evidence); 2 planning tools (train_start, export) that are always listed but refuse to run unless you pass '--allow-mutating', and even then only render the exact command that would run; and 2 executing tools (train_execute, export_execute) behind '--allow-execute', which was a reserved gate in v0.73.2 and opens in v0.73.3. An executing tool accepts only a confirmation_token: the token is server-generated, random, single-use, expires in five minutes and is bound to both the plan and the execution kind, so no command, argv, shell string or client-supplied environment ever reaches the process. The validated config is snapshotted at plan time and the run executes from that copy, protected inputs are revalidated by a content hash that walks a directory tree rather than by timestamp and size, the child is spawned with shell=False and stdin closed with output redirected to a log under .soup/mcp-runs, the token is consumed before the spawn so a failed spawn needs a fresh plan, only one execution runs at a time gated on a persisted run whose process is still alive so a restarted server cannot double-book a GPU, and the run goes through the normal experiment tracker so 'soup runs' sees what MCP started. '--allow-mutating' on its own can never spawn a process. One caveat for anyone auditing this against the binary: the shipped '--help' text for '--allow-execute' was not updated by the feature and still describes the pre-v0.73.3 reserved gate. Stdio is the default and opens no port; v0.74.0 added two network transports, '--transport sse' and '--transport http', each requiring an 'Authorization: Bearer' header compared in constant time with no opt-out, with the SDK's DNS-rebinding protection on (421 on a foreign Host, 403 on a foreign Origin) because a page the operator merely visits sends no auth header and its request still reaches the port, and with warnings on an off-loopback or wildcard bind. '--allow-execute' is refused with either network transport, in the command and again in the app factory, because behind a listener a leaked token would mean process execution rather than plan disclosure; '--host', '--port' and '--auth-token' are refused under stdio rather than silently ignored. On stdio, stdout is reserved for the JSON-RPC channel (all human text goes to stderr), every path argument re-enters cwd-containment plus symlink rejection, output is control-char sanitized so a malicious dataset string cannot smuggle terminal escapes into the client, errors are path-free, and string (<= 4096 chars) / JSON (<= 16 MiB) / dataset (<= 1 GiB) / int bounds are enforced. It installs behind a new '[mcp]' extra ('pip install "soup-cli[mcp]"', dependency mcp>=1.10.0,<3 since v0.74.0, which bridges both SDK majors by probing the server constructor rather than reading a version string) that is lazy-imported so the core CLI stays PyTorch-free and SDK-free. A client connects with { "mcpServers": { "soup": { "command": "soup", "args": ["mcp", "serve"] } } }. The server exposes MCP Tools only — no Resources or Prompts. A crash between spawning a training job and recording its process id used to let a restart double-book a second job, so an unresolved launch now counts as active and 'soup mcp runs reconcile --expunge-launching' recovers capacity from stale rows, refusing the whole operation if any candidate records a process that is still alive.

Question 20

What is the Fine-tune Doctor (soup data doctor / soup data lint) in Soup CLI v0.71.27?

Accepted Answer

v0.71.27 'Fine-tune Doctor' is a pure-CPU, zero-GPU pre-flight that kills the top silent fine-tune failures before a single training step — something no competitor (Unsloth, Axolotl, LLaMA-Factory) ships. 'soup data doctor --model ' runs 8 chat-template compatibility checks against the real tokenizer: chat_template present, template renders cleanly, {% generation %} markers present, eos_in_labels (the number-one 'model never stops generating' bug — every assistant turn's trained span must actually contain an EOS/EOT token, checked on every turn not just the last), bos_duplication (template and tokenizer both prepending BOS), system_role support (Mistral-style templates reject a leading system turn), unknown_roles, and truncation_risk (p95 rendered length vs max_length). It uses the same OK/MINOR/MAJOR taxonomy as 'soup diagnose' with exit 0 on OK/MINOR and exit 2 on MAJOR, and '--show-mask N' colours each token trained-versus-masked through the REAL collator path (answer-only / per-message train-field / RAFT span-mask), not a reimplementation, so an assistant-mask bug is visible instantly. 'soup data lint ' is a preference-data linter for dpo/orpo/simpo/ipo/bco/kto that flags length_bias (chosen systematically longer than rejected, the number-one silent DPO degradation, reported as a Cohen's d effect size), label_imbalance (KTO desirable:undesirable ratio), near_duplicates (MinHash/LSH, reusing the dedup kernel), identical_pairs (chosen == rejected, zero preference signal), and prompt_leak (prompt echoed verbatim in the completion); pass '--model' for exact token-length bias instead of word count.

Question 21

What is 'soup spectrum scan' and how does Spectrum targeted training work?

Accepted Answer

'soup spectrum scan --model --top-percent 50 [--modules mlp,attn|all] [--output patch.yaml]' (v0.71.23) is native Spectrum targeted training. It streams a model's safetensors shards one tensor at a time — no model load, so peak RAM is the largest single weight matrix and you can scan even a 70B on a CPU box — computes a singular-value SNR per weight matrix with a Marchenko-Pastur noise threshold (arXiv:2406.06623), ranks layers within each module-type group, and prints the top --top-percent as a ready-to-paste 'training.unfrozen_parameters' YAML block. The SNR kernel is pure-numpy and transpose-invariant, so GPT-2 Conv1D weights (c_attn/c_fc/c_proj) are recognised alongside Llama-style names; results cache at '~/.soup/spectrum/.json'. The new schema field 'training.unfrozen_parameters: list[str]' takes regex patterns of parameter names to keep trainable: the SFT trainer freezes every parameter then unfreezes the matched set (a full fine-tune of a subset of layers, LoRA off). It requires task=sft, backend=transformers, modality=text, quantization=none and is mutually exclusive with LoRA features / freeze_layers / freeze_ratio / train_router_only / expand_layers. Patterns are ReDoS-validated (nested-unbounded-quantifier reject, 50k count cap, 512-char cap); Hub downloads route through the SSRF-hardened namespace-pinned loader and matrices above a 2^31-element SVD cap are skipped.

Question 22

How does 'soup train --reward-hack-detector' detect reward hacking in GRPO?

Accepted Answer

'soup train --reward-hack-detector info_rm|rm_ensemble [--reward-hack-halt]' (v0.70.0) is an early-warning system for reward hacking in GRPO runs (a 'task: ppo' run refuses it since v0.76.0). Two detectors: 'info_rm' tracks the InfoRM Cluster-Separation Index (Wang et al. 2024, arXiv 2402.09345) — a calibrated metric on reward-model embeddings that drops when the policy collapses onto a degenerate reward-maximizing subspace; 'rm_ensemble' tracks the mean pairwise variance across an RM ensemble (cap 32) — when ensemble members disagree, the policy is exploiting one of them. Math kernels 'compute_cluster_separation', 'compute_rm_ensemble_divergence', and 'classify_hack_signal' are LIVE with OK/WARN/HACK bands at 0.10 / 0.30 relative drop from the baseline window. '--reward-hack-halt' stops training when the verdict reaches HACK. Cross-validator: the config accepts task grpo or ppo only (and 'soup train --help' still says GRPO/PPO), halt=True requires detector, rejects mlx backend; but since v0.76.0 'task: ppo' refuses the detector at trainer setup, before the model loads, because the trl PPO trainer takes no reward functions and the signal buffer was never fed, so use 'task: grpo'. Composes with v0.34 'soup why' for anomaly explanation. LIVE as of v0.71.11 — the GRPO TrainerCallback now reads per-step rewards via a thread-safe capture buffer, classifies OK/WARN/HACK, logs the verdict, and halts on HACK.

Question 23

What is closed-loop reward-hacking auto-mitigation in Soup CLI v0.71.26?

Accepted Answer

'soup train --reward-hack-mitigation off|log_only|kl_control|pid_lagrangian' (config field 'training.reward_hack_mitigation', v0.71.26) makes a GRPO trainer DETECT reward hacking mid-run and SELF-CORRECT instead of only halting. No open-source RLHF library (TRL, Unsloth, Axolotl, OpenRLHF, verl) closes the loop from detection to automatic correction to continue — that is the whitespace it fills. Four modes: 'log_only' streams a per-step mitigation_log.jsonl (drop_pct, OK/WARN/HACK verdict, reward mean/std, completion-length trend, n-gram repetition) and provably never touches training dynamics; 'kl_control' is a reversible bang-bang controller with hysteresis that, when a multi-signal vote (InfoRM cluster-separation or RM-ensemble divergence plus length-trend and repetition) trips for a dwell window, raises the KL coefficient beta (clamped to a positive floor and ceiling, never crossing zero, written to both trainer.beta and trainer.args.beta for stock and variant GRPO) and relaxes it once the signal recovers; 'pid_lagrangian' replaces bang-bang with a PID-Lagrangian controller (Stooke et al. 2020) that holds the hacking signal at a target with integral anti-windup, plus an escalation ladder (raise KL, then roll back to the last-good RL checkpoint, then early-stop with a plain-English give-up explanation via 'soup why'). Anti-gaming hardening adds per-signal EMA/median smoothing, conservative-on-disagreement voting, a reward-distribution-drift guard, and optional bounded reward shaping on the gamed proxy. It ships as proof-of-mechanism ONLY, validated on SmolLM2-135M plus a synthetic length-hacking task on a single RTX 3050 (all four modes live including a real mid-run rollback); since v0.76.0 'task: ppo' refuses 'reward_hack_mitigation' (and the detector and the echo-trap flags) before the model loads, because the trl PPO trainer cannot feed the signal buffer, so the mitigation runs on 'task: grpo' only, and scale validation on 7B+ with real reward models is an open community question tracked in issue #286. +184 tests in that release; the suite now spans 875 test files.

Question 24

What is 'training.uld_strategy' (Universal Logit Distillation) in Soup CLI v0.70?

Accepted Answer

'training.uld_strategy' in soup.yaml selects a Universal Logit Distillation loss (Boizard et al. 2024, arXiv 2402.12030) on 'task: distill', run with 'soup train --config soup.yaml'. There are three strategies and only one of them is cross-tokenizer, which matters because picking the wrong one used to train silently on the wrong text. 'wasserstein_aligned' is the cross-tokenizer strategy: it re-tokenizes and aligns decoded character spans, so Llama to Mistral, or GPT-2 BPE to Llama SentencePiece, is a real comparison. 'wasserstein' (a 1-D Wasserstein distance over sorted teacher / student logits, no alignment, cheap) and 'topk_align' (top-K teacher logits matched by a BPE-overlap heuristic, plus 'training.uld_top_k: N') both forward the student's own token ids to the teacher unchanged, so they are correct ONLY when the two tokenizers are interchangeable, such as two sizes in one family. Until v0.75.0 they clamped a mismatched id into the teacher's vocabulary range instead, which does not translate a token between vocabularies, so the teacher was conditioned on the wrong text while the loss stayed finite and plausible; since v0.75.0 both refuse a non-interchangeable pair at setup and name 'wasserstein_aligned' instead. v0.75.0 also made the ULD loss respect the response-only label mask and the causal shift, which it had ignored, so under the default 'train_on_responses_only' it had been optimising prompt tokens too. _MAX_VOCAB_SIZE=262144 covers multilingual SentencePiece + GPT-OSS 200K vocabularies. All three are gated to task='distill' and reject the mlx backend. Live as of v0.71.11.

Question 25

How does 'training.minillm_enabled' stabilize reverse-KL distillation?

Accepted Answer

The config keys 'training.minillm_enabled: true' plus 'minillm_teacher_mix_ratio: 0.3', 'minillm_length_normalize: true', 'minillm_pretrain_anchor_weight: 0.1' and 'minillm_pretrain_anchor_path: pre.jsonl' (v0.70.0) is on-policy reverse-KL distillation (Gu et al. 2024, arXiv 2306.08543) with all three §3 stability tricks bundled — teacher-mixed sampling (mix ratio of teacher samples into the on-policy rollout), length normalisation (per-token KL averaged), and pretrain-loss anchor (regularise toward an anchor distribution at weight α). Cross-validators reject silent no-ops: anchor_weight=0 with anchor_path set, or anchor_weight > 0 with path None. Gated to task='distill'. LIVE as of v0.71.11 — the teacher-mixed, length-normalised reverse-KL term plus the optional pretrain anchor now train end-to-end (the anchor reader is cwd-contained + symlink-rejecting).

Question 26

What does 'training.rl_checkpoint_save_every_steps' fix that TorchTune punts on?

Accepted Answer

The config keys 'training.rl_checkpoint_save_every_steps: N' plus 'rl_checkpoint_keep_last', 'rl_checkpoint_include_optimizer', 'rl_checkpoint_include_ref_model' and 'rl_checkpoint_include_rollout_buffer' (v0.70.0) adds mid-epoch checkpointing to PPO and GRPO — a feature TorchTune explicitly punts (their docs link out to 'restart the epoch on crash'). save_every_steps ∈ [1, 10M], keep_last ∈ [1, 100] (oldest pruned). Optional inclusions: optimizer state, reference model snapshot, rollout buffer. At v0.76.0 Soup restores these checkpoints automatically in one place only, the rollback rung of the 'pid_lagrangian' reward-hack mitigation ('training.reward_hack_rollback: true', which requires this cadence to be set). The v0.32 loss-spike recovery writes a hint file and does not read them back. LIVE as of v0.71.11 — it writes a real adapter + optimizer state + JSON manifest every N steps and prunes to 'rl_checkpoint_keep_last'. All of these are config keys under 'training.', not CLI flags on 'soup train'.

Question 27

What is 'soup iterative-dpo' and how does it differ from a single DPO run?

Accepted Answer

'soup iterative-dpo --base-model --reward-model --prompts --output-dir --rounds N --pairs-per-round N [--plan-only]' (v0.70.0) drives the iterative-DPO loop: sample from the current policy, score with the reward model, build new preference pairs, retrain DPO, repeat for N rounds. Frozen IterativeDPOPlan with a consecutive-round_index invariant and canonical per-round artifacts ('./out/round-00/pairs.jsonl', './out/round-00/adapter', numbered from zero). There is no resume: the runner does not skip a round whose adapter already exists, so a crashed run starts again from round-00. '--plan-only' renders the validated plan and exits 0; the live runner shipped in v0.71.11 — each round samples completions from the previous round's adapter, then trains a fresh LoRA from the base on that round's harvested pairs.

Question 28

How does 'training.echo_trap_enabled' detect multi-turn agent degeneration?

Accepted Answer

The config keys 'training.echo_trap_enabled: true' plus 'echo_trap_threshold: 0.6' and 'echo_trap_halt: true' (v0.70.0) is the RAGEN-style trajectory-degeneration detector for multi-turn agent RL (Zhu et al. 2025, arXiv 2504.14437). Pure-Python n-gram repetition rate per trajectory + a batch mean — when an agent's rollout collapses into 'echoing itself' (the same n-gram pattern appearing repeatedly within and across turns), this catches it before the reward model rewards the degenerate policy. OK / WARN / TRAP bands at 0.30 / 0.60. DoS caps _MAX_NGRAM_N=32, _MAX_TRAJECTORY_TOKENS=1M, _MAX_BATCH_TRAJECTORIES=100k. Gated to task in {grpo, ppo} non-mlx; since v0.76.0 'task: ppo' refuses 'echo_trap_enabled' before the model loads (the PPO trainer cannot feed the trajectory buffer), so use 'task: grpo'. Composes with v0.53.11 GRPOStabilityCallback. LIVE as of v0.71.11 — the GRPO callback now scores per-trajectory n-gram repetition, logs the verdict, and halts on TRAP when 'echo_trap_halt' is set. These are config keys, not CLI flags: the only echo-trap option on 'soup train' itself is '--echo-trap-tokenizer-aware'.

Question 29

What is 'soup build' and how is it dbt-shaped?

Accepted Answer

'soup build <manifest.yaml> [--dry-run]' (v0.69.0) is a dbt-shaped DAG of dataset transforms with incremental materialisation. Closed SUPPORTED_MODEL_KINDS = {incremental, table, view}. Topo-sort via Kahn's algorithm. The 're-tokenise only changed rows' kernel: compute_row_hash (SHA-256 over canonical-JSON, 'id' field excluded) + incremental_diff(prev, new) → {added, changed, removed, unchanged}. DoS caps _MAX_MODELS=256, _MAX_REFS_PER_MODEL=32, _MAX_FILE_BYTES=1 MiB. '--dry-run' validates topology and exits 0. LIVE as of v0.71.6 — run_build materialises datasets with five built-in transforms, and that is the whole list: identity, drop_empty, lowercase, add_field and token_count. 'table' rebuilds, 'view' re-derives (and is never written to a file, so end a build that feeds training in a 'table' or 'incremental' model), 'incremental' re-transforms only the rows whose content hash changed (SQLite state store keyed by row hash + transform fingerprint). A custom transform can be passed per-run via the Python API OR named in the manifest as a dotted import path, 'module.path:function_name', which the CLI resolves lazily and which is arbitrary code execution, so only run a manifest you trust. An unknown transform name is caught by the runner rather than the plan, so '--dry-run' passes a manifest whose real run fails.

Question 30

What does 'soup expect' assert about a chat dataset?

Accepted Answer

'soup expect ' (v0.69.0, LIVE) is a Great Expectations suite for chat data with a closed SUPPORTED_EXPECTATIONS allowlist: expect_no_pii (reuses v0.47 Presidio analyzer), expect_token_length_between (min_tokens / max_tokens bounds over whitespace-separated words, defaulting to 1 through 1,048,576 so an empty row fails), expect_no_refusal_pattern (reuses v0.56 refusal detector), expect_chosen_preferred_over_rejected_by_judge (it needs a judge: pass --judge or set args judge in the suite entry; with none it flags every preference row as a violation and exits 2, unless the entry sets args advisory: true, which opts back into only checking that every row has both chosen and rejected). Each suite entry must be a mapping with a name and optional args; a bare name is refused with exit 3. Walks 'text', 'content', 'output', 'prompt', 'instruction', 'response' top-level keys + 'messages[].content' nested arrays. _MAX_SUITE_LEN=64. Since v0.76.0: exit 0 = all expectations passed, 2 = an expectation failed, 3 = a usage or input error (an invalid or missing suite, a missing or unparseable dataset); exit 1 means the --judge backend was unreachable; before v0.76.0 a failure was 3 and a rejected suite was 2, so a CI script that compares the code needs a look. Drop into CI between 'soup data' and 'soup train'.

Question 31

How do 'soup data gen-magpie' and 'persona-mix' improve synthetic data diversity?

Accepted Answer

'soup data gen-magpie --base --provider ollama|vllm --target N [--quality-filter] [--plan-only]' (v0.69.0, LIVE as of v0.71.6) generates synthetic chat data via the Magpie chat-template-prefix harvest trick — prime the base model with just the assistant chat-template prefix (chatml / llama3 / gemma / mistral auto-detected) and let it complete the prompt itself. Live providers are 'ollama' and 'vllm' raw-completion endpoints (both SSRF-hardened, loopback-only); 'anthropic' is rejected (no raw-completion endpoint). '--quality-filter' drops low-quality rows via the v0.47 scorers; exact-duplicate instructions are de-duplicated. 'soup data persona-mix --prompts --n N --output [--personas ] [--styles ]' (v0.69.0, LIVE) is the Persona-Hub diversity sampler — bundled 12 personas × 5 writing styles by default (BYO Tencent 200k corpus via --personas / --styles). Deterministic by seed (random.Random(seed)); compute_topic_diversity = Shannon entropy over pooled whitespace tokens; atomic JSONL write with 100 MiB / 100k entries per loader caps.

Question 32

How does 'soup data brain-rot' detect AI slop in training data?

Accepted Answer

'soup data brain-rot <data.jsonl> [--strict] [--max-major-fraction 0.25]' (v0.69.0, LIVE) implements the arXiv 2510.13928 brain-rot detector with two pure-Python scorers: score_triviality (token-diversity inversion + '!!' / '??' punctuation runs + low-effort token density + length penalty) and score_popularity_signal (clickbait phrase scan + emoji U+1F300–U+1FAFF density). Worst-signal-wins composition: 1.0 − max(triviality, popularity). Bands match the v0.26 / v0.56 / v0.65 taxonomy: OK ≥ 0.85, MINOR ≥ 0.60, else MAJOR. refuse_if_rotten raises when MAJOR fraction exceeds threshold. '--strict' exits 3 on excessive MAJOR. English keywords by default; '--lang' selects a bundled per-language bundle (en, es, fr, de, ru).

Question 33

What is 'soup compile' and why does it hedge against prompt engineering winning over fine-tuning?

Accepted Answer

'soup compile --eval [--optimizer mipro|gepa|textgrad|copro|bootstrap_fewshot] [--max-iters N] [--output ] [--plan-only]' (v0.68.0) is a DSPy + GEPA + TextGrad prompt-program compiler — if prompt engineering paradigms keep winning over fine-tuning, you have the compilation surface inside Soup. Hard bound MAX_COMPILE_ITERS=1000. Closed optimiser allowlist of 5. validate_program_path requires '.py' extension, cwd containment, and os.lstat + S_ISLNK rejection via the shared paths.enforce_under_cwd_and_no_symlink helper. CompileResult enforces finite-score (no NaN / ±Inf) and iterations ≥ 0. Module is commands/compile_cmd.py because 'compile' is a Python builtin. LIVE as of v0.71.13 behind the '[compile]' extra (pip install "soup-cli[compile]") — run_compile dispatches DSPy / GEPA / TextGrad optimisation; '--plan-only' still renders the plan and exits 0.

Question 34

What does 'soup distill-prompt' do?

Accepted Answer

'soup distill-prompt --traces --teacher --student --strategy sft|preference|kl [--output ] [--plan-only]' (v0.68.0) distils prompt-heavy traces — long system prompts, in-context examples — into small fine-tunes that internalise the prompt. Closed strategy set {sft, preference, kl}. Teacher / student IDs capped at 512 chars; null-byte + empty rejected. Composes with v0.71 cross-tokenizer ULD — distill-prompt picks the dataset shape, ULD bridges teacher / student vocabularies. LIVE as of v0.71.13 — prepare_distill_dataset calls the teacher once per trace via the v0.20 providers (Ollama / Anthropic / vLLM) and emits sft/kl messages or preference chosen/rejected pairs. The teacher and student are provider model names (Ollama tags by default), not Hugging Face repo ids, and a failing teacher call is swallowed, so a wrong name ends with zero rows written and exit 0: read the row count.

Question 35

How does 'soup apple-adapter' target Apple FoundationModels?

Accepted Answer

'soup apple-adapter --direction hf-to-mlx|mlx-to-hf|hf-to-apple|mlx-to-apple --output [--sign/--no-sign] [--plan-only]' (v0.68.0) converts adapter formats between HuggingFace safetensors and Apple MLX npz, with optional Merkle-root signing; the two '*-to-apple' directions (Apple FoundationModels, iOS 26+) are accepted but refuse with exit 3 until Apple publishes a stable adapter format. Closed direction allowlist of 4. Explicit S_ISDIR + symlink rejection on source. 'sign' must be a real bool (bool-as-int defence). Reuses v0.60 Part B signing infrastructure and extends the v0.25 MLX backend. Live convert_apple_adapter ships in v0.68.1 (Apple's spec is still moving).

Question 36

What is 'soup local-rl' and how does the personal-LLM flywheel work?

Accepted Answer

'soup local-rl {init, status, record, harvest, train}' (v0.68.0) is a personal-LLM feedback flywheel daemon (LIVE except 'train'). 'soup local-rl init --db ' creates a POSIX 0o600 SQLite DB with interactions + thumbs tables (idempotent CREATE TABLE IF NOT EXISTS). 'soup local-rl record --db --prompt --response --thumb up|down' parameterised inserts with 16 KiB prompt + 16 KiB response caps and null-byte rejection. 'soup local-rl status --db ' renders a Rich table of interactions / up / down counters. 'soup local-rl harvest --db ' walks thumbs by ts ASC and emits one DpoPair{prompt, chosen, rejected} per prompt with both an up and down (last-writes-win dedup), atomic JSONL write via tempfile.mkstemp + os.replace — operators feed this straight into 'soup train with task: dpo'. 'soup local-rl train' is LIVE as of v0.71.13 — '--once' harvests the latest thumbs DPO pairs and runs a real nightly DPO/KTO/ORPO train via a 'soup train' subprocess (a 'state' table skips re-runs with no new feedback or fewer than --min-pairs); without '--once' it renders an injection-safe systemd .service/.timer + launchd .plist scheduler scaffold.

Question 37

What is 'soup adapters merge --strategy cmaes' and why is it different from linear / TIES / DARE / SVD?

Accepted Answer

'soup adapters merge -o --strategy cmaes --eval --budget 1h' (v0.67.0) runs Sakana-style evolutionary search over LoRA merge weights via pure-Python rank-mu CMA-ES (no 'cma' dependency). The optimiser parameterises N-1 logits, softmaxes them onto the simplex (sum=1, each ≥0), samples a population, keeps the elite half, and plateau-detects after 3 generations without improvement (converged=True). Bounds: 2–16 adapters, population [2, 256], generations [1, 10K], budget [60s, 24h] (reuses v0.57 blame.parse_budget). Operator-supplied eval_fn closure; failures swallowed with a sentinel -1e9 score so one broken eval doesn't crash the run (mirrors v0.40.3 proxy-failure isolation). Linear / TIES / DARE / SVD are still available as fast deterministic baselines; CMA-ES is for when you actually have a measurable eval and want the optimiser to search for you. LIVE as of v0.71.4 — the full CMA-ES loop now merges, materialises, and scores each candidate against the eval suite and writes the best-weighted merge to --output; the backdoor-scan + license gates run for cmaes too. '--eval' is a JSONL tasks file (the format 'soup eval custom' reads, not a YAML suite), and '--canary' / '--strict-verdict' are refused with --strategy cmaes (exit 2) because the eval suite already drives the search.

Question 38

What is the VeRA / VB-LoRA vector bank and why does it matter for multi-tenant serving?

Accepted Answer

'soup_cli.utils.vector_bank' (v0.67.0) is the storage format for VeRA / VB-LoRA — shared random projection matrix P (d_model × d_model) + per-user scaling vector v_u (vector_dim floats). A 128-D scaling vector at fp32 is ≈512 bytes per user vs. ~30 MB for a rank-16 LoRA on a 7B model — thousands of per-user adapters at MB-each instead of hundreds-of-MB per LoRA. Hosted vendors price by GPU-hour, not adapter count, so they structurally cannot offer this economics. Schema: bank name (kebab-case, ≤128 chars), base_model (≤512 chars), per-entry {user_id (≤256 chars), scaling_vector (floats, finite, no NaN/Inf)}. Bounds: vector_dim [1, 16K], up to 1M entries per bank, 16 MiB file-size cap, atomic JSON I/O via shared paths.atomic_write_text + cwd containment + symlink rejection. estimate_bank_size(num_users, vector_dim) for capacity planning. LIVE as of v0.71.12 — 'soup serve --bank <bank.json> [--bank-strength S]' reconstructs the shared projection + per-user scaling vectors and installs a decode-time forward hook; the active user is chosen per request via the 'X-User-Id' header (an unknown/absent id is a zero-delta no-op, so there is no cross-request leak), serving N personas at ~KB-per-user.

Question 39

What is MoLE per-token gating in Soup CLI v0.67?

Accepted Answer

task='moe_lora_routing' (v0.67.0) is Mixture of LoRA Experts: a gating network routes per-token activations to top-K task adapters via softmax over the hidden state. Config: MoleGatingConfig with num_task_adapters [2, 64], hidden_dim [1, 16K], temperature (1e-6, 100.0] finite, top_k ≤ num_task_adapters. Cross-validator rejects mlx backend. Beyond 64 adapters per-token softmax becomes the bottleneck; for more, hierarchy the gating. LIVE as of v0.71.12 — 'task: moe_lora_routing' with 'mole_task_adapters: [...]' trains a real per-token gating network that blends N frozen task LoRAs (mole_top_k / mole_temperature); only the router trains, and the gate is saved as 'mole_gate.pt' alongside the run.

Question 40

How do 'soup adapters pr' adapter pull requests work?

Accepted Answer

'soup adapters pr --base-sha <64-hex> --adapter <path> --eval <eval_delta.json> --samples <sample_diffs.json>' (v0.67.0) renders a GitHub-shaped pull request from an adapter triple {base SHA, dataset diff, adapter weights, eval-delta report} as review-friendly Markdown with an eval-delta table + per-sample baseline/candidate diffs. Bounds: ≤64 EvalDelta entries, ≤256 sample diffs, per-output cap 32 KiB to keep PRs reviewable. The _md_table_escape function neutralises backslash, pipe, newline, CR, and tab characters in operator-controlled cells against Markdown / Rich injection. JSON output is also available for the v0.68 GitHub Action that will post the PR comment. Composes with 'soup adapters diff' from v0.57.

Question 41

What does 'soup lock' do and how is it different from 'soup env lock'?

Accepted Answer

'soup lock write --base-model --base-sha <64hex> --dataset-sha <64hex> --env-hash <64hex>' (v0.67.0) computes closure_sha = SHA256(base_sha || dataset_sha || env_hash) and writes a JSON soup.lock with soup_version, base_model (≤512 chars), and the three SHAs. The point: commit soup.lock to git alongside soup.yaml so the entire team trains on identical (base, dataset, env). 'soup lock check' compares the 5 content fields and exits 2 on drift (3 when the lock file is missing or unparseable; drift was 3 before v0.76.0); soup_version + created_at are advisory only (legitimate operator upgrades don't trigger drift). 'soup env lock' is v0.64.0 — it produces the env_hash that feeds soup.lock; the two together close the reproducibility chain.

Question 42

What is 'soup adapters bisect' and how does it work?

Accepted Answer

'soup adapters bisect ... --eval-command "soup eval gate --suite gate.yaml --baseline baseline.json --model {ckpt}"' (v0.67.0) does binary search over an ordered checkpoint history to find the first regression boundary. The eval command has to exit non-zero on a failing checkpoint: soup eval gate does, soup eval custom does not (it exits 0 whenever the eval completes), so with it every checkpoint would pass. The operator supplies a shell template with a {ckpt} placeholder — Soup uses shlex.split after shlex.quote(ckpt), argv-list mode, no shell=True. Probes both endpoints first to short-circuit all-OK or all-broken, then ~log₂(n) midpoint probes. Verdicts: ALL_OK or BROKEN_AT (returns first_broken checkpoint ID). Exit 2 on BROKEN_AT for cron-friendly automation. Input: 2..4096 unique ordered checkpoint IDs.

Question 43

What does 'soup tunability' decide before you pick a base model?

Accepted Answer

'soup tunability --dataset <path> --candidates qwen3-1.7b,llama-3.2-3b,phi-4-mini --probe-steps 100 --holdout-size 64' (v0.64.0) ranks the candidate bases with a deterministic offline heuristic and reports the Pareto frontier (best efficiency for cost); add '--live' to probe each candidate with a lightweight LoRA on a held-out dataset slice and rank on measured training-loss deltas. The candidate allowlist is closed and ships with licensing metadata (Apache-2.0, MIT, LLaMA-3, etc.). '--plan-only' dry-runs without probing; '--list' catalogues all bundled candidates. Bounds: probe_steps [10, 10000], holdout_size [10, 100000] rows. Output: per-candidate delta (base_loss - probe_loss), wall-clock seconds, estimated USD cost, Pareto frontier membership. Safety: path containment (is_under_cwd), null-byte rejection, symlink-escape rejection on dataset and output paths.

Question 44

What is 'soup plan' / 'soup apply' and how is it Terraform-shaped?

Accepted Answer

'soup plan --config soup.yaml --state ./soup.tfstate' (v0.64.0) reads a training configuration, computes cost / ETA / peak-VRAM / SHA-256 hashes, and writes an immutable state file. 'soup apply' re-reads the config, detects any drift (changed batch size, dataset SHA, base SHA), and refuses to proceed (exit 3) if drift is found. State shape: plain JSON with 'plan' (cost, ETA, SHA hashes), 'applied' (bool), 'applied_at' (ISO timestamp), 'run_id' (reserved). TOCTOU defense: config read as YAML with os.lstat before open; symlinks explicitly rejected. Peak VRAM calculated with a 10% safety margin; estimated cost + minutes included with spot pricing per GPU tier. Composes with v0.60 license-check for adapter merge and v0.67 soup.lock for the full reproducibility chain.

Question 45

How does 'soup env lock' detect ABI drift?

Accepted Answer

'soup env lock --output ./soup-env.lock' (v0.64.0) snapshots the current Python environment (version, CUDA major version, platform, every installed package) into a JSON lockfile. 'soup env check --lock ./soup-env.lock' compares the current env against the lock and exits 3 on ABI-sensitive drift (e.g. Python minor version change, CUDA major version change) that would invalidate training. Fields: soup_version, python_version, platform, cuda_version (or 'none'), ISO timestamp, per-package {name, version, source}. Atomic write to lock file; reads have a file-size cap (no unbounded YAML). The env_hash from 'soup env lock' feeds the v0.67 soup.lock closure for full team reproducibility.

Question 46

What does 'soup license-advisor' decide for a deploy target?

Accepted Answer

'soup license-advisor --target b2c|defense|embedded --license <id> --monthly-active-users 0' (v0.64.0) recommends license-clean base models for a deployment target. Returns ok (safe), warn (yellow), or block (red; exits 3) with a reason string and recommended/forbidden license lists. Per-license risk scales with expected MAU (e.g., LLAMA-3 OK for <1M MAU, risky above). Composes with the v0.60 license_matrix.check_license_compat gate on 'soup adapters merge' — so a 33-entry SPDX matrix governs adapter merges, while license-advisor governs base-model selection upstream.

Question 47

What new failure modes does 'soup eval behavior' / 'eval capability' / 'eval checklist' add in v0.65?

Accepted Answer

v0.65.0 takes failure-mode coverage from 6 to 10 (7 to 11 counting the citation probe added in v0.71.10). 'soup eval behavior --battery xstest|harmbench|jailbreakbench|elephant|syceval --evidence ' diffs pre/post-FT on bundled safety / refusal / jailbreak / sycophancy probe sets with OK/MINOR/MAJOR via the v0.26/v0.56 thresholds (≥0.85 OK, ≥0.60 MINOR, else MAJOR). 'soup eval capability --suite full|fast|math|code' emits validated lm-eval-harness task IDs for 7 bundled benchmarks (MMLU-Pro, GPQA, BBEH, AIME, MATH-500, HumanEval+, SWE-bench-Verified). Add '--live --model ' to run the suite through lm-eval instead of only emitting task IDs; 'soup eval behavior' takes '--base-model' and '--adapter' for a live pre/post diff. 'soup eval checklist ' runs CheckList-style MFT (minimum functionality, keyword in response) / INV (invariance under paraphrase) / DIR (directional perturbation) tests from a 1 MiB-capped YAML spec with up to 1,000 tests. It loads no model: it scores the responses you pass with '--evidence', which is required since v0.76.0 ('soup eval checklist' without it exits 3, as does 'soup eval behavior' with neither --evidence nor --base-model). Evidence files capped at 16 MiB with O_NOFOLLOW open against symlink swap.

Question 48

What does 'soup eval irt-subset' do?

Accepted Answer

'soup eval irt-subset <responses.jsonl> --size full|small|tiny' (v0.65.0) fits a 1-parameter Rasch Item Response Theory model (the default; '--model 2pl' or '3pl' fit the richer models and need a respondent_id on each row) to per-item correctness signals and selects a minimum-cost subset that preserves ranking power. The math: P(correct | ability θ, difficulty β) = σ(θ - β); β_i = -log(p̂_i / (1 - p̂_i)); item information I(β) = σ(-β) · σ(β) which peaks at β≈0 (50/50 items are most discriminating). Profiles: full (100%), small (~30%), tiny (~10%). Input shape: JSONL rows {item_id (str, ≤256 chars, no null bytes), correct (bool), score? (float)} up to 1M rows. Pure-Python math (no numpy/scipy in v0.65.0); 256 MiB file cap against unbounded uploads.

Question 49

What is 'soup probe sae-diff' and which SAE repos are bundled?

Accepted Answer

'soup probe sae-diff --top-k 20' (v0.66.0) applies a Sparse Autoencoder encoder to pre- and post-FT activation batches, computes per-feature mean changes, and reports the top-K most-changed features. Purely descriptive (no verdict gate); designed for CI logging and model cards. The SAE repo allowlist is closed — Gemma Scope (2B/9B/27B residual-stream), EleutherAI Pythia SAEs, JBloomAus Llama SAEs, OpenAI GPT-2 SAE — and as of v0.71.8 '--auto-download' fetches an allowlisted SAE from the HF Hub into '~/.soup/sae-cache/' (validated against the allowlist BEFORE any network call, via an SSRF-hardened snapshot_download). Input shape: JSON with 'activations': [[...], ...] (2D float32 matrix [num_tokens, hidden_dim]) read via O_NOFOLLOW to prevent symlink swap. Output: SaeFeatureDiffReport with top-K features and per-feature {feature_id, delta, pre_mean, post_mean}. Bounds: top-k [1, 10K], up to 1M SAE features, up to 1M tokens per activation batch, 16 MiB evidence file cap.

Question 50

How does 'soup probe sleeper' work?

Accepted Answer

'soup probe sleeper --evidence ' (v0.66.0) applies a linear defection probe (synthetic, deterministic placeholder weights keyed on base name, so the bundled verdict exercises the plumbing and says nothing about the model) to a 2D activation tensor. Classifies per-token scores as defection or benign, returns flagged rate, and assigns OK/MINOR/MAJOR: ≤1% → OK, ≤5% → MINOR, >5% → MAJOR. Without '--weights' the base must be one of six exact ids (meta-llama/Llama-3-8B, meta-llama/Llama-2-7B, mistralai/Mistral-7B-v0.1, Qwen/Qwen2-7B, google/gemma-2-9b, google/gemma-2-2b) — metadata includes hidden_dim, threshold, description. With '--weights ' (v0.71.8) it uses your calibrated probe weights instead of the synthetic seed fallback and accepts any base id. Activation input: JSON {'activations': [[...], ...]} (2D float32 [num_tokens, hidden_dim]); 1M token cap. No evidence = OK report with 0 tokens (matches v0.56 diagnose neutral-mode policy). 16 MiB evidence cap, symlink rejection.

Question 51

What is 'soup probe interference' and when does it gate CI?

Accepted Answer

'soup probe interference <losses.json>' (v0.66.0) builds an N×N matrix of adapter pairwise compatibility scores from operator-measured losses. Input JSON has 'adapters' list and 'losses' dict with keys like 'a|b' (loss on domain A when both A and B loaded) and 'a|a' (baseline loss with A alone). Formula: score(A→B) = (loss(A_target | A+B) - loss(A_target | A alone)) / loss(A alone). Classification: |score| <5% → OK, <20% → MINOR, ≥20% → MAJOR. Bounds: 2–16 adapters (4–256 pairwise probes); adapter names ≤256 chars and markup-escaped against Rich/Markdown injection before render. Output: InterferenceMatrix with cells and worst-pair summary; exits 2 if worst ≥20% interference (gates CI merge).

Question 52

How does 'soup ingest' pull traces from observability vendors?

Accepted Answer

'soup ingest --source langfuse|langsmith|helicone|openpipe|otel|openai-stored --logs --output ' (v0.63.0) normalises any observability export into one TraceRecord schema (trace_id / prompt / output / source / signal / metadata as a MappingProxyType so callers can't mutate). By default there are no network calls: the operator exports from the SaaS dashboard, then 'soup ingest' parses the file. Since v0.75.0 Langfuse is the one source Soup can also fetch for itself: 'soup ingest --source langfuse --pull --since 24h' reads Langfuse's Observations API v2 directly and writes one row per GENERATION observation, with credentials taken only from the environment and never a flag, HTTPS enforced through the same SSRF validator as --slack-url, redirects refused rather than followed with credentials attached, 30 s per request, 64 MiB per response, and a --max-pages cap (default 100, max 10000) that exits 1 writing nothing rather than truncating. Six sources at launch: LANGFUSE_PUBLIC_KEY plus LANGFUSE_SECRET_KEY for Langfuse (the older single-key hint named LANGFUSE_KEY, a variable Langfuse does not read; corrected in v0.75.0), LANGSMITH_API_KEY for LangSmith, HELICONE_API_KEY for Helicone, OPENPIPE_API_KEY for OpenPipe, OTEL_EXPORTER_OTLP_HEADERS for OpenTelemetry OTLP, OPENAI_API_KEY for OpenAI Stored Completions. PII warning prints once per invocation. Output feeds directly into the v0.26 'soup data from-traces' preference-pair builder and the v0.58 'soup loop' HarvestFn — closing the production-traces → training loop without paying per-trace fees to any vendor.

Question 53

What does 'soup prune-prompt' do?

Accepted Answer

'soup prune-prompt --input --output --min-frequency 0.95' (v0.63.0) detects the longest shared system-prompt prefix across all rows via binary search over up to 32 candidate lengths and strips it. By removing the shared boilerplate (e.g., 'You are a helpful AI assistant...'), the fine-tuned model internalizes it into its weights instead of repeating it on every turn — OpenPipe's signature trick at the data layer. Two-pass file read; capped at 100k rows. v0.63 fixed an O(N²) early-exit bug from the prototype detect_common_prefix function.

Question 54

What is 'soup ab' and how is it different from a fixed-N A/B test?

Accepted Answer

'soup ab --input --metric latency|judge_score|retry_rate --alpha 0.05 --effect-size 0.1' (v0.63.0) is a sequential A/B test you can re-run after every new row without breaking its error guarantee: you can peek at any sample count and read reject_h0, accept_h0 or continue. Through v0.75.2 it was an mSPRT (a Gaussian-mixture likelihood ratio read against Wald's two boundaries) configured with --alpha and --beta. Since v0.76.0 it is two-sided, so a treatment that moves the wrong way is reported as worse through a 'direction' field read against the metric's polarity (judge_score: higher is better; latency and retry_rate: lower is better), and it decides with a normal-inverse-gamma Bayes factor that depends on the rows only through the t statistic, so its false-positive rate stays at most --alpha however often you re-run it. accept_h0 comes once a confidence sequence at level --alpha for the difference lies inside plus or minus --effect-size, so --alpha bounds both wrong verdicts. Upstream's record of that accept rule (benchmarks/gate-1418-ab-cs-accept.md) compares it only against designs that were not released, on simulated rows, so no speed-up or rows-needed figure is stated here. '--beta' is retired: it is ignored with a warning and upstream says Soup v0.77 will refuse it. Three metrics: latency, judge_score, retry_rate. The exit code is 0 on every decision, so read 'direction' to gate on a regression. Input rows: {arm: 'control'|'treatment', : }.

Question 55

What does 'soup drift-alarm' watch in production?

Accepted Answer

'soup drift-alarm --reference --live --threshold 0.2 [--slack-url ... --discord-url ...]' (v0.63.0) computes KL divergence between token-distribution snapshots at fine-tune time (reference) and production (live). Catches both behavioral drift ('model now outputs JSON when it didn't before') and vocabulary drift ('same 20 phrases on repeat'). Tokenization is whitespace only; there is no tokenizer option. Optional Slack / Discord webhooks fire on drift, SSRF-validated: HTTPS hosts are accepted, plain HTTP only for loopback, and RFC1918 / link-local / cloud-metadata IPs (169.254.0.0/16, 100.64.0.0/10, 198.18.0.0/15) are rejected on either scheme. Exit code 3 on drift for cron-friendly automation. Input rows: one JSON object per line with the model output under 'output' ('response' and 'text' are accepted too); a row without one of those keys is skipped, and if none qualifies the report shows n_reference 0 and an OK verdict, so check the counts.

Question 56

How does Soup CLI handle GDPR right-to-be-forgotten?

Accepted Answer

Soup CLI v0.61.0 ships 'task: unlearn' as a first-class trainer with three methods: NPO (Negative Preference Optimization — DPO-shaped negative-only loss, needs reference model + retain set), SimNPO (length-normalized NPO, no reference model needed — faster on long sequences), and RMU (Representation Misdirection Unlearning — residual-stream noise on forget inputs, best for concept-level removal). Config: set 'task: unlearn', 'training.unlearn_method: npo|simnpo|rmu', 'training.unlearn_alpha' (0.0–10.0, retain-set weight; unset resolves to 1.0 in the trainer, and an explicit 0.0 is also treated as 1.0), 'data.forget_set' (required JSONL or HF dataset ID), 'data.retain_set' (optional, strongly recommended for NPO). Verify with 'soup eval unlearning --benchmark tofu|muse|wmdp --evidence ' which scores Forget Quality / Model Utility / PrivLeak from the pre/post numbers you supply and emits an OK / MINOR / MAJOR verdict (exit 2 on MAJOR). It runs no benchmark itself: a metric you leave out, and the whole report when there is no --evidence, scores a neutral OK, so an all-green report without evidence means nothing was measured. The trainer is LIVE as of v0.71.9 — covariance-free NPO / length-normalised SimNPO / RMU representation-steering kernels run end-to-end, warning when run without a retain set.

Question 57

Can I patch a single fact in a model without retraining?

Accepted Answer

Yes. Soup CLI v0.61.0 ships 'soup edit set --base --method rome|memit|alphaedit --subject "" --target ""' for surgical fact patching. Three methods: ROME (Rank-One Model Editing, targets MLP layers — Meng 2022), MEMIT (Mass-Edit Memory in a Transformer — Meng 2023), and AlphaEdit (ROME projected orthogonal to the down-proj's top singular direction). LIVE as of v0.71.9 — covariance-free rank-1 weight-edit kernels apply the update and optionally save (on a tiny model a ROME edit moved P('Lyon' | 'The capital of France is') from 0.0016 → 0.96); a SQLite EditGovernor enforces the per-base edit cap + norm-blowup refusal across separate runs. '--method grace' is a fourth method: it appends a (key, value) pair to a grace_codebook.json sidecar and leaves the base weights untouched, and no Soup command at v0.76.0 applies that codebook at decode time. 'soup edit diff

--probes probes.jsonl --before-model --after-model ' runs each probe prompt through both models and lists the completions that changed; the two ids are labels for the report, not registry lookups, and since v0.76.0 the command requires both --before-model and --after-model when --probes is given (it exits 2 otherwise). The Sequential Edit Governor, consulted by 'soup edit set' and by the GRACE edit, refuses further edits on a base model once 50 have been recorded and after an edit whose measured weight-norm change reaches the blow-up threshold; it cannot stop the edit that causes the blow-up, because the change is only known afterwards, and its state lives in a SQLite store ('SOUP_EDIT_GOVERNOR_DB' moves it): a store path that is refused or cannot be created disables the governor with a 'Governor disabled' note and the edit goes ahead, but a corrupt database stops 'soup edit set' with an error.

Question 58

What is RAFT and how does Soup CLI implement it?

Accepted Answer

RAFT (Retrieval Augmented Fine-Tuning, Stanford 2024) teaches a model to use retrieved context — both golden and distractor documents — to answer queries. Soup CLI v0.62.0 ships 'data.format: raft' with per-row schema {query, golden_doc, distractor_docs[], answer}. Pair with 'training.ra_dit_stage: retriever|generator' for the RA-DIT two-stage pipeline: stage 1 trains a contrastive sentence-transformer ('ra_dit_retriever_model: sentence-transformers/all-MiniLM-L6-v2' default), stage 2 SFTs the main model on RAFT rows. LIVE as of v0.71.10 — RAFT span-mask training masks the prompt to -100 and labels each doc '[doc-N]' so the model learns to cite the supporting document (reproducible shuffle via 'data.raft_shuffle_seed'), and the 'soup ra-dit' one-shot orchestrator trains retriever then generator in a single command, recording the trained retriever as the generator's paired retriever (auto-linked from the Registry when unset). Add 'training.citation_faithful: true' + 'citation_style: bracket|inline|footnote' to raise the loss weight on citation spans during training. 'citation_recall_threshold' is accepted by the schema, but nothing in v0.76.0 reads it, so no save is ever refused on it (it warns at load, and upstream's stated deadline for refusing it is v0.77); measure recall afterwards with 'soup eval citation' or the diagnose citation probe. Three new recipes shipped: raft-llama3-8b, ra-dit-retriever, ra-dit-llama3-8b.

Question 59

Can I steer a model's behavior at inference time without fine-tuning?

Accepted Answer

Yes. Soup CLI v0.62.0 ships 'soup steer train --base --method caa|iti|repe --name --pairs ' for training activation steering vectors, then 'soup steer apply --name --strength ' to load a stored vector and print its metadata (|strength| ≤ 10 enforced); the decode-time intervention itself is 'soup serve --steer'. Three methods: CAA (Contrastive Activation Addition, adds a learned vector to the residual stream), ITI (Inference-Time Intervention, shifts specific attention heads), and RepE (Representation Engineering, PCA-based direction extraction). Pairs JSONL: {positive: '', negative: ''}. The Registry has a 'steering_vector' artifact kind (v0.26), and since v0.76.0 'soup steer train --registry-id ' attaches the trained steering_vector.safetensors as a steering_vector artifact (a failed attach exits 1 with the vector already on disk); through v0.75.2 it passed the directory, the Registry refused it, and the command warned and exited 0 with nothing recorded. LIVE as of v0.71.10 — 'soup steer train' fits the CAA/ITI/RepE control vector from {positive, negative} pairs and persists a safetensors + config artifact, and 'soup serve --steer --steer-strength ~~' applies it at decode time via a forward hook. Use case: nudge safety, formality, or domain focus without a full SFT/DPO run.~~

Question 60

Does Soup CLI generate compliance docs for procurement / regulators?

Accepted Answer

Yes (v0.59.0 'Governance & Provenance'). 'soup bom emit --name --version --base-model --base-sha <64-hex> --config-sha <64-hex> --task --license --format cyclonedx|spdx|both --output ' (with --format both, --output is a prefix and it writes .cdx.json and .spdx.json) produces machine-learning Bills of Material in CycloneDX 1.6 (with the ML-BOM extension) and SPDX 2.3 AI-profile formats — both, in one shot. 'soup attest emit --stage extract|train|eval|export|publish --subject --sha <64-hex> --builder --invocation --sign unsigned|ed25519|sigstore' writes SLSA-3 in-toto attestations per stage (ed25519 needs the [sign] extra and a key; since v0.76.0 sigstore signs keylessly through Fulcio and Rekor, needs the [sigstore] extra, an ambient OIDC identity such as GitHub Actions or the opt-in --interactive-oidc, and --output, writes the bundle into the .sig sidecar, and never falls back to an unsigned statement: without the extra it exits 2 and writes nothing; 'soup attest verify' then needs --cert-identity and --cert-oidc-issuer). 'soup audit-log tail/rotate' keeps a HIPAA/SOC2-shaped JSONL audit log at ~/.soup/audit.jsonl in which credentials are masked (hf_, sk- and Bearer values, and the values of credential-named options); every other argument (a webhook URL passed with '--slack-url' or '--discord-url' included), the operator's user name and the host name are recorded as typed, so it is not PII redaction. 'soup train --annex-xi --repro-receipt ' emits EU AI Act Annex XI/XII documentation plus SR 11-7 reproducibility receipts (every seed, kernel version, library version, dataset hash). All atomic writes (tempfile.mkstemp + os.replace) with symlink-rejection TOCTOU defense.

Question 61

How does Soup CLI prevent supply-chain attacks on LoRA adapters?

Accepted Answer

Six controls in v0.60.0. (1) 'soup adapters scan ' spectral-analyses LoRA weights for backdoors: rank-1 dominance (warn at 50×, fail at 200×), top-singular-value energy concentration (warn > 75%, fail > 95%), Frobenius outliers (warn > 4σ, fail > 8σ), NaN/Inf. Exit 0/1/3 for OK/WARN/FAIL — pure numpy, no torch. (2) 'soup adapters sign ' computes a Merkle root over all files and writes .soup-signature.json — real ed25519 detached signatures are LIVE as of v0.71.2 via the '[sign]' extra ('--backend ed25519 --key ' or '--generate-key'; 'soup adapters verify --public-key ' does a cryptographic verify, failing closed on any tamper or wrong key (and from v0.71.2 to v0.75.0 it did NOT: the trusted key was consulted only inside the ed25519 branch, so a record with any other backend verified with exit 0 even under --strict, which is fixed in v0.75.1)). Sigstore keyless signing is available from v0.76.0 ('soup adapters sign --backend sigstore', with the [sigstore] extra; 'soup adapters verify' then requires --cert-identity and --cert-oidc-issuer as an out-of-band trust policy). (3) 'soup adapters verify --strict' refuses to load on signature mismatch. (4) 'soup adapters check-safetensors --strict' refuses pickle / .bin / .pt — the single biggest LoRA attack vector. (5) 'soup adapters merge --license [--license ] --license-override ""' enforces SPDX-license compatibility via a 33-entry matrix. (6) 'soup airgap-bundle --model --output bundle.tar --dataset ... --wheel ... --kernel ... --bundle-size-cap 100' packs everything into one tarball for one-way data-diode transfer, ≤100 GiB cap, with a manifest of SHA-256 fingerprints per file. Plus a trust-on-first-use namespace pin used by 'soup spectrum scan', 'soup probe sae-diff --auto-download', the base model of a 'stream_layers: true' training run and, since v0.76.0, a Hub id given as 'training.prm_reward': it records the repo's author and creation time on first download and refuses a later download if the author or the creation time changed (there is no command-line flag to approve a change: the refusal text names '--allow-namespace-shift', which exists only as a Python argument). It does not cover the base model that an ordinary (non-streamed) training, chat, serve, merge or export run loads, and it fails open when the Hub metadata or the local store is unavailable.

Question 62

What is 'soup loop' and how does the production data flywheel work?

Accepted Answer

'soup loop' (v0.58.0) runs the full production data flywheel from a single CLI: harvest production traces → distill into preference pairs → eval-gated DPO train → canary deploy → auto-rollback on regression. 'soup loop init --eval --baseline registry:// --monthly-budget 50usd --max-runs-per-day 3' creates an atomic '.soup/loop.yaml' state file. 'soup loop watch' runs as a long-running daemon (SIGTERM/SIGINT safe, reloads state every iteration so external pause/resume takes effect immediately). 'soup loop canary --traffic 5% --autoroll-on-regress' splits traffic via deterministic SHA-256 hash routing (±0.01% split granularity, threading-locked BucketStats; through v0.75.2 it only recorded the canary and its share and 'soup serve' never read them, and since v0.76.0 'soup serve' on the transformers backend routes a request by hashing its conversation key, conversation_id in the body or an X-Conversation-Id header, reading the canary state from the .soup/loop.yaml in the directory it was started from, when the canary adapter was loaded through --adapters and the request names no adapter of its own); the verdict uses the v0.26 Quant-Lobotomy OK/MAJOR thresholds with a 30-sample minimum. 'soup loop pause' / 'soup loop resume' flip atomic status without killing the daemon. 'soup loop replay []' walks per-iteration manifests at '.soup-loops//iteration.json'. Budget guardrails enforce a daily-cap → estimate-sanity → monthly-budget check order; budget-skipped iterations produce no manifests. Every iteration is recorded as a frozen IterationRecord with gate verdict, canary verdict, shipped flag, rolled-back flag, and estimated cost.

Question 63

What does 'soup advise' decide before you fine-tune?

Accepted Answer

'soup advise --goal ""' (v0.54.0) is a pre-flight decision engine that classifies the task (keyword + structural signals — tool_calls → tool_use, '' → reasoning, chat messages → input-extraction) across 7 task categories, profiles the dataset (row_count, avg input/output chars, type-token diversity, label variance, has_chosen_rejected, has_reasoning_traces — capped at 2,000-row sample), and emits a rubric verdict among PROMPT_ENG / RAG / SFT / DPO / GRPO. Heuristics: preference pairs → DPO; reasoning + ≥500 rows → GRPO; <50 rows → PROMPT_ENG; high-variance factual → RAG; default → SFT. Optional '--probe' adds ROI estimates, a heuristic computed from the shape of your data by default; add '--probe-model ' to run a live probe that loads that model, scores zero-shot and few-shot on a held-out slice and LoRA-trains on a tiny one. '--record' appends an entry to '~/.soup/advise_history.jsonl' (atomic, file-locked, 16 MiB cap). A verdict is biased only by this project's own past entries that carry a recorded outcome, and '--record' writes none (nothing in v0.76.0 sets an outcome), so recording alone does not change later verdicts. 'soup advise explain' prints the full rubric. 'soup advise compare' lists the prior verdicts recorded with '--record' (newest first, '--limit N'). Outputs the literal next command ('soup autopilot --model … --data … --goal chat') so the handoff is one paste away.

Question 64

How does 'soup eval design' derive an eval suite from your data?

Accepted Answer

'soup eval design --goal "..."' (v0.55.0) uses TF-IDF salience over your training data (10,000-row DoS-capped subsample) plus goal-keyword dispatch to draft a goal-conditioned eval suite with up to 5 dimensions and a scorer per dimension chosen from {exact_match, regex, judge, rlvr}: 'json' / 'code' / 'math' keywords → rlvr; 'classify' → exact_match; 'extract' → regex; default → judge. 'soup eval discover ' runs greedy farthest-first Jaccard-distance clustering (10,000-row cap) to surface held-out canaries + adjacent-skill probes + 25%-prefix memorization probes. 'soup eval lock ' freezes the suite as a SHA-256-checksummed 'eval_suite' artifact (canonical-JSON for stable hashes; attached to a registry entry only when you pass --attach-to-registry, next to the new 'canaries' kind). 'soup eval coverage --task ' does gap analysis against the v0.54.0 task taxonomy (each task has a recommended scorer set — e.g. reasoning → rlvr/judge, format_conversion → regex/rlvr). 'soup eval gate-install --baseline ' writes a '.git/hooks/pre-push' that runs 'soup eval against' for the baseline run and the candidate run named by SOUP_CANDIDATE_RUN_ID (a paired-bootstrap regression verdict, n_samples in [100, 100_000], ci_level in (0, 1)) and blocks the push on regression; with SOUP_CANDIDATE_RUN_ID unset the hook skips itself and lets the push through.

Probe	What it catches
`forgetting`	Per-task Δ accuracy with tolerance band — extends v0.25 `eval/forgetting`. Live, it is scored on `--holdout` prompts or a built-in general-knowledge set, never on your training rows (v0.76.0)
`refusal`	advbench / xstest delta over caller-supplied generators (`_MAX_REFUSAL_SCAN = 8192`)
`format`	JSON / regex / tool-call validity over the RLVR verifier set, with an explicit ReDoS probe (`compiled.search("a" * 128)`)
`mode_collapse`	Pairwise n-gram-Jaccard distance over K completions (`k ∈ [2, 32]`, `ngram_n ∈ [1, 8]`)
`memorization`	Training-prefix echo via partial-prompt continuation (`_MAX_SCAN_ROWS = 1000`). Live, it scores clipped completion-bigram precision against the held-out suffix (v0.76.0)
`contamination`	n-gram overlap with public benchmarks (combined-complexity cap rejects when training rows × benchmark corpus > 1e9)
`citation`	citation-recall regression on RAFT-shaped rows: does the answer still cite the supporting `[doc-id]`? (v0.71.10)

Code	Means
0	Overall OK or MINOR, or `NOT_RUN` with `--allow-not-run`. With no evidence and no `--base-model` every mode is a neutral OK.
2	Overall MAJOR. Also an `--evidence` file that does not exist, an invalid `--citation-style` and a mistyped flag.
3	Overall `NOT_RUN` without `--allow-not-run`, `--attach-to-registry` without `--output`, or a bad `--tokenizer`.
1	An `--evidence` file that is not valid JSON, a live run that failed, an unwritable `--output`, or an attach that failed after the report was written.

`soup diagnose` — seven failure-mode probes, one verdict

Usage

`NOT_RUN`: a probe that did not run is not a pass (v0.76.0)

What the live probes read (v0.76.0)

`soup train --diagnose-gate`

Exit codes

See also

soup diagnose — seven failure-mode probes, one verdict