Skip to content

v0.78.0: a continual-learning loop, a per-row ship audit, native MLX chat and merge

v0.78.0 was tagged on 18 November 2026. Upstream gave it no codename for the tenth release running; the title reads: a continual-learning loop, soup ship per-row audit, native MLX chat and merge. It is a cut of the development branch after v0.77.2, not a backport. By the CHANGELOG's count that is 60 additions, 20 changes, 2 deprecations, 171 fixes and 3 security entries. The release body says: "This is not a fixes-only release, and it refuses some input v0.77.2 accepted."

218 of the 271 pull requests in this release came from 48 people outside the maintainer. That is the release body's own count; the names are under Credit.

Coming from v0.76.x or older? Read v0.77.0 and v0.77.2 first. Python 3.10 stays on v0.76.x (pip install "soup-cli[train]<0.77"); this release supports Python 3.11 to 3.13, and v0.76.x gets no further security fixes after it.

bash
pip install -U soup-cli
soup --version
soup doctor --config soup.yaml

With pipx, pipx upgrade soup-cli replaces the first line; soup --version prints soup v0.78.0. For Docker, docker pull ghcr.io/makazhanalpamys/soup:0.78.0 (also latest); the host driver must support CUDA 12.8. A soup.yaml that loads on v0.77.2 loads on v0.78.0 unless it sets one of the values under Refused now, a boolean in a numeric training field first among them. soup doctor --config does not read an eval-gate suite file; soup train --config soup.yaml --dry-run does.

Four things are worth a look before the first run: a config with a boolean in a numeric field or an enabled eval gate on a path that never ran one (Refused now); a CI step that tests an exit code (Exit codes that moved); anything that reads soup ship evidence, because soup-cli 0.77.x refuses a schema_version: 2 file (soup ship); and a soup train --push-as, which now creates a private repository.

Re-run this

What was wrongWho is affectedPR
Upstream: a merge of adapters whose ranks differ on some modules "exited 0 before and wrote an adapter without those modules". Run on both versions here: v0.77.2 exited 0 and wrote a file holding no tensors; v0.78.0 exits 2 and writes nothing.A soup adapters merge of adapters whose LoRA rank differs on a shared module, that share no tensor name, or whose shared tensor has two shapes, made on v0.77.2 (the version run). Check the output with soup adapters check; merge again with soup adapters arithmetic ... --rank N, or merge adapters of one rank.#2926 by @abdulwaarith0 (reported as #2907)
Upstream: soup merge --low-memory "refuses an adapter whose init_lora_weights rewrites the base at init (PiSSA / OLoRA from lora.init_strategy, CorDA, LoftQ, LoRA-GA) instead of silently writing wrong weights"; its advice is to use the regular merge for these.A checkpoint written by soup merge --low-memory from such an adapter. Checked in the code at v0.77.2: its low-memory merge had no check for this. Merge again without --low-memory.#2732 by @abdumutalipabay0 (reported as #2722)
Upstream: task: classifier, reranker, cross_encoder and asr full fine-tunes now use fp32 master weights, "where 16-bit weights barely moved at the default learning rate." Upstream measured it once: after 8 steps on one RTX PRO 5000 Blackwell, 100% of the norms and 69.7% of the attention and MLP weights were bit-identical to the checkpoint's.A full fine-tune (the default) of one of those tasks over a 16-bit checkpoint, at the default learning rate, that finished on v0.77.2 (the version checked). Train again on v0.78.0; it takes twice the memory, so a run that fit the card can now run out. classifier_lora: true and asr_lora: true runs are not in this row.#2949 by @abdumutalipabay0 (reported as #2941)
Upstream: an enabled eval gate on backend: mlx and on task: prm, unlearn and moe_lora_routing "never attached a gate and trained without one", and a training.eval_gate.suite that could not load trained without a gate.A run you believed was gated. The model is not wrong; it was never checked. Run soup eval gate or soup ship on its output.#2564 by @vaputa; #2264 by @deven1003
Upstream's commit message: soup runs clean "deleted the checkpoints it promises to keep whenever two runs shared an output directory", which output: ./output run twice in one project gives.Something you ran may be lost: soup runs clean on v0.77.2 where two runs recorded one output directory (checked in the code at v0.77.2: each run is cleaned against its own best checkpoint). Check which checkpoints you still have; nothing restores a deleted one.#2812 by @abdulwaarith0 (reported as #2724)
Upstream: for a soup loop whose served model was a registry://<id> reference, "the pre-wired train stage trained hf-internal-testing/tiny-random-gpt2 and reported success."An adapter from a pre-wired soup loop iteration for such a served model. Checked in the code at v0.77.2 (_render_dpo_yaml substitutes that model for a registry:// reference); the first affected release is not named. Run soup loop set base_model <hf-id-or-path> and train again.#2506

These six are the changes where a finished file or run is wrong, may be lost, or was never checked. A change that moves a number without a wrong artifact, such as the weights of soup adapters merge, is under Numbers that moved.

Refused now

Each message names the field and what to change; Settings Soup refuses at load has them, with exit codes.

  • A boolean in a numeric field of training or training.lora (lr: true, batch_size: true, lora.r: false) stops with training -> lr: lr must be a number, not a bool (#2462 by @Samearth17); v0.77.2 coerced false to 0 and true to 1. YAML's other boolean spellings are refused too, and data is not covered. lora.r: false used to mean full fine-tuning: write r: 0 if you meant that.
  • training.vllm_sleep_mode: true on task: grpo never reached a vLLM engine; delete the line (#2773).
  • The eval gate. An enabled gate (or --gate) on backend: mlx and on task: prm, unlearn and moe_lora_routing (#2564); baseline: previous, even with enabled: false (#2640: use the path of a baseline results file); and a training.eval_gate.suite that cannot load, which stops soup train and soup train --dry-run with exit 1 (#2264) while soup doctor --config exits 0. A baseline path that cannot be resolved is caught only at trainer setup, after the model loads.
  • Suites, rubrics, calibration and data. A boolean threshold, a repeated key in a task mapping, a non-string top-level key, duplicate rubric criteria, a non-finite rubric weight and a calibration file with calibrated: true and num_pairs: 0 are refused, and --write-baseline onto an input exits 3 (#2697, #2873, #2021, #2699, #2701, #2513, #2694). A query_id that is a boolean, a float, a list, an empty or over-long string, or of mixed types, for reranker and cross_encoder, is refused at data load, and --dry-run makes that check (exit 1; #2637). At trainer setup, which --dry-run does not reach, task: embedding rows where only some have a negative and a task: unlearn retain_set with no usable row are refused (#2563, #2577); a KTO row outside {prompt, completion, label} is refused before any model loads, and --dry-run makes that check (#2435).
  • Wrong types, unknown keys. A wrong-typed value in any of 38 training fields (upstream's count) is a config error naming training -> <field>, where it was a bare TypeError (#2382). An unknown key with no close match ends with a next step (#1823).
  • soup train. --push-as <repo> creates a private repository and --push-public (needs --push-as) a public one (#2802); if the Hub refuses the private create, the message says to pass --push-public, and Soup does not retry as public by itself. --cloud refuses local input files at plan time, and --push-as or --gate beside it (#2627). --resume and --hf-resume on task: ppo are refused (#2772).
  • One Ctrl+C no longer stops a Hugging Face Trainer run. The first saves and continues, the second saves and stops, the third interrupts; <output_dir>/.checkpoint_now is read at each optimizer-step boundary (#2414 by @kok-o). It covers a Hugging Face Trainer run, not MLX and not task: unlearn, and upstream lists the three-signal behaviour among what it did not exercise by hand.
  • soup serve and friends. --max-tokens above 16384 on soup serve, and below 1 on soup chat and soup bench infer (as is soup bench infer --num-prompts below 1 without --prompts-file), exit 2, as does soup chat or soup merge with a --backend other than transformers or mlx. --trace-log and --record-thumbs are refused on --backend vllm, sglang and mii, where they were silently ignored, and so is the new --feedback-log; drop the flag or serve with --backend transformers. These recording-flag checks and the soup serve --max-tokens range check run once the server extra is installed; a core install stops first with the missing-extra message, exit 1. The --adapter-dir refusals come before that check and exit 2 on any install.
  • Six options refuse nan, not every float option: soup data validate --min-valid-fraction, soup draft measure --min-acceptance, soup data toxicity, decontaminate and score --threshold, and soup data forge --uncertainty-threshold (#2917 by @Keshemir).
  • More, before anything is written: soup data sample with -n below 1 or a --pct outside (0, 100]; soup iterative-dpo with a prompts file that has no usable line (an empty file or only blank lines; rows whose prompt is a blank string still count); soup tokenizer train --output naming an existing file; soup can run on a can holding <redacted> values, with --yes (without it the confirmation panel exits 1 first); and soup ship with an --output or --emit-evidence that is one of its inputs or with a pickle adapter (soup ship). The rest are in the exit-code table.
  • soup data generate now asks again until --count is reached, and after three requests in a row with no valid row it saves what it has and exits 1 (#2418).

Numbers that moved

These change a number, a file or a default without a wrong artifact behind them, so they are not Re-run rows. Where upstream gives an instruction it is quoted.

  • soup adapters merge weights (#2927 by @abdulwaarith0). Each input's own term is scaled by its weight, where v0.77.2 used the weight squared: every linear, dare and svd merge of two or more inputs changes (an equal-weight pair doubles each own term, 0.25 to 0.5), and the weights an earlier --strategy cmaes run printed reproduce a different adapter. Upstream: "Re-run merges whose output you compare or ship." soup adapters arithmetic without --rank now gives a lora_B.bias tensor (adapters saved with lora_bias=True) its coefficient, not the square root of it.
  • scoring: "tool_call_args_subset" (#2939 by @AlibekBerik). A row counts as Correct only at full credit, so soup eval custom, soup ship in metric mode and soup eval retention report a lower Accuracy on suites that use it, and the exit status can change for an unchanged model: a soup ship that ended 0 can end 2. Upstream: "Record base and tuned scores again with this version before comparing."
  • training.curriculum: true trains in sorted order (#2924 by @abdumutalipabay0). Through v0.77.2 the default sampler dealt the sorted rows out in a seeded random order (upstream's words). Pre-tokenised data, packing: true and multipack: true keep the shuffled order.
  • The VRAM pre-flight prices differently (#2749, #2871, #2925, #2934, #2950): an 8-bit run with gradient checkpointing, layer streaming (its cuBLAS workspaces) and a task: distill run (its teacher) cost more, and a padded text SFT run that was refused because of data.max_length now starts when its longest row fits. A run that launched close to the card's limit can be refused. Upstream states that the former 0.85% accuracy bound no longer applies and that "the preprint's validation describes the estimator of v0.77.x"; the preprint is unchanged. The release body's remedy list is "--allow-oom-attempt, a lower batch_size or data.max_length", but --allow-oom-attempt does nothing for a layer-streamed run, because the hardware-fit pre-flight returns before it when stream_layers is on: lower training.batch_size or data.max_length, or use stream_vram_probe (task: sft only) or stream_vram_override. Detail: layer streaming at scale.
  • soup profile --config and the MCP profile tool do not take the seq_len and shape the release notes name; those are arguments of the Python function profile_from_config(seq_len=, shape=), and soup profile --model has --seq (which v0.77.2 already had). The function now returns a truthy FitUnavailable instead of None when a run cannot be priced, so test isinstance(result, dict).
  • Files and servers. ~/.soup/experiments.db gains nullable columns in place: loss, lr and speed may be NULL where v0.77.2 stored 0.0, and a database written by v0.78.0 stays readable by v0.77.2. soup ui and the MCP run tools return output_dir relative to the working directory, or only its last path component when the folder is outside it (#2807); soup ui sends non-finite numbers as null (#2750); and for a run id prefix that matches several runs the web UI and the MCP runs_show tool name the ambiguity (#2470, #2752); a prefix also resolves in the metrics and compare routes (#2794). soup audit-log tail skips a JSON line that is not an object (#2460). A --trace-log line is a version-1 conversation record (ts an ISO 8601 UTC string, tokens an object), and a line whose schema_version is not 1 is skipped by soup loop and soup data from-traces. Crash bundles and .can files are redacted before they are written, pattern-based and best-effort: read a bundle before sharing it (#2883). RL-checkpoint optimizer state loads with weights_only=True (#2916). soup export now adds artifact rows to the registry entry of the run that wrote --model, where it used to attach nothing (#2726; --registry-id overrides).
  • GGUF and MLX installs. The llama.cpp pin moved from b5270 to b12423 (#2910 by @1DMB1): delete ~/.soup/llama.cpp to have the next export clone the pinned tag, by tag and not by commit, since an existing checkout is reused as it is. "Qwen3.5 GGUF export works on a fresh checkout" is upstream's statement and was not exercised here. The mlx extra needs mlx>=0.31.2 (#2779).

Runs that are not bit-comparable with v0.77.2. The release body lists these; each changes the numbers of a run, not whether its adapter is valid.

  • Resident 4-bit and 8-bit runs on a bf16 card keep embeddings, norms and an untied head in bf16, not fp32 (#2683). The cast-back applies only to tensors that loaded as bf16. The record, gate-2609-kbit-fp32-ab.md, is an A/B on soup-cli 0.77.2 run before the change, not a run of v0.78.0 against v0.77.2: arm A the unpatched default, arm B the same run with the embeddings, lm_head and norms cast back to bf16 after PEFT's call. It reads, on one RTX 4090 at a 450 W power cap, Qwen2.5-7B-Instruct NF4, LoRA r16, batches of 8 x 512 tokens, gradient checkpointing, three seeds: unprofiled median step 2.2562 s (A) to 1.6949 s (B), 24.9% shorter, allocator peak 3.8 GiB lower (17,425 to 13,520 MiB), loss inside a tolerance written before the run. The record's verdict that the fp32 cast is the whole cause is PARTIAL; 8-bit, tied embeddings, other models and another card were not measured (entry).
  • fp16 auto_mixed_precision on a bf16 card loads the frozen base in fp16 (#2921); the BOS token is kept after a keep_end prompt cut in KTO, BCO, ORPO and SimPO, changing only rows whose prompt was cut (#2544, #2561, #2572); DPO, ORPO and MLX preference training cut a prompt only when the row overflows (#2555, #2659); soup data split --stratify and --stratify-semantic give the requested validation and test counts, so one seed can place different rows (#2954).
  • The Fast-LoRA 4-bit dtype boundary (#2383, #2546); soup ship on a CUDA card without bf16, which evaluates in float16 (#2669); task: pretrain resumed in 4-bit or 8-bit, which keeps its LoRA adapter in bf16 (#2859, #2936; an unresumed run is unchanged); and the four tasks of the third Re-run row.

Exit codes that moved

v0.78.0 keeps the eleven commands of v0.76.0 on one scheme (0 pass, 2 gate failed, 3 usage or input error, a mistyped flag 3) and adds soup eval retention and soup data leakage, which makes thirteen in the code; upstream's docs/commands.md still names eleven. A CI step that passed on v0.77.2 can fail. Each row was run on the v0.78.0 binary beside v0.77.2; the full table has the messages.

CommandConditionv0.77.2v0.78.0
soup data mix --applyYAML that does not parse, or a weight that is not a number12
soup expecta suite or dataset that exists and cannot be opened (permission denied)13
soup data validate--min-valid-fraction nan0, the gate was off3
the other five nan optionsnan0, 1 or accepted2, before any input is read
soup drift-alarmrows that hold no tokens0, OK1
soup adapters mergeinputs it cannot merge0, an adapter with no weights2
soup airgap-bundlea missing folder0, a bundle listing no files1
soup ci init --python03.11 or 3.01101
soup train--resume or --hf-resume on task: ppoaccepted1
soup train --cloudlocal input files, --push-as or --gate0, a plan printed2
soup ship--output or --emit-evidence that is one of its inputs0, and the input replaced by the verdict3
soup shipadapter weights that are a picklea warning, then the live run3
soup ship --evidencea schema_version field31 and 2 are read; 3 for any other value
soup serve--max-tokens outside 1 to 16384; recording flags on vllm, sglang, miiaccepted2, once the server extra is installed
soup eval gatea non-string top-level key, a boolean threshold, a repeated key in a task mapping, --write-baseline onto an inputloaded, or 13

Two rows of the release notes are wider than the code. soup expect moved from 1 to 3 only for an input that exists and cannot be opened: a missing file, invalid YAML, invalid UTF-8 and a directory were already 3, so exit 1 now means that the judge was unreachable (an unexpected error inside the run is not caught either, and also ends in exit 1). soup data mix --apply moved from 1 to 2 only for unparseable YAML and non-numeric weights: a negative, zero, NaN or infinite weight is still accepted, with exit 0.

A continual-learning loop

On the transformers backend, soup serve gives every answer on /v1/chat/completions and /v1/messages a response_id (also the X-Response-Id header). --trace-log writes one version-1 conversation record per answer, --feedback-log turns on POST /v1/feedback (a thumb, a rating, an edit or a comment, by id), and soup serve --adapter-dir with POST /v1/adapters/load loads a LoRA adapter into a running server without a restart. soup loop trains on those thumbs, ratings and edits with sft (the new default), dpo or kto (the feedback log also keeps comments, which nothing trains on or scores), paraphrases the prompts people corrected, judges a candidate with soup ship --gate loop-v2 by the corrections it learned, and starts a canary on 5% of keyed traffic. A canary takes 5% of keyed traffic and is judged twice: by request errors and by the ratings on its answers. The rating verdict withdraws it when its share of negative ratings (a thumb down, a rating of 1 or 2, an edit) is more than 5 points above the stable side's and a one-sided test calls the gap significant, which needs 30 rated answers on each side; a canary whose requests failed more than the stable side's is withdrawn too. Both withdrawals need auto-roll on regress: it is on for a canary the loop starts itself and by default for soup loop canary (--no-autoroll-on-regress turns it off). If the ratings have not arrived when --canary-max-wait runs out (24 hours by default), --canary-on-timeout decides: errors-only, the default, lets the request errors alone decide with however many answers there are, so a canary can be promoted with its quality unverified, and the iteration notes say so; rollback withdraws it; wait keeps waiting. A canary that got no answers at all is withdrawn, unless errors-only was given explicitly. Upstream plans to change that default from errors-only to rollback in v0.79.0, so that a canary without 30 rated answers on each side at the end of the wait is withdrawn; tracked as #3083. Both options belong to soup loop init. New with it: soup loop set, soup loop rollback, soup loop watch --detach (a PID file and a log) and soup ship --base-adapter. soup loop canary refuses an adapter whose newest iteration ended with a DON'T SHIP verdict that stopped the rollout, named or given as the path of its directory under .soup-loops/adapters (an iteration whose DON'T SHIP still went to a canary, or that was promoted or withdrawn, is not refused); --allow-dont-ship overrides it and the override is recorded. A new soup loop init makes an sft loop (existing loop.yaml files keep their method; --train-method dpo keeps preference pairs), and a hand-edited loop.yaml whose monthly_budget_usd or spent_this_month_usd is not finite or is above 1e15 is refused, exit 2. The six JSONL formats are in upstream's docs/loop-formats.md. The loop is the maintainer's work, folded from 17 stacked pull requests into #2897.

Upstream did not exercise by hand the deploy and the canary after a SHIP verdict (its one iteration ended DON'T SHIP), a canary decided by 30 real ratings, or a paraphrase endpoint other than the served model. This page makes no quality claim for the loop: upstream says its default recipe "regressed small evaluation suites in a study on one project", and a poisoned edit that the model learns passes the loop-v2 check (Known limitations). Detail: soup loop and serving.

soup ship

  • What the number is made of. Per task it counts the rows that got better, worse and stayed the same, the answers cut off at the token limit and the failed generations. --output verdict.json also writes verdict.task-rows.jsonl on a live --task-mode metric, judge_score or reward run (not pairwise, not a replay), and the panel prints how much of the base the adapter's update changes (display only) (detail).
  • Evidence schema_version: 2. The --output JSON always carries it; --emit-evidence writes it only when it has version-2 content ("a replayed version-1 file stays a version-1 file"), which is narrower than the release body's sentence. v0.78.0 reads versions 1 and 2; soup-cli 0.77.x refuses version 2 with exit 3, so upgrade whatever reads evidence (a CI gate, the MCP ship_evidence tool) first.
  • --task-mode reward --reward-fn <accuracy|format|verifiable:math|verifiable:json_schema|path.py> scores each answer from 0 to 1. A .py file is executed with your permissions, only when named, and its hash is recorded; the hash says which file scored a verdict and is not a defence against a hostile file. verifiable:code and judge_rank stay refused (detail).
  • --base-adapter compares two adapters on one base, --gate loop-v2 is the loop's verdict, and soup eval retention measures forgetting across versions against a ledger of pinned control tasks, outside the soup ship verdict (base-adapter, loop-v2, retention).
  • Refusals. An --output or --emit-evidence that is one of the run's inputs, or the other output, exits 3 before any model loads and before anything is written (it used to replace the input with the verdict). An adapter whose weights are a pickle (adapter_model.bin, .pt, .pth, .pkl, .pickle, .ckpt, or a pickle posing as .safetensors) is refused on --adapter and --base-adapter, exit 3: re-save it as safetensors. Upstream files this under hardening. A CUDA card without bf16 evaluates in float16 and stamps float16.

Serving

soup serve --backend transformers streams while it generates on /v1/chat/completions and /v1/messages, and honours stop and stop_sequences (a bad value is a 400; vLLM, SGLang and MII do not forward stop). --reasoning-parser, a draft model, n-gram lookup, MoLE and beam search still generate the whole reply first, and on Windows a client disconnect may not stop generation promptly. No latency figure is claimed. --tool-auth-token is checked on the tool, thumbs, feedback and adapter routes only; POST /v1/adapters/load needs a token on every bind, loopback included, and generation, /v1/models, /health and /metrics still carry no token on any backend (serving).

Layer streaming

  • Seven tasks stream at v0.78.0: sft, dpo, orpo, simpo, kto, bco and ipo. task: preference streams with one preference_loss (dpo, orpo, simpo, bco or ipo); a preference_loss_weights blend is refused (tracked as #2766). grpo and ppo are refused, and that refusal names no release. BCO, IPO and task: preference are by @here-2007 (#2896, #2915, #2843).
  • The reference model. DPO and IPO keep a frozen ref adapter beside the policy adapter on the same streamed base; KTO and BCO take the reference with adapters switched off; ORPO and SimPO have none. No second copy of the base weights is held. The v0.72.4 memory record was measured on trl 0.19.1 with a synthetic fixture and was not re-measured for the adapter snapshot (detail).
  • Records, with their scope. bco-streaming-t4.md and ipo-streaming-t4.md are one Kaggle Tesla T4 and a two-layer vocabulary-64 fixture: one optimizer step for BCO; for IPO, one step at an initial gradient scale of 1024 set in the harness, plus default-setting runs that skipped four updates and applied the fifth. They show execution, an update and adapter persistence only (for IPO, also finite updates and scaler recovery). IPO's four original default-scale cells failed (a NaN gradient norm, the first update skipped by the fp16 gradient scaler) and the record keeps them; on an fp16 card the scaler can skip IPO's first updates (resident IPO does the same, in upstream's words). No tok/s, peak VRAM or speed figure exists for BCO or IPO, and KTO still needs batch_size >= 2.
  • More on the streamed base. A double-quantised bitsandbytes NF4 checkpoint with bf16 compute imports byte for byte and trains under stream_layers: true with quantization: 4bit, on a dense decoder and a bf16 card; 8-bit, FP4, GPTQ and AWQ checkpoints stay refused (#2628, #2653). Forward-only streamed inference is a Python API with no KV cache, no generation and no command (#2509 stays open). soup prune-layer-cache removes stale cache slots and is a dry run unless --yes. The packed-state backward (#2655) and the Windows commit-headroom check (#2658) are on the scaling and disk-tier pages.
  • Kimi K2. A guarded resident route for Kimi-K2-Base and -Instruct on sft and grpo at quantization: none, on tiny-checkpoint evidence only: CPU parity against K2's own modelling code, no real K2 weights, no GPU, nothing trained on K2. The kimi-k2-sft recipe stays unavailable (soup recipes use exits 3), and Kimi-K2-Thinking and -Instruct-0905 have no supported route (#2759).
  • DeepSpeed, re-measured. Re-measured in v0.78.0 under a pre-registered protocol on one H100: against DeepSpeed ZeRO-3 with CPU parameter offload, set up in a plain Hugging Face Trainer and PEFT script, layer streaming ran at 1.56x the throughput (1.4x to 1.7x across the settings the record ran) at 1.7x less peak VRAM, not the 2.93x and 9.7x of the earlier record. Layer streaming is slower than every DeepSpeed arm that keeps the weights on the GPU. What it buys is VRAM on a card the model does not fit, not speed on a card it fits. One H100 host, 8B bf16, LoRA r=8, batch 1; absolute tok/s moved about 2.3x between two H100 hosts, so quote ratios; no run below 16 GiB. The earlier row stays on external validation as history, reached through Soup's own --deepspeed flag; the record did not isolate why the two differ. The preprint is unchanged.

Apple Silicon

soup chat --backend mlx and soup merge --backend mlx (a dense checkpoint; --dequantize for a 4-bit source) are new. MLX text SFT now takes lora.r: 0 (full fine-tuning of a dense Llama), rope_scaling_type: linear on unquantized bases, vocabulary expansion with trainable new tokens (its adapter is refused by soup infer, soup chat and soup merge), freeze_layers and freeze_ratio on a full fine-tune, the two selective loss masks and existing preprocessing caches. All of it is covered only by the hosted macOS MLX test job and fixtures, in upstream's words; no speed, memory or time figure is claimed. soup doctor --config now reports eleven backend and task combinations. See the MLX backend.

New commands, tracking and recipes

  • Commands. soup eval retention, soup data leakage (exact input-text overlap only), soup docs search (offline search of the bundled guides) and soup prune-layer-cache are new: 80 top-level commands, 30 of them groups with subcommands, and 261 help screens, with no command or option removed. Also new: soup eval quant-check --performance, soup export --format onnx --validate-onnx (an FP32 smoke check, not a quality threshold; it needs a separate export-only environment), soup diagnose <run_id> --evidence <file> --require-all-modes, soup train --push-public, soup train --diagnose-gate-require-all-modes and soup deploy hf-space --hardware. soup export --format torchao now merges the adapter first.
  • Tracking. The tracker records a GRPO run's per-step score and spread beside the loss; the live panel shows zero-spread groups and an estimated finish time; runs record the project's Git commit and clean or dirty state (a startup observation, not an attestation); soup runs compare shows full config differences and the latest stored evaluation scores. task: classifier reports accuracy and F1, and reranker and cross_encoder report ranking quality when validation rows carry query_id.
  • More fixes. soup train --deepspeed no longer stops before the first step under fp16 (#2276); the GRPO watchdog names the parameters whose gradient was not finite, read before the clip, and records nan_param_names_source (#2645); resuming a DPO, IPO, preference or GRPO run on transformers 5.23.0 reloads the adapter only when the Trainer did not, and upstream says no migration is needed (#2822); soup sweep applies the quantization guard of soup train and binds each arm to its base revision; RunPod jobs use REST API v2, because RunPod announced the retirement of REST API v1 for 2026-11-15 (upstream's statement); the built-in Gradio and Streamlit Space templates no longer raise TypeError when the Space runs, so redeploy (#2652, #2657); soup migrate --from axolotl with a custom prompt-format type: writes data.format: auto and one warning where it crashed; an explicit zero VRAM headroom in soup autopilot counts as zero (#2885); soup recipes search --size 7B no longer matches 17B (#2930); soup local-rl harvest, nightly local-RL training and the pre-wired loop skip a prompt whose latest up- and down-vote carry identical responses (#2521); soup iterative-dpo reads a UTF-8 BOM first prompt (#2943). More than 30 pull requests make commands print paths and names as typed (what is left: #2489).
  • Text shown as typed. More than 30 pull requests, by upstream's count, make commands print paths, names and messages as typed: brackets and :name: text are kept, and terminal control characters are removed from the display only. Some commands still print an exception's text unescaped; that is tracked as #2489.
  • Recipes. Eight, 181 in all: qwen3.5-4b-grpo, qwen3.5-27b-grpo, qwen3.5-122b-a10b-dpo, qwen3.5-122b-a10b-grpo, qwen3.5-397b-a17b-grpo, minimax-m3-dpo, qwen3.5-9b-pretrain and qwen3.5-2b-pretrain. They use inherited settings and are not tuned or measured (tracked as #275). soup doctor --config accepts the config 179 of the 181 recipes write; the two that refuse are kimi-k2-sft and kimi-k2-thinking-grpo (catalog). soup_cli.kernels.packed_lora (#2623) is a library of Triton kernels with no config key; soup train does not read it.

Measurement records

benchmarks/ holds 84 records at the tag, 29 of them new in v0.78.0 (index). One is worth naming: nf4-on-blackwell.md is a post-#1143 status record, not a gate. On one RTX 5070 Laptop the shipped streaming path matched a resident NF4 model on logits, LoRA gradients and a 20-step loss curve, for one 0.5B model at sequence length 64; #776 is closed. It does not establish exactness at other sizes, other Blackwell parts or older architectures.

Deprecated

  • training.warmup_auto: true and a training.forgetting_threshold set away from 0.1 load with a warning that they do nothing; upstream says v0.79.0 refuses them (#2814). Delete them (messages).
  • soup deploy hf-space with the Streamlit template prints a deprecation warning (#2834); the warning is all that is documented.

Security

This release carries 3 entries under Security in the CHANGELOG. No advisory is published with it. A security review of the code merged for this release, run on 2026-11-17, recorded 9 findings (0 critical, 1 high, 1 medium, 7 low): 5 are fixed in this release and 4 are listed under Known Limitations in upstream's release notes (this page's own list is below). Supported versions: 0.78.x is supported; 0.77.x gets security fixes only until v0.79.0 is released; below 0.77 is unsupported; v0.76.x gets no further security fixes after this release.

Known limitations

Upstream's list, shortened. "Tracked as #N" marks an open issue. Most of it is about the new loop; its page has the full list.

  • The loop's operations. .soup-loops/ grows with every training run and nothing is pruned or rotated (#3015). There is no soup loop stop, soup loop init --force does not check for a running daemon, and two foreground soup loop watch processes are not guarded against each other (#3016, #3017). A refused deploy is attempted once per poll, three times in all (the first included), with no back-off; then deploy by hand with soup loop canary <adapter directory> (#3018). Run the loop in a directory you control: a Python process that soup train or soup ship starts in turn does not drop the current directory from sys.path (#3019, #3020). quantization: auto picks 4-bit when the Hub is offline; pass --sft-quantization none (#3033).
  • The loop's feedback is training data. On a loopback server started without --tool-auth-token any local process can post to /v1/feedback: use a token on a shared machine. The loop does not check an edit for intent, and the loop-v2 gate checks a candidate against the corrections it was trained on, so a poisoned edit that the model learns passes that check; the canary's ratings and soup loop rollback are the other defences.
  • The loop's quality. The default recipe (all linear modules, r 64 / alpha 128, lr 2e-4, 8 epochs) "regressed small evaluation suites in a study on one project"; milder defaults and anchors are untested. A 40-row suite's base side, captured nine times on the same merged model between soup ship calls, read 0.925 eight times and 0.900 once, so on that suite one item is 0.025 and a 0.05 threshold is two items. The numeric scorer behind mini_arithmetic counts a correct answer written with a thousands separator (32,768) as wrong (#3014, #3012, #3013, #3065).
  • The manual canary check is narrow. The --allow-dont-ship check of soup loop canary applies to the adapter's own directory under .soup-loops/adapters, not to a checkpoint directory inside it, and on a case-insensitive file system to a run name written in the letter case the run was recorded with (tracked as #3068).
  • A live soup ship verdict is slow on a laptop card. A full verdict for a 0.6B model (Qwen3-0.6B) took 35 minutes for 20 task rows plus the eight bundled mini suites on an RTX 5070 Laptop (8 GB), with other work on the machine, at about 95% of one CPU core and 7-12% GPU utilisation. It prints nothing between loading the models and the result panel unless you pass --progress jsonl (tracked as #3011).
  • The VRAM estimate is fitted on small dense models. Its 8-bit checkpointing term is calibrated on three (Qwen2.5-0.5B, Qwen2.5-1.5B, SmolLM2-360M), gradient_checkpointing: medium is outside the range and only warns, a distillation step counts as outside it, and a micro-batch that fills 512 tokens can still need more than the estimate (#3021).
  • Layer streaming. Streamed inference is forward only (#2509 stays open); preference_loss_weights blends are refused (#2766); the feature remains BETA and opt-in. training.fast_lora has no speedup claim and its correctness half is open (#792).
  • Platforms and exports. The MLX features are covered only by the hosted macOS MLX test job and fixtures. The RunPod controller uses REST API v2 from the published reference and has not been run against a real account (#2877, #2098, #2192). The Kimi K2 route rests on tiny-checkpoint evidence only (#2759, #2104). GGUF export clones llama.cpp b12423 by tag, not by commit. soup merge --low-memory refuses DoRA, modules_to_save, rank or alpha patterns, embedding adapters and PiSSA, OLoRA, CorDA, LoftQ or LoRA-GA initialisations.
  • Partial fixes whose issues stay open. Settings a backend or task never applies are refused only on the paths fixed so far (#1811). The base-revision binding covers the transformers backend, soup merge, soup sweep and native MLX text training, not Unsloth loads or local and cached-path hints (#1902). soup data leakage finds exact input-text overlap only (#2672). The recipes above use inherited settings (#275). Text shown as typed (#2489), fields that reach no consumer (#1814) and soup doctor --config coverage (#1815) also stay open.
  • Not exercised by hand before this release, in upstream's list: the deploy and the canary after a SHIP verdict, a canary decided by 30 real ratings, a paraphrase endpoint other than the served model, soup diagnose <run_id> --evidence <file> --require-all-modes, soup eval quant-check --performance, soup eval retention with live model loads, soup runs compare, the three-signal Ctrl+C, and Triton, MLX, vLLM, SGLang, MII, ONNX, torchao, cloud submission and Hub writes.

Upstream lists as fixed and no longer limiting: stale layer-stream cache slots (#2196), freeze_layers with a string target_modules (#2201), config errors without a next step (#1816), GGUF export of Qwen3.5 (#1911), native MLX soup chat loading (#1986), the BOS token lost in KTO and BCO (#2487, #2486), Windows quoting in soup adapters bisect --eval-command (#2488), soup ci init --python 03.11 (#2534) and judge rows checked only during generation (#2535).

What was checked, and what was not

Run on the v0.78.0 binary (a light install with no Torch, MLX, server extra or GPU, offline, on tiny local inputs, beside the v0.77.2 tree where a before and after was wanted): the exit-code rows, the config refusals through soup doctor --config and soup train --dry-run, the soup ship option refusals, soup data leakage, soup docs search, soup prune-layer-cache, the state commands of soup loop, the soup serve startup refusals with the server imports stubbed, and soup adapters merge. Read in the code and not run: a live loop cycle, soup serve streaming, live soup ship runs, Ctrl+C, MLX, the pre-flight and layer streaming, and the Hub, RunPod and GGUF paths. Not measured, and not claimed: an MLX speed, memory or time figure; any figure for BCO or IPO; a soup serve latency; any quality gain from the loop. Test files go from 1220 to 1493 and the catalog from 173 to 181 recipes, counted at the tag.

Credit

218 of the 271 pull requests in this release came from 48 people outside the maintainer, in the release body's count, in its order of merged pull requests. The maintainer's pull requests, the loop among them, carry no credit line here.

WhoPRsWhat
Nariman Abdikarim (@Nari-Ab)45Input handling across the CLI: soup data mix --apply, soup expect, BOM-prefixed JSONL, soup iterative-dpo
AbayAbdumutalip (@abdumutalipabay0)18soup merge --low-memory refusal, curriculum order, classifier-family fp32, four recipes
Arman (@AubakirovArman)17soup ship in float16, quant-check --performance, ONNX validation, soup data leakage
shubham-dev-stack (@Shubham-dev-stack)12soup airgap-bundle checks, soup ci init, judge rows validated first
Harshit Sharma (@harshitthek)9Space templates that publish in one commit, a queued streamed request
Pernav Jain (@here-2007)9MLX vocabulary expansion, RoPE and dense SFT; soup docs search; streamed BCO, IPO and task: preference
Abdulwaarith Zakariyya (@abdulwaarith0)8soup runs clean, --max-tokens bounds, soup adapters merge
CapyRiver (@capyriver)8cuBLAS workspaces in layer streaming, Space hardware, two records
Yang Bai (@ybai08)8BOS kept after a prompt cut, partial negative rows, retain_set
Faisal Fayaz (@Faisal-Fayaz)7LoRA dtype at the 4-bit boundary, fp32 SwiGLU add, --max-tokens below 1
David (@1DMB1)6The Kimi-K2 resident route, the llama.cpp pin, tokenless inputs
Nurkhan Esenbek (@kok-o)6Manual checkpoints and Ctrl+C saves, identical thumbs pairs, three Qwen3.5 recipes
Tal (@taljeon)6Eval-gate input hardening
Alibek Berik (@AlibekBerik)5Ranking metrics, classifier accuracy and F1, tool_call_args_subset
Aditya Pratap Singh (@akkupratap323)4Run id prefix messages, malformed curriculum history, Hub shards for --low-memory
AmixDigital (@Amix29)4Native MLX chat, merge, loss masks and preprocessing caches
Rymbek Nazerke (@nazek52)4Full local run comparisons, estimated finish time, prompts kept when rows fit
nurik (@noncalan)4Paths, ids and errors shown as typed
vaputa (@vaputa)4Refused eval gates on unsupported paths, base-revision binding

Two pull requests each: Ahmet (@ANO3457; #2418, #2572), artikulus1301 (@artikulus1301; #2761, #2831), DEVEN KALATHIYA (@deven1003; #2264, #2267), Samarth Mukhija (@Samearth17; #2462, #2954), Srinivasan R (@Srinivasan8888; #2555, #2688). One each: Aaron Probha (@AaronProbha18; #2647), Batyrkhan (@Batyrkhantzy111; #2269), Dinmukhamed Bekeshev (@BDaaac; #2640), Alexis (@bernalalexis-try; #1823), zgy (@bfs2021; #2310), Can Akyuz (@canakyuz; #2252), Salil Mhatre (@Deadpool2000; #2807), Erizoned (@Erizoned; #2857), George Tonmoy Roy (@george07-t; #2021), Izumx (@Izumx; #2918), Emil Jafarov (@jafarovvvvvvv; #2681), Keshemir (@Keshemir; #2917), nurpeisovnurbolofficial (@nurpeisovnurbolofficial; #2794), PANDIARAJ (@Pandiaraj-1; #2435), Sajjad P. Savoji (@SajjadPSavoji; #2513), Shivansh Shukla (@ShivanshShukla; #2378), Shu Chen (@shu-chen-dot-ca; #2382), Shutaru (@Shutaru; #2644), Siddhardha Nanda (@SID-6921; #2690), Sofia5050 (@Sofia5050; #2779), Vansh Sharma (@VanshSharmaPES; #2276), Kratos (@wanjinhao1; #2259), xiemocun (@xiemocun; #2289) and Ziyadinkhan Kudaibergenuly (@Ziyadin23; #2687).

Reports that led to fixes, in the release body: @abdumutalipabay0 (#2787, #2788, #2794, #2796, #2812, #2855, #2885, #2898, #2928), @Batyrkhantzy111 (#2414), @baurvtm-stack (#2918), @duysengali-kuanysh (#2689, #2695, #2819, #2833), @Faisal-Fayaz (#2781), @noncalan (#2877), @Samearth17 (#2418) and @taljeon (#2513, #2838, #2900, #2911, #2955, #2979).

See also

Soup is free and Apache-2.0. If it saved you a training run, starring the repo costs nothing and helps most. You can also fund the GPU time behind the work a 4 GB laptop cannot reach.