soup shrink: make a model smaller (v0.71.29)

v0.71.29 ships soup shrink: one command that makes a model permanently smaller and faster by removing its least-useful decoder layers, optionally healing the damage by distillation, and printing a single SHIP / DON'T-SHIP perplexity verdict. The output is one dense smaller model, not a base plus an adapter.

It implements "The Unreasonable Ineffectiveness of the Deeper Layers" (Gromov et al.), with a Minitron-style distillation heal in place of the paper's plain fine-tune.

We have not found one-command importance-ranked depth pruning, with a distill-heal and a binary perplexity verdict, in another fine-tuning CLI.

How it works

1. Importance scan. For each contiguous block of decoder layers, Soup runs one forward pass per calibration prompt and measures the angular distance the residual stream travels from the block's input to its output, averaged over every non-pad token. A block that barely changes the residual stream is doing little work and is the safest to drop.
2. Select + prune. The least-important block is removed. The first and last decoder layers are always protected (they carry the most transformation). The model is sliced, its num_hidden_layers patched, saved, and reloaded so the surviving layers re-index cleanly.
3. Distill-heal (optional). With --heal, the full-depth original (teacher) is distilled into the pruned student via a LoRA logit-distillation soup train subprocess. The resulting adapter is then fused back into the pruned base, so the shipped artifact stays a single dense model.
4. Verdict. Perplexity is measured before and after. Soup SHIPs only if the perplexity ratio stays within your tolerance (default 10%), else DON'T SHIP.

The rendered heal config is validated before the subprocess starts: a config that does not load stops the command with Invalid rendered heal config: ... and exit 1. A --trust-remote-code you passed to soup shrink is forwarded to that child soup train (#1449); before v0.76.0 the heal step told you to re-run with a flag you had already given. If the merge of the healed adapter fails part-way, the previous output is moved aside and restored, instead of being deleted before the new one is in place (#1450).

Quick start

# Plan only: print the layer-importance table + chosen block, write nothing.
soup shrink --model HuggingFaceTB/SmolLM2-135M-Instruct --drop-ratio 0.25 \
    --calib calib.jsonl --device cpu --plan-only

# Prune 25%, distill-heal, get a verdict, attach the report to a registry entry.
soup shrink --model HuggingFaceTB/SmolLM2-135M-Instruct --drop-ratio 0.25 \
    --calib calib.jsonl --heal heal.jsonl --heal-steps 200 \
    --tolerance 0.10 -o shrunk --device cpu --attach-to-registry <id>

The shrunk model lands in <out>/model, and the verdict is written to <out>/shrink_report.json.

Flags

The first and last layers are always protected, so the resolved drop count is clamped to [1, num_layers - 2].

The verdict + exit codes

soup shrink drops into CI exactly like soup ship and soup diagnose:

Pruning always raises perplexity, so shrink uses this dedicated ratio rule rather than soup ship's absolute-improvement gate. The perplexity numbers are the same procedure applied before and after, so the ratio is valid, but they are not directly comparable to soup eval / lm-eval absolute scores.

Supported architectures (v1)

Llama, Qwen, SmolLM. Anything else is a friendly reject with a pointer to open an issue. The family is checked twice: fail-fast from the config before weights load, and again on the loaded model.

Models whose config keeps one entry per decoder layer are pruned correctly since v0.76.0 (#1241). layer_types (Qwen2, Qwen3, Qwen2-MoE, SmolLM3) and SmolLM3's no_rope_layers are sliced with the kept layers, so a mixed sliding/full-attention list keeps the right entry for each survivor. (Since closed in v0.76.0: through v0.75.2, shrinking one of those models scored the layers and then exited 1 on save with an empty model/ directory, because transformers 5 refuses to save a config whose layer_types length differs from num_hidden_layers.)

A config that places its MoE layers by layer number is refused before the prune, because the kept layers would reload with MoE and dense MLPs on the wrong layers: mlp_only_layers and decoder_sparse_step (Qwen-MoE family), and interleave_moe_layer_step and moe_layers (Llama4-text) when they are not at their defaults. The message reads cannot prune a model whose config sets <attr>=...: it holds layer indices that would point at the wrong layers after the drop.

Validated results

Live-validated on SmolLM2-135M on a single RTX 3050 (4 GB):

• Drop 25%, unhealed: 30 → 22 layers (~21% of params gone), perplexity x2.98.
• Drop 4 + CPU heal: perplexity recovered to x1.35.

The importance pass loads the model, so it is live-validated on models up to ~3B on that hardware; larger models work but are unvalidated on the reference card.

The second figure (the healed one) was measured at v0.71.29, before the distillation gradient-accumulation fix of v0.76.0 (#742, #943); the unhealed figure involves no distillation. The heal step renders task: distill without setting gradient_accumulation_steps, so the schema default of 4 applies to it. Upstream's wording of the defect: "the same effective batch split four ways previously produced four times the gradient and changed the effective learning rate". No re-measured figure exists, so none is given.

Security

Every path (--calib, --heal, --output-dir, and every derived write path like <out>/model and the fuse staging dir) is held under cwd with realpath + commonpath + O_NOFOLLOW + symlink rejection, re-validated right before each write (TOCTOU defence, including after the potentially hours-long heal). The heal runs as an argv-list subprocess (no shell) with a timeout and a schema-validated config; its output is control-char-stripped before it reaches your terminal. Input files are capped at 64 MiB and 10,000 calib rows. --model defaults trust_remote_code=False.

See also

• soup ship — the SHIP / DON'T-SHIP verdict for a fine-tune; shrink mirrors its exit codes.
• Spectrum targeted training — the complementary move: instead of removing layers, freeze all but the high-signal ones.
• Fine-tune Doctor — pre-flight your data before you prune-and-heal.