Data Moat II (v0.71.36)

A semantic layer over your training data, plus two tools for what a fine-tune forgets and leaks. Four commands: dedup the reworded duplicates MinHash misses, map which topics your data actually covers, plant canaries and later prove whether the model memorized them, and replay a slice of the old task so a new one does not erase it.

Zero new dependencies. The embedding kernel reuses transformers from the [train] extra (mean-pooled all-MiniLM-L6-v2, no sentence-transformers), so dedup --semantic and topics need [train] plus a one-time model download; plain MinHash dedup stays on the light core.

soup data dedup --semantic — near-duplicates over meaning, not shingles

soup data dedup train.jsonl --semantic -o clean.jsonl
soup data dedup train.jsonl --semantic --threshold 0.85 --field text -o clean.jsonl
soup data dedup train.jsonl --semantic --embed-model sentence-transformers/all-mpnet-base-v2

MinHash shingling compares token sets, so a reworded duplicate reads as distinct. Semantic dedup compares embedding cosine instead and catches those: measured, rewordings score 0.88 to 0.91 cosine where MinHash scored them apart, while genuinely-distinct-but-similar instructions are kept.

Honest limit: this is not a paraphrase detector. Measured with all-MiniLM-L6-v2, paraphrase cosines (0.49 to 0.76) overlap genuinely-distinct rows (0.54 to 0.76). "Add two numbers" vs "Multiply two numbers" scores 0.759, higher than the true paraphrase "reverse a string" / "invert the order of characters" at 0.491, so no threshold separates them. Lowering --threshold to chase paraphrases deletes real training rows. The default 0.8 is deliberately conservative (max distinct-pair cosine measured was 0.759) and still strictly beats MinHash.

soup data topics — a coverage map of your dataset

soup data topics train.jsonl                    # 'auto' picks the cluster count
soup data topics train.jsonl --clusters 8 -o topics.json

Clusters the dataset (k-means over the same embeddings), labels each cluster with its distinctive terms (c-TF-IDF), and prints a coverage table plus a warning for thin topics:

82% code · 9% chat · 6% math · 2% sql · 1% legal
topic 'legal' is thin: 1% of rows — likely under-represented

Since v0.76.0 the tokenizer behind the labels keeps digits and words in non-Latin scripts, which it used to discard (#1233), so those can now appear in a label. Only topics under 2% of the data are flagged (a 6% topic like math above is left alone). Labels are emergent term clusters, not a fixed taxonomy, and there is no automatic join to an eval-coverage suite; read the table as "where my data concentrates", not as classification against an ontology.

soup data canary insert|check — prove what the model memorized

# 1. insert K unique high-entropy secrets (keep the manifest OUT of your repo)
soup data canary insert train.jsonl -o canaried.jsonl --count 16 --manifest secrets.json

# 2. train on canaried.jsonl as usual, then rank each secret's loss vs controls:
soup data canary check --manifest secrets.json --base ./my-model --adapter ./lora

A Secret-Sharer memorization probe (Carlini et al.): insert K secrets, train, then check any model or adapter by ranking each secret's loss against never-inserted controls drawn from the same space. Measured on SmolLM2-135M, a memorized set lands at percentile 0.0 (loss 1.7 to 2.5) against a clean model's 4.1 to 6.2. The verdict is a binomial-tail test (α = 0.05) over the count of memorized canaries, and exit 2 on MAJOR lets CI gate a leak.

The verdict is a binomial tail on purpose. The obvious rule, "MAJOR if any single canary ranks in the bottom 1%", fires MAJOR on a clean model about 15% of the time at K = 16 (a CI gate crying wolf one run in seven); the count-based test drops that false-MAJOR rate to ~1.2% while still catching a partial leak of 2 of 10. "No exposure" is the sampled-control approximation, not full-space rank enumeration, so it is evidence of no memorization, not a proof. The manifest is the sensitive artifact and is written 0600 on POSIX.

Flags worth knowing: insert --seed N (default 0) makes the canaries and the rows they go to reproducible, and insert --format auto|alpaca|sharegpt|chatml (default auto) picks the format they are written in (below). check --controls N (default 128) sets how many never-inserted controls each secret is ranked against, and check --device auto|cpu|cuda picks where the loss is computed. check -o report.json writes a report that embeds every secret, so treat it like the manifest: keep it out of the repo.

insert writes the canaries in the dataset's own format, so the loader keeps them. --format auto (the default) detects it from the first 100 rows, as data.format: auto does; --format alpaca, sharegpt or chatml names it. In each, the carrier is the prompt and the secret is the trained response. Every other format is refused before anything is written, naming the format and exiting 1: dpo and kto have no single supervised response, plaintext trains on raw text rather than the chat turn check scores, tool-calling puts a tool-schema system turn before the prompt, and the multimodal formats need a real image or audio file per row. An explicit --format that is not the file's own is refused too. -o takes .jsonl, or .json for a JSON array that loads back.

Canaries are spread through the file, not appended: the file is cut into one equal stretch per canary and each canary goes to a random row of its own stretch (--seed fixes the rows), while the dataset's own rows keep their order. The manifest records the format, the file's row count and each canary's row (counted from 0). insert also prints how many canaries the default data.val_split of 0.1 trains on. On a 200-row ChatML file with --count 8 --seed 1 it printed Inserted 8 chatml canaries: 200 -> 208 rows and 7 of 8 canaries are in the training split; the rest are in the held-out tail. Set data.val_split: 0 for a run where every canary must be trained on.

Re-run this: an earlier "OK" from soup data canary check may have measured nothing. Upstream's changelog says two things about insert before v0.76.0. It "wrote chatml canaries into every dataset, so on any other format the loader dropped all of them and check could only report OK" (#1216). And it appended the canaries, while the loader holds out the file's last rows as validation, so "from 135 rows on every appended canary was held out, no model trained on it, and soup data canary check could only report OK" (#1331). The 135 matches the default of 16 canaries and the default val_split of 0.1: 135 original rows plus 16 canaries is 151 rows, and the held-out tail is 16 of them. A clean verdict on a model trained from a canary file made by an earlier release is therefore not evidence of no memorization if the dataset was in a format other than ChatML, or had 135 or more rows with the default 16 canaries and the default data.val_split of 0.1. In a ChatML file under 135 rows at least some canaries were still in the training split, and with data.val_split: 0 all of them were. Re-insert with v0.76.0, retrain, and check again.

soup train --replay — rehearse the old task so the new one does not erase it

soup train --config new_task.yaml --replay old_task.jsonl --replay-ratio 0.1

Continual-learning rehearsal: interleave a seeded sample of an old dataset into training so a new task does not overwrite the old one. --replay-ratio is the fraction of the final mixed set (n_replay = round(r/(1-r) · n_new)), rows are interleaved rather than appended, an undersized pool reports the shortfall instead of repeating rows, and the mix touches train only, so validation stays pure new-task.

Honest scope: proof-of-mechanism. On SmolLM2-135M + LoRA, replay retained the old task 7% better than a no-replay control (the correct direction), but forgetting without it was only about +4%, i.e. mild, so the effect size at full fine-tuning or 7B+ is unproven on a 4 GB box. Replay v1 is sft / pretrain only and is incompatible with packing / multipack.

Fixes that shipped with it

• The hardware-fit gate refused to train any locally-merged model. A soup merge -o ./mymodel output has no size marker in its name, so the size guesser returned its 7B default, predicted ~16 GB of VRAM, and refused, blocking the exact merge-then-train-from-merged flow that --replay serves. Local checkpoints are now measured from their safetensors header (0.135B actual versus 7.0B guessed). This is the third instance of that class after the v0.71.32 (Whisper) and v0.71.33 (M suffix) fixes, and all three were caught by a live smoke, never a unit test.
• Replay rows now get the same path-traversal validation as the primary dataset (a llava-shaped {"image": ...} replay row previously bypassed the vision/audio guard).

See also

• Data Tools — inspect, validate, convert, merge, and plain MinHash dedup.
• Data Engineering Pro — the dbt-shaped build DAG, soup expect, Magpie, brain-rot (Data Moat I).
• soup ship — the SHIP / DON'T-SHIP verdict a canary leak should block.
• Supply-chain security — the other side of "what leaves the model".