Data Forge & Quality Moat (v0.47.0)

soup data forge — synthetic data pipeline

Chunk → judge → active-prune → JSONL with provenance.

# Smoke test of the plumbing only: no model runs, every answer is a canned placeholder line.
soup data forge --docs ./docs --output ./synth.jsonl --task sft --target-rows 100

# Real synthesis needs a live judge backend.
soup data forge --docs ./docs --output ./synth.jsonl --task sft --target-rows 5000 \
  --judge-provider ollama --judge-model llama3.1 --provenance ./provenance.json

Without --judge-provider nothing is generated. The default judge is a deterministic offline stub that writes Synthesised answer (offline): ... as every assistant turn and exits 0. --teacher is only a label recorded in the provenance manifest (as judge_id); it never loads or calls a model, so a stub run still names whatever you passed. --judge-provider takes ollama (localhost only), anthropic (key from ANTHROPIC_API_KEY) or vllm. --hub modelscope|modelers pre-fetches --teacher only when it is an owner/name repo id.

--task picks the shape of what comes out: sft, preference or tool. --provenance writes a separate manifest, a single JSON document (not JSONL) of {version, row_count, records} where each record is {row_id, source_doc, judge_id, chunk_id, filter_score}, so a generated row can be traced back to the document it came from. The rest tune the pipeline: --uncertainty-threshold for the active-prune step, --max-chunk-chars for chunking, --judge-model / --judge-base-url for the judge, and --hub hf|modelscope|modelers for where the teacher is pulled from.

Each row carries provenance (source doc, chunk id, judge id, filter score).

A failed judge call never becomes a row (v0.76.0). A call that fails (a transport error, a non-200 status, a malformed reply) or returns an empty reply is dropped, whatever --uncertainty-threshold is. The summary reports N of M judge calls failed for --judge-provider <name> (<endpoint>); first error: .... A partial outage keeps the rows that succeeded and prints the panel as Data Forge — synth complete with judge failures, with a Judge calls: N of M failed line. If no usable row was produced the command prints No usable rows produced and exits 1 without writing the dataset or the provenance file. --judge-base-url has no effect for anthropic: every call goes to https://api.anthropic.com, and the error label names that origin.

Re-run this. Through v0.75.2 a failed judge call came back as empty text, was scored as maximally uncertain so that no --uncertainty-threshold could prune it, and was written as a row with an empty answer, after which the command printed synth complete and exited 0. A dataset made by soup data forge --judge-provider ... while the provider was down or rate-limiting can therefore hold rows with an empty assistant turn. Filter for them, or regenerate. (The same release closed the equivalent hole in soup data augment and soup distill-prompt; see Data augmentation.)

soup data score — composite quality scorecard

PII + an abuse-keyword heuristic (not a toxicity classifier) + langdetect + educational value. The Decontaminated cell reads not run (no benchmark texts) unless you supply comparison texts.

soup data score --input ./train.jsonl
soup data score --input ./train.jsonl --benchmark-file ./benchmark.jsonl --threshold 0.8

The plain run prints a table with the rows PII flagged, Abuse-keyword flagged, Edu mean, Decontaminated (not run (no benchmark texts) until you supply a file) and one lang:<code> row per language found. The scorecard reports PII matches, abuse-keyword matches, the language mix and mean educational value. It reads only text, content and messages[].content; an Alpaca row with instruction and output scores as empty.

Decontamination is opt-in and breaking (v0.76.0, #1448). Through v0.75.2 the cell was a fixed 0, which read as a measurement. Now it is not run unless --benchmark-file <jsonl> supplies your own comparison texts (a file under the working directory; rows with text, content or messages[].content, each with usable 8-gram text; --threshold is the overlap threshold, default 0.8). --benchmarks / -b (mmlu, gsm8k, humaneval, truthfulqa, arc, hellaswag) are labels that are validated and nothing more: they do not select rows of that file, and they now require --benchmark-file, so soup data score --input f.jsonl -b mmlu alone exits 1 with score --benchmarks labels need --benchmark-file; benchmark corpora are not bundled. Soup bundles no benchmark corpora. A zero obtained with a file describes only those texts, not the absence of contamination against public benchmarks. Use soup data decontaminate --benchmark-file for real overlap removal.

Each filter is also addressable individually:

• soup data decontaminate --input ./train.jsonl --benchmark-file ./benchmarks/eval_sets.jsonl — drop rows whose n-grams overlap a benchmark file you supply (rows with text, content or messages[].content; a prompt field is not read, and a file with none of those loads zero texts and removes nothing, with exit 0). Soup bundles no benchmark corpora: --benchmarks gsm8k alone only records a label and removes nothing, and with neither option the command exits 1. The kept rows go to clean.jsonl by default.
• soup data toxicity — an abuse-keyword heuristic, labelled as one since v0.76.0 (--help: "Run an abuse-keyword heuristic (not a toxicity classifier)"); the panel is titled Abuse-keyword heuristic. It is not a toxicity classifier and no classifier backend ships. Ambiguous technical terms such as a process kill no longer count on their own. In upstream's published maintainer review of this scorer, 9 of 10 held-out abusive examples scored zero and 10 of 12 benign technical or editorial examples were flagged at the default threshold, so use it for keyword triage only, never as a safety decision. The [data-pro] extra adds langdetect and Presidio only; it does not install Llama Guard or FineWeb-Edu
• soup data langdetect — 2-letter language code per row
• soup data pii — flag email / phone / SSN / credit-card patterns
• soup data educational — educational-value score per row [0, 1]