Data Pipeline Pro (v0.42.0)

18 features across 6 parts. Axolotl + LLaMA-Factory data-layer parity.

5 new data formats

data:
  train: ./data/train.jsonl
  format: prm            # PRM stepwise-supervised
  # | pre_tokenized      # LF tokenized_path / Axolotl empty
  # | input_output       # Axolotl template-free segments+labels
  # | video
  # | multimodal         # axolotl content-parts schema

Remote URIs

7-entry allowlist for object stores. Bucket regex ^[a-zA-Z0-9][a-zA-Z0-9._\-]{0,62}$. Userinfo / fragment / query rejected.

data:
  train: s3://my-bucket/train.jsonl    # s3 | gs | gcs | az | abfs | abfss | oci
  streaming: true
  buffer_size: 10000
  shards: 16

Live fsspec loaders shipped in v0.53.8 (schema-only in v0.42.0).

A single Hugging Face Hub name streams too, since v0.75.0. data.streaming: true with data.train: org/dataset is forwarded to datasets.load_dataset(..., streaming=True), buffer_size shuffles that stream, and Soup then materialises up to 1,000,000 rows, printing a warning if the source is larger. Before v0.75.0 the flag was ignored for a Hub name and the whole dataset was loaded. buffer_size shuffles the train split only; a validation split is taken in order.

data.prompt_strategy — your own row transform

An Axolotl-style module.path:function_name spec. The import is resolved once at setup, so a bad path fails fast rather than mid-epoch, and the function is applied to each row before template rendering.

data:
  train: ./data/train.jsonl
  prompt_strategy: my_pkg.strategies:to_chat

It runs code you name, so it is operator-controlled by design: only point it at a module you trust. It must have the module.path:function_name shape, and format: pre_tokenized refuses it, because soup data preprocess never applies it and the cached ids would not contain its effect.

Multi-dataset interleave

data.interleave selects the strategy, and it is validated at config load. It attaches to a list-valued data.train and is refused outright against a single path, so the sub-block never appears on its own:

# A FRAGMENT, not a config: this block goes under the data: key, beside
# a list-valued data.train. On its own at the root it is refused as an
# unknown key, because only a root-level lora: block is remapped.
data:
  interleave:
    strategy: probs      # concat | under | over | probs
    probs: [0.7, 0.3]    # one entry per dataset, in data.train's order

Probs are validated as 2–32 entries summing to 1.0 ± 1e-6, each in (0, 1], and the list must be exactly as long as data.train. The full config is below.

This was a known gap until v0.74.0, and the history is worth one paragraph, because the site published the gap while it stood. data.interleave had validated, documented and parsed since v0.42.0 while nothing read it at training time, so every multi-dataset mixture silently trained on data.train's single path. data.train also declared a single string, so naming several datasets was rejected outright at config load. v0.73.3 repaired only the visible half, the recipe soup data mix --optimize writes, which had been emitting a list form its own loader rejected.

v0.74.0 wired it. data.train accepts str | list[str]; a list of two or more entries is combined by data.interleave into one row set, and a single path stays byte-identical to what it was.

v0.75.0 then changed where val_split sits in that order, and the consequence is worth knowing before you trust a validation number. Under over or probs the same row could land in both train and val, because the split ran after the padding and a padded copy is the same row. On the eager local-file and all-Hub-name paths val_split now runs first, per source, so that cannot happen. Two things follow, both upstream's own:

• The requested fraction is no longer exact under over or probs, because it is taken from each source's smaller, unpadded row count. Two sources of 1000 and 100 rows with over and val_split: 0.1 give 110 validation rows out of 1910, which is 5.8%, not 200 out of 2000.
• Train and validation end up with different source mixtures. If you oversampled specifically to correct a source imbalance, your validation set still reflects the original, un-rebalanced skew.

concat and under never pad, so neither applies to them. At v0.75.0 the streaming path was not fixed and could still place one row on both sides of the split under over. (Since closed in v0.76.0 for streaming over, #702: see below.)

The streaming path reaches the same guarantee for over by a different route (v0.76.0). A stream cannot be counted ahead of time, so there is nothing to take a fraction of before interleaving starts. Once over has been materialised, the split is sized over the distinct rows and validation is taken from the end of the stream, preferring rows whose content occurs only once, so train keeps every row and all of its oversampling. Only if there are too few such rows is repeated content moved to validation; its other copies are then withheld from train and the number withheld is printed as a warning. A split that would leave train empty raises instead. Validation on this path is not balanced across sources: with 100 rows against 10 under val_split: 0.1, every validation row comes from the larger source, because the smaller one's rows are all recycled. The eager path's per-source carve-out is balanced; this one is not. concat, under and probs add no duplicates on the streaming path and keep the ordinary positional split. That is a statement about interleaving, not about your data: rows already duplicated inside a source can still land on both sides of the split under any strategy, on either path.

data:
  train:
    - ./data/support.jsonl
    - ./data/code.jsonl
  interleave:
    strategy: probs        # concat | under | over | probs
    probs: [0.7, 0.3]

Streaming and Hugging Face Hub names work too, and the strategies are deliberately mapped so a name means the same thing on both paths: concat concatenates, under and over stop at the first exhausted or the last, and probs passes the probabilities through. That equivalence was verified by running one probs config through both paths and comparing the resulting proportions, rather than by two tests that each pass alone.

Two shapes are still refused by name: an all-Hub list with data.streaming: true, and any list mixing Hub names with local or remote entries. training.packing and training.multipack are rejected with a list too, at config-parse time, with a message naming the reason.

How entries are classified

Each entry of a data.train list is classified on its own. It is a local file if its suffix is .jsonl, .json, .csv, .parquet or .txt, a remote URI if it contains ://, and otherwise a Hugging Face Hub name. So teknium/OpenHermes-2.5 is a Hub name, not a file with a .5 suffix. A list may mix local files and remote URIs, or hold only Hub names, but Hub names beside the others are refused at config load. Remote entries need data.streaming: true, and streaming picks the file type per entry from its suffix.

A :// entry counts as remote whatever its scheme, and only the seven object-store schemes above load. An https:// or ftp:// entry passes config load and is refused by name when the data is loaded. Under streaming, probs samples from the sources, so the proportions are approached rather than met exactly.

When every entry is a Hub name, a Hub entry's own validation split becomes the combined validation set only if every entry has one. If only some do, Soup prints a warning, ignores all of them, and data.val_split applies instead.

Advanced masking + vocab expansion

add_new_tokens and new_special_tokens add tokens to the tokenizer and resize the embeddings whenever a token was actually added. They are applied on backend: transformers by these tasks: sft (text, vision and audio), dpo, kto, orpo, ipo, simpo, bco, grpo, online_dpo, preference (which hands off to the paired-loss trainers) and tts. Since v0.76.0 a config that sets either key anywhere else is refused at load, naming the reason: any other task (pretrain, embedding, ppo, reward_model, prm, classifier, reranker, cross_encoder, distill, asr, unlearn, moe_lora_routing), training.stream_layers: true, backend: mlx, and backend: unsloth (SFT vision and audio are the exception, because they go through the transformers setup whatever the backend says). Through v0.75.2 those configs loaded and the tokens were silently dropped, so they never reached the tokenizer (#1358). format: pre_tokenized refuses both keys too, together with data.prompt_strategy, because the cache was built without them.

train_on_prompt is mutually exclusive with train_on_responses_only, which defaults to true; on the transformers backend, putting prompt tokens in the loss means setting train_on_responses_only: false, since nothing else reads train_on_prompt.

mask_history: train only the last assistant turn

mask_history: true keeps only the last assistant turn in the loss. Every earlier assistant turn is masked, alongside the user and system turns that the assistant-only path already masks. It never adds tokens to the loss, and it only means something for multi-turn chat shapes. It was accepted and ignored through v0.75.2; it is read as of v0.76.0 (#761).

data:
  train_on_responses_only: true    # the default, and required by mask_history
  mask_history: true

• It requires train_on_responses_only: true. With false the config is refused at load, which also rules out train_on_messages_with_train_field, itself exclusive with train_on_responses_only.
• It is read on the text label path that task: sft and task: distill use.
• backend: mlx ignores it. MLX SFT supervises every assistant turn, so one config trains the last turn on transformers and every turn on MLX; soup train says so on its MLX backend ignores: line.
• On task: sft with modality: vision or audio it is accepted but not applied: every text token trains, and soup doctor --config reports it as ignored.
• soup data doctor --mask-history previews the same mask. It is refused (exit 3) together with --no-train-on-responses-only or --train-on-messages-with-train-field. See Fine-tune Doctor.

data.chat_template

data.chat_template takes a registered name (chatml, deepseek-r1, gemma-3, gemma3, llama-3, llama3, mistral, phi-4, phi4, qwen, qwen2.5) or a raw Jinja string. A name that is not registered is refused immediately (exit 1, Invalid data.chat_template, listing the known names), including under soup train --dry-run. Through v0.75.2 only the SFT path applied the override. As of v0.76.0 the other conversational trainers apply it as well (DPO, ORPO, SimPO, KTO, BCO, IPO, PPO, reward-model, GRPO and online DPO), and a config that sets it on a task that never renders chat is refused at load: pretrain, embedding, classifier, reranker, cross_encoder, prm, asr, moe_lora_routing and unlearn.

Keys that are accepted and read by nothing

Accepted by the schema, but nothing in v0.76.0 reads it; each warns at load, and upstream's printed deadline is that v0.77 refuses it (#808): video_dir, video_fps, video_maxlen, image_min_pixels, image_max_pixels, image_resize_algorithm, split_thinking, eval_on_each_dataset, resize_vocab, extend_conversation and skip_prepare_dataset. Setting one to a non-default value prints Warning: data.<name> is accepted but read by nothing; v0.77 will refuse it and the load continues; the schema default is silent. Delete the key to keep the config loading after v0.77. They do not change training, but they still count in the config hash that soup ship --config compares, so editing one makes --evidence from before the edit read as stale. In particular, split_thinking does not turn on any <think> masking. See Config refusals for the full list across sections.

remove_unused_columns is a separate case. It defaults to false since v0.76.0 (it declared true before, which misdescribed every run), because no trainer reads it: the SFT, pretrain, DPO, ORPO, SimPO, KTO, IPO, BCO, GRPO, reward-model, embedding, distill, PRM, ASR and MoLE-routing trainers all set remove_unused_columns=False themselves as a constant. An explicit true still loads, is ignored, and prints a warning ending Soup v0.77 will refuse it; delete the line. soup autopilot wrote remove_unused_columns: true into the configs it generated before v0.76.0.

data:
  add_new_tokens: ["<thought>", "</thought>"]
  new_special_tokens: ["<|im_end|>"]

soup data ingest

Convert PDF / DOCX / MD / TXT into JSONL.

soup data ingest mybook.pdf --output mybook.jsonl

Install pypdf for PDFs and python-docx for DOCX; neither is in a Soup extra, and without them the command exits with pypdf not installed. Run: pip install pypdf. Markdown and plain text need nothing. A PDF becomes one row per page, a DOCX one row per non-empty paragraph, Markdown one row per heading ({"text", "section", "level", "source"}) and a text file one row. Input and output must sit under the current directory, and the output defaults to ./ingested.jsonl.

AOT tokenize cache (rebuilt in v0.76.0)

soup data preprocess soup.yaml --output ./cache

soup data preprocess tokenizes the training split once. A later run with data.format: pre_tokenized and data.tokenized_path pointing at the cache loads those rows instead of tokenizing again. The config file and --output must both sit under the current directory (exit 1 otherwise), and the command prints the dataset, tokenizer, max_length, Loss mask:, Cache key: and Target: it resolved before it starts.

Through v0.75.2 this round trip did not work. The training gate hashed pre_tokenized while preprocess hashed the source format, so every cache that preprocess had written from a chat or Alpaca source was refused with cache hash mismatch. Those caches also held no labels column, so a cache loaded without its metadata.json (the path that prints a yellow "assuming compatible" advisory) trained on the prompt as well as the answer, with a normal-looking loss curve. v0.76.0 is the first release where the round trip works (#1038, #1054).

What the 16-character key covers. It is a SHA-256 over the cache schema version (v7) and these inputs, as PREPROCESS_KEY_FIELDS in utils/data_pipeline.py declares them:

A cache built under one value of any of these is refused when the pre_tokenized run uses another: pre_tokenized cache hash mismatch, naming the fields that changed when it can. That includes data.chat_template (preprocess now renders with it; before, it used the tokenizer's shipped template) and the masking mode. A cache written by v0.75.2 or earlier is refused too, and the message says it predates the keying; re-run soup data preprocess. Target already exists (exit 0, nothing tokenized) appears only when the same key is already in the output directory; --yes tokenizes again and replaces it.

Cached rows carry labels. They are built by the same helpers as the live training path, so a row masks what train_on_responses_only would mask. A leading BOS that the live path would not train on is dropped, and a truncated row no longer gains an EOS. With task: sft, a pre_tokenized dataset you built yourself must carry a labels column (-100 on every token not to train on); one without is refused. task: pretrain has no such check, because training on every token is the pretraining objective.

It stops on a row it cannot tokenize. A chat template that rejects the roles, an empty messages, a row that renders as empty text, or a tokenizer error ends the command with exit 1 and Train row N cannot be tokenized, where N counts from 1 among the rows that loaded, quoting the start of the row's first message. Nothing is written. A tokenizer with no chat template is reported once, pointing at data.chat_template. Earlier releases dropped such rows and exited 0, so the cache silently held fewer rows than the training path would train on (#1180).

Re-run this. A pre_tokenized run from a cache built before v0.76.0 either never started (the hash mismatch) or, when the cache had no metadata.json, optimised full-sequence loss instead of the assistant-only mask. Rebuild the cache and retrain. A cache built on a template that renders no end-of-sequence token may also have trained without a stop token; the v0.76.0 cache applies the same append-EOS rule as the live path (#938).

pre_tokenized pairs with data.tokenized_path, and a pre_tokenized config with a non-text modality (vision, audio) is refused at load. The pre_tokenized training config must keep data.val_split, data.replay, data.replay_ratio, data.replay_seed, data.streaming, data.buffer_size, data.image_dir and data.audio_dir as they were when the cache was built, and must name the same chat template.