Skip to content

Contributing to Soup

This page is a contributor guide for a first pull request. It restates upstream's CONTRIBUTING.md and AGENTS.md, docs/issue-difficulty.md, CODEOWNERS, CODE_OF_CONDUCT.md, CONTRIBUTORS.md, the changelog fragment README, the pull request template, the issue forms and the CI workflows, all read at the v0.77.0 tag, except that CONTRIBUTING.md was read again at v0.77.2 (the recipe-count step, the order of the recipe checklist and the Docker tag note below follow that version; the maintained-areas holder table lost its MLX row there, and this page lists no holders). It adds what the test suite enforces. Where this page and those files differ, the files win.

The short version. Claim an issue by commenting on it. Branch, write the tests first, run ruff check and pytest, add one changelog fragment, and open a pull request. A merged pull request puts your name in CONTRIBUTORS.md.

Why contribute, and where work is handed out

Soup is Apache-2.0, and most of the pull requests in recent releases came from outside. All 24 pull requests in v0.73.3 came from people other than the maintainer, 116 of the 120 merged into v0.74.0 did, and so did all 60 in v0.75.0. The two releases after that, v0.75.1 and v0.75.2, were security backports the maintainer wrote alone. For v0.76.0 upstream reports it differently, as a window rather than a tag-to-tag count: of the 490 pull requests merged since 2026-09-12, the day v0.75.0 shipped, 418 came from 80 people outside the maintainer (the v0.76.0 release). v0.76.1 was a maintainer-only backport. For v0.77.0, in the release body's words, "151 of the 226 pull requests in this release came from 44 people outside the maintainer" (the v0.77.0 release).

Work comes from four places:

  • GitHub issues. Start with the good first issue label. The help wanted filter is where upstream's README lists the work that is blocked on hardware, so it suits you if you have a bigger GPU. Issues labelled infra-blocked are waiting on hardware or infrastructure that the person picking them up may not have.
  • Soup Tasters, the Telegram channel where unclaimed issues, hardware still to be tested on and benchmarks still to be run are posted. Its app keeps score in XP; see The Soup Tasters app.
  • Discord, which upstream recommends for quick questions and for pairing on a pull request. Anything worth finding later still belongs in Issues or Discussions, and a security report never goes in Discord.
  • GitHub Discussions, for questions.

Upstream names four good first contributions: a new recipe (a ready-made config for a popular model), documentation (docstrings, README examples, example configs), tests (more coverage for existing commands) and bug fixes (issues labelled bug). Measurements count as well: two of the measurement records published in v0.74.0 came from contributors, on hardware the project does not own.

Before you write code

  • Read the feature's page first. The README and the matching page under docs/ in the repository describe each feature; docs/commands.md is the full command list. The Pydantic schema in src/soup_cli/config/schema.py is the single source of truth for every config field; Configuration is the user-facing reference.
  • Read the code of conduct. It is the Contributor Covenant, and it applies in every project space, Discord included. Report a violation to team@trysoup.dev or makazanalpamys@gmail.com.
  • Know who reviews. CODEOWNERS names the maintainer, @MakazhanAlpamys, as the default owner of every path. It lists src/soup_cli/config/ and src/soup_cli/trainer/ separately, and src/soup_cli/ui/, src/soup_cli/data/providers/ and src/soup_cli/utils/ollama.py under a heading called "Security-sensitive areas".
  • AI coding agents. AGENTS.md is upstream's tool-agnostic entry point for Codex, Cursor, Aider, Claude Code and similar tools. It repeats the build commands and the conventions on this page in a shorter form. Its lint line, ruff check src/soup_cli/ tests/, leaves out scripts/ and benchmarks/, which CI also lints, so use the four-directory form from Set up your environment.

Set up your environment

You need Python 3.11, 3.12 or 3.13. Upstream declares requires-python = ">=3.11,<3.14" (3.10 was dropped in v0.77.0, #1805), CI runs exactly those three versions (3.13 in the nightly and release runs, not on a pull request), and tests/test_requires_python_bound.py derives the declared bound from the CI matrix, so widening one without the other fails the suite.

Fork the repository on GitHub and clone it. Upstream's guide shows the main repository's URL; use your fork's URL if you want to push to it. Then work inside a virtual environment and install in editable mode with the dev extra:

bash
git clone https://github.com/MakazhanAlpamys/Soup.git
cd Soup
python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

These are upstream's commands, quoted from its guide and not run for this page. On Windows, activate with .venv\Scripts\activate. The virtual environment is not optional on Debian 12 or Ubuntu 23.04 and later: without it pip refuses with error: externally-managed-environment (PEP 668). pipx is the right tool for installing Soup and the wrong one here, because the test suite needs an editable checkout it can import.

In pyproject.toml at the tag, dev is the train, mcp and data extras plus jsonschema (new in v0.76.0, for the CycloneDX schema validation of the BOM), cryptography (so the ed25519 signing tests run), reportlab (so the Annex XI and XII PDF tests run), pytest, pytest-cov, pytest-timeout (new in v0.77.0), httpx, mypy, ruff, pre-commit and fastapi. That includes the full training stack, torch among it, so expect a large install. Every soup command writes under ~/.soup/; Environment variables and state files says how to keep a scratch session out of your real home.

Confirm the setup by running the suite and the linter. If both pass, you are ready:

bash
pytest tests/ -v --tb=short
ruff check src/soup_cli/ scripts/ tests/ benchmarks/

Some unit tests load small public models from the Hugging Face Hub (CI pre-downloads a handful of small repositories such as sshleifer/tiny-gpt2), so the first full run needs network access. CI forces UTF-8 on every platform with PYTHONUTF8=1 and PYTHONIOENCODING=utf-8, because Windows defaults to cp1252 and some upstream packages read their own files without naming an encoding. If a test dies on Windows with a charmap error, set the same two variables.

Git hooks

Upstream's .pre-commit-config.yaml installs with one command (the dev extra already provides pre-commit):

bash
pre-commit install

The hooks run ruff --fix, actionlint over the workflow files, and checks for end-of-file newlines, trailing whitespace, YAML and TOML syntax, merge-conflict markers and files over 5 MB. The actionlint hook builds with Go; the config says to swap it for the actionlint-docker hook id if you have no Go toolchain. The ruff-format hook is disabled on purpose and CI never runs ruff format --check: the config explains that enabling it would rewrite nearly every file and destroy git blame. Keep your diff to the lines you change.

How the repository is laid out

This map is built from the tree at v0.77.0, which adds or removes no package against v0.76.x. Upstream's own directory listing in CONTRIBUTING.md is older and omits several of these packages.

Inside src/soup_cli

PathWhat lives there
cli.pyThe entry point. It imports every command module, registers each on the Typer app, and defines the root options (--verbose, --log-level, --no-audit-log, --no-telemetry).
commands/One module per command or command group. commands/train.py is where a task is routed to its trainer.
config/schema.py, the Pydantic v2 models that define every config field and default, loader.py, which loads a soup.yaml and refuses unknown keys and the staged fields, and staged_fields.py, the table of declared fields that nothing reads (since v0.77.0 one set away from its default refuses the load; 31 fields at the tag).
trainer/One wrapper per training task around the TRL, Hugging Face or MLX trainer (sft.py, dpo.py, grpo.py and so on), plus stream_setup.py, the shared layer-streaming setup.
data/Loading, format detection and conversion (formats.py, loader.py), validation, collators, chat templates, loss masks, the synthetic-data providers and trace harvesting.
bench/The collector and the training benchmark behind soup bench train (new in v0.76.0).
recipes/catalog.py: the ready-made configs, one RecipeMeta entry each.
templates/The YAML files behind soup init --template, and manifest.json, which lists them.
eval/The eval platform: custom tasks, the LLM judge, the eval gate, the leaderboard, forgetting detection, quantization checks and the Aider benchmark.
experiment/The SQLite experiment tracker.
monitoring/Training callbacks and the live display.
registry/ and cans/The model registry (hashing, store, diff, attach) and the Soup Can artifact format (pack, unpack, verify, publish, run).
migrate/Config conversion from LLaMA-Factory, Axolotl and Unsloth.
autopilot/The zero-config decision engine behind soup autopilot.
cloud/The Modal, Lambda and RunPod backends for soup train --cloud (RunPod new in v0.77.0).
envs/Bundled toy rollout environments (calculator, guess_number, retrieval_qa), used through training.rollout_func.
mcp_server/The MCP server behind soup mcp serve: server.py, the tool registry in registry.py, and execution.py.
plugins/The plugin and hook registry, including entry-point discovery and the persisted opt-in.
ui/The Web UI: a FastAPI app and its static single-page front end. (soup_cli.ui.plugins, a tab registry nothing used, was removed in v0.76.0.)
tui_app.pyThe terminal UI.
autodistill/The artifact contract for AutoDistill. Upstream's docs/autodistill.md calls it a design-and-contract slice that adds no task or training.
utils/Shared helpers: GPU detection, path containment (paths.py), the shared outbound-address guard (net_guard.py), the gate exit codes (exit_codes.py), hardware fit, quantization, layer streaming and the rest.

Everywhere else

PathWhat lives there
tests/The test suite, with tests/conftest.py for shared fixtures and tests/fixtures/ for data.
docs/Upstream's per-topic feature guides, with docs/commands.md as the command list.
examples/Example configs and datasets.
benchmarks/The published measurement records and a harness/ directory. CI lints it.
scripts/Among others, assemble_changelog.py, generate_recipe_snapshot.py and check_recipe_repo_ids.py.
changelog.d/Pending changelog fragments.
notebooks/Notebooks, including the reproduction notebook.
.github/The workflows, the issue forms, the pull request template and the constraints files under .github/constraints/: transformers-floor.txt, and since v0.77.0 core-floor.txt for the core-floor job and pinned-upstream.txt, the stack pull-request CI installs.

Pick work and claim it

Comment on the issue saying you are taking it. GitHub only lets upstream set the formal assignee badge for repository collaborators, so the comment is the claim: the issue will look unassigned either way, and the thread is the record. Since v0.77.0 the current board is the pinned issue #1945, which a scheduled job regenerates daily from the open issues labelled help wanted, claimed and infra-blocked. It reports the rules below rather than setting them: the claim is still your comment in the issue.

Upstream makes two commitments in return:

  • If a maintainer ends up doing the work instead, you are told in the thread, before or when it lands.
  • A claim that goes quiet is released with a comment, not silently. A maintainer checks in first, and "life got in the way" needs no explanation.

If an issue needs hardware you do not have, say so in the claim. Several open issues carry the infra-blocked label for exactly that reason, and knowing early is more useful than a stalled branch.

To file an issue, use one of the two forms. The bug report asks for a description, steps to reproduce, your soup.yaml if relevant, the error output, and your environment. The description, the steps and the environment are required, and the environment is the output of soup doctor. For the error output the form asks for the full error with a traceback, which comes from the root option --verbose, so it goes before the command name: soup --verbose train --config soup.yaml. The feature request asks for the problem, your proposed solution, alternatives you considered and an area; the problem, the solution and the area are required. Both forms carry an optional Suggested difficulty dropdown (1 to 10); a maintainer sets the final difficulty:N label. Blank issues are off (blank_issues_enabled: false in the issue config since v0.76.0); the chooser links questions to Discussions and a private security report (Report a security vulnerability (private), which goes to GitHub Security Advisories), and the bug report form tells you not to open an issue for a security problem.

How work is queued

Review time and hosted CI are the scarce resources, so upstream (v0.77.0) states a few rules. They are applied with judgment, not mechanically, and nothing is closed or released as a penalty.

  • Reviews go in issue-number order. A pull request that is green and whose review rounds are answered moves up; one that waits on its author does not hold the others back.
  • Open pull requests per author. With fewer than 3 merged pull requests you can have 2 open at a time; with 3 to 9 merged, 3; with 10 or more, 5. Extra ones are not closed: a comment says what they wait behind, and they may be converted to drafts. Mark one ready for review when another lands.
  • Claims. Hold at most 2 claims whose pull request is not open yet. A comment without a plan is read as a question, not as a claim. A claim blocked on another issue (infra-blocked, or "lands after #N" in the thread) does not take one of the two slots, and neither does one whose pull request is open and waiting only for the merge line.
  • Regression tests. A bug-fix pull request carries a test that fails on main without the fix; the review checks that, and checks that undoing the fix makes the test fail. Merged pull requests that meet this get the regression-test label.
  • A quiet claim. With no pull request and no word for 7 days you get a check-in in the thread; if another 7 days pass without an answer, the claim is released with a comment. Nothing you wrote is lost and you can take it back.
  • A quiet pull request. A pull request waiting on its author for 14 days gets a ping; if another 7 days pass it is closed with a note that it can be reopened at any time.
  • Full CI. The required test cells run when a maintainer adds the ci:full label to a pull request that is ready to merge, not on every push; other pushes run a quick subset, and a draft pull request runs no CI at all (details under What CI runs).

Maintained areas

Since v0.77.0, contributors with 3 or more merged pull requests can hold an area: a part of the project to own end to end, from triage to the last follow-up, with its own area:<name> label and a tracker issue. Holding one means first refusal on new issues with that label (they stay with the holder for 3 days before they are open to anyone), an open-pull-request limit two higher than your usual one inside the area, and no new powers: reviews and merges stay with the maintainers. An area is released the way a claim is, after 14 quiet days and a 7-day check-in. The current holders are listed in upstream's CONTRIBUTING.md.

Issue difficulty

Maintainers rate every triaged issue from 1 to 10 with a difficulty:N label, so you can pick something that fits your time and experience. The label estimates how much work and risk the fix carries; it is not a statement about how important the bug is. An issue with no difficulty label is not triaged yet (a workflow adds needs-triage to a new one and removes it when a level is set, and a run fails if one issue gets two levels). Upstream's docs/issue-difficulty.md defines the levels, and the table there wins over a label that disagrees with it:

LevelWhat it looks like
1Wording, a typo or a one-line fix. No new logic.
2One non-test file, under about 15 changed lines; the issue spells the fix out; one test.
3One non-test file, up to about 50 changed lines; tests run on CPU without torch.
4Two to four source files in one subsystem, following a pattern that already exists; tests run on CPU without torch.
5Five or more source files, or a trainer, or the config schema; tests need torch.
6Several tasks, trainers or backends at once, or non-trivial numerics; each affected one needs its own regression test.
7A new mechanism or a deep change to a core path (trainer internals, layer streaming, the data pipeline); proving it takes a measurement.
8The cause is not known yet or is hard to reproduce, or the result must be bit-exact.
9A new subsystem, or support for several model families at once.
10Research-grade: hand-written kernels or backward passes with a numerical gate. Rare; most hard work stops at 8 or 9.

If the fix cannot be verified without special hardware or a paid service (a large GPU, several GPUs, a paid cloud), add 3 levels to what the work itself would get, up to 10: a one-module change that can only be checked on an H100 is level 6, not level 9. Say so in your claim. Issues closed before the system existed were rated afterwards from the pull request that closed them, so those older labels are approximate. If you disagree with a number, say so in the issue thread with the reason.

Make the change

  1. Create a branch. Upstream's examples are feature/your-feature-name and fix/your-bug-fix.
  2. Write the tests first, then the code that passes them. Keep commits focused and logical.
  3. Add a changelog fragment if the change is visible to users. The next section has the rules. If you add or remove a CLI command, option, argument or soup.yaml field, or change its type, default or bounds, run python scripts/gen_reference_docs.py and commit the two pages it rewrites (docs/cli-reference.md, docs/config-reference.md, generated since v0.77.0); the tests fail until you do and name what differs. Rewording a help string or a description needs nothing: maintainers regenerate the wording at release.
  4. Lint, then test. Upstream's guide omits benchmarks/ from its --fix line, but CI lints it, so the form below covers all four directories.
  5. Commit with a Conventional Commits message. The types are feat, fix, refactor, docs, test, chore, perf and ci. Stage specific files rather than everything.
  6. Push and open a pull request. CI runs on pull requests to main.

The commands for steps 4 to 6:

bash
ruff check --fix src/soup_cli/ scripts/ tests/ benchmarks/
pytest tests/ -v --tb=short
git add <specific-files>
git commit -m "feat: add support for X"
git push origin feature/your-feature-name

Give the pull request a clear title, a description of what changed and why, any related issue (for example "Closes #123"), and your test results. The template then asks you to pick a type of change (bug fix, new feature, refactor or cleanup, documentation, tests) and to tick four boxes:

  • ruff check passes. The template's command omits benchmarks/; use the CI form above.
  • pytest tests/ -v passes.
  • The relevant docs are updated (README.md and the matching page under docs/), if needed.
  • A changelog fragment is added, or the change has no user-visible impact.

CONTRIBUTING.md keeps its own version of the checklist, which adds two items: new tests for new functionality, and no breaking changes, or any that are documented in the pull request description.

Changelog fragments

A user-visible pull request adds one Markdown file instead of editing CHANGELOG.md:

text
changelog.d/<latest-release>/<number>.<category>.md
  • <latest-release> is the newest released version heading at the top of CHANGELOG.md on the branch you are building on, so read it there and do not copy a version from this page. A fragment filed under any other version fails validation, and tests/test_issue487_changelog_fragments.py runs that validation against the repository, so it fails the test suite and not only the release. After a release, an older fragment forces its pull request to rebase and move the file.
  • <number> should be the pull request number rather than the issue it closes, so two pull requests for the same issue stay distinct. The validator rejects a second category with the same number.
  • <category> is one of added, changed, deprecated, removed, fixed or security.
  • The content is the complete changelog list item: it must start with - , mention the same #<number>, be non-empty UTF-8, and may include paragraphs, tables and fenced code blocks, which assembly copies verbatim. Symlinks are not allowed in changelog.d/.

A fragment for pull request 1234 that fixes a bug would be changelog.d/<latest-release>/1234.fixed.md, starting like this:

text
- Fix the crash when a config names a missing dataset (#1234).

Before a release, a maintainer runs python scripts/assemble_changelog.py. It validates every fragment, inserts the entries under [Unreleased] and consumes the files. Publishing to PyPI independently refuses to continue while any fragment remains.

Code standards

Ruff is the style gate in CI. Its settings in pyproject.toml are a line length of 100, the rule sets E, F, I, N and W, and a Python 3.11 target (3.10 until v0.77.0).

  • Imports are sorted (ruff's I rules).
  • Names: no single-letter variables. Ruff's E741 rejects l, O and I; use entry, part or length.
  • Lazy imports. Import heavy dependencies (torch, transformers, peft, trl, mlx) inside functions, never at module level, so the CLI stays fast. A test guards the result for the training stack: tests/test_cli_startup_is_light.py imports soup_cli.cli in a fresh interpreter and fails if torch, transformers, accelerate, peft, trl, datasets or bitsandbytes came with it.
  • Config validation uses Pydantic v2 with BaseModel and Field.
  • Output goes through rich.console.Console, never a bare print().
  • Type hints on every parameter and return value. CI runs mypy as a non-blocking baseline: it reports issues as a warning and does not fail the job. Since v0.77.0 that type-check job runs in the nightly and release runs, not on a pull request.
  • Path containment uses os.path.realpath plus os.path.commonpath, not Path.resolve() plus relative_to(), which breaks on Windows 8.3 short names. The shared helpers in src/soup_cli/utils/paths.py are is_under, is_under_cwd and enforce_under_cwd_and_no_symlink. A test scans src/soup_cli/ and fails on the banned idiom used as a containment check.
  • No third-party licence headers. The project is Apache-2.0 and ships to PyPI, so an SPDX-License-Identifier or a copyright line naming anyone else is a licensing claim it cannot make. If you generate a file from a template, including with automated tooling, strip its header before you push. tests/test_no_foreign_license_headers.py fails the build on it.

Upstream's own example of the lazy-import and output rules, lightly adapted:

python
# WRONG: a heavy import at module level and a bare print
import transformers

def train():
    print("Starting training")

# CORRECT
from rich.console import Console

def train():
    import transformers

    console = Console()
    console.print("Starting training")

Tests

bash
pytest tests/ -v --tb=short
pytest tests/test_config.py -v
pytest tests/test_data.py::test_detect_alpaca_format -v
pytest tests/ --cov=soup_cli --cov-report=html

The four commands run everything, one file, one test, and a coverage report. The default options in pyproject.toml deselect the smoke marker and measure coverage of soup_cli against a floor of 77 percent. CI's own subset runs pass --no-cov, and some also clear the defaults with -o addopts=; the Recipe Validation workflow runs its two files with --no-cov. The markers are smoke (slow tests that download models and train; run them with pytest -m smoke), unit (fast and isolated: no subprocess, network, filesystem or real model load), integration (real subprocess, SQLite, filesystem or HTTP), gpu (needs a CUDA device, see below), requires_symlink and requires_hardlink (the test creates a link and is skipped when the account or filesystem cannot). At the v0.77.0 tag tests/ holds 1070 files named test_*.py, counted from the git tree; upstream publishes no test total, so this page gives none. Since v0.77.0 CI cuts a hung test off after ten minutes (--timeout=600, from pytest-timeout).

GPU tests

Every CI runner is GPU-less, so the tests that need a CUDA device skip there and only mean something on a real card. They carry the gpu marker (#833):

bash
pytest tests/ -m gpu --no-cov -v

--no-cov matters because the default --cov-fail-under gate is computed over the whole package and fails a subset run. To gate a new test on CUDA, decorate it with @pytest.mark.gpu (or @pytest.mark.gpu(reason="...")); do not write your own torch.cuda.is_available() check. tests/conftest.py holds the one probe, and tests/test_issue833_gpu_marker.py fails on a private one. Use tests.conftest.cuda_available() only to pick a device, never to skip.

If you have a card, a run is a contribution: upstream says both recorded runs so far found defects CI could not see. Open an issue (or comment on #833) with the card, its compute capability (torch.cuda.get_device_capability()), its VRAM, the driver version, the torch build (torch.__version__ and torch.version.cuda), the bitsandbytes version if installed, the commit you ran (git rev-parse HEAD), the summary line (N passed, N failed, N skipped) and the name and first error line of every failure. Accepted runs are recorded in benchmarks/gpu-test-runs.md.

CI runs the tests in parallel

Every test (...) cell runs the suite with pytest-xdist, one test file per worker process (-n logical --dist loadfile). The tests of one file still run in order in one process, but two files can run at the same moment and in any order. So a test must not depend on another file having run first, on the working directory another test left behind, or on the terminal width of the process: give the console an explicit width in the test instead of setting COLUMNS. To reproduce a failure that shows only in CI, install the plugin and run the same selection:

bash
pip install "pytest-xdist[psutil]"
pytest tests/ -n logical --dist loadfile

Conventions that the tree shows:

  • File names. Name a new test file after the module or feature it covers (test_<module>.py), not after an issue number or a version (upstream's rule since v0.77.0). Existing files keep their names, so many are still tests/test_issue<number>_<topic>.py.
  • Shared helpers live in tests/conftest.py. Import strip_ansi from it rather than writing another copy. An autouse fixture points SOUP_DB_PATH at a per-test temporary file, so a test never sees runs in your real ~/.soup/experiments.db. tmp_data_dir, sample_alpaca_data and sample_config give you sample data and a config.
  • Strip ANSI before asserting on --help output. Typer renders --help through Rich, which on a colour-capable stream puts escape codes inside the flag name, so "--noise-floor" in result.output fails. Windows passes because Rich turns colour off there, which is how an assertion that passed on a Windows machine has turned CI red before.
  • The Web UI escaping tests run safe_html.js under node and skip when node is absent. CI installs node 22 so they cannot vanish silently.

Tests that guard repository-wide rules

A pull request can break one of these without touching the code a test is named after.

Test fileIt fails when
test_cli_startup_is_light.pyImporting soup_cli.cli loads the training stack.
test_cli_dependency_floors.pyThe Typer or Click floor declared in pyproject.toml differs from its pin in .github/constraints/transformers-floor.txt, or, in the transformers-floor job, the CLI does not build its command tree, render every help page and run soup version on those pins (new in v0.76.1, #1802; since v0.77.0 the core-floor job installs the declared core floors and starts the CLI on them too). Move a floor and its pin together.
test_yaml_reads_are_limited.pyA module other than utils/yaml_limits.py parses YAML (another module may only write YAML or catch its error class), or a YAML reader does not report a document past the node or nesting limit in its own error contract (new in v0.76.1).
test_requires_python_bound.pyThe declared requires-python and the CI Python matrix disagree.
test_no_foreign_license_headers.pyA file carries a third-party licence header.
test_issue775_path_containment_ratchet.pyrelative_to or is_relative_to is used as a containment check under src/soup_cli/.
test_issue748_config_fields_reach_a_consumer.pyA field is declared in config/schema.py and nothing under src/ reads it, unless it is allowlisted with a reason. The check matches names globally, so a field named like a common attribute can pass without being wired.
test_cli_help_assertions_are_ansi_safe.pyA test asserts on raw --help output.
test_readme_translation_sync.pyREADME.md changed and a README.<lang>.md translation no longer matches it. There are four translations at v0.77.0 (Turkish, Arabic, Japanese and Simplified Chinese), each linked from a language banner above the logo; the test also checks that the banner links exactly the READMEs that exist. Update each translation, then replace its stamp with the line the failure prints.
test_typer_defaults_do_not_leak.pyA Typer command is called as a plain Python function without every parameter filled. Typer fills a command's parameters only when it invokes the command, so an unfilled one keeps its default, an OptionInfo, which is truthy (#752).
test_issue1547_outbound_http_sinks.pyAn outbound HTTP call (httpx, urllib.request, requests) in src/soup_cli is not in the declared ledger of known gates.
test_issue833_gpu_marker.pyA test module probes CUDA itself instead of using the gpu marker.
test_recipe_count_is_synced.pyThe recipe count written in the documentation differs from the catalog.
test_issue621_recipe_snapshot.pyA recipe's resolved config, or a schema default, differs from tests/fixtures/recipe_config_snapshots.json.

What CI runs on your pull request

The CI workflow runs on pushes to main and release/**, on pull requests to main, once a night on main and on demand. A first job, plan, picks one of four matrices, and every job except lint follows it:

  • Quick, for a push to main and for any pull request run that does not carry the ci:full label: lint plus test on Ubuntu with Python 3.12 only.
  • Approval, for a pull request labelled ci:full: the four required test cells (Ubuntu on 3.11 and 3.12, Windows and macOS on 3.12) plus four gated jobs: mlx-smoke (macOS, .[mlx] only with the PyTorch stack asserted absent: one real MLX SFT step, then the MLX tests that need the real library), pytorch-smoke (Ubuntu, the smoke-marked tests against the real PyTorch pipeline), core-floor (new in v0.77.0: the declared core dependency floors, with the CLI started on them) and transformers-floor (Ubuntu, .[dev] under .github/constraints/transformers-floor.txt, then pip check, the pin assertions and a compatibility test set; since v0.76.1 that file also pins Typer and Click to the floors pyproject.toml declares, and the job starts the CLI on them). type-check and torchao-contract no longer run on a pull request. Adding the label starts this run, and it also runs on every later push while the label stays.
  • Full, nightly on main at 03:23 UTC or by manual dispatch: seven test cells across Python 3.11 to 3.13 (no Windows or macOS on 3.11) plus the six gated jobs (the four above, type-check and torchao-contract). Python 3.13 runs only in the nightly and release runs, so a failure that exists only there shows on main the next night.
  • Release, for a push to release/**: all nine test cells plus the six gated jobs.

Coverage is measured on main, not on a pull request (since v0.77.0): every pull-request test cell runs --no-cov, so a change that takes coverage below 77% merges green and turns the next nightly run red; run plain pytest tests/ locally before you push a change that removes tests or adds untested code. Pull-request CI installs a pinned upstream stack: on a pull request and on a push to main, the jobs that install torch or transformers take them at the exact versions in .github/constraints/pinned-upstream.txt (transformers-floor installs its own floor instead). A separate typer-vendored job, on schedule and manual dispatch only, checks the CLI against a Typer that vendors Click.

A draft pull request runs no CI at all; marking it ready for review starts the quick run. Until the approval run starts, the other test (...) checks show as Expected and the smoke jobs as skipped, and a skipped job passes branch protection, so the test cells are what block a merge. Neither is a failure on your side. Add any other labels before ci:full, because a label added afterwards starts a second approval run. lint is ruff check src/soup_cli/ scripts/ tests/ benchmarks/ and then actionlint over the workflow files; the test cells run pip install -e ".[dev]" and the whole suite with coverage. Which checks are required for a merge is a repository setting, not a file in the tree, so the list above is what runs rather than what blocks. The other workflows are:

  • merge freshness re-evaluates a pull request against the live main without writing to your fork. Lag behind main is reported as a neutral check; a conflict, or a ruff F821 (undefined name) in the merge, fails it.
  • issue triage labels keeps the difficulty:N and needs-triage labels consistent.
  • recipe attach runs soup recipes verify over every shipped recipe (#1116), so a recipe whose target modules match nothing is caught before a user downloads the weights. Upstream's workflow comment says it is not a required check and must never become one, because a Hub outage would then block every merge.
  • CodSpeed is a regression benchmark in its own workflow. Since v0.77.0 it is off: the workflow starts only by manual dispatch, so it adds no check to your pull request, and even when it is turned on it never runs on a fork pull request. It blocks nothing.
  • claim board rewrites the pinned issue #1945 once a day (new in v0.77.0), and CodeQL scans the package weekly or on demand; neither is a pull-request check you have to pass.
  • Recipe Validation runs the two recipe test files on pull requests and pushes that touch recipes, config, the data formats or loader, or those tests.
  • dependency drift runs weekly, on Mondays at 06:00 UTC or on demand. It installs the latest resolvable stack, writes a resolved-versus-declared table into the run summary and runs the suite against it.
  • recipe repo ids runs weekly, on Mondays at 07:00 UTC or on demand, and resolves every recipe's model id on the Hub anonymously. It is deliberately not a pull request check.
  • Publish to PyPI runs on a v* tag, and Build and Publish Docker Image runs when a GitHub release is published.

A red or green mark can be stale

A pull_request run pins refs/pull/N/merge when the run is created, so a mark can describe a main that no longer exists, and gh run rerun replays the frozen merge and cannot produce a different answer (upstream's #1017). If your pull request is red for failures you did not touch, merge or rebase onto the current origin/main and push: that is the only way to rebuild the merge the test matrix runs on. If it is green and main has moved, the merge-freshness check is re-posted when main moves (a burst of pushes is collapsed to the newest tip), and gh pr checks returning all green is necessary but not sufficient before a merge.

How credit works

Upstream credits every contributor in three places:

  • CONTRIBUTORS.md. Every merged pull request, or one adopted in-tree, adds you there with your work and the pull request number. The file is maintained by hand and listed by first contribution. If you merged a pull request and are not in it, open a pull request adding your line; that counts too.
  • CHANGELOG.md. From the release that ships your change, the entry is tagged (#NNN by @you).
  • Commit trailers. Pull requests land with a Co-Authored-By: line for you, so GitHub's contributors graph reflects your work even after a squash merge.

If you contributed and are not credited somewhere, upstream calls that a bug: open a pull request or an issue. A security reporter is credited by name in the release notes unless they ask for anonymity.

The licence of your contribution

In upstream's words since v0.77.0: "Contributions are accepted under the Apache License, Version 2.0, the project's own terms (section 5 of the license): by opening a pull request you agree that your contribution is licensed that way and you keep your copyright; there is no CLA and no DCO sign-off for now." The NOTICE file at v0.77.0 reads Copyright 2026 Makazhan Alpamys and the Soup contributors.

The Soup Tasters app

The Soup Tasters app runs inside Telegram and turns contributions into XP, levels and a public leaderboard. It sits beside the three credits above, not in place of them.

  1. Open @souptasters_bot and tap Open app.
  2. Attach your CV and fill in five fields, which earns 10 XP.
  3. Earn XP with code or with posts.
  • Code. Link your GitHub account and every pull request merged into Soup counts on its own, for 10 to 300 XP. Pull requests merged before you joined count too.
  • Posts. Writing about Soup on any network earns up to 100 XP.

There are 16 levels, from Taster to Core. Higher levels unlock harder and paid tasks, and Core leads to an internship interview. Your CV stays with the Soup team.

Report a vulnerability

Do not open a public issue for anything security-sensitive, and do not report it in Discord. Report it privately through GitHub Security Advisories, which is the preferred route, or, if you cannot use GitHub, by email to team@trysoup.dev. The supported versions, the scope, what to expect and the fact that there is no bug bounty are in Reporting a vulnerability.

Security expectations for code

None of CONTRIBUTING.md, AGENTS.md or SECURITY.md contains a separate security checklist for pull requests. What they state is:

  • Path containment follows the rule in Code standards, through the helpers in utils/paths.py.
  • The classes of bug SECURITY.md treats as in scope are the ones to keep in mind when code reads or writes anything a user names: path traversal or arbitrary file read or write from config, dataset or artifact paths; SSRF in the synthetic-data providers, the inference server or the hub and endpoint validators; command, Modelfile, Jinja chat-template or systemd and launchd unit injection; secret leakage in logs, crash bundles or generated artifacts; and sandbox escape in the code-execution reward path.
  • The paths CODEOWNERS marks as security-sensitive are src/soup_cli/ui/, src/soup_cli/data/providers/ and src/soup_cli/utils/ollama.py.

How to add a recipe

  1. Add a RecipeMeta entry to the RECIPES dict in src/soup_cli/recipes/catalog.py. Its fields are model, task, size, tags, description and yaml_str, and the dict key is the recipe's name. yaml_str is a complete soup.yaml that must load as a SoupConfig. In the existing recipes the model id and the YAML base: both name a Hugging Face repository, and a weekly workflow resolves every id.
  2. Add tests in tests/test_recipes.py.
  3. Move the recipe count everywhere it is written. tests/test_recipe_count_is_synced.py checks five places: a comment in catalog.py, CONTRIBUTING.md, docs/commands.md and two lines in docs/serving-and-export.md. Since v0.76.0, python scripts/sync_recipe_count.py rewrites them for you (--check only reports what is out of sync, with exit 1, and writes nothing). It deliberately does not touch EXPECTED_RECIPE_COUNT in tests/recipe_count.py, the milestone pin, which stays a hand edit; upstream's v0.77.2 checklist sets the pin first and runs the script after it, and notes that deriving the pin from len(RECIPES) would reduce the milestone assertions to a comparison that can never fail (#1016). Several tests also pin the exact catalog size, in tests/test_recipes.py and in three older release-milestone test files. Pull request 864, which added two recipes, touched all of them.
  4. Regenerate the config snapshot with python scripts/generate_recipe_snapshot.py and review the diff. The new recipe's resolved config has to be in the committed snapshot, or test_fixture_and_catalog_have_the_same_recipe_names fails.
  5. Add a changelog fragment in the added category. Upstream's recipe checklist up to v0.77.1 also listed an update of the recipes section of README.md; its v0.77.2 checklist is the four steps above and no longer does.

You can check the result with soup recipes show <name>, which prints the recipe's YAML, and soup recipes list. A new recipe is one of the first contributions upstream suggests; see Recipes for what the catalog holds.

How to add a command

  1. Create src/soup_cli/commands/your_command.py with a handler function. A command group is a typer.Typer() app in the module.
  2. Register it in src/soup_cli/cli.py. Import the module in cli.py (the oldest commands share one from soup_cli.commands import (...) block; newer ones are imported one at a time, often under an alias such as as _ship_cmd), then register it the way its neighbours are: app.command()(your_command.your_command) for one command (pass name= to set the CLI name), or app.add_typer(your_command.app, name="your-group", help="...") for a group. Upstream's guide says @app.command(); the file registers by calling app.command()(function).
  3. Import heavy dependencies inside the handler, so soup --help stays fast.
  4. Add tests in tests/test_your_command.py, using strip_ansi for any --help assertion.
  5. Update the help text and README.md, and add the command to docs/commands.md, the full command list. Add a changelog fragment.

If the command is a gate or a verdict that CI branches on, use the shared exit codes in soup_cli.utils.exit_codes: EXIT_OK 0, EXIT_RUNTIME_ERROR 1, EXIT_GATE_FAILED 2 and EXIT_USAGE_ERROR 3. Register it with cls=GateCommand (or GateGroup for a group), so that a Click usage error, a mistyped flag, also exits 3 instead of 2. Eleven commands follow this scheme at v0.76.0; the list is on Exit codes.

How to add a training task

  1. Write src/soup_cli/trainer/your_trainer.py, a wrapper class around the appropriate trainer. commands/train.py builds it with config, device, report_to, deepspeed_config, fsdp_config and trust_remote_code, then calls setup(dataset) and train(...). trainer/orpo.py shows the shape.
  2. Add the task name and its config fields to src/soup_cli/config/schema.py: the name goes in the task literal on SoupConfig. Every new field needs a consumer, or the repository-wide test above fails, and since v0.75.0 a key that is not in the schema is refused when a config loads. A field that is declared and read by nothing is either allowlisted in that test with a reason or listed in config/staged_fields.py (31 fields at the v0.77.0 tag): since v0.77.0 one set to a non-default value refuses the load (v0.76 only warned).
  3. Add a template: a YAML file under src/soup_cli/templates/ and its entry in manifest.json.
  4. Route it in src/soup_cli/commands/train.py with an elif cfg.task == "your_task": branch that imports the wrapper inside the branch. Runs on the MLX backend go through trainer/mlx_routing.py first.
  5. Add tests in tests/test_your_trainer.py. Upstream's guide asks for at least thirty for a new trainer.
  6. Update README.md, the relevant page under docs/, and add a changelog fragment.

For a new data format, the steps are shorter: add detection and conversion in src/soup_cli/data/formats.py, add the name to the format literal in config/schema.py, update data/loader.py if needed, add tests in tests/test_formats.py, and document it in the relevant docs/ page. See Data formats for the formats that exist.

What happens after your pull request merges

Release steps are the maintainer's side of the process, and upstream's guide lists them under "Releases". The project uses semantic versioning, MAJOR.MINOR.PATCH.

  1. Refresh the dev stack with pip install -U -e ".[dev]", or read the last dependency drift run.
  2. Run python scripts/assemble_changelog.py and review the assembled CHANGELOG.md.
  3. Update the version in pyproject.toml and src/soup_cli/__init__.py.
  4. Run the full test suite and the linter, including the recipe snapshot test.
  5. Update README.md, the relevant page under docs/, and SECURITY.md if the release is security-related.
  6. Commit as Release v0.X.0 and tag v0.X.0.
  7. GitHub Actions verifies that no fragment was left behind, then publishes to PyPI on the v* tag through a Trusted Publisher (OIDC).

Docker version tags are published for every release, while latest follows the highest stable vX.Y.Z tag (#2214, by @Nari-Ab). Before tagging a maintenance-line Docker release, the maintainer backports .github/workflows/docker.yml and scripts/docker_release_tags.py so it keeps the newer line's latest image. A manual dispatch uses the workflow at its selected ref, and updating main does not change workflows on existing historical tags. That is upstream's CONTRIBUTING.md as of v0.77.2; it is a maintainer step, not something a pull request does.

See also

Soup is free and Apache-2.0. If it saved you a training run, starring the repo costs nothing and helps most. You can also fund the GPU time behind the work a 4 GB laptop cannot reach.