Reward Forge: synthesize a reward verifier, then prove it cannot be gamed (v0.71.40 → v0.71.41)

An RL run trusts its reward verifier more than anything else and inspects it least. If the verifier pays out for the wrong reasons, the policy learns to hack it, and you find out after the GPU hours. Reward Forge is two commands that turn Soup's reward-hacking detector on the verifier itself: soup reward synth builds a deterministic verifier from your data and refuses a bad one; soup reward stress attacks a verifier to see if degenerate junk gets paid.

Both are pure and offline, and both ride the existing load_reward_fn .py path, so they add no new trusted-execution surface. No RL library (TRL, Unsloth, Axolotl, OpenRLHF) synthesizes a reward or tests one for gameability.

soup reward synth — a verifier from your gold outputs (v0.71.40)

soup reward synth golds.jsonl -o reward.py                      # auto-detect the family
soup reward synth golds.jsonl -o reward.py --kind numeric --tolerance 1e-6
soup reward synth golds.jsonl --plan-only                       # show the induced spec, write nothing
soup reward synth golds.jsonl -o reward.py --output-report calib.json

Point it at a JSONL of reference (gold) outputs and it infers one of four verifier families, auto-detected or forced with --kind, each grounded in your golds. Each row is read from its answer field (--field names another one) or, for chat rows, from the assistant turn of messages; a file with neither exits 1 with no gold outputs found.

The emitted reward.py is readable and committable — you read it, edit it, diff it in review, and check it into git, exactly like a hand-written reward. It is not a trained-weights black box. Since v0.76.0 the numeric kind imports Soup's answer parser (soup_cli.utils.final_answer), so the file runs where Soup is installed and is not fully self-contained; its header says so.

Regenerate a numeric verifier written before v0.76.0 (#1350). The old file pulled the last number with a local regex. A new one reads both sides with the shared parser, so 1,000-style golds and The answer is ... completions score where they scored 0 before, and it stops paying some completions the old regex paid: a #### or \boxed{} answer carrying units, % or a non-dollar currency symbol or code (#### 42 apples, \boxed{42\%}, #### €42), a hedged or parenthetically justified answer phrase (The answer is either 41 or 42, The answer is 42 (6*7=42).), and a completion whose last \boxed{} is wrong after a right first one. A bare 42% and The answer is 42%. still pay, as under the built-ins. Do not compare runs across an old and a regenerated verifier.

The moat: mandatory calibration

The differentiator is that synth refuses to emit a verifier that cannot tell good from bad. Before writing the file, the synthesized verifier is loaded back and run against:

• its own references — it must accept at least 90%, and
• auto-generated bad answers — it must reject them.

A degenerate always-accept verifier is refused with exit 2, never silently written. --plan-only reports the induced spec without writing, and --output-report persists the calibration JSON, which now names every reference the verifier rejects (rejected_references); a refused run lists the first ten of them in the panel and counts the rest. Since v0.76.0 -o is not touched until the candidate has been loaded back and accepted: the candidate is written to a temporary sibling file and moved onto -o only on acceptance, so a refusal, an error or a crash leaves an existing reward.py byte-identical. (Through v0.75.2, soup reward synth -o reward.py --force overwrote the existing verifier first and deleted the file outright on a refusal or error, destroying a previously working verifier (#1450).)

--min-discrimination (default 0.5, allowed 0.0 to 1.0, otherwise exit 1) sets how far apart those two groups must be. Discrimination is the share of its own references the verifier accepts minus the share of the auto-generated bad answers it also accepts; below the bar the verifier is refused with exit 2. --min-discrimination 0 does not switch the check off: a verifier whose discrimination is 0 or less is refused regardless.

soup reward stress — is the verifier gameable? (v0.71.41, six attacks since v0.76.0)

soup reward stress reward.py --references golds.jsonl
soup reward stress reward.py --references golds.jsonl --attacks empty,length,repetition,sentinel --sentinel GOLD
soup reward stress verifiable --verifiable-domain math --references golds.jsonl   # probe a builtin

Where synth proves a verifier separates references from friendly perturbations, stress asks the adversarial question a reward-hacking model asks at train time: does the verifier pay out for degenerate junk? It feeds six attack families, each scored against the real gold (references are cycled), so a numeric, tool_call or json_schema verifier still gets a valid target and must reject the garbage. --attacks takes a comma list and defaults to all six:

• empty — an empty completion
• length — a length-padded completion
• repetition — a repeated fragment
• sentinel — spam containing a magic token (--sentinel)
• wrapped_junk — structure-preserving junk: junk wrapped in the structure a reward looks for (the built-in format reward paid 1.0 to <think></think>a)
• answer_spray — a completion that embeds the real gold among distractor answers

The last two were added in v0.76.0 (#814, #937). Each family is expanded into several distinct variants and judged per family, so a clean family cannot dilute a gameable one. A run costs up to 23 batched verifier invocations, or 4,600 scored completions at the 200-gold cap, so a slow or model-based verifier takes proportionally longer than a string check.

Real output for the reward.py synthesized above from ten numeric golds (42, 43, ...):

$ soup reward stress reward.py --references golds.jsonl
attack         n  accepted  accept-rate
empty         30         0           0%
length        30         0           0%
repetition    30         0           0%
sentinel      30         0           0%
wrapped_junk  70         0           0%
answer_spray  40         0           0%

reference accept: 100%   gameability: 0%   verdict: robust (not gameable)

It reports a per-attack accept-rate table plus an overall gameability verdict, tuned by --max-gameable and --threshold, and --output-report persists the JSON. It probes a synthesized .py and a builtin (accuracy, format, verifiable) through the same loader. Exit codes: 0 = robust, 2 = gameable, 1 = error. A gold-requiring verifier probed with no --references is a hard error naming the flag, never a false "robust".

The built-in format reward is gameable. Probed against the same golds, soup reward stress format --references golds.jsonl accepts 57% of wrapped_junk and 25% of answer_spray completions, prints verdict: GAMEABLE and exits 2, while accuracy and verifiable with --verifiable-domain math come back robust (the format run also prints Using built-in reward function: format). format pays 0.5 for a <think>...</think> block and 0.5 for any text after it, so it rewards structure and says nothing about whether the answer is right. Train against it alone and a policy can collect it with an empty think block; combine it with a reward that checks the answer. (A robust verdict from v0.75.x for the built-in format reward, or for a custom verifier that rewards structure only, was made with four attack families that could not exercise that structure: re-probe before you train GRPO against it.)

--field (default answer) names the gold field, and for verifiable --verifiable-domain json_schema it can name a field holding the schema as a JSON object, which is decoded before scoring (--field schema). One limit: the built-in attacks are plain text and not valid JSON, so json_schema rejects them during parsing before any schema-specific constraint is evaluated. The result does not yet distinguish a strict schema from a permissive one, and its reference accept is not a meaningful control for that domain (it is normally 0%).

Reward ensembles now train (v0.71.40)

A comma-separated training.reward_fn finally trains as a reward ensemble:

task: grpo                       # root-level, NOT under training
training:
  reward_fn: "accuracy,format"   # resolves to GRPOTrainer(reward_funcs=[accuracy, format])
  gradient_checkpointing: true   # since v0.77.0 the default false reaches TRL; true keeps the old memory

This also unlocks the rm_ensemble reward-hack detector, which needs at least two rewards, and it fixes a recipe (deepseek-v3-reasoning) that shipped exactly this form and previously crashed with Unknown reward function. The reward_fn field also gained a validator that rejects null bytes, blanks, oversize strings and empty comma segments (verifiable in a comma list without a verifiable_domain now fails at parse time, like the bare form). Since v0.77.0 an ensemble can also include judge_rank, the LLM-judge ranking reward (format,judge_rank, for example; #2047, #2117), which is not a verifier: it sends prompts and completions to the judge host you configure. See RLVR and GRPO Plus.

Re-run this

Where it fits

soup reward synth  →  reward.py  →  soup reward stress  →  soup train (task: grpo)
   (build)             (commit)        (prove robust)         (train against it)

Synthesize a verifier from data you already have, stress it until the accept-rates are zero, commit the .py, then train GRPO against a reward you have actually audited.

See also

• RL & preference training — GRPO, online DPO and the judge in the loop the reward feeds.
• Closed-loop reward-hacking auto-mitigation — the train-time half: detect a policy gaming the reward and self-correct.
• soup ship — the SHIP / DON'T-SHIP verdict that gates the trained model.