Reward Forge: synthesize a reward verifier, then prove it cannot be gamed (v0.71.40 → v0.71.41)

An RL run trusts its reward verifier more than anything else and inspects it least. If the verifier pays out for the wrong reasons, the policy learns to hack it, and you find out after the GPU hours. Reward Forge is two commands that turn Soup's reward-hacking detector on the verifier itself: soup reward synth builds a deterministic verifier from your data and refuses a bad one; soup reward stress attacks a verifier to see if degenerate junk gets paid.

Both are pure and offline, and both ride the existing load_reward_fn .py path, so they add no new trusted-execution surface. No RL library (TRL, Unsloth, Axolotl, OpenRLHF) synthesizes a reward or tests one for gameability.

soup reward synth — a verifier from your gold outputs (v0.71.40)

soup reward synth golds.jsonl -o reward.py                      # auto-detect the family
soup reward synth golds.jsonl -o reward.py --kind numeric --tolerance 1e-6
soup reward synth golds.jsonl --plan-only                       # show the induced spec, write nothing
soup reward synth golds.jsonl -o reward.py --output-report calib.json

Point it at a JSONL of reference (gold) outputs and it infers one of four verifier families, auto-detected or forced with --kind, each grounded in your golds:

The emitted reward.py is self-contained, readable and committable — you read it, edit it, diff it in review, and check it into git, exactly like a hand-written reward. It is not a trained-weights black box.

The moat: mandatory calibration

The differentiator is that synth refuses to emit a verifier that cannot tell good from bad. Before writing the file, the synthesized verifier is loaded back and run against:

• its own references — it must accept at least 90%, and
• auto-generated bad answers — it must reject them.

A degenerate always-accept verifier is refused with exit 2, never silently written. --plan-only reports the induced spec without writing, and --output-report persists the calibration JSON.

soup reward stress — is the verifier gameable? (v0.71.41)

soup reward stress reward.py --references golds.jsonl
soup reward stress reward.py --references golds.jsonl --attacks empty,length,repetition,sentinel --sentinel GOLD
soup reward stress verifiable --verifiable-domain math --references golds.jsonl   # probe a builtin

Where synth proves a verifier separates references from *friendly* perturbations, stress asks the adversarial question a reward-hacking model asks at train time: does the verifier pay out for degenerate junk? It feeds four attack families, each scored against the real gold (references are cycled), so a numeric, tool_call or json_schema verifier still gets a valid target and must reject the garbage:

• empty — an empty completion
• length — a length-padded completion
• repetition — a repeated fragment
• sentinel — spam containing a magic token (--sentinel)

$ soup reward stress reward.py --references golds.jsonl
attack       accept-rate
empty        0%
length       0%
repetition   0%
sentinel     0%
verdict: ROBUST   0 / 4 attacks gamed   exit 0

It reports a per-attack accept-rate table plus an overall gameability verdict, tuned by --max-gameable and --threshold, and --output-report persists the JSON. It probes a synthesized .py and a builtin (accuracy, format, verifiable) through the same loader. Exit codes: 0 = robust, 2 = gameable, 1 = error. A gold-requiring verifier probed with no `--references` is a hard error naming the flag, never a false "robust".

Reward ensembles now train (v0.71.40)

A comma-separated training.reward_fn finally trains as a reward ensemble:

training:
  task: grpo
  reward_fn: "accuracy,format"   # resolves to GRPOTrainer(reward_funcs=[accuracy, format])

This also unlocks the rm_ensemble reward-hack detector, which needs at least two rewards, and it fixes a recipe (deepseek-v3-reasoning) that shipped exactly this form and previously crashed with Unknown reward function. The reward_fn field also gained a validator that rejects null bytes, blanks, oversize strings and empty comma segments (verifiable in a comma list without a verifiable_domain now fails at parse time, like the bare form).

Where it fits

soup reward synth  →  reward.py  →  soup reward stress  →  soup train --task grpo
   (build)             (commit)        (prove robust)         (train against it)

Synthesize a verifier from data you already have, stress it until the accept-rates are zero, commit the .py, then train GRPO against a reward you have actually audited.

See also

• RL & preference training — GRPO, online DPO and the judge in the loop the reward feeds.
• Closed-loop reward-hacking auto-mitigation — the train-time half: detect a policy gaming the reward and self-correct.
• soup ship — the SHIP / DON'T-SHIP verdict that gates the trained model.