Closed-loop reward-hacking auto-mitigation (v0.71.26)

The reward-hack and echo-trap detectors from loop hardening can halt a degenerate RL run. v0.71.26 closes the loop: the GRPO trainer now detects reward hacking mid-run and self-corrects — it raises the KL penalty, shapes the reward, rolls back to the last-good checkpoint, and only early-stops as a last resort. (v0.71.26 wired the signal buffer for PPO too. At v0.76.0 task: ppo refuses these flags before the model loads, because the trl PPO trainer this build supports takes no reward functions and the buffer was never written; see below. This feature is GRPO-only.)

We have not found an open-source RLHF library that closes the loop from detection → automatic correction → continue. The ones we checked detect, or they halt. Soup steers the run back on course. That is the whitespace this release aims at.

The one flag

soup train --config grpo.yaml \
  --reward-hack-detector info_rm \
  --reward-hack-mitigation kl_control

The flag takes one of off, log_only, kl_control or pid_lagrangian; the example picks kl_control. It requires a reward_hack_detector (info_rm or rm_ensemble, set by the second flag above or in the config) on a grpo transformers run. The config-file equivalent is training.reward_hack_mitigation. The schema also accepts the key on task: ppo, but the PPO trainer then refuses reward_hack_detector, reward_hack_mitigation and echo_trap_enabled at setup, before any model loads (#1441); the soup train --help text for these flags still says "GRPO/PPO".

Four modes, escalating

log_only — the eyes (no control action)

Streams a per-step mitigation_log.jsonl under the run directory: raw signal, health, drop_pct, OK / WARN / HACK verdict, current β / kl_coef, reward mean/std, completion-length mean and trend, and n-gram repetition. It provably never mutates β or kl_coef — run it first to see hacking happen before you let the controller act. The log writer is cwd-contained with rotation, secret redaction, and symlink-reject-on-rotate.

kl_control — reversible bang-bang controller with hysteresis

When a multi-signal vote (InfoRM cluster-separation or RM-ensemble divergence, plus completion length_trend and n-gram repetition) stays above the trip band for a dwell window, the controller raises β (the KL coefficient) multiplicatively, clamped to [reward_hack_beta_floor, reward_hack_beta_ceil], never crossing zero. Once the signal recovers below the release band for the patience window, it relaxes β back toward the floor. Dwell + release-patience are the hysteresis that stops it flapping.

β is written to both trainer.beta and trainer.args.beta, so it takes effect on stock GRPO and Soup's GRPO variants (gspo / dapo / dr_grpo / …); the variants read trainer.beta as trl does, and fall back to args.beta for a base that exposes only the config. (The code also writes trainer.args.kl_coef for PPO, which is now unreachable because task: ppo refuses the flags.)

(Since closed in v0.76.0: the sentence above was only true for stock GRPO through v0.75.2. The variants validated grpo_beta and dropped it, so the controller moved a β that the variant loss ignored, while trl still ran the reference forward each step (#1232). From v0.76.0 every non-standard variant applies the KL penalty, so the controller's β now takes effect on them. See GRPO Plus.)

pid_lagrangian — principled control + a safety net

Replaces bang-bang with a PID-Lagrangian controller (Stooke et al. 2020, "Responsive Safety in RL via PID Lagrangian Methods"): it treats "hacking signal ≤ target" as a constraint whose Lagrange multiplier (the β / KL coefficient) is updated by a PID law with integral anti-windup and an output clamp. On persistent hacking it climbs an escalation ladder:

1. Raise KL continuously toward the target.
2. Roll back to the last-good RL checkpoint (requires training.rl_checkpoint_save_every_steps), dropping LR / raising β on resume.
3. Early-stop cleanly with a plain-English give-up explanation — which signal, for how long, and what the controller tried — surfaced through soup why.

Anti-gaming hardening (any control mode)

So the controller itself can't be gamed: per-signal EMA or median smoothing, conservative-on-disagreement voting (when detectors disagree, keep KL high), a reward-distribution-drift guard (catches a policy shifting its reward distribution to fool the top/bottom split), and optional bounded reward shaping — a capped penalty on the gamed proxy (length / repetition / sentinel token) applied over the reward-function seam and always logged.

Example config

# soup.yaml
base: HuggingFaceTB/SmolLM2-135M
task: grpo
data:
  train: ./prompts.jsonl
training:
  reward_hack_detector: info_rm
  reward_hack_mitigation: pid_lagrangian  # off | log_only | kl_control | pid_lagrangian
  rl_checkpoint_save_every_steps: 50      # required by reward_hack_rollback below
  reward_hack_beta_floor: 0.02            # positive floor, so β never crosses 0
  reward_hack_beta_ceil: 1.0
  reward_hack_trip_band: 0.30             # raise β above this drop_pct
  reward_hack_release_band: 0.10          # relax β below this
  reward_hack_dwell_steps: 2              # consecutive trips before acting (hysteresis)
  reward_hack_release_patience: 3
  reward_hack_signals: ["info_rm", "length_trend"]
  reward_hack_kl_gain: 1.5                # how hard a trip raises beta
  # pid_lagrangian only. These are REFUSED at config load under any other
  # mitigation mode, so commenting out the mode rather than these keys does
  # not work: a YAML comment does not disable a key that is still set.
  reward_hack_pid_kp: 0.5
  reward_hack_pid_ki: 0.05
  reward_hack_pid_kd: 0.1
  reward_hack_signal_target: 0.15         # the setpoint the controller holds
  reward_hack_integral_clamp: 1.0         # anti-windup bound on the I term
  reward_hack_rollback: true              # enable the rollback rung
  reward_hack_rollback_patience: 3        # trips before rolling back
  reward_hack_max_recovery_attempts: 2    # then early-stop with a soup why
  # optional hardening
  reward_hack_signal_smoothing: ema       # none | ema | median
  reward_hack_conservative_on_disagreement: true

The config above is the pid_lagrangian shape, because the PID tunables and reward_hack_rollback are refused at config load under kl_control rather than ignored (when set to a non-default value), and reward_hack_rollback: true separately requires rl_checkpoint_save_every_steps. For plain kl_control, keep everything down to reward_hack_kl_gain and delete the PID block.

Defaults and bounds

Setting any key below to a non-default value is refused at config load while reward_hack_mitigation is off. The example above is not a set of defaults: it sets reward_hack_pid_ki: 0.05 and reward_hack_pid_kd: 0.1, where the schema's own defaults are 0.1 and 0.05.

Reward shaping is therefore off twice over: the switch defaults to false, and the strength defaults to 0.0, so turning on reward_hack_reward_shaping without also raising reward_hack_shaping_strength is refused rather than silently doing nothing. Every key marked pid_lagrangian only is likewise refused, when set to a non-default value, under log_only and kl_control, as the paragraph above says.

pid_lagrangian has never had a valid run (upstream #371). In the H100 gate record all three pid_lagrangian attempts crashed (at steps 6, 25 and 8), and upstream attributes those crashes to the non-finite-gradient failure filed as #342. v0.76.0 added the gradient watchdog for #342 on task: grpo, but benchmarks/ holds no pid_lagrangian run recorded after it. The mechanism is implemented and the mode is selectable; its efficacy is not established. kl_control is the mode with a demonstrated mid-run rollback behind it.

Honest limitations

This is an RL feature, so it cannot ship "schema-only" — a controller that mutates training dynamics is only real if a logged run shows it working. On the maintainer's single RTX 3050 that means proof-of-mechanism only:

• Validated on SmolLM2-135M + a synthetic length-hacking task, with all four modes live including a real mid-run rollback.
• The mitigation is refused on task: ppo at v0.76.0 (task: ppo itself still runs; it refuses the reward-hack detector, the mitigation and the echo-trap flags). v0.71.26 shipped it on PPO as BETA, with the signal buffer and kl_coef mutation unit-tested, but the on-GPU proof was always GRPO-only, and the trl PPO trainer this build supports never fed the buffer (#1441). Use task: grpo for these flags.
• Whether the loop suppresses hacking without collapsing true reward on 7B+ models with real reward models is an open, community-validatable question, tracked in issue #286. Contributors with larger GPUs (or soup train --cloud modal) can reproduce the experiments and report back.
• reward_hack_mitigation in {kl_control, pid_lagrangian} is mutually exclusive with a DPO-style β schedule / ref_model_ema_alpha (both drive the KL / ref dynamics).
• It is also mutually exclusive with grpo_beta: 0 (KL-free GRPO), because the controller needs a positive β to steer (#1247). The config is refused at load: grpo_beta: 0 is mutually exclusive with reward_hack_mitigation='kl_control' (the mitigation controller requires a positive beta to steer); use grpo_beta > 0 or disable mitigation.
• info_rm needs a reward that varies on both sides of the batch median. On a two-valued reward such as reward_fn: accuracy (0 or 1), a step whose median split leaves either half constant casts no vote and logs one warning naming the cause (#1438). Through v0.75.2 a first step near 50% success recorded a phantom baseline, so a healthy later step read as a collapse and the controller could raise β, roll back or halt on a false signal; mitigation_log.jsonl from earlier runs is not comparable. On a two-valued reward use rm_ensemble or the length_trend and repetition signals. See Loop hardening.
• Every task: grpo run now also skips non-finite optimizer steps through a gradient watchdog, whether or not a mitigation mode is set (#342); see GRPO Plus.

Numbers

+184 tests in tests/test_v07126.py, taking the suite to 14,788 across ~298 files.

See also

• Loop hardening (v0.70 → live v0.71.11) — the detectors this feature turns from "halt" into "self-correct".
• Lean install + live wiring (v0.71) — the release line this caps.
• GRPO Plus — the GRPO variants β mutation reaches on both code paths.
• soup ship — the post-training SHIP / DON'T-SHIP verdict that pairs with a hardened run.