Closed-loop reward-hacking auto-mitigation (v0.71.26)

The reward-hack and echo-trap detectors from loop hardening can *halt* a degenerate RL run. v0.71.26 closes the loop: the GRPO/PPO trainer now detects reward hacking mid-run and self-corrects — it raises the KL penalty, shapes the reward, rolls back to the last-good checkpoint, and only early-stops as a last resort.

No open-source RLHF library — TRL, Unsloth, Axolotl, OpenRLHF, verl — closes the loop from detection → automatic correction → continue. They detect, or they halt. Soup steers the run back on course. That is the whitespace this release fills.

The one flag

bash
soup train --task grpo --config grpo.yaml \
  --reward-hack-mitigation off | log_only | kl_control | pid_lagrangian

It requires a reward_hack_detector (info_rm or rm_ensemble) on a grpo / ppo transformers run. The config-file equivalent is training.reward_hack_mitigation.

Four modes, escalating

log_only — the eyes (no control action)

Streams a per-step mitigation_log.jsonl under the run directory: raw signal, health, drop_pct, OK / WARN / HACK verdict, current β / kl_coef, reward mean/std, completion-length mean and trend, and n-gram repetition. It provably never mutates β or kl_coef — run it first to *see* hacking happen before you let the controller act. The log writer is cwd-contained with rotation, secret redaction, and symlink-reject-on-rotate.

kl_control — reversible bang-bang controller with hysteresis

When a multi-signal vote (InfoRM cluster-separation or RM-ensemble divergence, plus completion length_trend and n-gram repetition) stays above the trip band for a dwell window, the controller raises β (the KL coefficient) multiplicatively, clamped to [reward_hack_beta_floor, reward_hack_beta_ceil], never crossing zero. Once the signal recovers below the release band for the patience window, it relaxes β back toward the floor. Dwell + release-patience are the hysteresis that stops it flapping.

β is written to both trainer.beta and trainer.args.beta, so it takes effect on stock GRPO and Soup's GRPO variants (gspo / dapo / dr_grpo / …), and to trainer.args.kl_coef on PPO.

pid_lagrangian — principled control + a safety net

Replaces bang-bang with a PID-Lagrangian controller (Stooke et al. 2020, "Responsive Safety in RL via PID Lagrangian Methods"): it treats "hacking signal ≤ target" as a constraint whose Lagrange multiplier (the β / KL coefficient) is updated by a PID law with integral anti-windup and an output clamp. On persistent hacking it climbs an escalation ladder:

  1. Raise KL continuously toward the target.
  2. Roll back to the last-good RL checkpoint (requires --rl-checkpoint-save-every-steps), dropping LR / raising β on resume.
  3. Early-stop cleanly with a plain-English give-up explanation — which signal, for how long, and what the controller tried — surfaced through soup why.

Anti-gaming hardening (any control mode)

So the controller itself can't be gamed: per-signal EMA or median smoothing, conservative-on-disagreement voting (when detectors disagree, keep KL high), a reward-distribution-drift guard (catches a policy shifting its reward distribution to fool the top/bottom split), and optional bounded reward shaping — a capped penalty on the gamed proxy (length / repetition / sentinel token) applied over the reward-function seam and always logged.

Example config

yaml
# soup.yaml
base: HuggingFaceTB/SmolLM2-135M
task: grpo
data:
  train: ./prompts.jsonl
training:
  reward_hack_detector: info_rm
  reward_hack_mitigation: kl_control      # off | log_only | kl_control | pid_lagrangian
  reward_hack_beta_floor: 0.02            # positive floor, so β never crosses 0
  reward_hack_beta_ceil: 1.0
  reward_hack_trip_band: 0.30             # raise β above this drop_pct
  reward_hack_release_band: 0.10          # relax β below this
  reward_hack_dwell_steps: 2              # consecutive trips before acting (hysteresis)
  reward_hack_release_patience: 3
  reward_hack_signals: ["info_rm", "length_trend"]
  # optional hardening
  reward_hack_signal_smoothing: ema       # none | ema | median
  reward_hack_conservative_on_disagreement: true

Switch reward_hack_mitigation to pid_lagrangian and add rl_checkpoint_save_every_steps to unlock the rollback rung of the ladder.

Honest limitations

This is an RL feature, so it cannot ship "schema-only" — a controller that mutates training dynamics is only real if a logged run shows it working. On the maintainer's single RTX 3050 that means proof-of-mechanism only:

  • Validated on SmolLM2-135M + a synthetic length-hacking task, with all four modes live including a real mid-run rollback.
  • PPO ships BETA — the signal buffer and kl_coef mutation are wired and unit-tested, but the on-GPU proof is GRPO-only.
  • Whether the loop suppresses hacking without collapsing true reward on 7B+ models with real reward models is an open, community-validatable question, tracked in issue #286. Contributors with larger GPUs (or soup train --cloud modal) can reproduce the experiments and report back.
  • reward_hack_mitigation in {kl_control, pid_lagrangian} is mutually exclusive with a DPO-style β schedule / ref_model_ema_alpha (both drive the KL / ref dynamics).

Numbers

+184 tests in tests/test_v07126.py, taking the suite to 14,788 across ~298 files.

See also