DPO training guide: align LLMs with human preferences

Direct Preference Optimization (DPO) aligns language models with human preferences without needing a reward model — it's simpler and more stable than RLHF/PPO.

When to use DPO

• You have a dataset of "chosen" vs "rejected" responses
• You want to reduce hallucinations and off-topic answers
• You want alignment without the complexity of PPO

Use DPO after SFT. A typical pipeline: Pretrain → SFT → DPO.

1. DPO dataset format

[
  {
    "prompt": "Explain quantum entanglement.",
    "chosen": "Quantum entanglement is a physical phenomenon where...",
    "rejected": "Idk, something quantum."
  }
]

Save as preferences.json.

2. Config

base: ./runs/my-sft-model/latest   # Start from SFT checkpoint
task: dpo
backend: transformers             # root-level, NOT under training

data:
  train: preferences.json
  format: dpo
  max_length: 2048                # sequence length lives under data

training:
  epochs: 1
  lr: 5.0e-7                      # the key is lr, not learning_rate
  batch_size: 2
  gradient_accumulation_steps: 8
  dpo_beta: 0.1                   # the key is dpo_beta, not beta
  lora:
    r: 16                         # LoRA turns on when r > 0; there is no enabled flag
    alpha: 32

Spell these two exactly. Unknown config keys are ignored rather than rejected, so learning_rate leaves you on the lr default of 2e-5 — about 40x too high for DPO, which is the difference between alignment and a wrecked model. Likewise beta is not read; the field is dpo_beta.

Key DPO hyperparameters:

• dpo_beta: 0.1 — KL penalty weight. Higher = stay closer to reference model.
• lr: 5e-7 — DPO needs a much smaller LR than SFT.
• epochs: 1 — DPO overfits quickly, rarely needs more than 1–2 epochs.

3. Train

soup train --config dpo.yaml

4. Evaluate

Compare the DPO model against the SFT baseline:

soup eval compare <sft-run-id> <dpo-run-id>

DPO variants in Soup CLI

Soup supports several preference-optimization methods — swap task: to change algorithm:

• task: dpo — Direct Preference Optimization
• task: orpo — ORPO (combines SFT + DPO in one step, no reference model)
• task: simpo — SimPO (length-normalized, no reference model)
• task: ipo — IPO (IPO loss, more stable than DPO on noisy data)
• task: kto — KTO (works with unpaired binary labels)

Related

• Training methods reference
• Fine-tune Llama 3.1 with LoRA