DPO training guide: align LLMs with human preferences

Direct Preference Optimization (DPO) aligns language models with human preferences without needing a reward model — it's simpler and more stable than RLHF/PPO.

When to use DPO

• You have a dataset of "chosen" vs "rejected" responses
• You want to reduce hallucinations and off-topic answers
• You want alignment without the complexity of PPO

Use DPO after SFT. A typical pipeline: Pretrain → SFT → DPO.

1. DPO dataset format

[
  {
    "prompt": "Explain quantum entanglement.",
    "chosen": "Quantum entanglement is a physical phenomenon where...",
    "rejected": "Idk, something quantum."
  }
]

Save as preferences.json. That one row only shows the shape of the file: Soup holds out 10% of the rows for validation by default (data.val_split), so a one-row file leaves nothing to train on. Use your real dataset.

2. Config

base: ./merged                    # the SFT adapter merged into its base (see below)
task: dpo
backend: transformers             # root-level, NOT under training

data:
  train: preferences.json
  format: dpo
  max_length: 2048                # sequence length lives under data

training:
  epochs: 1
  lr: 5.0e-7                      # the key is lr, not learning_rate
  batch_size: 2
  gradient_accumulation_steps: 8
  dpo_beta: 0.1                   # the key is dpo_beta, not beta
  lora:
    r: 16                         # LoRA turns on when r > 0; there is no enabled flag
    alpha: 32

An SFT run saves a LoRA adapter, not a full model, and Soup does not create a latest directory. Merge the adapter into its base first and start DPO from the merged model:

soup merge --adapter ./runs/my-sft-model --output ./merged

Spell these two exactly. learning_rate is not the field, so it leaves you on the lr default of 2e-5 — about 40x too high for DPO, which is the difference between alignment and a wrecked model. Likewise beta is not read; the field is dpo_beta. Since v0.75.0 a misspelled key is refused when the config loads (soup train exits 1): unknown config key 'training.learning_rate' - refused. and unknown config key 'training.beta' - did you mean 'kto_beta' or 'dpo_beta'? Refused. Since v0.74.0 it was at least reported; before that it was dropped in silence, which is how a 40x learning rate reaches a real run.

Key DPO hyperparameters:

• dpo_beta: 0.1 — KL penalty weight. Higher = stay closer to reference model.
• lr: 5e-7 — DPO needs a much smaller LR than SFT.
• epochs: 1 — DPO overfits quickly, rarely needs more than 1–2 epochs.

Before you train

Save the config as dpo.yaml and run soup doctor --config dpo.yaml first. Since v0.76.0 many more settings are refused when the config loads, each naming the field and what to change, so a config that loaded on v0.75 can stop loading. soup doctor --config prints the error and exits 2 without loading a model (it stops at the first cross-field refusal, so fix that setting and run it again until it prints no error); a setting that nothing reads yet, or a deprecated value, prints a warning and the config still loads. See Config refusals.

3. Train

soup train --config dpo.yaml

During training the held-out rows (10% by default, data.val_split) are evaluated at the end of each epoch and recorded as eval_loss; training.eval_steps: N evaluates every N optimizer steps instead. That is new in v0.76.0. Upstream's changelog says that through v0.75.2 no trainer on the transformers backend scheduled an evaluation, so the held-out rows were never used and val_loss stayed empty (#1223).

4. Evaluate

Compare the DPO model against the SFT baseline:

soup eval compare <sft-run-id> <dpo-run-id>

DPO variants in Soup CLI

Soup supports several preference-optimization methods — swap task: to change algorithm:

• task: dpo — Direct Preference Optimization
• task: orpo — ORPO (combines SFT + DPO in one step, no reference model)
• task: simpo — SimPO (length-normalized, no reference model)
• task: ipo — IPO (IPO loss, more stable than DPO on noisy data)
• task: kto — KTO (works with unpaired binary labels). Since v0.76.0 training.batch_size must be at least 2 on every KTO run (batch_size: auto floors at 2), not only on streamed ones (#1420)

SimPO has one more check. Through v0.75.2, on trl 0.29, a row whose chosen outgrows data.max_length lost its whole rejected, the length-normalised log-probability became 0/0, every LoRA tensor went NaN and the logged loss read 0.0. Since v0.76.0 task: simpo refuses such a dataset before the first step, naming how many rows are affected and the data.max_length involved (#1208). If you trained SimPO on an earlier version with long chosen answers, check the adapter.

Schedules, blends and streaming

• training.dpo_beta_schedule is wired from v0.76.0 and needs dpo_beta_end. Through v0.75.2 a run that set it on dpo, ipo or preference stopped at step 0, because the callback was not a Hugging Face TrainerCallback (#1229, #1294); there is no finished run to re-check. training.dpo_ref_regen_epochs is wired under LoRA (#1345, #1636): it copies the active adapter into the frozen ref adapter every Nth epoch, and it is refused with lora.r: 0. Through v0.75.2 a run that set it on dpo, ipo or preference stopped when training started, for the same reason. This config loads:

training:
  dpo_beta: 0.1
  dpo_beta_schedule: linear
  dpo_beta_end: 0.05          # required with dpo_beta_schedule
  dpo_ref_regen_epochs: 2     # every 2 epochs; needs lora.r >= 1

• task: preference with preference_loss_weights trains only a blend that names simpo and orpo alone; a blend naming dpo or ipo stops at the first step. See Preference Variety for the details and the re-run rows.
• DPO also runs over a layer-streamed base (training.stream_layers: true). On an untied checkpoint such as Llama-3.1-8B, dpo and kto failed at the first backward through v0.75.2 and hold a private copy of the output head per step from v0.76.0; see Streaming preference losses. That page also explains why the memory pre-flight refuses a Llama-3.1-8B preference run on a 4 GB card.

Related

• Training methods reference
• Fine-tune Llama 3.1 with LoRA