Preference losses over layer streaming (v0.72.4)

Until v0.72.4, a model too big for your card could only be supervised fine-tuned on it. task had to be sft, and dpo, orpo, simpo and kto were all refused when the config was read. All four run against a streamed base now, with the same keys SFT already used, so preference alignment stops being the step that forces you to rent a bigger card.

Nothing else about streaming changed. If you have not read it yet, start with Layer streaming for the mechanism and Scaling a streaming run for architectures, batches and the disk tier. This page is only about what the preference losses add on top.

The five accepted tasks

sft   dpo   orpo   simpo   kto

That list is exact. ipo, bco and the unified task: preference dispatcher are not in the streaming allowlist, so "all preference losses stream" would be false. grpo and ppo are excluded permanently, and their refusal deliberately names no release: generation rollouts re-read every layer once per generated token, which destroys the amortisation streaming exists to buy.

DPO's reference model is free

DPO scores the policy against a frozen reference. The obvious implementation builds a second copy of the model, which doubles the weights and defeats the entire premise of streaming: you would be back to needing the model twice over on a card that could not hold it once.

Soup takes the reference from the same streamed base with its LoRA adapters switched off. Policy and reference are one set of weights, one RAM store and one buffer pool, and the adapter is the only difference between the two forward passes.

Measured on the reference box, a 4 GB RTX 3050 Laptop, in bf16 with a 365.2M-parameter model (730.4 MB of weights, 24 decoder layers, vocabulary 260, sequence length 64, batch 1):

The RAM store was byte-identical between the SFT and DPO arms (729.91 MB each), and so was the VRAM buffer pool (60.83 MB each).

The control arm is what makes the headline mean anything. Forcing a real second instance moved the peak by 730.44 MB against 730.44 MB of weights, which is exactly one copy. As the gate record puts it: "DPO is not 2x SFT" means nothing unless the same harness can show what 2x looks like.

Read 0.914x as "no second copy", not as a saving

DPO is not cheaper than SFT in general. That row sits below 1.0 because of the fixture, not because of the feature: the SFT arm computes its loss over all 64 positions, while the DPO arm splits the same 64 into a 32-token prompt and a 32-token completion, so its logits tensor is smaller. A negative delta is simply the strongest available form of "there is no second copy of the weights".

These numbers also come from a synthetic 365.2M-parameter model, not from a real checkpoint and not from Llama-3.1-8B. They are a memory-accounting result, not a throughput result.

Why this was measured rather than asserted

A passing loss curve cannot detect the failure this feature is exposed to. The streamed layer substitutes base weights through functional_call rather than through the module's own forward, so it is entirely plausible that disabling the adapter is a silent no-op through that path. If it were, the reference would be the policy, every log-ratio would be zero, and the DPO loss would sit at 0.6931 forever, which reads as slow training rather than as a bug.

So three things were checked that a loss curve cannot check:

1. The memory table above, with its forced-second-instance control.
2. A policy-versus-reference gap of 8.219452e-01 on chosen completions, proving the two passes really differ.
3. A zero-adapter control measuring exactly 0.000e+00, proving that gap comes from the adapter and not from some other difference between the two forwards.

KTO is not reference-free

However it is usually described, KTO picks its reference exactly the way DPO does, so it gets exactly the same treatment: the same streamed base with adapters disabled, no second copy. ORPO and SimPO genuinely are reference-free and never construct one at all.

KTO carries one extra requirement, and Soup enforces it when the config is read rather than minutes into a run after the checkpoint has already been sharded:

task='kto' requires training.batch_size >= 2 (TRL's KL term is
degenerate at batch 1).

Note that KTO is streamable only because v0.72.3 lifted streaming's own batch-1 restriction. Under v0.72.0 through v0.72.2 it could not have shipped at all.

Free in memory is not free in time

DPO walks the layer stack more often than supervised fine-tuning does, because it adds a reference forward on top of the policy forward and the gradient-checkpoint recompute. Measured on the same 24-layer model: 46 layer loads per step for SFT against 70 for DPO, which is 1.52x.

That is the honest cost, and it is the one to plan around. Streaming makes the reference free in memory; it does not make it free.

Correctness

Every one of the four losses is bit-exact against a resident run of the same loss, difference exactly 0.0. DPO's arm ran in float32 on CPU precisely so that "bit-exact" means literally zero rather than bf16 noise: streamed 0.67242944 against resident 0.67242944 over 16 synchronised adapter tensors. Layer-0 adapter gradients are non-zero, and two runs at the same seed differ by 0.000e+00.

End to end through the released CLI, streaming SmolLM2-135M under NF4 (30 layers, a 0.05 GB pinned RAM store, two 2 MB VRAM buffers):

All four saved adapters are ordinary LoRA files: 120 tensors, zero keys carrying the streaming wrapper segment, and 60 of 60 lora_B tensors non-zero. They load into a non-streaming model like any other adapter.

A config that runs

DPO over a streamed, NF4-quantized base:

base: Qwen/Qwen2.5-3B
task: dpo
backend: transformers        # streaming requires it

data:
  train: ./data/preferences.jsonl
  format: dpo
  max_length: 512            # see the pre-flight note below

training:
  epochs: 3
  dpo_beta: 0.1
  batch_size: 1              # a concrete value; "auto" is refused
  gradient_accumulation_steps: 1
  quantization: 4bit         # or none
  gradient_checkpointing: true
  stream_layers: true
  stream_source: auto        # RAM when it fits, NVMe when it does not
  stream_buffers: 2
  lora:
    r: 64
    alpha: 16

output: ./output

The other three are the same file with one or two lines changed:

• ORPO: task: orpo, and orpo_beta: 0.1 in place of dpo_beta.
• SimPO: task: simpo, with simpo_gamma: 0.5 and cpo_alpha: 1.0.
• KTO: task: kto, format: kto, kto_beta: 0.1, and batch_size: 2 or more.

Before you configure it: the pre-flight is a loose upper bound

The VRAM pre-flight predicts peak VRAM and refuses a run that will not fit. For preference losses it is sound but conservative. It charges the supervised loss's 14 bytes per logit element over twice the rows, while TRL reduces preference logits to per-token log-probabilities with selective_log_softmax instead of holding a full-vocabulary fp32 upcast. The gap is large: DPO's entire above-resident cost measured 51.76 MB where the charge for the same shape is roughly 458 MB.

In practice, on a 4 GB card with a 128k-vocabulary 1B model, DPO is allowed at `max_length: 512` and refused from `768` up, even though it would probably fit.

That shipped deliberately, because under-predicting is the strictly worse failure. On Windows an overcommit is not an exception you can catch: WDDM silently spills into shared host memory, so the run completes an order of magnitude slower with no error at all. A run completing is not evidence that its configuration fits. Refusing is.

If the pre-flight refuses a preference run you believe fits, lower data.max_length first. Tightening the estimate for these losses is tracked upstream.

A packaging bug fixed along the way

Six preference trainers pass a field that TRL removed across several of its own releases, so on a fresh install soup train --task orpo could fail at import before training ever started. It was invisible to CI because the TRL imports live inside the trainer's setup(), and no test had ever called setup() on those wrappers. The dependency bound that pins it was then settled by constructing all six configs against each candidate version rather than by reading source, which is the general lesson worth keeping: a version bound derived by reading source is a hypothesis, and the experiment that tests it is constructing the object.

This was a pre-existing defect, not one v0.72.4 introduced. A contract test now drives the real setup() for all six on the ordinary non-streaming path, and derives its covered set from the trainer sources so a seventh trainer joins automatically.

What is still out of scope

Layer streaming remains BETA.

• task must be one of sft, dpo, orpo, simpo, kto. ipo, bco and task: preference are refused.
• GRPO and PPO are permanently excluded, not pending a release.
• backend: transformers, modality: text, plain LoRA, and quantization of none or 4bit.
• No throughput figure is claimed for any preference loss, at any model size, because none was measured.
• Nothing above 8B has been measured for streaming at all, so there is no 14B or 70B claim.

Measurement records and citation

The gate record behind this release is published in full, including the checks that failed, the diagnoses that turned out wrong, and the numbers that were measured and then discarded: the benchmarks directory. The layer-streaming work also has a preprint: 10.5281/zenodo.21771064.

See also

• Layer streaming — the mechanism, the measured numbers and the full refusal table.
• Scaling a streaming run — architectures, batch versus gradient accumulation, the pre-flight, resume and the disk tier.
• Preference variety — the preference losses themselves, and how they differ.
• Online DPO and the judge in the loop — preference training with a judge or reward model on-policy.