Preference losses over layer streaming (v0.72.4)

Until v0.72.4, a model too big for your card could only be supervised fine-tuned on it. task had to be sft, and dpo, orpo, simpo and kto were all refused when the config was read. All four run against a streamed base now, with the same keys SFT already used, so preference alignment stops being the step that forces you to rent a bigger card.

Nothing else about streaming changed. If you have not read it yet, start with Layer streaming for the mechanism and Scaling a streaming run for architectures, batches and the disk tier. This page is only about what the preference losses add on top.

The five accepted tasks

sft   dpo   orpo   simpo   kto

That list is exact. ipo, bco and the unified task: preference dispatcher are not in the streaming allowlist, so "all preference losses stream" would be false. grpo and ppo are excluded permanently, and their refusal deliberately names no release: generation rollouts re-read every layer once per generated token, which destroys the amortisation streaming exists to buy.

DPO's reference model needs no second copy

DPO scores the policy against a frozen reference. The obvious implementation builds a second copy of the model, which doubles the weights and defeats the entire premise of streaming: you would be back to needing the model twice over on a card that could not hold it once.

Soup takes the reference from the same streamed base with its LoRA adapters switched off. Policy and reference are one set of weights, one RAM store and one buffer pool, and the adapter is the only difference between the two forward passes. On an untied checkpoint dpo and kto do still hold one private copy of the output head's weight per step, since v0.76.0 a documented and pre-flight-charged cost; see Untied checkpoints.

Measured on the reference box, a 4 GB RTX 3050 Laptop, in bf16 with a 365.2M-parameter model (730.4 MB of weights, 24 decoder layers, vocabulary 260, sequence length 64, batch 1):

The RAM store was byte-identical between the SFT and DPO arms (729.91 MB each), and so was the VRAM buffer pool (60.83 MB each).

The control arm is what makes the headline mean anything. Forcing a real second instance moved the peak by 730.44 MB against 730.44 MB of weights, which is exactly one copy. As the gate record puts it: "DPO is not 2x SFT" means nothing unless the same harness can show what 2x looks like.

Read 0.914x as "no second copy", not as a saving

DPO is not cheaper than SFT in general. That row sits below 1.0 because of the fixture, not because of the feature: the SFT arm computes its loss over all 64 positions, while the DPO arm splits the same 64 into a 32-token prompt and a 32-token completion, so its logits tensor is smaller. A negative delta is simply the strongest available form of "there is no second copy of the weights".

These numbers also come from a synthetic 365.2M-parameter model, not from a real checkpoint and not from Llama-3.1-8B. They are a memory-accounting result, not a throughput result.

Why this was measured rather than asserted

A passing loss curve cannot detect the failure this feature is exposed to. The streamed layer substitutes base weights through functional_call rather than through the module's own forward, so it is entirely plausible that disabling the adapter is a silent no-op through that path. If it were, the reference would be the policy, every log-ratio would be zero, and the DPO loss would sit at 0.6931 forever, which reads as slow training rather than as a bug.

So three things were checked that a loss curve cannot check:

1. The memory table above, with its forced-second-instance control.
2. A policy-versus-reference gap of 8.219452e-01 on chosen completions, proving the two passes really differ.
3. A zero-adapter control measuring exactly 0.000e+00, proving that gap comes from the adapter and not from some other difference between the two forwards.

KTO is not reference-free

However it is usually described, KTO picks its reference exactly the way DPO does, so it gets exactly the same treatment: the same streamed base with adapters disabled, no second copy. ORPO and SimPO genuinely are reference-free and never construct one at all.

KTO carries one extra requirement, and Soup enforces it when the config is read rather than minutes into a run after the checkpoint has already been sharded. Since v0.76.0 that applies to every KTO config, streamed or resident (#1420):

task='kto' requires training.batch_size >= 2: TRL's KL term is degenerate at a per-device batch of 1. Set batch_size to 2 or more and raise gradient_accumulation_steps to keep the effective batch size.

Note that KTO is streamable only because v0.72.3 lifted streaming's own batch-1 restriction. Under v0.72.0 through v0.72.2 it could not have shipped at all.

No second copy is not free in time

DPO walks the layer stack more often than supervised fine-tuning does, because it adds a reference forward on top of the policy forward and the gradient-checkpoint recompute. Measured on the same 24-layer model: 46 layer loads per step for SFT against 70 for DPO, which is 1.52x.

That is the honest cost, and it is the one to plan around. Streaming means the reference needs no second copy of the weights; it does not make the reference free. On an untied checkpoint there is a memory cost as well, a private copy of the output head per step, described under Untied checkpoints.

Correctness

Upstream's CHANGELOG states the loss is bit-exact against a resident run of the same loss for all four, and the benchmark record shows it for DPO: max|loss_streamed - loss_resident| = 0.0 over 16 synchronised adapter tensors (streamed 0.67242944, resident 0.67242944), in float32 on CPU on a 4-layer model, precisely so that "bit-exact" means literally zero rather than bf16 noise. That is a forward-side comparison of the loss; layer-0 adapter gradients were checked to be non-zero, not equal. Two runs at the same seed differ by 0.000e+00.

End to end through the released CLI, streaming SmolLM2-135M under NF4 (30 layers, a 0.05 GB pinned RAM store, two 2 MB VRAM buffers):

All four saved adapters are ordinary LoRA files: 120 tensors, zero keys carrying the streaming wrapper segment, and 60 of 60 lora_B tensors non-zero. They load into a non-streaming model like any other adapter.

A config that runs

DPO over a streamed, NF4-quantized base:

base: Qwen/Qwen2.5-3B
task: dpo
backend: transformers        # streaming requires it

data:
  train: ./data/preferences.jsonl
  format: dpo
  max_length: 512            # see the pre-flight note below

training:
  epochs: 3
  dpo_beta: 0.1
  batch_size: 1              # a concrete value; "auto" is refused
  gradient_accumulation_steps: 1
  quantization: 4bit         # or none
  gradient_checkpointing: true
  stream_layers: true
  stream_source: auto        # RAM when it fits, NVMe when it does not
  stream_buffers: 2
  lora:
    r: 64
    alpha: 16

output: ./output

The other three are the same file with one or two lines changed:

• ORPO: task: orpo, and orpo_beta: 0.1 in place of dpo_beta.
• SimPO: task: simpo, with simpo_gamma: 0.5 and cpo_alpha: 1.0.
• KTO: task: kto, format: kto, kto_beta: 0.1, and batch_size: 2 or more.

The rows go through the same data loader and converters as a resident run, so the v0.76.1 change to preference rows applies here unchanged: the dpo and kto converters hand the trainer every row in one layout, all strings or all message lists with the prompt turns in prompt only, and a row that cannot be written that way is dropped at load with a reason that names the field. orpo and simpo prepare the same tokens as before. What changed in v0.76.1. Since v0.77.0 a KTO row whose label is not a boolean, a true / false-style string, or exactly -1, 0 or 1 is dropped at load with a reason too, instead of being trained as a desirable or undesirable row.

Before you configure it: the pre-flight is a loose upper bound

The VRAM pre-flight predicts peak VRAM and refuses a run that will not fit. It is sound but conservative, and a preference run feels that most because it sends twice the rows through the same charge of 14 bytes per logit element, while TRL reduces preference logits to per-token log-probabilities with selective_log_softmax instead of holding a full-vocabulary fp32 upcast. The gap is large: DPO's entire above-resident cost measured 51.76 MB where the charge for the same shape is roughly 458 MB.

One correction worth carrying, from a later series of 72 runs, three repeats per configuration, with no spread: the over-prediction is not a preference-loss problem. A supervised control at the same effective row count misses by the same margin, so doubling the rows for a paired loss is the right thing to do and one shared coefficient is simply too high. Preference losses are where you notice it, not where it comes from.

In practice, on a 4 GB card with a 128k-vocabulary 1B model, DPO is allowed at max_length: 512 and refused from 768 up, even though it would probably fit. On an untied checkpoint, dpo and kto are also charged a second large-layer slot (v0.76.0).

That shipped deliberately, because under-predicting is the strictly worse failure. On Windows an overcommit is not an exception you can catch: WDDM silently spills into shared host memory, so the run completes an order of magnitude slower with no error at all. A run completing is not evidence that its configuration fits. Refusing is.

If the pre-flight refuses a preference run you believe fits, lower data.max_length first. Tightening the estimate for these losses is tracked upstream.

A packaging bug fixed along the way

Six preference trainers pass a field that TRL removed across several of its own releases, so on a fresh install soup train with task: orpo could fail at import before training ever started. It was invisible to CI because the TRL imports live inside the trainer's setup(), and no test had ever called setup() on those wrappers. The dependency bound that pins it was then settled by constructing all six configs against each candidate version rather than by reading source, which is the general lesson worth keeping: a version bound derived by reading source is a hypothesis, and the experiment that tests it is constructing the object.

This was a pre-existing defect, not one v0.72.4 introduced. A contract test now drives the real setup() for all six on the ordinary non-streaming path, and derives its covered set from the trainer sources so a seventh trainer joins automatically.

What is still out of scope

Layer streaming remains BETA.

• task must be one of sft, dpo, orpo, simpo, kto. ipo, bco and task: preference are refused.
• GRPO and PPO are permanently excluded, not pending a release.
• backend: transformers, modality: text, plain LoRA, and quantization of none or 4bit.
• No tokens-per-second figure is claimed for any preference loss, at any model size, because none has been measured. A later run on borrowed hardware timed all four in samples per second, which is a different unit and a directional one; see the section below.
• The sizes above 8B that have been measured for streaming were measured on server hardware, not on the card this feature targets. What those runs establish.
• The v0.77.0 speed and context keys stream_recompute (off, auto) and stream_loss_chunk are task: sft only, and are refused at config load on dpo, orpo, simpo and kto (what they do).
• deepseek_v3 bases (and Kimi-K2-Base and -Instruct through their config check, verified on CPU with a tiny K2-shaped checkpoint only; nothing has been trained on K2), added in v0.77.0, are admitted on all five streaming tasks, but their parity gate is a causal-LM loss and its LoRA gradients, on a tiny from-config model on CPU; no real DeepSeek-V3 checkpoint was compared against a resident run, and there is no NF4 parity check for it. No preference-loss parity was measured on a DeepSeek-V3-family base, and none is claimed. Streaming a Mixture-of-Experts base.
• A LoRA target that matched the streamed head of an untied checkpoint through a regex used to fail in peft during DPO setup (Target module StreamedLargeLayer(...) is not supported); since v0.77.0 it is refused by name when the model is built (#2082).

Since measured on a real 8B

The numbers above come from a 365.2M-parameter synthetic fixture, which is the right size to make a second copy of the weights impossible to miss but the wrong size to be convincing on its own. The same claim was later checked on Llama-3.1-8B in NF4, streamed, through the shipped CLI, on an 8x H100 box:

Peak VRAM is flat across all five, within 64 MiB or 1.7%. DPO costs 44 MiB more than supervised fine-tuning, against roughly 5.6 GB for a second resident copy of an NF4 8B. That is the claim of this page at the layout it was measured on, at a size where a second copy could not hide. The next section says which layout that was.

The throughput column needs three cautions. These are samples per second, not tokens per second, so they do not translate into the tok/s figures quoted elsewhere on this site. They come from 64-row single-epoch runs where setup is a large share of an 11 to 14 second step, so the ordering is reproducible and the absolute ratios are directional. And kto ran a different dataset, unpaired, so only its VRAM column compares directly.

Note that DPO's 0.77x here and the 1.52x layer reads quoted above are not the same metric and neither corrects the other: one counts samples per second end to end, the other counts weight reads per optimizer step.

One scoping note on a rule stated elsewhere: a preference loss on Llama-3.1-8B is refused by the pre-flight on a 4 GB card, and that is a property of the card, not of the model. On an 80 GB card the same configuration passes the pre-flight and trains, which is exactly what the table above is.

Untied checkpoints (v0.76.0)

The table above was measured before untied embeddings and output heads started sharing one device slot (v0.74.0). On an untied checkpoint (Llama-3.1-8B is one), dpo and kto run a reference forward before the backward. That forward refilled the shared slot the output head's autograd view pointed into, and the policy backward then failed with one of the variables needed for gradient computation has been modified by an inplace operation (#1049). That was a hard error and not a silent one, so there is nothing to re-run. Upstream's changelog calls untied "the common shape" (Llama-3-8B-class checkpoints are untied). The shared slot arrived in v0.74.0, so that is the release from which the failure was possible, through v0.75.2.

v0.76.0 fixes it by giving those two losses a private copy of the output head's weight for the step, one head-sized buffer, and only when the slot is shared. The VRAM pre-flight charges the copy as a second large-layer slot for exactly the runs that take it (about 1.05 GB for Llama-3.1-8B in bf16, per upstream's docs). That charge is a bound the pre-flight uses, not a measured peak, and no new peak was measured. sft, orpo, simpo and tied checkpoints pay nothing.

So "DPO costs 44 MiB more than SFT" describes the measured layout and should not be read as a statement about an untied base on v0.76.0. It also does not change the card-level rule stated just above: a preference loss on Llama-3.1-8B is refused by the pre-flight on a 4 GB card, and the second large slot only widens that.

Measurement records and citation

The gate record behind this release is published in full, including the checks that failed, the diagnoses that turned out wrong, and the numbers that were measured and then discarded: the benchmarks directory. The layer-streaming work also has a preprint, 10.5281/zenodo.21771064, summarised on the paper page. Note the scope: v1 of the preprint measured the mechanism as it shipped in v0.72.3, v2 adds the 8x H100 session and the correctness repair that shipped in v0.73.0, and v3 retracts one explanation v2 gave without changing a measured number. The preference losses on this page came after v1 and are still outside what the paper measures. The preprint is unchanged at v0.77.0.

See also

• Layer streaming — the mechanism, the measured numbers and the full refusal table.
• Scaling a streaming run — architectures, batch versus gradient accumulation, the pre-flight, resume and the disk tier.
• Layer streaming disk tier — v0.76.0: the rewritten NVMe tier and every streaming knob.
• Preference variety — the preference losses themselves, and how they differ.
• Online DPO and the judge in the loop — preference training with a judge or reward model on-policy.