LISA (v0.71.34)

LISA (Layerwise Importance Sampled AdamW, arXiv:2403.17919) reaches for full-fine-tuning quality without a full fine-tune's memory. On the quality half it delivers, measured at 3B and 8B. On memory it beats full fine-tuning and loses to LoRA, by a margin that widens with scale, which is measured below rather than glossed. Where Spectrum picks the layers once and trains that fixed set, LISA re-samples a small random set of decoder layers every N steps and freezes the rest. The input embeddings, the LM head, and the final norm stay trainable throughout.

task: sft
backend: transformers
modality: text
training:
  quantization: none       # LISA is a full fine-tune of the active layers
  lisa_enabled: true
  lisa_num_layers: 2       # decoder layers active per interval (clamped to model depth)
  lisa_interval_steps: 20  # re-sample cadence, in global steps

Config fields

Setting any lisa_* field away from its default while lisa_enabled is false is a hard error, not a silent no-op.

Why it saves memory, and how much

Only a handful of layers train at any moment, and their optimizer state is cleared when they are re-frozen, so peak optimizer memory is roughly embeddings + head + lisa_num_layers, far below a full fine-tune. Unlike a static freeze, the active set keeps moving, so updates spread across the depth of the model instead of staying pinned to one slice. The set is drawn at random each interval, so that is a tendency rather than a guarantee: a short run fires few intervals and will not reach every layer.

Measured at 3B and 8B, and half the pitch did not survive

LISA is often described, here included, as full fine-tuning quality at something like LoRA's memory. On an H100 80 GB, over Alpaca with 200 steps and three interleaved repeats per arm, the quality half holds and the memory half does not.

LISA beat full fine-tuning at both learning rates, so the quality claim stands. But it costs 1.22x LoRA's memory at 3B and 1.51x at 8B, and the gap widens with scale, because the embeddings, the LM head and the final norm stay trainable through every interval and account for 70.7% of everything LISA trains at 8B. The arithmetic in the paragraph above is correct and still reads as a win over LoRA. It is not one.

Stated positively, which is the honest version of the pitch: LISA trains an 8B on a single 80 GB card where full fine-tuning needs roughly 120 GB and cannot run at all. Reach for it when you want closer-to-full-fine-tune behaviour and LoRA is not enough, not when you want to save memory over LoRA.

Two caveats on the table. Held-out quality here is in-distribution loss and token accuracy on an Alpaca validation split, not a downstream benchmark. And raising lisa_num_layers costs quality as well as memory on this data: at 3B the held-out loss went 1.2504 at 2 layers, 1.2673 at 8 and 1.2950 at 16, while at 8B anything above 8 exhausts an 80 GB card. lisa_interval_steps was indistinguishable anywhere from 1 to 50.

Gates

LISA is sft + transformers + text + quantization: none only. It is mutually exclusive with LoRA features, freeze_layers / freeze_ratio, and Spectrum's unfrozen_parameters, because each of those independently decides what trains and stacking them is a footgun, not a feature. It is also mutually exclusive with train_router_only, expand_layers, freeze_trainable_layers, moe_lora, relora_steps, loraplus_lr_ratio and stream_layers (a streamed base is frozen on the meta device, so per-step layer sampling has nothing to sample). Every reject config fails with a specific message.

Implementation note: the model is left fully trainable at trainer-setup time so Hugging Face's optimizer (built before the first callback fires) contains every decoder parameter; the LISA callback then toggles requires_grad per interval, and frozen parameters produce no gradient so the optimizer skips them. This ordering invariant is what makes the callback work at all.

Live on a 4 GB GPU for small models: a 4-epoch SmolLM2-135M-Instruct run completes with no callback crash (loss 0.66, token accuracy 0.80).

See also

• Spectrum targeted training — the static, SNR-ranked counterpart.
• PEFT & efficiency — LoRA, PiSSA, ReLoRA, per-pattern rank.
• Adapter algebra — the other half of the v0.71.34 release.