LISA (v0.71.34)
LISA (Layerwise Importance Sampled AdamW, arXiv:2403.17919) targets full-fine-tuning quality at LoRA-like memory. Where Spectrum picks the layers once and trains that fixed set, LISA re-samples a small random set of decoder layers every N steps and freezes the rest. The input embeddings, the LM head, and the final norm stay trainable throughout.
task: sft
backend: transformers
modality: text
training:
quantization: none # LISA is a full fine-tune of the active layers
lisa_enabled: true
lisa_num_layers: 2 # decoder layers active per interval (clamped to model depth)
lisa_interval_steps: 20 # re-sample cadence, in global stepsConfig fields
| Field | Default | Meaning |
|---|---|---|
lisa_enabled | false | Turns LISA on. |
lisa_num_layers | 2 | Decoder layers trainable per interval, clamped to the model's depth. |
lisa_interval_steps | 20 | How often, in global steps, the active set is re-sampled. |
lisa_reset_optimizer | true | Clears optimizer state for layers as they are re-frozen. |
Setting any lisa_* field away from its default while lisa_enabled is false is a hard error, not a silent no-op.
Why it saves memory
Only a handful of layers train at any moment, and their optimizer state is cleared when they are re-frozen, so peak optimizer memory is roughly embeddings + head + lisa_num_layers, far below a full fine-tune. Unlike a static freeze, the active set keeps moving, so updates spread across the depth of the model instead of staying pinned to one slice. The set is drawn at random each interval, so that is a tendency rather than a guarantee: a short run fires few intervals and will not reach every layer.
Gates
LISA is sft + transformers + text + quantization: none only. It is mutually exclusive with LoRA features, freeze_layers / freeze_ratio, and Spectrum's unfrozen_parameters, because each of those independently decides what trains and stacking them is a footgun, not a feature. It is also mutually exclusive with train_router_only, expand_layers, freeze_trainable_layers, moe_lora, relora_steps and loraplus_lr_ratio. Every reject config fails with a specific message.
Implementation note: the model is left fully trainable at trainer-setup time so Hugging Face's optimizer (built before the first callback fires) contains every decoder parameter; the LISA callback then toggles
requires_gradper interval, and frozen parameters produce no gradient so the optimizer skips them. This ordering invariant is what makes the callback work at all.
Live on a 4 GB GPU for small models: a 4-epoch SmolLM2-135M-Instruct run completes with no callback crash (loss 0.66, token accuracy 0.80).
See also
- Spectrum targeted training — the static, SNR-ranked counterpart.
- PEFT & efficiency — LoRA, PiSSA, ReLoRA, per-pattern rank.
- Adapter algebra — the other half of the v0.71.34 release.