LoRA Quality — PiSSA, ReLoRA, Per-Pattern Rank, Surgical Patches (v0.39.0)

Five PEFT-surface improvements that LlamaFactory and Axolotl maintain.

yaml
training:
  lora:
    init_strategy: pissa          # 'random' (default), 'pissa', 'olora'
    rank_pattern:                 # per-target-module rank override
      q_proj: 8
      v_proj: 16
    alpha_pattern:                # per-target-module alpha override
      q_proj: 16
  relora_steps: 500               # magnitude-prune LoRA every 500 steps
  relora_warmup_ratio: 0.1        # skip first 10% of training
  relora_prune_ratio: 0.9         # zero out smallest 90% by magnitude
  relora_reset_optimizer: true    # clear optimizer state on each fire

PiSSA

PiSSA initializes the LoRA pair from the SVD of the base weight, giving faster early convergence than random init at the cost of one extra SVD pass on the first epoch.

init_strategy: olora is also accepted; setting the legacy use_olora: true auto-aligns for back-compat.

ReLoRA

ReLoRA fires every N global steps, magnitude-prunes the LoRA adapter weights (keeping the top 1 - relora_prune_ratio by absolute value), and optionally clears optimizer state for the pruned parameters so momentum doesn't fight the new sparse weights.

Useful for very long training runs where the LoRA capacity saturates.

Per-pattern rank/alpha

Map module name patterns to integer ranks. Useful in MoE configs where expert FFNs need lower rank than attention.

Caps: 256 keys × value 1024. bool rejected on values. Incompatible with use_vera (VeRA shares one rank).

Surgical patches

  • Gemma 4 ClippableLinear swap (auto-detected via word-boundary regex on model name)
  • Fused-MoE 3-D expert lora_dropout strip

Both auto-fire when the model name and architecture match. Both are gated and silent on unrelated models.

Template registry

The 17 built-in templates now live as soup_cli/templates/*.yaml with a manifest.json index. soup init --template <name> reads the YAML; the inline copies in schema.py stay as a back-compat fallback, deprecated in favour of the YAML registry.

load_template containment-checks the resolved path against _templates_dir() so a tampered manifest.json cannot read files outside the package directory.

Scope

Wired into the SFT trainer + transformers backend in v0.39.0. Multi-trainer expansion of ReLoRA (DPO/GRPO/KTO/...) is tracked for v0.39.1. MLX backend gets a distinct error message.