Fine-tune Llama 3.1 with LoRA using Soup CLI

This guide shows how to fine-tune Meta Llama 3.1 8B with LoRA adapters on a custom dataset using Soup CLI. End-to-end from install to inference on a single GPU; how long the training step takes depends on your dataset size, sequence length and card, and no timing is promised here.

Why LoRA on Llama 3.1?

LoRA (Low-Rank Adaptation) trains only a small fraction (~0.1%) of model parameters, which means:

• Train 8B parameter Llama 3.1 on a single 24GB GPU (RTX 4090, A10)
• Checkpoints are tiny (~100MB instead of 16GB)
• Faster training and easier experimentation

1. Install

pip install "soup-cli[train,fast]"

Since v0.71.0 the base soup-cli package is a light, PyTorch-free CLI. [train] adds the training stack; [fast] adds Unsloth on top. Installing [fast] alone cannot train.

The [fast] extra adds Unsloth for 2–5× training speedup on Llama-family models.

2. Prepare your dataset

Use the Alpaca format (JSON list of instruction/input/output triples):

[
  {
    "instruction": "Summarize the following text.",
    "input": "Soup CLI is a fine-tuning toolkit...",
    "output": "Soup CLI is an open-source LLM fine-tuning tool."
  }
]

Save as train.json. One row only shows the shape of the file. Soup holds out 10% of the rows for validation by default (data.val_split), and a one-row file leaves nothing to train on: since v0.76.0 soup train --dry-run on it exits 1 with No training rows: train.json loaded 0 train samples as format 'alpaca'. Use your real dataset.

3. Create the config

Save as llama31.yaml:

base: meta-llama/Llama-3.1-8B-Instruct
task: sft
backend: unsloth            # root-level, NOT under training

data:
  train: train.json
  format: alpaca
  max_length: 2048          # sequence length lives under data

training:
  epochs: 3
  lr: 2.0e-4                # the key is lr, not learning_rate
  batch_size: 2
  gradient_accumulation_steps: 8
  lora:
    r: 16                   # LoRA turns on when r > 0; there is no enabled flag
    alpha: 32
    dropout: 0.05
    target_modules: [q_proj, k_proj, v_proj, o_proj]

output: ./runs/llama31

Spelling matters, and until v0.74.0 nothing told you so. Writing learning_rate instead of lr, or nesting backend under training, used to be dropped in silence and the default used instead. Since v0.74.0 loading a config walks the whole tree and reports every key it cannot place, and since v0.75.0 the config is refused: soup train exits 1 before the training stack is imported and prints unknown config key 'training.learning_rate' - refused. When the key is close to a real one the message adds a suggestion (bacth_size gets did you mean 'batch_size'?); learning_rate is not close to lr, so you get the refusal alone. Check the spelling against Configuration.

Before you train

Run soup doctor --config llama31.yaml first. Since v0.76.0 many more settings are refused when the config loads, each naming the field and what to change, so a config that loaded on v0.75 can stop loading. soup doctor --config prints the error and exits 2 without loading a model (it stops at the first cross-field refusal, so fix that setting and run it again until it prints no error); a setting that nothing reads yet, or a deprecated value, prints a warning and the config still loads. See Config refusals.

4. Train

soup train --config llama31.yaml

Soup detects your GPU and trains the LoRA adapters through Unsloth (backend: unsloth). FlashAttention is not switched on for you: training.use_flash_attn defaults to false. How long it takes depends on your sequence length, batch size and hardware; soup profile --config llama31.yaml estimates memory and speed before you start.

5. Chat with your fine-tuned model

soup chat --model ./runs/llama31

6. Export for deployment

# Merge LoRA into base model and export GGUF for Ollama
soup export --model ./runs/llama31 --format gguf --quant q4_k_m

Soup writes the training output (the adapter and tokenizer) straight into the output directory, so ./runs/llama31 above is the config's output: value, and there is no latest subdirectory. Without --output, the GGUF is written next to the model directory as ./runs/llama31.q4_k_m.gguf.

Troubleshooting

Out of memory? Enable QLoRA (a 4-bit base model). Note 4bit is already the default, so this is only worth setting explicitly if you changed it:

training:
  quantization: 4bit
  lora:
    r: 16

Slow training? Ensure backend: unsloth is set at the root of the config (not under training) and that you installed pip install "soup-cli[train,fast]".

Next steps

• Export to GGUF and Ollama
• Multi-GPU training with DeepSpeed
• Training methods reference