SFT · Instruction TuningPractice
1 min read
◆ The PatternTurning a base model into a helpful assistant
A pre-trained model predicts next tokens but doesn't follow instructions. Supervised Fine-Tuning (SFT) trains on curated (instruction, response) pairs — teaching the model to converse, follow directions, and output structured answers.
L_SFT = −Σ log P(response_t | instruction, response_{<t})
Only compute loss on the response tokens — the instruction tokens are context, not targets.
Dataset: ~10K–100K high-quality (instruction, response) pairs
Quality >> quantity. Careful curation matters more than scale for SFT.
Catastrophic forgetting is the main risk: fine-tuning too aggressively overwrites pre-training knowledge. Mitigations: low learning rate (1e-5 to 5e-5), few epochs (1–3), mixing pre-training data.
The gap between a base model (random-feeling completions) and a fine-tuned model (helpful assistant) is dramatic — SFT is what makes "chat" models work.
Interactive — fine-tuning effect on output distribution
Python — SFT with masking#
Pattern bridge: Adapting a pre-trained model to a specific task. In markets, adapting a general moving average strategy to a specific asset class.
Performance in practice
- LoRA trains well under 1% of the parameters. Full fine-tuning of a 7B model with Adam needs about 112 GB for weights, gradients and optimizer state alone (16 bytes per parameter); QLoRA fits it on a single GPU
- Hosted fine-tuning APIs are cheap per token but tie you to the provider’s base model, pricing and deprecation schedule
- Quality beats quantity in instruction data: LIMA reached strong results with 1,000 curated examples (Zhou et al. 2023)
- For production: fine-tune on your domain, then evaluate on held-out examples. If perplexity improves but task metrics don’t, data quality is the bottleneck
When to use this
✓ Use when: You need a model to follow specific output formats. Domain-specific knowledge that the base model doesn't have. Consistent style/tone requirements. Reducing prompt length (teach the model once instead of repeating instructions).
✗ Skip when: Prompt engineering with examples (few-shot) already works well enough. Your data is small (<100 examples) — few-shot is better. You need to switch between many tasks dynamically — keep the generalist model.
Quick start — LoRA fine-tuning
pip install transformers peft datasets accelerate bitsandbytes
# ────────────────────────────────────────
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
from peft import LoraConfig, get_peft_model
from trl import SFTTrainer
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3-8B", load_in_4bit=True)
lora = LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj","v_proj"])
model = get_peft_model(model, lora) # ~0.5% trainable params
trainer = SFTTrainer(model, train_dataset=dataset,
args=TrainingArguments(num_train_epochs=2, learning_rate=2e-4, per_device_train_batch_size=4))
trainer.train()Worked example
50,000 instruction examples of about 500 tokens are 25M tokens — 0.0002% of a 15T-token pre-training run. LIMA got a strong assistant from just 1,000 carefully chosen examples. Fine-tuning mostly teaches format and behaviour; the knowledge comes from pre-training.
Where it misleads
- Fine-tuning on facts the model does not already know can increase hallucination rather than teach the facts.
- Aggressive fine-tuning erodes general abilities (catastrophic forgetting); low learning rates and mixing in general data help.
- Small, curated datasets outperform large, noisy ones; quality matters more than count.
In code
sft_tokens = 50_000 * 500 # 50k examples of ~500 tokens pretrain_tokens = 15e12 share = sft_tokens / pretrain_tokens lima_examples = 1_000 # the LIMA paper's whole SFT set
Assumes import numpy as np. Configurations named after real models use their published shapes; other numbers are made up or simulated, as the comments say.
The same idea with a library
# SFT training: only compute loss on response tokens
def sft_loss(model, input_ids, response_start_idx):
logits = model(input_ids[:, :-1])
# Create labels: -100 for instruction tokens (ignored by loss)
labels = input_ids[:, 1:].clone()
labels[:, :response_start_idx] = -100
loss = CrossEntropyLoss(ignore_index=-100)(
logits.reshape(-1, vocab_size),
labels.reshape(-1)
)
return loss
# Typical hyperparameters
# lr: 2e-5, epochs: 2-3, batch: 128, warmup: 3%Sources
- L. Ouyang et al., “Training Language Models to Follow Instructions with Human Feedback”, NeurIPS, 2022
- C. Zhou et al., “LIMA: Less Is More for Alignment”, NeurIPS, 2023
- Y. Luo et al., “An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning”, arXiv:2308.08747, 2023
Last reviewed October 2026 · How this site checks its content · Report a mistake
Linked from 1