SFT · Instruction Tuning✓ Mathematical
1 min read
◆ The PatternTurning a base model into a helpful assistant
A pre-trained model predicts next tokens but doesn't follow instructions. Supervised Fine-Tuning (SFT) trains on curated (instruction, response) pairs — teaching the model to converse, follow directions, and output structured answers.
L_SFT = −Σ log P(response_t | instruction, response_{<t})
Only compute loss on the response tokens — the instruction tokens are context, not targets.
Dataset: ~10K–100K high-quality (instruction, response) pairs
Quality >> quantity. Careful curation matters more than scale for SFT.
Catastrophic forgetting is the main risk: fine-tuning too aggressively overwrites pre-training knowledge. Mitigations: low learning rate (1e-5 to 5e-5), few epochs (1–3), mixing pre-training data.
The gap between a base model (random-feeling completions) and a fine-tuned model (helpful assistant) is dramatic — SFT is what makes "chat" models work.
Interactive — fine-tuning effect on output distribution
Python — SFT with masking#
# SFT training: only compute loss on response tokens
def sft_loss(model, input_ids, response_start_idx):
logits = model(input_ids[:, :-1])
# Create labels: -100 for instruction tokens (ignored by loss)
labels = input_ids[:, 1:].clone()
labels[:, :response_start_idx] = -100
loss = CrossEntropyLoss(ignore_index=-100)(
logits.reshape(-1, vocab_size),
labels.reshape(-1)
)
return loss
# Typical hyperparameters
# lr: 2e-5, epochs: 2-3, batch: 128, warmup: 3%Pattern bridge: Adapting a pre-trained model to a specific task. In markets, adapting a general moving average strategy to a specific asset class.
Performance in practice
- LoRA reduces trainable parameters by 99%+ — fine-tune a 7B model on a single GPU (16GB) in hours. Full fine-tuning of 7B needs 4× A100 80GB
- OpenAI's fine-tuning API: ~$8/million tokens for GPT-4o-mini. Cost-effective for domain-specific tasks, but you lose control over the base model
- 10K high-quality examples often outperform 1M noisy examples. Alpaca (52K examples) made LLaMA competitive with ChatGPT on many tasks
- For production: fine-tune on your domain, then eval on held-out examples. If perplexity improves but task metrics don't, your data quality is the bottleneck
When to use this
✓ Use when: You need a model to follow specific output formats. Domain-specific knowledge that the base model doesn't have. Consistent style/tone requirements. Reducing prompt length (teach the model once instead of repeating instructions).
✗ Skip when: Prompt engineering with examples (few-shot) already works well enough. Your data is small (<100 examples) — few-shot is better. You need to switch between many tasks dynamically — keep the generalist model.
Quick start — LoRA fine-tuning
pip install transformers peft datasets accelerate bitsandbytes
# ────────────────────────────────────────
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
from peft import LoraConfig, get_peft_model
from trl import SFTTrainer
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3-8B", load_in_4bit=True)
lora = LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj","v_proj"])
model = get_peft_model(model, lora) # ~0.5% trainable params
trainer = SFTTrainer(model, train_dataset=dataset,
args=TrainingArguments(num_train_epochs=2, learning_rate=2e-4, per_device_train_batch_size=4))
trainer.train()