Perplexity · MMLU · Arena✓ Mathematical
◆ The PatternMeasuring what matters — and what doesn't
How do you know if an LLM is good? Perplexity measures language modeling quality. Benchmarks test specific skills. Human evaluation and arena rankings capture overall helpfulness. Each has blind spots.
Perplexity = e^(−1/T · Σ log P(token_t))
Average surprisal. Lower = better at predicting text. Only measures language modeling, not task ability.
MMLU: 57 subjects, multiple choice (humanities, STEM, social sciences)
Knowledge breadth. GPT-4: ~86%, LLaMA-3-70B: ~82%. But multiple-choice ≠ open-ended ability.
Key benchmarks: MMLU (knowledge), HumanEval (code generation), GSM8K (math), HellaSwag (common sense), ARC (reasoning), TruthfulQA (hallucination). Chatbot Arena uses live ELO rankings from anonymous human votes — currently the most trusted evaluation.
Benchmark contamination is rampant — if test questions leak into training data, scores are meaningless. Private held-out test sets and Arena rankings are more reliable than public benchmark scores.
Interactive — benchmark comparison radar chart
Python — evaluation with lm-harness#
# Using EleutherAI lm-evaluation-harness
# pip install lm-eval
# Run MMLU benchmark
# lm_eval --model hf --model_args pretrained=meta-llama/Llama-3.1-8B \
# --tasks mmlu --batch_size 8 --output_path results/
# Programmatic usage
from lm_eval import evaluator
results = evaluator.simple_evaluate(
model="hf",
model_args="pretrained=meta-llama/Llama-3.1-8B",
tasks=["mmlu", "hellaswag", "arc_challenge", "gsm8k"],
batch_size=8,
)
for task, metrics in results["results"].items():
print(f"{task}: {metrics['acc,none']:.3f}")Pattern bridge: Benchmarking models with metrics and human evaluation. In ML, precision/recall/F1 are the toolkit. In statistics, hypothesis testing evaluates claims.