LLM Engineering

86% accuracy on a 1,000-question benchmark has a 95% margin of about ±2.15 points; on MMLU’s 14,042 test questions, ±0.57. Two models scoring 86.0 and 87.5 on the 1,000 questions differ by 0.97 standard errors — not distinguishable. Leaderboard gaps of a point or two are often within the noise.

• Both models answer the same questions, so a paired test is more sensitive than this independent one; report it (Miller 2024).
• Contamination — test items in the training data — inflates scores in ways no error bar captures.
• Benchmarks measure what is easy to score; arena-style human preference captures other things, and has its own biases.

def ci95(acc, n):
    return 1.96 * np.sqrt(acc * (1 - acc) / n)

small = ci95(0.86, 1_000)                       # a 1,000-question benchmark
mmlu = ci95(0.86, 14_042)                       # MMLU's test set
gap = 0.875 - 0.86
se_diff = np.sqrt(2) * small / 1.96             # two independent models on the 1,000 questions
z = gap / se_diff

Assumes import numpy as np. Configurations named after real models use their published shapes; other numbers are made up or simulated, as the comments say.

# Using EleutherAI lm-evaluation-harness
# pip install lm-eval

# Run MMLU benchmark
# lm_eval --model hf --model_args pretrained=meta-llama/Llama-3.1-8B \
#   --tasks mmlu --batch_size 8 --output_path results/

# Programmatic usage
from lm_eval import evaluator
results = evaluator.simple_evaluate(
    model="hf",
    model_args="pretrained=meta-llama/Llama-3.1-8B",
    tasks=["mmlu", "hellaswag", "arc_challenge", "gsm8k"],
    batch_size=8,
)
for task, metrics in results["results"].items():
    print(f"{task}: {metrics['acc,none']:.3f}")Copy