Quality · Dedup · Mixing✓ Mathematical
◆ The PatternThe most impactful and least glamorous part of LLM training
Data quality determines model quality. The pipeline: crawl → filter → deduplicate → classify → mix. Common Crawl provides ~250B pages, but only a small fraction is high-quality. Aggressive filtering and deduplication are essential.
Quality filter pipeline: URL → language ID → perplexity → toxicity → heuristic rules
Each stage drops data. LLaMA 3 starts with 15T+ raw tokens and uses a classifier trained on quality signals.
Dedup: MinHash + LSH for fuzzy, exact-match for verbatim
Duplicates hurt training: models memorize repeated passages, wasting capacity. 30–50% of web crawl is near-duplicate.
Data mixing is critical: model capabilities depend on training data composition. Typical mix: ~50% web, ~25% code, ~10% academic, ~5% books, ~5% math, ~5% conversation. Overloading on code improves reasoning.
Benchmark contamination is a real problem — if test questions appear in training data, benchmarks are meaningless. Modern data pipelines include decontamination stages that remove known benchmarks.
Interactive — data filtering funnel
Python — MinHash deduplication#
from datasketch import MinHash, MinHashLSH
def minhash_doc(text, num_perm=128):
m = MinHash(num_perm=num_perm)
for word in text.lower().split():
m.update(word.encode('utf8'))
return m
# Build LSH index for near-duplicate detection
lsh = MinHashLSH(threshold=0.8, num_perm=128)
for doc_id, text in documents:
mh = minhash_doc(text)
if not lsh.query(mh): # not similar to existing
lsh.insert(doc_id, mh)
else:
print(f"Dropping duplicate: {doc_id}")Pattern bridge: Quality over quantity in training data — deduplication, filtering, mixing. In statistics, sampling methodology determines everything.