18 — Training

Data Curation#

Quality · Dedup · MixingStatistical 1 min read
◆ The PatternThe most impactful and least glamorous part of LLM training

Data quality determines model quality. The pipeline: crawl → filter → deduplicate → classify → mix. Common Crawl’s archive spans more than 250 billion pages collected since 2008, but only a small fraction is high-quality. Aggressive filtering and deduplication are essential.

Quality filter pipeline: URL → language ID → perplexity → toxicity → heuristic rules
Each stage drops data. Llama 3 was trained on over 15T tokens selected with heuristic filters, deduplication and model-based quality classifiers.
Dedup: MinHash + LSH for fuzzy, exact-match for verbatim
Duplicates hurt training: models memorize repeated passages, wasting capacity. A large share of web text is near-duplicate (Lee et al. 2022).

Data mixing is critical: model capabilities depend on training data composition. Meta reported Llama 3’s mix as roughly 50% general knowledge, 25% maths and reasoning, 17% code and 8% multilingual text. More code has been linked to better reasoning, though the evidence is mixed.

Benchmark contamination is a real problem — if test questions appear in training data, benchmarks are meaningless. Modern data pipelines include decontamination stages that remove known benchmarks.
Interactive — data filtering funnel

Python — MinHash deduplication#

Pattern bridge: Quality over quantity in training data — deduplication, filtering, mixing. In statistics, sampling methodology determines everything.
Worked example

Two sentences that differ in two words share 0.817 of their five-character shingles (Jaccard). MinHash estimates this from 256 hash minima as 0.785, without comparing the texts directly — which is how near-duplicates are found among billions of web pages. Removing them makes models better and less prone to memorising (Lee et al. 2022).

Where it misleads
  • A threshold that removes boilerplate can also remove legitimately similar documents (laws, templates, code).
  • Quality filters trained on “good” reference text encode its tastes and can under-represent dialects and topics.
  • Deduplication against benchmarks (decontamination) needs fuzzy matching; exact matching misses paraphrased test items.
In code
import zlib
def shingles(text, k=5):
    t = text.lower()
    return {t[i:i + k] for i in range(len(t) - k + 1)}

a = 'The quick brown fox jumps over the lazy dog near the river bank today.'
b = 'The quick brown fox jumped over the lazy dog near the river bank today!'
A, B = shingles(a), shingles(b)
jaccard = len(A & B) / len(A | B)

rng = np.random.default_rng(0)
coef = rng.integers(1, 2**31 - 1, size=(256, 2))       # 256 random hash functions
h = lambda s: np.array([zlib.crc32(x.encode()) for x in s], dtype=np.int64)
minhash = lambda s: ((coef[:, :1] * h(s) + coef[:, 1:]) % (2**31 - 1)).min(axis=1)
estimate = (minhash(A) == minhash(B)).mean()            # the share of equal minima estimates Jaccard
Python, in your browser — edit the code and run it again

Assumes import numpy as np. Configurations named after real models use their published shapes; other numbers are made up or simulated, as the comments say.

The same idea with a library
from datasketch import MinHash, MinHashLSH

def minhash_doc(text, num_perm=128):
    m = MinHash(num_perm=num_perm)
    for word in text.lower().split():
        m.update(word.encode('utf8'))
    return m

# Build LSH index for near-duplicate detection
lsh = MinHashLSH(threshold=0.8, num_perm=128)
for doc_id, text in documents:
    mh = minhash_doc(text)
    if not lsh.query(mh):  # not similar to existing
        lsh.insert(doc_id, mh)
    else:
        print(f"Dropping duplicate: {doc_id}")
Sources
  • A. Z. Broder, “On the Resemblance and Containment of Documents”, Compression and Complexity of Sequences, 1997
  • K. Lee et al., “Deduplicating Training Data Makes Language Models Better”, ACL, 2022
  • G. Penedo et al., “The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale”, NeurIPS Datasets and Benchmarks, 2024

Last reviewed October 2026 · How this site checks its content · Report a mistake

Linked from 1
← Previous
DPO
Open in the full reader, with the topic sidebar