What does a modern pre-training data pipeline look like?

Pre-training looks less like “download the internet and train” and more like an industrial process: many stages that progressively turn noisy web-scale text into a shaped mixture of tokens that matches a training plan. Modern pipelines spend serious effort on deduplication, filtering, and curriculum-style mixing because model quality depends on what you feed it, not just how much you feed it.

The modern pipeline in one view

A typical pre-training pipeline can be summarized as: collect → normalize → deduplicate → filter → label/score → mix → pack → audit → iterate. Each stage tries to improve the signal-to-noise ratio and reduce waste so compute goes to useful tokens.

1) Collection and ingestion

Most pipelines start with a large pool of sources: web crawls, books, code, forums, academic text, and licensed datasets. The first ingestion pass focuses on:

• Parsing and extraction (HTML to text, boilerplate removal, code extraction)
• Language identification (often at document and paragraph level)
• Basic normalization (Unicode cleanup, whitespace normalization, sentence boundary heuristics)
• Metadata capture (source, timestamp, domain, content type, length)

Metadata becomes important later for stratified sampling, audits, and curriculum.

2) Deduplication: removing waste and reducing memorization risk

Dedup is no longer optional at scale. Without it, token budgets get spent repeating near-identical pages, mirrors, spam, and syndicated content. Dedup also reduces the chance a model memorizes and regurgitates large chunks.

Common strategies include:

• Exact dedup: hash whole documents or chunks; drop identical copies.
• Near-duplicate dedup: use shingling (n-grams) plus similarity search (MinHash/LSH) to remove pages that are mostly the same but not byte-identical.
• Chunk-level dedup: dedup within documents (repeated headers/footers) and across documents, often after splitting into fixed-size text blocks.
• Cross-source dedup: dedup the combined pool, not just per dataset, because the same content appears in multiple corpora.

A key design choice is how aggressively to dedup. Too weak and you waste compute; too strong and you may delete legitimate repetitions (definitions, common phrases, code templates). Many pipelines treat dedup as “drop near-identical blocks above a threshold” rather than “make everything unique.”

3) Filtering: raising average quality

Filtering aims to remove low-value tokens: spam, boilerplate, machine-translated sludge, SEO pages, scraped menus, and garbled text. Modern filtering typically layers multiple signals:

• Heuristic filters
  • Minimum/maximum document length
  • Ratio of alphabetic to non-alphabetic characters
  • Excessive repetition or keyword stuffing
  • Bad HTML residue, corrupted encodings
• Language and script checks
  • Mixed-language noise detection
  • Script mismatches (e.g., Latin text in a “CJK” bucket)
• Model-based quality scoring
  • A small classifier trained to separate “high-quality prose/code” from junk
  • Perplexity-based filtering using a reference language model (too high can indicate garbage; too low can indicate boilerplate or duplicated content)
• Safety and policy filters
  • Remove clearly disallowed categories for the intended model
  • Redact certain patterns (personal data, credential dumps) when feasible

Filtering is often iterative: run a first pass, train a small model or classifier on the result, use that model to score the next pass, then repeat.

4) Labeling and routing: turning a pile into a mixture

After dedup and filtering, many pipelines label data so it can be mixed intentionally:

• Domain/topic tags (math, legal, medical, fiction, chat, news)
• Format tags (tables, code, Q&A, dialogue, markdown)
• Difficulty proxies (reading level, symbol density, problem/solution structure)
• Quality scores (a scalar used for sampling weights)

This routing enables targeted sampling: for example, keeping enough conversational text for instruction-tuning later, or maintaining a stable fraction of code to preserve coding ability.

Curriculum and mixing: shaping what the model sees over time

Curriculum in pre-training rarely means a strict “easy-to-hard” sequence. It more often means dynamic mixing: controlling proportions of data types across training phases.

Common patterns:

• Quality-first warm start: early training uses a higher share of clean, high-quality text (books, curated sources) to stabilize grammar, facts, and style.
• Broadening phase: later phases increase diversity (web, niche domains) to expand coverage and reduce overfitting to polished prose.
• Skill-focused ramps: adjust fractions for code, math, and reasoning-heavy text depending on target capabilities.
• Anti-forgetting refresh: periodically re-inject high-value domains so late-stage training on noisy data doesn’t erode earlier gains.

Mixing is usually done through weighted sampling, sometimes with temperature adjustments (flattening or sharpening distribution over sources) and caps to prevent any single domain from dominating.

Packing and training-time efficiency

Once the dataset is decided, pipelines pack it for training:

• Tokenization-aware chunking to minimize padding waste
• Document boundary strategies (keep docs intact vs. pack multiple docs per sequence)
• Shuffling with constraints (avoid long runs of one domain)
• Decontamination splits (hold out evaluation sets and remove overlaps)

Packing can matter more than it sounds: wasted padding is wasted compute, and poor shuffling can produce training instabilities or unintended curricula.

Audits and iteration: the pipeline is a loop

Modern teams treat dataset building as an ongoing process:

• Sample audits for spam and pathological content
• Domain balance checks
• Near-duplicate rate monitoring
• Memorization probes on known duplicated passages
• Regression tests: “after a filter change, do we lose math ability or code formatting?”

Small changes in filtering thresholds or source weights can shift model behavior, so pipelines often keep dataset versions and run controlled ablations.

Sensitivity: data quality vs sheer token count

LLMs benefit from more tokens, but the relationship is not “more is always better.” A useful mental model is: token count buys coverage; data quality buys efficiency and behavior.

When token count dominates

More tokens help when:

• The model is under-trained relative to its size (too few tokens per parameter).
• The dataset is already reasonably clean and diverse.
• You’re chasing long-tail knowledge and rare patterns.

In these regimes, scaling tokens tends to improve perplexity and downstream performance fairly reliably.

When data quality dominates

Quality matters disproportionately when:

• The dataset contains lots of repetition, spam, or templated pages (dedup and filtering can yield “free” gains).
• You care about instruction-following style, factuality, and reasoning traces (format and source choice matters).
• You’re near compute limits (better data gives more performance per token).
• You’re trying to reduce toxic outputs or unstable behavior (bad data can imprint unwanted patterns).

Low-quality tokens can be worse than neutral: they can teach the model to produce boilerplate, evasive spammy phrasing, or incorrect “confident” answers. They also burn budget that could have gone to better examples.

The practical takeaway

Most modern results come from a combination: use a very large token pool, then aggressively remove waste and shape the mixture. Dedup can reduce training compute without hurting quality; filtering can raise capability at the same token count; curriculum-style mixing can shift strengths (code, math, dialogue) without changing model size. In practice, teams chase a “clean enough, big enough” dataset—then spend the rest of the time tuning what the model sees, when it sees it, and how often.