10 — Monitor & Observe

Data Quality Gates#

DataPractice 1 min read
◆ The PatternAutomated checks that stop bad data before it reaches your model

Garbage in, garbage out — but in production, garbage arrives silently. Data quality gates enforce schema validation, range checks, freshness constraints, and completeness thresholds. They sit in your pipeline before feature engineering and before inference.

// Interactive — data quality pipeline flow
Pattern bridge: Data quality gates are the production equivalent of missing data strategies — catching the problem before it corrupts your analysis. In markets, data hygiene (adjusting for splits, dividends, survivorship) serves the same protective role.
Worked example

Six rows checked against five rules before they reach the model. Row count and freshness pass. Three fail: 17% of ages are missing against a 5% limit, one age is 230, and a third of incomes are missing. A gate that stops this batch costs a delayed run; letting it through costs silently wrong predictions.

Where it misleads
  • Rules written once go stale; review thresholds when the data legitimately changes.
  • Hard failures on every rule stop pipelines for trivia; separate blocking checks from warnings.
  • Valid-looking data can still be wrong (a unit change from euros to cents passes a range check); compare distributions too.
In code
batch = pd.DataFrame({'age': [34, 51, None, 29, 230, 45],
                      'income': [52_000, 61_000, 48_000, None, None, 75_000],
                      'ts': pd.to_datetime(['2026-09-30 22:00'] * 6)})
now, expected_rows = pd.Timestamp('2026-10-01 09:00'), 6
checks = {
    'row count': len(batch) >= 0.5 * expected_rows,
    'age nulls <= 5%': batch.age.isna().mean() <= 0.05,
    'age in 0-120': batch.age.dropna().between(0, 120).all(),
    'income nulls <= 5%': batch.income.isna().mean() <= 0.05,
    'fresh within 24 h': (now - batch.ts.max()) <= pd.Timedelta('24h'),
}
failed = [name for name, ok in checks.items() if not ok]
Python, in your browser — edit the code and run it again

Assumes import numpy as np and import pandas as pd. Each snippet carries its own example numbers; the comments say which are made up or simulated.

The same idea with a library
# Python — Great Expectations data quality check
import great_expectations as gx

context = gx.get_context()
ds = context.sources.add_pandas("prod_data")
asset = ds.add_dataframe_asset("batch", dataframe=df)

expectations = [
    gx.expectations.ExpectColumnValuesToNotBeNull(column="user_id"),
    gx.expectations.ExpectColumnValuesToBeBetween(
        column="price", min_value=0, max_value=100000),
]
Sources
  • E. Breck, N. Polyzotis, S. Roy, S. E. Whang & M. Zinkevich, “Data Validation for Machine Learning”, Proceedings of Machine Learning and Systems 1, 2019
  • S. Schelter et al., “Automating Large-Scale Data Quality Verification”, Proceedings of the VLDB Endowment 11(12), 2018
  • D. Sculley et al., “Hidden Technical Debt in Machine Learning Systems”, Advances in Neural Information Processing Systems 28, 2015

Last reviewed October 2026 · How this site checks its content · Report a mistake

Check yourself

1. Where should data quality gates sit?

Show answer

C. A gate is useful only if a failure stops something. Checks that just log warnings get ignored.

2. A column’s null rate jumps from 1% to 30% overnight. What should the pipeline do?

Show answer

C. Silent imputation turns a broken feed into quietly wrong predictions.

3. Schema checks pass, but prices arrive in cents instead of euros. What would have caught it?

Show answer

C. The type is still a number. Expectations on ranges, means or quantiles catch unit changes.

Linked from 3
← Previous
Shadow Mode & Champion/Challenger
Open in the full reader, with the topic sidebar