Lineage tracking records the full provenance chain: which data → which features → which code → which model → which prediction. When a model misbehaves, you can trace backwards to find the root cause. Regulatory compliance (GDPR, finance) often requires this.
One raw data partition turns out to be corrupt. Following the lineage graph downstream finds 6 things built from it: the feature set, two models, two prediction tables and a retention dashboard. Without lineage the question “what did this break?” is answered by memory.
- Lineage captured by hand goes out of date; it has to be recorded automatically by the tools that run the jobs.
- Coarse lineage (whole tables) over-reports the impact; row- or partition-level lineage is costlier but precise.
- Lineage shows what depends on what, not whether the result changed.
edges = {'raw/2026-09-28': ['features/v12'], 'raw/2026-09-29': ['features/v12'],
'features/v12': ['model/fraud-v9', 'model/churn-v4'],
'model/fraud-v9': ['preds/fraud-0930'], 'model/churn-v4': ['preds/churn-0930', 'dashboard/retention']}
def downstream(node): # everything built from a node
seen, stack = set(), [node]
while stack:
for nxt in edges.get(stack.pop(), []):
if nxt not in seen: seen.add(nxt); stack.append(nxt)
return sorted(seen)
affected = downstream('raw/2026-09-29') # this partition turned out to be corruptAssumes import numpy as np and import pandas as pd. Each snippet carries its own example numbers; the comments say which are made up or simulated.
- OpenLineage specification, openlineage.io
- M. Vartak et al., “ModelDB: A System for Machine Learning Model Management”, HILDA Workshop at SIGMOD, 2016
- D. Sculley et al., “Hidden Technical Debt in Machine Learning Systems”, Advances in Neural Information Processing Systems 28, 2015
Last reviewed October 2026 · How this site checks its content · Report a mistake