An ML pipeline is a directed acyclic graph (DAG) of steps: data ingestion → preprocessing → training → evaluation → deployment. Each step is versioned, cacheable, and independently retriable. Kubeflow, Airflow, Vertex, and SageMaker Pipelines are the main orchestrators.
| Platform | Strengths | Best for |
|---|---|---|
| Kubeflow Pipelines | K8s-native, portable | Cloud-agnostic teams |
| Apache Airflow | Mature, huge ecosystem | Data engineering + ML |
| Vertex AI Pipelines | Managed, GCP-integrated | Google Cloud shops |
| SageMaker Pipelines | Managed, AWS-integrated | AWS shops |
Seven steps that take 185 minutes one after another. Run in parallel where dependencies allow, the pipeline still takes 170 — features and labels overlap, but everything waits for training. Reuse cached ingestion, validation and features and a retrain takes 120. The critical path, not the number of workers, sets the time.
- Caching is only safe when the cache key includes the code and data versions; otherwise stale outputs are reused silently.
- Pipelines that retrain automatically can also degrade automatically; keep an evaluation gate before deploy.
- A pipeline nobody can run locally is hard to debug; keep steps runnable on their own.
minutes = {'ingest': 20, 'validate': 5, 'features': 40, 'labels': 15,
'train': 90, 'evaluate': 10, 'deploy': 5}
needs = {'validate': ['ingest'], 'features': ['validate'], 'labels': ['validate'],
'train': ['features', 'labels'], 'evaluate': ['train'], 'deploy': ['evaluate']}
def finish(step, cache=()): # earliest finish, with unlimited parallel workers
start = max((finish(d, cache) for d in needs.get(step, [])), default=0)
return start + (0 if step in cache else minutes[step])
total = finish('deploy')
cached = finish('deploy', cache={'ingest', 'validate', 'features'}) # retrain on cached features
serial = sum(minutes.values())Assumes import numpy as np and import pandas as pd. Each snippet carries its own example numbers; the comments say which are made up or simulated.
- D. Baylor et al., “TFX: A TensorFlow-Based Production-Scale Machine Learning Platform”, KDD, 2017
- D. Sculley et al., “Hidden Technical Debt in Machine Learning Systems”, Advances in Neural Information Processing Systems 28, 2015
- Designing Machine Learning Systems, C. Huyen, O’Reilly, 2022
Last reviewed October 2026 · How this site checks its content · Report a mistake