MLOps & Production ML

A task that fails transiently one time in five succeeds within three retries with probability 99.84%. Spacing the retries at 30, 60 and 120 seconds lets a struggling dependency recover instead of being hammered. And when a bug is fixed, a backfill re-runs the daily DAG for each of the last 90 days.

• Retries only help transient failures; a bug fails four times instead of once.
• Retries without jitter synchronize clients into waves that can take a service down again.
• Backfills of non-idempotent tasks double-write; make each run safe to repeat.

p_fail = 0.20                          # a task fails transiently one time in five
retries = 3
p_success = 1 - p_fail ** (retries + 1)
backoff = [30 * 2 ** k for k in range(retries)]           # seconds between attempts: exponential
backfill_runs = 90                                         # a daily DAG re-run for the last 90 days

Assumes import numpy as np and import pandas as pd. Each snippet carries its own example numbers; the comments say which are made up or simulated.

# Python — Airflow DAG for retraining
from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

dag = DAG("retrain_model",
          schedule_interval="0 2 * * *",
          start_date=datetime(2025, 1, 1))

ingest = PythonOperator(task_id="ingest", python_callable=ingest_data, dag=dag)
train  = PythonOperator(task_id="train",  python_callable=train_model,  dag=dag)
eval_  = PythonOperator(task_id="eval",   python_callable=evaluate,     dag=dag)
deploy = PythonOperator(task_id="deploy", python_callable=deploy_model, dag=dag)

ingest >> train >> eval_ >> deploy