MLOps & Production ML

A million customers could be scored; fifty thousand are looked up each day. A nightly batch scores all 1,000,000, and the average answer is 12 hours old when read. Online serving scores only the 50,000 that are asked for, with fresh inputs. Here 95% of the batch work is never read — but batch needs no low-latency service, and online does.

• The choice is about freshness and cost, not fashion: a credit limit that changes monthly gains nothing from online serving.
• Batch and online paths built separately drift apart (training–serving skew); share the feature code.
• Streaming is not free real time: it adds state, ordering and replay problems that batch never has.

entities = 1_000_000                   # customers that could be scored
requested = 50_000                     # customers actually looked up per day
batch = dict(scored=entities, avg_age_h=24 / 2)            # nightly batch: everyone, results half a day old on average
online = dict(scored=requested, avg_age_h=0)               # online: only who is asked for, scored on the spot
waste = 1 - requested / entities                           # share of batch scores nobody reads

Assumes import numpy as np and import pandas as pd. Each snippet carries its own example numbers; the comments say which are made up or simulated.

# Python — FastAPI online serving
from fastapi import FastAPI
import onnxruntime as ort
import numpy as np

app = FastAPI()
session = ort.InferenceSession("model.onnx")

@app.post("/predict")
async def predict(features: list[float]):
    inp = np.array([features], dtype=np.float32)
    result = session.run(None, {"features": inp})
    return {"prediction": result[0].tolist()}