Architecture✓ Mathematical
◆ The PatternReal-time vs batch vs streaming — choosing your inference architecture
How you serve predictions matters as much as model accuracy. Online serving returns predictions in milliseconds via API calls. Batch serving scores entire datasets on a schedule. Streaming processes events as they arrive. Each pattern has different latency, cost, and complexity profiles.
// Interactive — compare serving patterns
Request Volume50k/day
| Pattern | Latency | Use Case |
|---|---|---|
| Online (REST/gRPC) | <100ms | User-facing predictions, recommendations |
| Batch | Minutes–hours | Nightly scoring, report generation |
| Streaming | Seconds | Fraud detection, real-time pricing |
| Embedded | <1ms | On-device, edge inference |
# Python — FastAPI online serving from fastapi import FastAPI import onnxruntime as ort import numpy as np app = FastAPI() session = ort.InferenceSession("model.onnx") @app.post("/predict") async def predict(features: list[float]): inp = np.array([features], dtype=np.float32) result = session.run(None, {"features": inp}) return {"prediction": result[0].tolist()}
Pattern bridge: Online vs batch serving mirrors the timeframe choice in trading — intraday (real-time) vs daily/weekly (batch). Both force you to match your system's response time to the decision frequency.