MLOps & Production ML

1,200 requests a second at 80 ms each means 96 in flight (Little’s law). With 8 concurrent requests per replica and a 70% utilisation target, that needs 18 replicas. Kubernetes’ own rule gets there step by step: 10 replicas at 92% CPU against a 70% target become 14 on the next evaluation.

• Scaling reacts after load arrives; with cold starts of tens of seconds, a spike is served by the replicas you already had.
• CPU is a poor signal for GPU or I/O-bound models; scale on queue length or concurrency.
• Scale-to-zero saves money at the price of a slow first request; not every endpoint can afford it.

import math
rps, latency_s, per_replica = 1_200, 0.080, 8       # traffic, time per request, concurrent requests per replica
in_flight = rps * latency_s                         # Little's law
replicas = math.ceil(in_flight / (per_replica * 0.70))   # target 70% utilisation
current, cpu, target = 10, 0.92, 0.70
hpa = math.ceil(current * cpu / target)             # Kubernetes HPA rule

Assumes import numpy as np and import pandas as pd. Each snippet carries its own example numbers; the comments say which are made up or simulated.