PerformancePractice
1 min read
◆ The PatternDon't predict what you've already predicted
If 80% of your requests are repeated inputs (product recommendations, credit scores), a prediction cache (Redis, Memcached) can serve them in microseconds. Prediction stores precompute scores for all known entities on a schedule. TTL and invalidation strategies keep results fresh.
// Interactive — cache hit rate vs latency
Cache Hit Rate %70%
Avg Latency—
Pattern bridge: Prediction caching is memoization at system scale. In markets, precomputed support/resistance levels serve the same purpose — calculate once, reference many times.
Worked example
A million keys whose popularity follows Zipf’s law. Caching just the top 1% — 10,000 keys — serves 68% of requests. At 1 ms from cache and 50 ms from the model, the average falls to 16.7 ms. Skewed traffic is why small caches work so well.
Where it misleads
- A cached prediction is only as fresh as its TTL; for fast-changing inputs a cache serves stale answers confidently.
- If traffic is not skewed — every request unique — a cache adds cost and gains nothing.
- Cache keys must include the model version, or a rollout keeps serving the old model’s answers.
In code
keys = 1_000_000 rank = np.arange(1, keys + 1) share = (1 / rank) / (1 / rank).sum() # Zipf: the k-th most popular key gets traffic ~ 1/k hit_rate = share[: keys // 100].sum() # cache the top 1% of keys latency = hit_rate * 1 + (1 - hit_rate) * 50 # ms: 1 from cache, 50 from the model
Assumes import numpy as np and import pandas as pd. Each snippet carries its own example numbers; the comments say which are made up or simulated.
Sources
- L. Breslau, P. Cao, L. Fan, G. Phillips & S. Shenker, “Web Caching and Zipf-like Distributions: Evidence and Implications”, IEEE INFOCOM, 1999
- Designing Data-Intensive Applications, M. Kleppmann, O’Reilly, 2017
Last reviewed October 2026 · How this site checks its content · Report a mistake
Linked from 1