Some of the most important numbers in the world cannot be measured directly. The temperature deep inside a spinning generator. The exact charge left in a battery. The true health of a patient between hospital visits. In each case the quantity that matters is hidden — wrapped in insulation, sealed in a cell, hidden inside a body — and all we have are noisy, indirect signals from the outside.
In 1960, Rudolf Kálmán described a way to estimate such hidden quantities. The idea is a rhythm of two steps. First predict: use a model of how the system behaves to guess where the hidden state should be a moment from now. Then correct: when a noisy measurement arrives, nudge the guess toward it — but only partway, in proportion to how much you trust the sensor versus the model. Repeat forever. The amount of that nudge is the famous Kalman gain, and it automatically settles on the optimal blend of prediction and measurement.
The classic filter assumes you can write the system’s physics down by hand as neat linear equations. Reality is rarely so polite. The Deep Kalman Filter keeps the same predict-then-correct rhythm but replaces the hand-written model with neural networks that learn how the hidden state evolves and how it shows up in the sensors — straight from data. It is the same ancient pattern of disciplined guessing, now able to handle systems too tangled to derive on paper.
Case 1 — The generator’s hidden hotspot
Inside a large generator or power transformer, the part that fails first is the hottest spot in the copper winding, buried under layers of insulation where no thermometer can sit. Run too hot for too long and the insulation ages and cracks. Engineers cannot read that hotspot directly — instead they measure load current, coolant and oil temperature, and ambient air, then let a thermal model infer the rest. A Kalman-style estimator fuses those noisy outside readings with a model of how heat builds and dissipates to track the unseen hotspot in real time. A deep version learns the messy, nonlinear thermal behaviour of a specific machine — something the textbook IEC loading curves only approximate — so operators can push the machine harder when it is safe and back off before insulation life is quietly spent.
Case 2 — How much charge is really left?
The “state of charge” on an electric car or laptop is not measured — it cannot be. There is no fuel gauge inside a lithium-ion cell. The battery management system only sees voltage, current, and temperature, and must estimate the charge from them. The relationship is nonlinear and drifts as the battery ages, so a raw reading jumps around with every change in load. Kalman filters have become the industry workhorse here: predict the charge from current drawn, correct using the measured voltage, and average out the noise. Deep Kalman variants learn the cell’s aging chemistry, giving a steadier, more honest percentage — the difference between a car that strands you and one you can trust.
Case 3 — A patient between visits
A person’s underlying health is a hidden state that moves continuously, yet we only glimpse it through scattered, irregular measurements — a blood test here, a blood-pressure reading there, a symptom noted weeks apart. Krishnan, Shalit and Sontag introduced the Deep Kalman Filter precisely for this setting: to estimate a latent patient state from sparse electronic health records and even reason about how a medication would change its trajectory. The filter fills the gaps between observations with a learned model of how the disease progresses, then snaps back to reality each time a real measurement arrives — the same predict-and-correct heartbeat, applied to a human being.
Three very different worlds — a power plant, a battery pack, a hospital ward — share one structure. Something essential is hidden; the sensors are noisy and indirect; and the way forward is not to trust the model alone, nor the measurement alone, but to weigh them against each other, moment by moment. That weighing is the pattern.