Case studies

01Missing minutes look like low demand

A day has 1,440 minutes. Count the days where some are missing, and look at the days with the lowest consumption.

import numpy as np
import pandas as pd

d = pd.read_csv('power-daily.csv', index_col='date', parse_dates=True)
partial = d.minutes < 1440
print(len(d), 'days |', partial.sum(), 'with missing minutes |',
      (d.minutes == 0).sum(), 'with none at all')
complete = d[~partial]
floor = complete.kwh.min()
print('lowest complete day:', round(floor, 1), 'kWh, on', complete.kwh.idxmin().date())
below = d[(d.minutes > 0) & (d.kwh < floor)]
print('part-days below that:', below.kwh.round(1).tolist(),
      '| minutes recorded:', below.minutes.tolist())

1442 days | 84 with missing minutes | 9 with none at all
lowest complete day: 4.2 kWh, on 2008-08-25
part-days below that: [0.4, 0.2, 2.2, 1.6, 1.8] | minutes recorded: [21, 30, 232, 152, 236]

No complete day used less than 4.2 kWh — the lowest was in August 2008, when the house was evidently empty: about 174 W around the clock, the appliances that never switch off. Yet 5 part-days fall below that, down to 0.2 kWh, and the 9 days without a single reading sum to exactly 0. They look like the household switching everything off; in fact the meter was not recording. Train on them and the model learns collapses that never happened; test on them and it is blamed for missing them.

02Repair the days, and keep track of which ones

Days with at least 90% of their minutes are scaled up to a full day. The rest are treated as missing and filled by linear interpolation between the days around them — but marked, so that no forecast is scored against an invented value.

mostly_there = d.minutes >= 0.9 * 1440
kwh = (d.kwh * 1440 / d.minutes).where(mostly_there)  # scale near-complete days to 24 h
observed = kwh.notna()
kwh = kwh.interpolate()                                # fill the gaps (model input only)
print('scaled days:', (mostly_there & partial).sum(), '| filled days:', (~observed).sum())
print('mean kWh per day:', round(kwh[observed].mean(), 1),
      '| std:', round(kwh[observed].std(), 1))

scaled days: 60 | filled days: 24
mean kWh per day: 26.2 | std: 10.0

60 days are scaled up and 24 filled. A typical day uses about 26 kWh, give or take 10 — a lot of day-to-day noise for one household.

03Season and weekday

Average consumption by month and by day of the week, over the observed days.

by_month = kwh[observed].groupby(kwh[observed].index.month).mean().round(1)
by_weekday = kwh[observed].groupby(kwh[observed].index.dayofweek).mean().round(1)
print('by month (Jan-Dec):', by_month.tolist())
print('by weekday (Mon-Sun):', by_weekday.tolist())

by month (Jan-Dec): [35.1, 31.2, 29.6, 25.2, 24.7, 21.8, 16.8, 13.7, 23.4, 27.3, 31.0, 35.7]
by weekday (Mon-Sun): [24.0, 25.7, 26.0, 23.5, 25.0, 29.9, 29.3]

Winter days use more than twice as much as August days (35 against 14 kWh) — probably heating and lighting in winter, and most likely the long French summer holiday in August. Weekends are about 19% higher than weekdays, presumably because people are at home.

04Six naive forecasts, tested on the last year

Forecast each day of the final twelve months (December 2009 – November 2010) using only the days before it — a walk-forward test — and score each forecast with the mean absolute error on the days that were actually observed.

forecasts = pd.DataFrame({
    'yesterday': kwh.shift(1),
    'same day last week': kwh.shift(7),
    'same weekday, last 4 weeks': sum(kwh.shift(7 * k) for k in range(1, 5)) / 4,
    'mean of last 7 days': kwh.shift(1).rolling(7).mean(),
    'mean of last 28 days': kwh.shift(1).rolling(28).mean(),
    'same day last year': kwh.shift(364),
})
test = (kwh.index >= '2009-12-01') & observed
mae = forecasts[test].sub(kwh[test], axis=0).abs().mean().round(2).sort_values()
print(mae.to_string())
print('test days:', test.sum())

mean of last 7 days           4.36
mean of last 28 days          4.99
yesterday                     5.19
same weekday, last 4 weeks    5.48
same day last week            5.98
same day last year            6.18
test days: 345

The best of them is the plain mean of the last 7 days, 4.36 kWh off on an average day of 26. Yesterday alone is noisier (5.19); the weekday-matched forecasts are worse still, because one Tuesday says little about the next. Same day last year is the worst of all, despite the strong seasonal pattern: one day a year ago is too noisy a guide.

05A model against the average

A ridge regression on the recent level (yesterday, last week, the 7- and 28-day means), the day of the week and the time of year, refitted at the start of each month on everything before it.

from sklearn.linear_model import Ridge

doy = 2 * np.pi * kwh.index.dayofyear / 365.25
X = pd.DataFrame({
    'yesterday': kwh.shift(1), 'last_week': kwh.shift(7),
    'mean7': kwh.shift(1).rolling(7).mean(), 'mean28': kwh.shift(1).rolling(28).mean(),
    'season_sin': np.sin(doy), 'season_cos': np.cos(doy)}, index=kwh.index)
X = X.join(pd.get_dummies(kwh.index.dayofweek, prefix='wd', dtype=float).set_index(kwh.index))
usable = X.notna().all(axis=1)

pred = pd.Series(np.nan, index=kwh.index)
for month in pd.date_range('2009-12-01', '2010-11-01', freq='MS'):
    train = usable & (kwh.index < month)
    now = usable & (kwh.index >= month) & (kwh.index < month + pd.offsets.MonthBegin(1))
    model = Ridge(alpha=1.0).fit(X[train], kwh[train])
    pred[now] = model.predict(X[now])
ridge_mae = (pred[test] - kwh[test]).abs().mean()
print('ridge MAE:', round(ridge_mae, 2), '| mean of last 7 days:', mae['mean of last 7 days'])

ridge MAE: 4.29 | mean of last 7 days: 4.36

13 features, refitted every month: 4.29 kWh against 4.36 for the 7-day mean, a 2% improvement. That is the honest result for this household: most of tomorrow is unpredictable from the meter alone, and the little that is predictable a moving average already captures. The obvious next input is the weather, which drives heating, rather than a more complex model of the same meter readings.

Last reviewed: 2 October 2026 · How this site checks its content · Report a mistake