{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# One household’s electricity: the average that is hard to beat\n",
"\n",
"How much electricity will this household use tomorrow? Four years of minute-by-minute readings — and the question of how much any model adds to simply averaging the last week.\n",
"\n",
"*From [Pattern is Everything](https://patterniseverything.com/cases/energy-forecast/). Data: Individual Household Electric Power Consumption — UCI Machine Learning Repository, donated by Georges Hébrail and Alice Bérard (EDF R&D).*"
]
},
{
"cell_type": "code",
"metadata": {},
"source": [
"# Fetch the data file if it is not next to this notebook (in JupyterLite it already is)\n",
"import os, urllib.request\n",
"if not os.path.exists('power-daily.csv'):\n",
" urllib.request.urlretrieve('https://patterniseverything.com/cases/data/power-daily.csv', 'power-daily.csv')"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Missing minutes look like low demand\n",
"\n",
"A day has 1,440 minutes. Count the days where some are missing, and look at the days with the lowest consumption."
]
},
{
"cell_type": "code",
"metadata": {},
"source": [
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"d = pd.read_csv('power-daily.csv', index_col='date', parse_dates=True)\n",
"partial = d.minutes < 1440\n",
"print(len(d), 'days |', partial.sum(), 'with missing minutes |',\n",
" (d.minutes == 0).sum(), 'with none at all')\n",
"complete = d[~partial]\n",
"floor = complete.kwh.min()\n",
"print('lowest complete day:', round(floor, 1), 'kWh, on', complete.kwh.idxmin().date())\n",
"below = d[(d.minutes > 0) & (d.kwh < floor)]\n",
"print('part-days below that:', below.kwh.round(1).tolist(),\n",
" '| minutes recorded:', below.minutes.tolist())"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"No complete day used less than 4.2 kWh — the lowest was in August 2008, when the house was evidently empty: about 174 W around the clock, the appliances that never switch off. Yet 5 part-days fall below that, down to 0.2 kWh, and the 9 days without a single reading sum to exactly 0. They look like the household switching everything off; in fact the meter was not recording. Train on them and the model learns collapses that never happened; test on them and it is blamed for missing them."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Repair the days, and keep track of which ones\n",
"\n",
"Days with at least 90% of their minutes are scaled up to a full day. The rest are treated as missing and filled by linear interpolation between the days around them — but marked, so that no forecast is scored against an invented value."
]
},
{
"cell_type": "code",
"metadata": {},
"source": [
"mostly_there = d.minutes >= 0.9 * 1440\n",
"kwh = (d.kwh * 1440 / d.minutes).where(mostly_there) # scale near-complete days to 24 h\n",
"observed = kwh.notna()\n",
"kwh = kwh.interpolate() # fill the gaps (model input only)\n",
"print('scaled days:', (mostly_there & partial).sum(), '| filled days:', (~observed).sum())\n",
"print('mean kWh per day:', round(kwh[observed].mean(), 1),\n",
" '| std:', round(kwh[observed].std(), 1))"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"60 days are scaled up and 24 filled. A typical day uses about 26 kWh, give or take 10 — a lot of day-to-day noise for one household."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Season and weekday\n",
"\n",
"Average consumption by month and by day of the week, over the observed days."
]
},
{
"cell_type": "code",
"metadata": {},
"source": [
"by_month = kwh[observed].groupby(kwh[observed].index.month).mean().round(1)\n",
"by_weekday = kwh[observed].groupby(kwh[observed].index.dayofweek).mean().round(1)\n",
"print('by month (Jan-Dec):', by_month.tolist())\n",
"print('by weekday (Mon-Sun):', by_weekday.tolist())"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Winter days use more than twice as much as August days (35 against 14 kWh) — probably heating and lighting in winter, and most likely the long French summer holiday in August. Weekends are about 19% higher than weekdays, presumably because people are at home."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Six naive forecasts, tested on the last year\n",
"\n",
"Forecast each day of the final twelve months (December 2009 – November 2010) using only the days before it — a walk-forward test — and score each forecast with the mean absolute error on the days that were actually observed."
]
},
{
"cell_type": "code",
"metadata": {},
"source": [
"forecasts = pd.DataFrame({\n",
" 'yesterday': kwh.shift(1),\n",
" 'same day last week': kwh.shift(7),\n",
" 'same weekday, last 4 weeks': sum(kwh.shift(7 * k) for k in range(1, 5)) / 4,\n",
" 'mean of last 7 days': kwh.shift(1).rolling(7).mean(),\n",
" 'mean of last 28 days': kwh.shift(1).rolling(28).mean(),\n",
" 'same day last year': kwh.shift(364),\n",
"})\n",
"test = (kwh.index >= '2009-12-01') & observed\n",
"mae = forecasts[test].sub(kwh[test], axis=0).abs().mean().round(2).sort_values()\n",
"print(mae.to_string())\n",
"print('test days:', test.sum())"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The best of them is the plain mean of the last 7 days, 4.36 kWh off on an average day of 26. Yesterday alone is noisier (5.19); the weekday-matched forecasts are worse still, because one Tuesday says little about the next. Same day last year is the worst of all, despite the strong seasonal pattern: one day a year ago is too noisy a guide."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5. A model against the average\n",
"\n",
"A ridge regression on the recent level (yesterday, last week, the 7- and 28-day means), the day of the week and the time of year, refitted at the start of each month on everything before it."
]
},
{
"cell_type": "code",
"metadata": {},
"source": [
"from sklearn.linear_model import Ridge\n",
"\n",
"doy = 2 * np.pi * kwh.index.dayofyear / 365.25\n",
"X = pd.DataFrame({\n",
" 'yesterday': kwh.shift(1), 'last_week': kwh.shift(7),\n",
" 'mean7': kwh.shift(1).rolling(7).mean(), 'mean28': kwh.shift(1).rolling(28).mean(),\n",
" 'season_sin': np.sin(doy), 'season_cos': np.cos(doy)}, index=kwh.index)\n",
"X = X.join(pd.get_dummies(kwh.index.dayofweek, prefix='wd', dtype=float).set_index(kwh.index))\n",
"usable = X.notna().all(axis=1)\n",
"\n",
"pred = pd.Series(np.nan, index=kwh.index)\n",
"for month in pd.date_range('2009-12-01', '2010-11-01', freq='MS'):\n",
" train = usable & (kwh.index < month)\n",
" now = usable & (kwh.index >= month) & (kwh.index < month + pd.offsets.MonthBegin(1))\n",
" model = Ridge(alpha=1.0).fit(X[train], kwh[train])\n",
" pred[now] = model.predict(X[now])\n",
"ridge_mae = (pred[test] - kwh[test]).abs().mean()\n",
"print('ridge MAE:', round(ridge_mae, 2), '| mean of last 7 days:', mae['mean of last 7 days'])"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"13 features, refitted every month: 4.29 kWh against 4.36 for the 7-day mean, a 2% improvement. That is the honest result for this household: most of tomorrow is unpredictable from the meter alone, and the little that is predictable a moving average already captures. The obvious next input is the weather, which drives heating, rather than a more complex model of the same meter readings."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 5
}