31 — Data Analytics

Exploratory Data Analysis#

AnalyticsPractice 1 min read
◆ The PatternThe first hour with any dataset — a repeatable profiling workflow

EDA is the disciplined version of "poking at the data": shape and types first, then missingness, then each variable's distribution, then relationships between variables. The goal is not pretty plots — it is a list of surprises and decisions: columns to fix, outliers to investigate, transformations to apply, hypotheses worth testing.

shape → types → missing → distributions → relationships
The EDA loop — always in this order. Each step decides what the next one means.
// Interactive — skew the data and inject outliers, watch mean vs median diverge
Skew0.40
Outliers %0%
Mean—
Median—
SteppandasYou're looking for
Shape & typesdf.shape, df.info()Row count, wrong dtypes, IDs read as numbers
Missingnessdf.isna().mean()Columns to drop, impute, or flag — see missing data
Numeric profiledf.describe()Impossible values, skew (mean ≠ median), scale
Categorical profilevalue_counts()Cardinality, typos ("NY" vs "N.Y."), rare levels
Distributionsdf.hist(), boxplotShape, outliers — see distribution shape
Relationshipsdf.corr(), scatter_matrixRedundancy, leakage candidates, target signal
Mean vs median is a free skew detector: when they disagree badly (as in the visual above), the distribution is skewed or contaminated by outliers — report medians, and consider a log transform before modeling.
EDA before splitting? Careful. Profile structure on everything, but tune decisions (imputation values, outlier caps, transformations) on the training split only — otherwise test-set information leaks into your pipeline.
Try it on real data
Kaggle: Titanic (classic messy mix of numeric, categorical, and missing) UCI: Adult Income (48K rows, categorical-heavy, "?" as missing marker)
Adult hides its missing values as the string "?" — a perfect first test of whether your EDA actually catches what df.isna() misses.
When to use this
✓ Use when: Every new dataset, every refreshed pipeline, before every model. Ten minutes of EDA routinely saves days of debugging silent data problems.
✗ Skip when: Never entirely — but for a stable, monitored pipeline the ongoing version of EDA is drift detection, not manual re-profiling.
Quick start — copy to notebook
pip install pandas seaborn matplotlib
# ────────────────────────────────────────
import seaborn as sns
import matplotlib.pyplot as plt

df = sns.load_dataset('titanic')

print(df.shape)
print(df.describe(include='all').T)
print(df.isna().mean().sort_values(ascending=False).head())

df.hist(bins=40, figsize=(12, 6))
sns.heatmap(df.isna(), cbar=False)   # missingness at a glance
plt.show()
Pattern bridge: The first hour with a dataset is where you catch what data-quality checks later automate, and where a time series first shows its trend and seasons — the starting point for decomposition.
Worked example

Five rows that look harmless. Ages stored as text become numbers only after conversion, with 1 entry (“n/a”) turning into a missing value. Incomes use −999 for “unknown”: the naive mean is 31,800, the mean without the placeholders 53,667. The first hour of EDA is mostly about finding what the numbers really are.

Where it misleads
  • Summary statistics hide placeholders, duplicated rows and unit changes; look at value counts and the extremes.
  • Every chart you look at during EDA is a test; the patterns you find need confirming on fresh data.
  • Cleaning rules decided after seeing results can quietly favour the answer you wanted.
In code
df = pd.DataFrame({'age': ['34', '51', '29', 'n/a', '45'],      # numbers stored as text
                   'income': [52_000, -999, 61_000, 48_000, -999]})   # -999 means "unknown"
age = pd.to_numeric(df.age, errors='coerce')
naive_mean = df.income.mean()
clean_mean = df.income.replace(-999, np.nan).mean()
Python, in your browser — edit the code and run it again

Assumes import numpy as np and import pandas as pd (and SciPy where imported). Each snippet simulates or makes up its own data, as the comments say.

The same idea with a library
# Python — the 10-line EDA opener
import pandas as pd

df = pd.read_csv('data.csv')

print(df.shape)
print(df.info())                          # dtypes + non-null counts
print(df.describe(include='all').T)      # numeric + categorical profile
print(df.isna().mean().sort_values(ascending=False).head(10))

for col in df.select_dtypes('object'):
    print(col, df[col].nunique(), df[col].value_counts().head(3).to_dict())

df.hist(bins=40, figsize=(12, 8))     # every numeric at once
Sources
  • Exploratory Data Analysis, J. W. Tukey, Addison-Wesley, 1977
  • K. W. Broman & K. H. Woo, “Data Organization in Spreadsheets”, The American Statistician 72(1), 2018
  • H. Wickham, “Tidy Data”, Journal of Statistical Software 59(10), 2014

Last reviewed October 2026 · How this site checks its content · Report a mistake

Linked from 5
← Previous
Correlation, Causation & Simpson's Paradox
Open in the full reader, with the topic sidebar