MLOps & Production ML

Prune 80% of a layer’s weights, keeping the largest fifth, and measure how much its output changes. If the weights are all of similar size, the output error is 59% — there is nothing small to remove. If a few weights are large and most near zero, the error is 11%. Compression works when a model is redundant, and how redundant it is has to be measured, not assumed.

• Output error is not task accuracy; fine-tune after pruning and evaluate on the real task.
• Unstructured sparsity rarely speeds up standard hardware; structured pruning (whole channels or heads) or distillation usually pays off more.
• Compressed models can lose accuracy unevenly, hurting rare classes most.

rng = np.random.default_rng(4)
x = rng.normal(0, 1, (1_000, 256))                 # a batch of inputs

def prune_error(W, keep=0.20):                     # keep the largest 20% of weights by size
    cut = np.quantile(np.abs(W), 1 - keep)
    Wp = np.where(np.abs(W) >= cut, W, 0)
    return np.linalg.norm(x @ Wp - x @ W) / np.linalg.norm(x @ W)

even = prune_error(rng.normal(0, 1, (256, 256)))            # weights all of similar size
peaked = prune_error(rng.standard_t(1.5, (256, 256)))       # a few large weights and many near zero

Assumes import numpy as np and import pandas as pd. Each snippet carries its own example numbers; the comments say which are made up or simulated.