ML Math

A one-unit network: h = ReLU(w₁x), y = w₂h, loss ½(y − t)². With x = 2, t = 1, w₁ = 0.5 and w₂ = −1.5, the chain rule gives ∂L/∂w₁ = 7.5 and ∂L/∂w₂ = −2.5. Nudging w₁ by a millionth and measuring the change gives 7.4999999999: the gradient check every hand-written backward pass should pass.

• Backprop computes gradients exactly; it says nothing about whether following them leads somewhere good.
• At ReLU’s kink the derivative is undefined; frameworks pick a value, which rarely matters but breaks naive gradient checks exactly at zero.
• Long chains multiply many factors, which is where vanishing and exploding gradients come from (see RNN).

x, t = 2.0, 1.0                                  # input and target
w1, w2 = 0.5, -1.5

def loss(w1, w2):
    h = max(w1 * x, 0)                             # ReLU hidden unit
    return 0.5 * (w2 * h - t) ** 2

h = max(w1 * x, 0); y = w2 * h
dL_dy = y - t
grad_w2 = dL_dy * h                                # chain rule, backwards
grad_w1 = dL_dy * w2 * (1 if w1 * x > 0 else 0) * x
eps = 1e-6                                         # check against finite differences
num_w1 = (loss(w1 + eps, w2) - loss(w1 - eps, w2)) / (2 * eps)

Assumes import numpy as np. Each snippet carries its own example numbers; the comments say which are made up or simulated.