ML Math

Scores of 2, 1 and 0.1 become probabilities 0.659, 0.242, 0.099. Halving the temperature sharpens them to 0.864, 0.117, 0.019; doubling it flattens them to 0.502, 0.304, 0.194. Scores of 1000 and 1001 overflow a naive implementation (nan); subtracting the largest score first gives the correct 0.269, 0.731.

• Softmax outputs look like probabilities but are often over-confident; temperature scaling on held-out data calibrates them (Guo et al. 2017).
• Only differences between scores matter; adding a constant to all of them changes nothing.
• A high softmax probability is not evidence that the input is like the training data.

def softmax(z, T=1.0):
    z = np.asarray(z, float) / T
    e = np.exp(z - z.max())                        # subtract the max: same result, no overflow
    return e / e.sum()

p = softmax([2.0, 1.0, 0.1])
cold, hot = softmax([2.0, 1.0, 0.1], T=0.5), softmax([2.0, 1.0, 0.1], T=2.0)
with np.errstate(over='ignore', invalid='ignore'):
    naive = np.exp([1000.0, 1001.0]) / np.exp([1000.0, 1001.0]).sum()   # overflows
stable = softmax([1000.0, 1001.0])

Assumes import numpy as np. Each snippet carries its own example numbers; the comments say which are made up or simulated.