ClassificationMathematical
◆ The PatternThe bridge from regression to classification — sigmoid turns scores into probabilities
Logistic regression wraps a linear model in a sigmoid function, producing a probability between 0 and 1. Despite the name, it's a classifier, not a regressor. It's the simplest neural network — a single neuron.
P(y=1|x) = σ(w·x + b) = 1/(1 + e^(−(w·x+b)))
σ = sigmoid | outputs probability of class 1 | decision boundary at 0.5
BCE = −[y·log(ŷ) + (1−y)·log(1−ŷ)]
Binary Cross-Entropy — penalises confident wrong predictions exponentially
// Decision boundary — adjust weight and bias
Weight w2.0
Bias b0.0
Numerical stability: Never use nn.Sigmoid() + nn.BCELoss(). Use nn.BCEWithLogitsLoss() which combines them with the log-sum-exp trick to avoid overflow/underflow.
Pattern bridge: The sigmoid that squeezes values into [0,1] reappears as probability curves in statistics and mirrors the S-curve of market sentiment cycles — gradual build, rapid shift, saturation.
Worked example
With w = 1.5, b = −2 and x = 3, the score is 2.5 and the sigmoid turns it into P = 0.924. Cross-entropy punishes confident mistakes hard: predicting 0.9 for a true 1 costs 0.105, predicting 0.1 costs 2.303 — 22 times more. The gradient is simply (p − y)·x, here −0.228.
Where it misleads
- The output is a probability only if the model is calibrated; strong regularization or class rebalancing shifts it.
- The decision boundary is linear in the inputs; curved boundaries need features or a different model.
- With perfectly separable data the weights grow without limit unless regularized.
In code
sigmoid = lambda z: 1 / (1 + np.exp(-z)) bce = lambda y, p: -(y * np.log(p) + (1 - y) * np.log(1 - p)) w, b, x = 1.5, -2.0, 3.0 p = sigmoid(w * x + b) # P(y = 1 | x) loss_right = bce(1, 0.9) # confident and right loss_wrong = bce(1, 0.1) # confident and wrong grad_w = (p - 1) * x # dBCE/dw for a true label y = 1
Assumes import numpy as np. Each snippet carries its own example numbers; the comments say which are made up or simulated.
The same idea with a library
# PyTorch logistic regression model = nn.Sequential( nn.Linear(784, 1), nn.Sigmoid() ) loss_fn = nn.BCELoss() # or BCEWithLogitsLoss (more stable) loss = loss_fn(model(x), y)
Sources
- D. R. Cox, “The Regression Analysis of Binary Sequences”, Journal of the Royal Statistical Society B 20(2), 1958
- The Elements of Statistical Learning (2nd ed.), T. Hastie, R. Tibshirani & J. Friedman, Springer, 2009 — chapter 4
Last reviewed October 2026 · How this site checks its content · Report a mistake
Linked from 1