Demystifying Logistic Regression: From Mathematics to Production-Ready Python

Despite the word “regression” in its name, Logistic Regression is one of the foundational, interpretable algorithms used for binary and multi-class classification tasks in modern machine learning. Whether assessing loan default risk, predicting patient diagnoses, or detecting spam emails, logistic regression serves as a transparent and efficient baseline.

---

1. Core Intuition & Mathematical Foundation

In standard linear regression, our objective is to model a continuous outcome $y$ using a linear combination of input features $X$:

$$z = w^T X + b = w_1 x_1 + w_2 x_2 + \dots + w_n x_n + b$$

However, linear regression can output arbitrary real values ranging from $-\infty$ to $+\infty$, making it unsuited for calculating probabilities (which must strictly lie within the closed interval $[0, 1]$).

Logistic Regression resolves this limitation by mapping the continuous log-odds $z$ through a Sigmoid (logistic) activation function $\sigma(z)$:

$$\hat{y} = P(Y=1 \mid X) = \sigma(z) = \frac{1}{1 + e^{-z}}$$

Properties of the Sigmoid Function:

• As $z \to +\infty$, $\sigma(z) \to 1$
• As $z \to -\infty$, $\sigma(z) \to 0$
• When $z = 0$, $\sigma(z) = 0.5$

The Decision Boundary

To convert continuous probabilities into discrete class decisions ($0$ or $1$), we define a decision threshold (by default, $\tau = 0.5$):

$$\hat{C} = \begin{cases} 1 & \text{if } \hat{y} \ge 0.5 \\ 0 & \text{if } \hat{y} < 0.5 \end{cases}$$

---

2. The Cost Function: Binary Cross-Entropy (Log Loss)

Using Mean Squared Error (MSE) on a non-linear sigmoid transformation leads to a non-convex optimization landscape with many local minima. To guarantee convexity, Logistic Regression minimizes the Binary Cross-Entropy (Log Loss):

$$J(w, b) = -\frac{1}{m} \sum_{i=1}^{m} \Big[ y^{(i)} \log(\hat{y}^{(i)}) + (1 - y^{(i)}) \log(1 - \hat{y}^{(i)}) \Big]$$

Gradient Descent Updates

The partial derivatives of the cost function with respect to the weights $w_j$ and bias $b$ simplify to:

$$\frac{\partial J}{\partial w_j} = \frac{1}{m} \sum_{i=1}^{m} (\hat{y}^{(i)} - y^{(i)}) x_j^{(i)}$$ $$\frac{\partial J}{\partial b} = \frac{1}{m} \sum_{i=1}^{m} (\hat{y}^{(i)} - y^{(i)})$$

Parameters are then iteratively updated via:

$$w_j \leftarrow w_j - \alpha \frac{\partial J}{\partial w_j}$$ $$b \leftarrow b - \alpha \frac{\partial J}{\partial b}$$

---

3. Regularization: L1 (Lasso) vs. L2 (Ridge)

To combat overfitting and manage multicollinearity in high-dimensional feature spaces, penalty terms are incorporated directly into the objective function:

L1 Regularization (Lasso)

$$J_{L1}(w, b) = J(w, b) + \lambda \sum_{j=1}^{p} \vert{}w_j\vert{}$$

• Behavior: The diamond-shaped contour constraint drives uninformative coefficients to exactly zero, acting as automated feature selection.

L2 Regularization (Ridge)

$$J_{L2}(w, b) = J(w, b) + \lambda \sum_{j=1}^{p} w_j^2$$

• Behavior: The spherical contour shrinks weight magnitudes proportionally without forcing them to absolute zero, distributing predictive influence smoothly.

Note on Scikit-Learn: The hyperparameter $C$ is the inverse of regularization strength ($C = \frac{1}{\lambda}$). Smaller $C$ values enforce stronger penalties.

---

4. End-to-End Implementation in Python

Here is a full end-to-end Python pipeline using NumPy, Pandas, Scikit-Learn, and Bokeh for the plots.

This one runs right here — press Run below. Unlike the other runnable blocks on this blog, which use a bare, fully self-hosted Pyodide with no extra packages, this one pulls NumPy, Pandas, scikit-learn, and Bokeh from Pyodide’s own CDN on demand (first run only, cached after — expect a sizeable download, upwards of 100 MB, since that’s the actual size of a WASM-compiled SciPy/scikit-learn stack). Bokeh’s plots are genuinely interactive here — pan, zoom, and hover tooltips all work — via BokehJS rendering the same JSON spec Bokeh would otherwise hand to a notebook.