Logistic Regression

Despite its name, Logistic Regression is a classification algorithm. Unlike generative models which model the class-conditional distributions, logistic regression directly models the posterior probability of the classes.

1. The Probabilistic Model

Logistic Regression assumes a Discriminative approach. We model the probability of a binary outcome $y \in \{0, 1\}$ as a Bernoulli distribution:

$y \sim \text{Bernoulli}(y \mid \sigma(\theta^T x))$

Where $\sigma(a)$ is the Sigmoid (Logistic) Function: $\sigma(a) = \frac{1}{1 + \exp(-a)} = h(x)$

The probability for a single sample is: $P(y \mid x; \theta) = h(x)^y (1 - h(x))^{1 - y}$

2. Training: Maximum Likelihood

Given a dataset $D = \{(x^{(i)}, y^{(i)})\}_{i=1}^m$, the Likelihood is the product of individual probabilities (assuming i.i.d.):

$L(\theta; D) = \prod_{i=1}^m P(y^{(i)} \mid x^{(i)}; \theta) = \prod_{i=1}^m h(x^{(i)})^{y^{(i)}} (1 - h(x^{(i)}))^{1 - y^{(i)}}$

Log-Likelihood Derivation

To find the optimal $\theta$, we maximize the Log-Likelihood:

$\log L(\theta; D) = \sum_{i=1}^m \log \left[ h(x^{(i)})^{y^{(i)}} (1 - h(x^{(i)}))^{1 - y^{(i)}} \right]$

$\log L(\theta; D) = \sum_{i=1}^m \left[ y^{(i)} \log h(x^{(i)}) + (1 - y^{(i)}) \log(1 - h(x^{(i)})) \right]$

In practice, we often minimize the Negative Log-Likelihood (NLL), which is the same as the Binary Cross-Entropy Loss: $J(\theta) = - \frac{1}{m} \sum_{i=1}^m \left[ y^{(i)} \log h(x^{(i)}) + (1 - y^{(i)}) \log(1 - h(x^{(i)})) \right]$

3. Decision Boundary

The decision boundary is the set of points where the probability of both classes is equal: $P(y=1 \mid x) = 0.5 \iff \theta^T x = 0$

This results in a linear decision boundary.