Probabilistic Generative Models

Generative models take a different approach to classification: they try to model how the data was generated for each class. Instead of just learning a boundary, they learn the distribution of the features for each class.

1. The MLE Landscape

In Maximum Likelihood Estimation (MLE), we can categorize algorithms based on how they model the likelihood $L(\theta; D) = P(D; \theta)$:

• Discriminative $P(y \mid x)$: Directly models the mapping from inputs to outputs.
  • Linear: Gaussian noise $\to$ Linear Regression.
  • Logistic: Bernoulli noise $\to$ Logistic Regression.
• Generative $P(x, y)$: Models the joint probability. Both $x$ and $y$ are treated as random variables.
  • Example: Naive Bayes.

2. The Generative Approach

A generative model learns:

1. Class-Conditional Densities: $P(x \mid y)$ (How does the data for class $y$ look?)
2. Class Priors: $P(y)$ (How common is class $y$?)

To classify a new point $x$, we use Bayes' Theorem to find the posterior probability $P(y \mid x)$:

$P(y \mid x) = \frac{P(x \mid y) P(y)}{P(x)}$

Dependent Events and Decomposition

For dependent events $x$ and $y$, the joint probability is: $P(x, y) = P(x \cap y) = P(x \mid y)P(y) = P(y \mid x)P(x)$

Generative ML assumes $P(D; \theta) = P(x, y; \theta)$, while Discriminative ML assumes $P(D; \theta) = P(y \mid x; \theta)$.

3. Gaussian Class Densities

In models like Gaussian Discriminant Analysis (GDA), we assume classes follow a Gaussian distribution.

Example: Drawing Samples If we have a class with distribution $P(x) \sim \mathcal{N}(2, 1.5)$, most values will fall within the range $[2 - 1.5, 2 + 1.5]$ (the high-density region).

• If all classes share the same covariance matrix $\Sigma$, the decision boundary is linear (LDA).
• If classes have different covariance matrices $\Sigma_k$, the decision boundary is quadratic (QDA).

Comparison: Generative vs. Discriminative