Probability Theory & Statistical Inference

A rigorous survey of probability theory, ranging from foundational axioms and counting principles to multivariate calculus, limit theorems, and Information Theory. This guide bridges the gap between intuitive concepts and the formal mathematics that underpin modern Data Science and Machine Learning.

How this guide is organized

1. Foundations

The core language of uncertainty. We establish the Kolmogorov axioms, Independence, and the critical debate between Frequentist and Bayesian interpretations. We also introduce Maximum Likelihood Estimation (MLE).

2. Discrete Distributions

Modeling "countable" events. From the simple Bernoulli trial to the Poisson limit of rare events.

3. Continuous Distributions

Modeling measurements. We break down the anatomy of the Normal distribution and the memoryless nature of the Exponential distribution.

4. Multivariate Probability

Moving from one variable to many. We explore Joint Distributions, the "slicing" of Conditional Distributions, and the Covariance Matrix used to map relationships in high-dimensional space.

5. Mathematical Tools

The engine room of probability. We use Moments & MGFs to uniquely identify distributions, and the Jacobian to map complex variable transformations.

6. Inference Distributions

The distributions used for testing and sampling. This includes the Chi-Square, Student's t, and the flexible Beta/Gamma families.

7. Limit Theorems

The "God-Tier" theorems that bring order to chaos. We prove how the Law of Large Numbers forces averages to converge, and how the Central Limit Theorem makes the Bell Curve appear everywhere.

8. Information Theory

Probability as a measure of surprise. We define Entropy and KL-Divergence, the fundamental metrics for measuring information loss and training modern AI.