Joint Distributions: The Multivariate World

A Joint Distribution allows us to calculate the probability of multiple events occurring simultaneously.

1. Discrete Joint Distributions (PMFs)

For discrete random variables $X$ and $Y$, the Joint Probability Mass Function (PMF) is defined as: $p(x, y) = P(X = x, Y = y)$

The Joint PMF Table

The most intuitive way to visualize discrete joint distributions is through a table. Each cell represents the probability of a specific $(x, y)$ pair.

Example: Let $X$ be the number of heads in 2 coin flips and $Y$ be whether the first flip was a head ($1$ for Head, $0$ for Tail).

Key Rules for Joint PMFs:

1. Normalization: The sum of all cells in the table must equal $1$: $\sum_x \sum_y p(x, y) = 1$.
2. Positivity: $p(x, y) \ge 0$ for all $x, y$.

2. Marginal Distributions: "Summing Out"

If you have the joint distribution but only care about one variable, you calculate its Marginal Distribution. You do this by summing across the rows or columns of the table.

• Marginal PMF of $X$: Sum over all possible $y$ values. $p_X(x) = \sum_y p(x, y)$
• Marginal PMF of $Y$: Sum over all possible $x$ values. $p_Y(y) = \sum_x p(x, y)$

3. Expectations of Joint Variables

What is the average value of some function $g(X, Y)$? For example, the product $XY$ or the sum $X+Y$.

$E[g(X, Y)] = \sum_x \sum_y g(x, y) \cdot p(x, y)$

Special Case: $E[XY]$

The expected value of the product $XY$ is a critical component for calculating Covariance. $E[XY] = \sum_x \sum_y (x \cdot y) \cdot p(x, y)$

4. Continuous Joint Distributions (PDFs)

For continuous variables, we use the Joint Probability Density Function (PDF), $f(x, y)$.

• Normalization: The total volume under the 3D surface must be $1$: $\int_{-\infty}^{\infty} \int_{-\infty}^{\infty} f(x, y) \, dx \, dy = 1$
• Marginal PDF: Instead of summing, we integrate out the variable we don't want: $f_X(x) = \int_{-\infty}^{\infty} f(x, y) \, dy$

5. Statistical Independence

Two random variables $X$ and $Y$ are independent if knowing the value of one provides no information about the other. Mathematically, the joint distribution must be the product of the marginals:

$p(x, y) = p_X(x) \cdot p_Y(y) \quad \text{(Discrete)}$ $f(x, y) = f_X(x) \cdot f_Y(y) \quad \text{(Continuous)}$

Table Visual for Independence

In an independence table, every row must be proportional to every other row. If you see a zero in a cell where both marginals are non-zero, the variables cannot be independent.

Test Your Knowledge