Covariance & Correlation

While Joint Distributions tell us where the probability is, Covariance and Correlation tell us how two variables move together.

1. Covariance: The "Raw" Movement

Covariance measures the direction of the linear relationship between two variables.

$\text{Cov}(X, Y) = E[(X - \mu_x)(Y - \mu_y)]$

The Computational Formula (The Shortcut)

In practice, we rarely use the formula above. We use this much simpler "shortcut": $\text{Cov}(X, Y) = E[XY] - E[X]E[Y]$

Properties of Covariance:

1. Symmetry: $\text{Cov}(X, Y) = \text{Cov}(Y, X)$.
2. Covariance with Self: $\text{Cov}(X, X) = \text{Var}(X)$.
3. Linearity: $\text{Cov}(aX + b, cY + d) = ac \cdot \text{Cov}(X, Y)$.
4. Variance of a Sum: $\text{Var}(X + Y) = \text{Var}(X) + \text{Var}(Y) + 2\text{Cov}(X, Y)$.

2. Correlation: The "Standardized" Movement

The problem with Covariance is that its units are "units of $X$ times units of $Y$." If you measure height in meters vs. centimeters, your covariance will change.

Correlation ($r$) solves this by dividing Covariance by the standard deviations, creating a unitless number between $-1$ and $1$.

$\rho_{X,Y} = \text{Corr}(X, Y) = \frac{\text{Cov}(X, Y)}{\sigma_X \sigma_Y}$

3. The Covariance Matrix ($\Sigma$)

In fields like Finance and AI, we deal with vectors of random variables. We organize all their relationships into a square matrix:

In a large matrix $\Sigma$, the diagonal represents the variances of the individual variables, and the off-diagonal entries represent the relationships between them.

Interactive Correlation Explorer

Test Your Knowledge