Correlation (Statistical)

For two variables $x$ and $y$,

$$\mathop{\rm cor}\nolimits (x,y) \equiv {{\rm cov}(x,y)\over \sigma_x\sigma_y},$$ (1)

where $\sigma_x$ denotes Standard Deviation and $\mathop{\rm cov}\nolimits (x,y)$ is the Covariance of these two variables. For the general case of variables $x_i$ and $x_j$, where $i,j=1$, 2, ..., $n$,

$$\mathop{\rm cor}\nolimits (x_i,x_j) = {{\rm cov}(x_i,x_j)\over \sqrt{V_{ii}V_{jj}}},$$ (2)

where $V_{ii}$ are elements of the Covariance Matrix. In general, a correlation gives the strength of the relationship between variables. The variance of any quantity is alway Nonnegative by definition, so

$$\mathop{\rm var}\nolimits \left({{x\over\sigma_x} + {y\over\sigma_y}}\right)\geq 0.$$ (3)

From a property of Variances, the sum can be expanded

$$\mathop{\rm var}\nolimits \left({x\over\sigma_x}\right)+\mat... ...imits \left({{x\over\sigma_x} , {y\over\sigma_y}}\right)\geq 0$$ (4)

$${1\over{\sigma_x}^2} \mathop{\rm var}\nolimits (x)+{1\over{\... ...{2\over\sigma_x\sigma_y}\mathop{\rm cov}\nolimits (x,y) \geq 0$$ (5)

$$1 + 1 + {2\over\sigma_x\sigma_y}\mathop{\rm cov}\nolimits (x... ...\over\sigma_x\sigma_y} \mathop{\rm cov}\nolimits (x,y) \geq 0.$$ (6)

Therefore,

$$\mathop{\rm cor}\nolimits (x,y) = {\mathop{\rm cov}\nolimits (x,y)\over\sigma_x\sigma_y} \geq -1.$$ (7)

Similarly,

$$\mathop{\rm var}\nolimits \left({x\over\sigma_x}\right)- \left({y\over\sigma_y}\right)\geq 0$$ (8)

$$\mathop{\rm var}\nolimits \left({x\over\sigma_x}\right)+\mat... ...mits \left({{x\over\sigma_x} , -{y\over\sigma_y}}\right)\geq 0$$ (9)

$${1\over{\sigma_x}^2} \mathop{\rm var}\nolimits (x) + {1\over... ...2\over\sigma_x\sigma_y} \mathop{\rm cov}\nolimits (x,y) \geq 0$$ (10)

$$1 + 1 - {2\over\sigma_x\sigma_y} \mathop{\rm cov}\nolimits (... ...\over\sigma_x\sigma_y} \mathop{\rm cov}\nolimits (x,y) \geq 0.$$ (11)

Therefore,

$$\mathop{\rm cor}\nolimits (x,y) = {\mathop{\rm cov}\nolimits (x,y)\over\sigma_x\sigma_y} \leq 1,$$ (12)

so $-1 \leq \mathop{\rm cor}\nolimits (x,y) \leq 1$. For a linear combination of two variables,

$$\mathop{\rm var}\nolimits (y-bx) = \mathop{\rm var}\nolimits (y)+\mathop{\rm var}\nolimits (-bx)+2\mathop{\rm cov}\nolimits (y,-bx)$$

$$= \mathop{\rm var}\nolimits (y)+b^2\mathop{\rm var}\nolimits (x)-2b\mathop{\rm cov}\nolimits (x,y)$$

$$= {\sigma_y}^2+{\sigma_x}^2-2b\mathop{\rm cov}\nolimits (x,y).$$ (13)

Examine the cases where $\mathop{\rm cor}\nolimits (x,y) = \pm 1$,

$$\mathop{\rm cor}\nolimits (x,y) \equiv {\mathop{\rm cov}\nolimits (x,y)\over\sigma_x\sigma_y} = \pm 1$$ (14)

$$\mathop{\rm var}\nolimits (y-bx) = b^2{\sigma_x}^2+{\sigma_y}^2\mp 2b\sigma_x\sigma_y = (b\sigma_x\mp\sigma_y)^2.$$ (15)

The Variance will be zero if $b\equiv\pm {\sigma_y/\sigma_x}$, which requires that the argument of the Variance is a constant. Therefore, $y-bx = a$, so $y = a+bx$. If $\mathop{\rm cor}\nolimits (x,y) = \pm 1$, $y$ is either perfectly correlated ($b>0$) or perfectly anticorrelated ($b<0$) with $x$.

See also Covariance, Covariance Matrix, Variance