Correlation Coefficient

The correlation coefficient is a quantity which gives the quality of a Least Squares Fitting to the original data. To define the correlation coefficient, first consider the sum of squared values ${\rm ss}_{xx}$, ${\rm ss}_{xy}$, and ${\rm ss}_{yy}$ of a set of $n$ data points $(x_i, y_i)$ about their respective means,

$${\rm ss}_{xx} \equiv \Sigma (x_i-\bar x)^2 = \Sigma x^2-2\bar x\Sigma x+\Sigma {\bar x}^2$$

$$= \Sigma x^2-2n{\bar x}^2+n{\bar x}^2 = \Sigma x^2-n{\bar x}^2$$ (1)

$${\rm ss}_{yy} \equiv \Sigma (y_i-\bar y)^2 = \Sigma y^2-2\bar y\Sigma y+\Sigma {\bar y}^2$$

$$= \Sigma y^2-2n{\bar y}^2+n{\bar y}^2 = \Sigma y^2-n{\bar y}^2$$ (2)

$${\rm ss}_{xy} \equiv \Sigma (x_i-\bar x)(y_i-\bar y) = \Sigma (x_iy_i-\bar xy_i-x_i\bar y+\bar x\bar y)$$

$$= \Sigma xy-n\bar x\bar y-n\bar x\bar y+n\bar x\bar y= \Sigma xy-n\bar x\bar y.$$ (3)

For linear Least Squares Fitting, the Coefficient $b$ in

$$y=a+bx$$ (4)

is given by

$$b={n\sum xy-\sum x\sum y\over n\sum x^2-\left({\sum x}\right)^2} = {{\rm ss}_{xy}\over {\rm ss}_{xx}},$$ (5)

and the Coefficient $b'$ in

$$x=a'+b'y$$ (6)

is given by

$$b'={n\sum xy-\sum x\sum y\over n\sum y^2-\left({\sum y}\right)^2}.$$ (7)

The correlation coefficient $r^2$ (sometimes also denoted $R^2$) is then defined by

$$r\equiv \sqrt{bb'} = {n\sum xy-\sum x\sum y\over \sqrt{\left... ...)^2}\right]\left[{n\sum y^2-\left({\sum y}\right)^2}\right]}},$$ (8)

which can be written more simply as

$$r^2={{{\rm ss}_{xy}}^2\over {\rm ss}_{xx}{\rm ss}_{yy}}.$$ (9)

The correlation coefficient is also known as the Product-Moment Coefficient of Correlation or Pearson's Correlation. The correlation coefficients for linear fits to increasingly noisy data are shown above.

The correlation coefficient has an important physical interpretation. To see this, define

$$A\equiv (\Sigma x^2-n{\bar x}^2)^{-1}$$ (10)

and denote the ``expected'' value for $y_i$ as $\hat y_i$. Sums of $\hat y_i$ are then

$$\hat y_i = a+bx_i=\bar y-b\bar x+bx_i=\bar x+b(x_i-\bar x)$$

$$= A(\bar y\Sigma x^2-\bar x\Sigma xy+x_i\Sigma xy-n\bar x\bar yx_i)$$

$$= A[\bar y\Sigma x^2+(x_i-\bar x)\Sigma xy-n\bar x\bar yx_i]$$ (11)

$$\Sigma \hat y_i = A(n\bar y\Sigma x^2-n^2\bar x^2\bar y)$$ (12)

$$\Sigma {\hat y_i}^2 = A^2[n\bar y^2(\Sigma x^2)^2-n^2\bar x^2\bar y^2(\Sigma x^2)$$

$$\phantom{=} -2n\bar x\bar y(\Sigma xy)(\Sigma x^2)+2n^2\bar x^3\bar y(\Sigma xy)$$

$$\phantom{=} +(\Sigma x^2)(\Sigma xy)^2-n\bar x^2(\Sigma xy)]$$ (13)

$$\Sigma y_i{\hat y}_i = A\Sigma[y_i\bar y\Sigma x^2+y_i(x_i-\bar x)\Sigma xy-n\bar x\bar yx_i y_i]$$

$$= A[n\bar y^2\Sigma x^2+(\Sigma xy)^2-n\bar x\bar y\Sigma xy-n\bar x\bar y(\Sigma xy)]$$

$$= A[n\bar y^2\Sigma x^2+(\Sigma xy)^2-2n\bar x\bar y\Sigma xy].$$ (14)

The sum of squared residuals is then

$${\rm SSR} \equiv \Sigma({\hat y}_i-\bar y)^2=\Sigma({\hat y_i}^2-2\bar y{\hat y_i}+\bar y^2)$$

$$= A^2(\Sigma xy-n\bar x\bar y)^2(\Sigma x^2-n\bar x^2) = {(\Sigma xy-n\bar x\bar y)^2\over \Sigma x^2-n\bar x^2}$$

$$= b\,{\rm ss}_{xy} = {{{\rm ss}_{xy}}^2\over {\rm ss}_{xx}} = {\rm ss}_{yy}r^2=b^2{\rm ss}_{xx},$$ (15)

and the sum of squared errors is

$${\rm SSE} \equiv \Sigma (y_i-\hat y_i)^2 = \Sigma (y_i-\bar y+b\bar x-bx_i)^2$$

$$= \Sigma [y_i-\bar y-b(x_i-\bar x)]^2$$

$$= \Sigma (y_i-\bar y)^2+b^2\Sigma (x_i-\bar x)^2-2b\Sigma (x_i-\bar x)(y_i-\bar y)$$

$$= {\rm ss}_{yy}+b^2{\rm ss}_{xx}-2b{\rm ss}_{xy}.$$ (16)

But

$$b = {{\rm ss}_{xy}\over {\rm ss}_{xx}}$$ (17)

$$r^2 = {{{\rm ss}_{xy}}^2\over {\rm ss}_{xx}{\rm ss}_{yy}},$$ (18)

so

$${\rm SSE} = {\rm ss}_{yy}+{{{\rm ss}_{xy}}^2\over {{\rm ss}_{xx}}^2} {\rm ss}_{xx}-2{{\rm ss}_{xy}\over {\rm ss}_{xx}}{\rm ss}_{xy}$$

$$= {\rm ss}_{yy}-{{{\rm ss}_{xy}}^2\over {\rm ss}_{xx}}$$ (19)

$$= {\rm ss}_{yy}\left({1-{{{\rm ss}_{xy}}^2\over {{\rm ss}_{xx}}^2}}\right)= {\rm ss}_{yy}(1-r^2)$$

$$= {s_y}^2-{s_{\hat y}}^2,$$ (20)

and

$${\rm SSE}+{\rm SSR}={\rm ss}_{yy}(1-r^2)+{\rm ss}_{yy}r^2={\rm ss}_{yy}.$$ (21)

The square of the correlation coefficient $r^2$ is therefore given by

$$r^2\equiv {{\rm SSR}\over {\rm ss}_{yy}} = {{{\rm ss}_{xy}... ... x\bar y)^2\over(\Sigma x^2-n\bar x^2)(\Sigma y^2-n\bar y^2)}.$$ (22)

In other words, $r^2$ is the proportion of ${\rm ss}_{yy}$ which is accounted for by the regression.

If there is complete correlation, then the lines obtained by solving for best-fit $(a, b)$ and $(a', b')$ coincide (since all data points lie on them), so solving (6) for $y$ and equating to (4) gives

$$y=-{a'\over b'}+{x\over b'} = a+bx.$$ (23)

Therefore, $a=-a'/b'$ and $b=1/b'$, giving

$$r^2=bb'=1.$$ (24)

The correlation coefficient is independent of both origin and scale, so

$$r(u,v) = r(x,y),$$ (25)

where

$$u \equiv {x-x_0\over h}$$ (26)

$$v \equiv {y-y_0\over h}.$$ (27)

See also Correlation Index, Correlation Coefficient--Gaussian Bivariate Distribution, Correlation Ratio, Least Squares Fitting, Regression Coefficient

References

Acton, F. S. Analysis of Straight-Line Data. New York: Dover, 1966.

Kenney, J. F. and Keeping, E. S. ``Linear Regression and Correlation.'' Ch. 15 in Mathematics of Statistics, Pt. 1, 3rd ed. Princeton, NJ: Van Nostrand, pp. 252-285, 1962.

Gonick, L. and Smith, W. The Cartoon Guide to Statistics. New York: Harper Perennial, 1993.

Press, W. H.; Flannery, B. P.; Teukolsky, S. A.; and Vetterling, W. T. ``Linear Correlation.'' §14.5 in Numerical Recipes in FORTRAN: The Art of Scientific Computing, 2nd ed. Cambridge, England: Cambridge University Press, pp. 630-633, 1992.