Correlation
Pearson Correlation
Measures linear association between two numeric variables:
\[r = \frac{\text{cov}(X,Y)}{\sigma_x \sigma_y}\]or the dot product:
\[r = \Big < \frac{x - \bar{x}}{\Vert x - \bar{x}\Vert}, \frac{y - \bar{y}}{\Vert y - \bar{y}\Vert}\Big>\]Given paired data $(x_1, y_1), \dots, (x_n,y_n)$, we standardize and get
\[r = \frac{\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum_{i=1}^n (x_i - \bar{x})^2} \sqrt{\sum_{i=1}^n(y_i - \bar{y})^2}}\]Spearman Correlation
Spearman is Pearson Correlation applied to ranks. Invariant to distortion, only affected by order.
Let
\[R_i = \text{rank}(x_i), \qquad S_i = \text{rank}(y_i)\]Then,
\[\rho = \frac{\sum_{i=1}^n (R_i - \bar{R})(S_i - \bar{S})}{\sqrt{\sum_{i=1}^n (R_i - \bar{R})^2} \sqrt{\sum_{i=1}^n(S_i - \bar{S})^2}}\]Theorem. If there are no ties, there is a simplied formula:
\[\rho = 1 - \frac{6 \sum_{i=1}^n d_i^2}{n(n^2 -1)}\]
where $d_i = R_i - S_i$.
Proof. Without ties, $R_i$ and $S_i$ are permutations of $\{1, 2, \dots, n\}$. So $\bar{R} = \bar{S} =\frac{n+1}{2}$. Set that as $\mu$.
So
\[\rho = \frac{\sum_i (R_i - \mu)(S_i - \mu)}{\sum_i (R_i-\mu)^2}\]Now note that
\[\sum_i d_i^2 = \sum_i (R_i - \mu)^2 + \sum_i (S_i - \mu)^2 - 2 \sum_i (R_i - \mu)(S_i - \mu) = 2 \sum_i (R_i - \mu)^2 - 2 \sum_i (R_i - \mu)(S_i - \mu)\]or
\[\sum_i (R_i - \mu)(S_i - \mu) = \sum_i (R_i - \mu)^2 - \frac{1}{2} \sum_i d_i^2 \]which in $\rho$ gives:
\[\rho = 1 - \frac{\sum_i d_i^2}{2\sum_i (R_i - \mu)^2}\]Note that $\sum_i (R_i - \mu)^2 = \frac{n(n^2-1)}{12}$, so we get
\[\rho = 1 - \frac{6 \sum_{i=1}^n d_i^2}{n(n^2 -1)}\]
Kendall's Tau
Kendall's tau compares all pairs of points.
Per pair $i < j$, it compares the sign of the difference:
\[\text{sgn}(x_i - x_j) \quad \text{and} \quad \text{sgn}(y_i - y_j) \]We define a pair as concordant if order agrees $(x_i - x_j)(y_i - y_j) > 0$, else it is discordant.
Let
\[C = \#\text{ concordant pairs, \qquad D = \#\text{ discordant pairs}}\]Without ties,
\[\tau = \frac{C-D}{n \choose 2} = \frac{C-D}{C+D}\]It is also equivalent to
\[\frac{2}{n(n-1)} \sum_{i < j}\text{sgn}(x_i - x_j) \text{sgn}(y_i - y_j)\]IMPORTANT:
With ties: we use Kendall's tau-b:
\[\tau_b = \frac{C-D}{\sqrt{(C+D+T_x)(C+D+T_y)}}\]
where
\[T_x = \#\text{ pairs tied only in }x, \qquad T_y = \#\text{ pairs tied only in }y\]NOTE For checking leaderboard stability, Kendall's tau is often cleanest as it checks whether the ordering stays the same among pairs of models.