Correlation Coefficients in Python (Pearson, Spearman, Kendall)

Correlation coefficients quantify association between two variables (features) in a dataset. They are widely used in statistics, data science, and engineering because they give a compact measure of how two variables move together. Python supports correlation analysis efficiently through NumPy, SciPy, and pandas, and you can visualize relationships and correlation matrices with Matplotlib.

---

1) Correlation: What It Measures (and What It Doesn’t)

A dataset typically has:

• Observations: individual records (rows), such as a person, product, day, or trip.
• Features/variables: measured attributes (columns), such as height, salary, speed, fare, or temperature.

When you compare two variables $x$ and $y$, correlation describes how strongly they are associated.

Common qualitative patterns

• Positive association: as $x$ increases, $y$ tends to increase.
• Negative association: as $x$ increases, $y$ tends to decrease.
• Weak/no obvious association: changes in $x$ do not systematically align with changes in $y$.

Critical interpretation rule

Correlation does not imply causation. A high correlation can occur because:

• $x$ causes $y$
• $y$ causes $x$
• both are driven by a third variable $z$
• the correlation is accidental (especially in small samples or with outliers)

---

2) Three Core Correlation Coefficients

Python users most commonly use:

• Pearson’s $r$ (linear association)
• Spearman’s $ρ$ (rank-based monotonic association)
• Kendall’s $τ$ (rank-based concordance/discordance)

They all lie in the range $[-1, 1]$.

---

3) Pearson Correlation (Linear Association)

Meaning

Pearson’s correlation measures how close the relationship between $x$ and $y$ is to a straight line.

• $r=1$: perfect positive linear relationship
• $r=-1$: perfect negative linear relationship
• $r=0$: no linear relationship (but non-linear association may still exist)

Formula (sample Pearson correlation)

If you have paired observations $(x_i, y_i)$ for $i=1,\dots,n$, Pearson’s $r$ can be written as:

$r=\dfrac{\sum_{i=1}^{n}(x_i-\bar{x})(y_i-\bar{y})}{\sqrt{\sum_{i=1}^{n}(x_i-\bar{x})^2}\sqrt{\sum_{i=1}^{n}(y_i-\bar{y})^2}}$

This is “covariance divided by the product of standard deviations.”

Practical notes

• Sensitive to outliers (a few extreme points can inflate or destroy $r$).
• Best suited when the relationship is roughly linear and variables are approximately well-behaved (often assumed approximately normal in classical inference contexts).

---

4) Spearman Correlation (Rank / Monotonic Association)

Meaning

Spearman’s correlation is Pearson correlation applied to ranks rather than raw values. It captures monotonic relationships:

• If $y$ tends to increase whenever $x$ increases (not necessarily in a straight line), Spearman can be high even when Pearson is moderate.
• Robust against outliers relative to Pearson (because ranks reduce the influence of extreme magnitudes).

Conceptually: compute ranks $R(x_i)$ and $R(y_i)$ and then compute Pearson correlation on those ranks.

• $ρ=1$: perfect increasing monotonic relationship
• $ρ=-1$: perfect decreasing monotonic relationship

---

5) Kendall Correlation (Concordance-Based Rank Association)

Meaning

Kendall’s $τ$ compares all pairs of observations and asks whether the ordering of $x$ agrees with the ordering of $y$.

Given two observations $(x_i,y_i)$ and $(x_j,y_j)$ with $i<j$:

• Concordant if $(x_i-x_j)(y_i-y_j)>0$
• Discordant if $(x_i-x_j)(y_i-y_j)<0$
• Ties require special handling

A common form used in software (tie-adjusted) is:

$τ=\dfrac{n^+-n^-}{\sqrt{(n^++n^-+n_x)(n^++n^-+n_y)}}$

where $n^+$ is concordant pairs, $n^-$ discordant pairs, $n_x$ ties only in $x$, $n_y$ ties only in $y$.

Practical notes

• Often more conservative than Spearman.
• Especially meaningful when you care about pairwise ranking agreement and when ties exist.

---

6) How to Compute Correlation in Python

Below is a conceptual “tool map” of what each library does best. (You provided code examples already; I’m focusing on what to use and why, without repeating large code blocks.)

NumPy

• Use when you want fast Pearson correlation matrices on numeric arrays.
• Main function: np.corrcoef(...)
• Output: correlation matrix where diagonals are 1 and off-diagonals are pairwise correlations.

Important convention:

• By default, np.corrcoef treats rows as variables and columns as observations.
• If your data is shaped like “rows = observations, columns = features” (common in ML), you often set rowvar=False.

NaNs:

• NumPy typically propagates NaNs into any correlation that involves the NaN-containing variable/segment.

SciPy (scipy.stats)

Best when you want:

• correlation coefficient and a p-value (inference-oriented routines)
• functions:
  • pearsonr(x, y) → returns $(r, p)$
  • spearmanr(x, y) → returns $(ρ, p)$ (or an object with .correlation, .pvalue)
  • kendalltau(x, y) → returns $(τ, p)$

NaNs:

• SciPy routines often error or propagate NaNs unless you specify a nan_policy such as 'omit' (where supported).

pandas

Best when you work with tabular data (DataFrames):

• Series.corr(other, method='pearson'|'spearman'|'kendall')
• DataFrame.corr(method=...) → correlation matrix across columns
• DataFrame.corrwith(...) → correlate matching columns (or rows) against another Series/DataFrame

NaNs:

• pandas generally handles missing values by pairwise deletion: it uses only the pairs where both values are present.

This makes pandas correlation convenient for real-world datasets with missing entries.

---

7) Correlation vs Linear Regression (and Why They’re Often Used Together)

Correlation measures “how strongly associated,” while regression estimates a best-fitting line for prediction or explanation.

A simple linear regression line is:

$\hat{y}=b_0+b_1x$

In SciPy, linregress(x, y) returns:

• slope $b_1$
• intercept $b_0$
• correlation-related value $r$ (reported as rvalue)
• p-value (testing slope different from 0 under classical assumptions)
• standard error of slope

Important conceptual caution:

• A large $r$ does not guarantee a good predictive model if the relationship is nonlinear, heteroscedastic, or driven by outliers.
• A regression line can look “good” because of a single extreme point—always inspect plots.

---

8) Visualization: What to Plot and Why

(A) Scatterplot (x–y plot)

Use it to confirm:

• whether the relationship looks linear, monotonic, clustered, or curved
• whether outliers dominate the association
• whether variance changes with $x$ (heteroscedasticity)

(B) Regression line overlay

Plot the line $\hat{y}=b_0+b_1x$ to visualize the linear fit and annotate with $r$.

(C) Correlation matrix heatmap

For many features, a correlation table becomes hard to read.
A heatmap helps you see:

• strong positive blocks (collinear groups)
• strong negative pairs
• near-zero relationships

Interpretation reminder:

• Correlation matrices reflect pairwise associations only; they do not account for confounding.

---

9) Choosing Pearson vs Spearman vs Kendall: Practical Decision Rules

Use Pearson when:

• You care about linear association
• Data are roughly continuous and not dominated by outliers
• The relationship is plausibly linear

Use Spearman when:

• Relationship is monotonic but not linear
• Data contain outliers or are skewed
• Variables are ordinal or rankings matter

Use Kendall when:

• You care about ordering agreement and pairwise concordance
• Ties are common and you want a robust rank-based measure
• Sample sizes are small to moderate and interpretability in terms of concordant/discordant pairs is useful

---

10) Common Failure Modes (High-Value Warnings)

1. Outliers can dominate Pearson’s $r$
   Always check scatterplots.
2. Nonlinear relationships can yield $r \approx 0$
   You may have strong dependence that is not linear.
3. Multiple comparisons in correlation matrices
   If you compute hundreds of correlations, many will look “significant” by chance unless you adjust for multiplicity.
4. Missing values handling differs across libraries
   • pandas: usually pairwise deletion
   • SciPy: may error or require explicit NaN policy
   • NumPy: often propagates NaNs