Gaussian Processes (GPs)

1. Big Picture

• A Gaussian Process is a distribution over functions.
• Just like a Gaussian distribution is defined by a mean and variance for numbers, a GP is defined by a mean function and a covariance function (kernel) for functions.
• You can think of it as: instead of guessing parameters (like weights in a neural net), a GP directly says: “I believe the function looks like this, with uncertainty around it.”

---

2. Formal Definition

A Gaussian Process is defined as:

$f(x) \sim \mathcal{GP}(m(x), k(x, x’))$

• $m(x) = \mathbb{E}[f(x)]$: mean function
• $k(x, x’) = \text{Cov}(f(x), f(x’))$: kernel (covariance function)

Properties:

• Any finite set of points $(x_1, \dots, x_n)$ has a multivariate Gaussian distribution:
  • $(f(x_1), \dots, f(x_n)) \sim \mathcal{N}(m, K)$ where $K$ is the covariance matrix built from the kernel.

---

3. Intuition

• Suppose you want to predict a function that passes through some noisy data points.
• A GP doesn’t give you a single function.
• Instead, it gives a distribution of possible functions that fit the data.
• This distribution is shaped by the kernel function, which encodes assumptions like smoothness, periodicity, or linearity.

---

4. Kernels (Covariance Functions)

The kernel determines how inputs are correlated:

• RBF / Squared Exponential kernel:
  • $k(x, x’) = \exp \left(-\frac{||x-x’||^2}{2\ell^2}\right)$ → functions are smooth and close points are strongly correlated.
• Linear kernel: produces straight-line trends.
• Periodic kernel: captures repeating patterns.
• You can combine kernels to model complex functions (e.g., trend + seasonality).

---

5. Posterior Inference

Given training data $(X, y)$ and test points $X_*$​:

• The GP prior says: both $y$ and $f_*$ (function values at test points) follow a joint Gaussian distribution.
• Conditioning on $y$, we get the posterior distribution of $f_*$​.

This gives:

• A mean prediction (like regression).
• A variance (uncertainty) around the prediction.

So predictions look like a smooth curve with a confidence band.

---

6. Advantages

• Captures uncertainty naturally.
• Flexible: you just need to choose a kernel.
• Works well on small/medium datasets (e.g., scientific regression problems).

---

7. Disadvantages

• Computationally expensive: inversion of covariance matrix is $O(n^3)$.
• Hard to scale to very large datasets.
• Kernel choice is critical: a bad kernel = poor performance.

---

8. Comparison with Bayesian Neural Nets

• BNNs: uncertainty comes from distributions over weights.
• GPs: uncertainty comes from distributions over functions directly.
• Fun fact: with certain limits, infinite-width neural networks converge to Gaussian Processes!

---

Summary:
Gaussian Processes are a way to do Bayesian regression and prediction by putting a probability distribution over functions. Instead of giving one best function, GPs say: “Here’s the likely shape of the function, plus uncertainty around it.”
They’re powerful for small data and uncertainty estimation, but expensive for big data.