1. Big Picture

  • A Gaussian Process is a distribution over functions.
  • Just like a Gaussian distribution is defined by a mean and variance for numbers, a GP is defined by a mean function and a covariance function (kernel) for functions.
  • You can think of it as: instead of guessing parameters (like weights in a neural net), a GP directly says: “I believe the function looks like this, with uncertainty around it.”

2. Formal Definition

A Gaussian Process is defined as:

$f(x) \sim \mathcal{GP}(m(x), k(x, x’))$

  • $m(x) = \mathbb{E}[f(x)]$: mean function
  • $k(x, x’) = \text{Cov}(f(x), f(x’))$: kernel (covariance function)

Properties:

  • Any finite set of points $(x_1, \dots, x_n)$ has a multivariate Gaussian distribution:
    • $(f(x_1), \dots, f(x_n)) \sim \mathcal{N}(m, K)$ where $K$ is the covariance matrix built from the kernel.

3. Intuition

  • Suppose you want to predict a function that passes through some noisy data points.
  • A GP doesn’t give you a single function.
  • Instead, it gives a distribution of possible functions that fit the data.
  • This distribution is shaped by the kernel function, which encodes assumptions like smoothness, periodicity, or linearity.

4. Kernels (Covariance Functions)

The kernel determines how inputs are correlated:

  • RBF / Squared Exponential kernel:
    • $k(x, x’) = \exp \left(-\frac{||x-x’||^2}{2\ell^2}\right)$ → functions are smooth and close points are strongly correlated.
  • Linear kernel: produces straight-line trends.
  • Periodic kernel: captures repeating patterns.
  • You can combine kernels to model complex functions (e.g., trend + seasonality).

5. Posterior Inference

Given training data $(X, y)$ and test points $X_*$​:

  • The GP prior says: both $y$ and $f_*$ (function values at test points) follow a joint Gaussian distribution.
  • Conditioning on $y$, we get the posterior distribution of $f_*$​.

This gives:

  • A mean prediction (like regression).
  • A variance (uncertainty) around the prediction.

So predictions look like a smooth curve with a confidence band.

6. Advantages

  • Captures uncertainty naturally.
  • Flexible: you just need to choose a kernel.
  • Works well on small/medium datasets (e.g., scientific regression problems).

7. Disadvantages

  • Computationally expensive: inversion of covariance matrix is $O(n^3)$.
  • Hard to scale to very large datasets.
  • Kernel choice is critical: a bad kernel = poor performance.

8. Comparison with Bayesian Neural Nets

  • BNNs: uncertainty comes from distributions over weights.
  • GPs: uncertainty comes from distributions over functions directly.
  • Fun fact: with certain limits, infinite-width neural networks converge to Gaussian Processes!

Summary:
Gaussian Processes are a way to do Bayesian regression and prediction by putting a probability distribution over functions. Instead of giving one best function, GPs say: “Here’s the likely shape of the function, plus uncertainty around it.”
They’re powerful for small data and uncertainty estimation, but expensive for big data.