Gaussian Process Regression

A Gaussian Process defines a probability distribution over functions, where any finite collection of function values has a multivariate Gaussian distribution:

$$f(x) \sim \mathcal{GP}(m(x), k(x,x'))$$

where $m(x)$ is the mean function and $k(x,x')$ is the covariance (kernel) function.

Prior Distribution

Given training inputs $X$ and test inputs $X_*$, the joint prior distribution is:

$$\begin{bmatrix} f \\ f_* \end{bmatrix} \sim \mathcal{N}\left(\begin{bmatrix} m(X) \\ m(X_*) \end{bmatrix}, \begin{bmatrix} K(X,X) & K(X,X_*) \\ K(X_*,X) & K(X_*,X_*) \end{bmatrix}\right)$$

Posterior Distribution

Given observations $y = f(X) + \epsilon$ where $\epsilon \sim \mathcal{N}(0, \sigma_n^2)$, the posterior distribution is:

$$f_* | X_*, X, y \sim \mathcal{N}(\bar{f}_*, \text{cov}(f_*))$$

where:

$$\bar{f}_* = m(X_*) + K(X_*,X)[K(X,X) + \sigma_n^2I]^{-1}(y - m(X))$$ $$\text{cov}(f_*) = K(X_*,X_*) - K(X_*,X)[K(X,X) + \sigma_n^2I]^{-1}K(X,X_*)$$

Kernel Functions

RBF (Squared Exponential):

$$k(x,x') = \sigma^2 \exp\left(-\frac{(x-x')^2}{2\ell^2}\right)$$

Periodic:

$$k(x,x') = \sigma^2 \exp\left(-\frac{2\sin^2(\pi|x-x'|/p)}{\ell^2}\right)$$

Matérn 3/2:

$$k(x,x') = \sigma^2(1 + \frac{\sqrt{3}|x-x'|}{\ell})\exp(-\frac{\sqrt{3}|x-x'|}{\ell})$$

where $\ell$ is the length scale, $\sigma^2$ is the variance, and $p$ is the period.