Variational and Approximate GPs¶
Variational and approximate Gaussian processes are used in a variety of cases:
- When the GP likelihood is non-Gaussian (e.g. for classification).
- To scale up GP regression (by using stochastic optimization).
- To use GPs as part of larger probablistic models.
With GPyTorch it is possible to implement various types approximate GP models. All approximate models consist of the following 3 composible objects:
VariationalDistribution, which define the form of the approximate inducing value posterior \(q(\mathbf u)\).
VarationalStrategies, which define how to compute \(q(\mathbf f(\mathbf X))\) from \(q(\mathbf u)\).
_ApproximateMarginalLogLikelihood, which defines the objective function to learn the approximate posterior (e.g. variational ELBO).
(See the strategy/distribution comparison for examples of the different classes.) The variational documentation has more information on how to use these objects. Here we provide some examples which highlight some of the common use cases:
- Large-scale regression (when exact methods are too memory intensive): see the stochastic variational regression example.
- Variational inference with natural gradient descent (for faster/better optimization): see the ngd example.
- Variational inference with contour integral quadrature (for large numbers of inducing points): see the ciq example.
- Variational distribution options for different scalability/expressiveness: see the strategy/distribution comparison.
- Alternative optimization objectives for the GP’s predictive distribution: see the approximate GP objective functions notebook. This example compares and contrasts the variational ELBO with the predictive log likelihood of Jankowiak et al., 2020.
- Classification: see the non-Gaussian likelihood notebook and the Polya-Gamma Binary Classification notebook.
- Multi-output variational GPs (when exact methods are too memory intensive): see the variational GPs with multiple outputs example.
- Uncertain inputs: see the GPs with uncertain inputs example.