Bayesian Statistical Methods

Centre of Excellence in Computational Complex Systems Research

 

Bayesian Methods for Neural Networks

Researchers: Jouko Lampinen, Aki Vehtari, Paula Litkey, Ilkka Kalliomäki, Simo Särkkä and Jani Lahtinen

We are studying full Bayesian approch for neural networks, where the high-dimensional integrals required in computation of various marginal distributions are approximated by Markov Chain Monte Carlo methods.

We are developing methods for using more general prior distributions for model parameters and noise models than currently available. Examples of recent results are non-Gaussian noise models with arbitrary correlations between model outputs, outlier tolerant noise models with adaptable tailness, and Bayesian bootstrap methods for analyzing the performance of the models.

We have applied Bayesian neural networks in a number of modelling tasks. In practical applications the Bayesian approach usually requires more expert work than the standard error minimization approach, to build the probability models and priors, and to integrate out all the hyperparameters. The obtained results in our experience have been consistently better than with other statistical estimation methods, and the possibility of compute reliable confidence intervals of the results is necessary in real world applications.

Figures below show two examples of Bayesian neural networks in function approximation and classification tasks.

Figure 1a
Figure 1b

Example of Bayesian neural network for image reconstruction in Electrical Impedance Tomography (EIT). The left figure shows a cross section of a pipe filled with liquid and some gas bubbles (marked by dark green contours). The color shade shows the potential field due to injection of electric current from the redmost electrode, with the bluemost electrode grounded. The right figure shows the reconstruction of the conductivity image from the potential measurements of the 16 electrodes, using Bayesian neural network. The color indicates the bubble probability and blue contour the detected bubble boundary.

Figure 2

Example of a classifying forest scene to tree trunks and background. The figures from left are: the forest image; CART (Classification and Regression Tree); k-Nearest Neighbor classifier with k chosen by leave-one-out cross-validation; Committee of early-stopped MLP neural networks; Bayesian MLP; Bayesian MLP with ARD prior.

References

  • Jouko Lampinen and Aki Vehtari (2001). Bayesian approach for neural networks - review and case studies. Neural Networks, 14(3):7-24. (Invited article. Note that unfortunately the paper version has some printer's errors). (PostScript) (PDF)

  • Jouko Lampinen, Aki Vehtari, and Kimmo Leinonen (1999). Application of Bayesian neural network in electrical impedance tomography. In IJCNN'99: Proceedings of the 1999 International Joint Conference on Neural Networks [CD-ROM], paper number 375. (PDF)

  • Aki Vehtari and Jouko Lampinen (2000). Bayesian MLP neural networks for image analysis. Pattern Recognition Letters, 21(13-14):1183-1191. (Special Issue - Selected Papers from The 11th Scandinavian Conference on Image Analysis.) (PostScript) (PDF)