Posterior Inference on Shallow Infinitely Wide Bayesian Neural Networks under Weights with Unbounded Variance

Overview

• The paper investigates the scaling limits of Bayesian neural networks, which describe the behavior of these networks as the number of hidden units approaches infinity.
• When the network weights have bounded prior variance, the scaling limit is a Gaussian process, which simplifies posterior inference and uncertainty quantification.
• However, when the weights have unbounded variance, the scaling limit is an α-stable process, posing unique challenges for posterior inference.
• The authors propose an efficient procedure for posterior inference under this non-Gaussian scaling limit, using a conditionally Gaussian representation.

Plain English Explanation

Bayesian neural networks are a type of machine learning model that can quantify the uncertainty in their predictions. As the number of hidden units in these networks gets very large, their behavior can be described by a mathematical process.

When the weights (the parameters that determine the network's behavior) have a bounded amount of variation, this limiting process is a Gaussian process. Gaussian processes have handy mathematical properties that make it easy to analyze the network's behavior and uncertainties.

But if the weights have an unbounded amount of variation, the limiting process is not a Gaussian process. Instead, it's a different type of mathematical process called an α-stable process. These α-stable processes are more challenging to work with.

The authors of this paper developed a new method that can still perform efficient and interpretable analysis of Bayesian neural networks, even when the weights have unbounded variation and the limiting process is non-Gaussian. Their key insight was to represent the non-Gaussian process in a special way that allows them to leverage the tools of Gaussian processes.

Technical Explanation

The paper builds on the classical work of Neal (1996), which showed that the infinite width scaling limit of a Bayesian neural network with one hidden layer is a Gaussian process, when the network weights have bounded prior variance.

This result has been extended to networks with multiple hidden layers and convolutional neural networks, all of which have Gaussian process scaling limits. The tractable properties of Gaussian processes then simplify posterior inference and uncertainty quantification, compared to analyzing a network of finite width.

However, when the network weights have unbounded variance, the classical central limit theorem breaks down. In this case, the scaling limit is an α-stable process under suitable conditions, as shown in recent work.

While forward simulations under these α-stable processes have been studied, the problem of posterior inference remains largely unaddressed, unlike in the Gaussian process case. To address this, the authors propose an interpretable and computationally efficient procedure for posterior inference, using a conditionally Gaussian representation.

This allows them to leverage the machinery of Gaussian processes to perform tractable posterior inference and uncertainty quantification, even in the non-Gaussian regime.

Critical Analysis

The paper provides a valuable contribution by addressing the challenge of posterior inference under the non-Gaussian scaling limit of Bayesian neural networks with unbounded weight variances.

While forward simulations under α-stable processes have been explored, the authors rightly note that the problem of posterior inference in this setting has remained largely unaddressed. Their proposed method of using a conditionally Gaussian representation is an ingenious solution that allows them to harness the well-understood tools of Gaussian processes.

One potential limitation is that the method may be sensitive to the specific choice of the conditional Gaussian representation. The authors acknowledge this and mention that further research is needed to fully understand the properties and limitations of their approach.

Additionally, the paper focuses on the theoretical properties of the scaling limit and the proposed inference method, but does not provide extensive empirical evaluations. It would be valuable to see how the method performs on realistic benchmark tasks, especially in comparison to other approaches for Bayesian inference in deep neural networks, such as those explored in other papers in this area.

Overall, this paper makes an important contribution by addressing a significant gap in the literature and providing a promising solution for posterior inference in Bayesian neural networks with non-Gaussian scaling limits. Further empirical validation and exploration of the method's properties would help solidify its value and potential impact.

Conclusion

This paper tackles the challenge of posterior inference in Bayesian neural networks with unbounded weight variances, where the scaling limit is a non-Gaussian α-stable process. By proposing an efficient procedure that leverages a conditionally Gaussian representation, the authors enable the use of well-understood Gaussian process tools for tractable posterior inference and uncertainty quantification in this setting.

This work represents an important step forward in understanding and analyzing the behavior of Bayesian neural networks in the limit of infinitely many hidden units. The authors' approach has the potential to significantly simplify the study of these powerful machine learning models, with implications for a wide range of applications where accurate uncertainty quantification is crucial.