Model Matters with Restricted Boltzmann Machines

A restricted Boltzmann machine (RBM) is a type of undirected graphical model with one hidden and one visible layer of variables, where all variables in one layer are conditionally independent given variables in the other layer.  In recent years, RBMs have risen to prominence due to their connection to deep learning.  By treating a hidden layer of one RBM as the visible layer in a second RBM, a deep architecture can be created.  RBMs are thought to thereby have the ability to encode very complex and rich structures in data, making them attractive for supervised learning.  However, the generative behavior of RBMs has been largely unexplored and an examination raises some serious concerns about RBMs as statistical models.  In this talk, we discuss the relationship between RBM parameter specification in the binary case and the tendency to undesirable model properties such as degeneracy, instability and un-interpretability.  We also describe the difficulties that arise in likelihood-based and Bayes fitting of such (highly flexible) models, especially as Gibbs sampling (quasi-Bayes) methods are often advocated for the RBM model structure.