KIP publications

An increasing number of experiments suggest that the brain performs stochastic inference when dealing with incomplete and noisy sensory information. This, in turn, has led to the development of various theoretical models that attempt to explain how this could be achieved with spiking neural networks. One candidate theory interprets spiking activity as sampling from distributions over binary random variables (Buesing et al., 2011) and has been shown to be compatible with the ensemble dynamics of noise-driven LIF neurons in the high-conductance state (Petrovici et al., 2013, 2015; Probst et al., 2015). Based on this theory, we constructed hierarchical LIF networks that sample from restricted Boltzmann distributions and compared their performance with conventional restricted Boltzmann machines (RBMs) on a commonly used dataset (MNIST). An important result is that LIF networks can achieve similar classification rates (95.1 % with 1994 neurons) as their machine-learning counterparts of equal size (95.2 %). In classical RBMs however, statistics are typically gathered by Gibbs sampling. This algorithm has a distinct disadvantage when dealing with high-dimensional multimodal distributions, where it often gets trappedin a local minimum due to deep troughs in the energy landscape that appear during training. It is for this reason that conventional RBMs that may perform very well as discriminative models are, at the same time, rather poor generative models of the learned data. While various methods exist that alleviate this problem (such as AST, see Salakhutdinov, 2010) they usually come at a highly increased computational cost. In the second part of our study, we show how short-term plasticity enables LIF networks to travel efficiently through the energy landscape and
thereby attain a generative performance that far surpasses the one achievable by conventional Gibbs sampling. This distinct advantage of biological neural networks allows them to simultaneously become good generative and discriminative models of learned data.