learning units, such as LSTMs and GRUs, on tasks including image classification (Figure 1c, upper part), language modelling, music prediction and weather prediction. Importantly, we demonstrated that although SNUs have the lowest number of parameters, they can surpass the accuracy of these units and provide a significant increase in speed (Figure 1c, lower part). Our results established the SNN state of the art with the best accuracy of 99.53% +/0.03%, for the handwritten digit classification task based on the rate-coded MNIST dataset using a convolutional neural network. Moreover, we even demonstrated the first-of-a-kind temporal generative adversarial network (GAN) based on an SNN.
The increasing adoption of ANNs has sparked the development of hardware accelerators to speed up the required computations. Because SNUs cast the dynamics of spiking neurons into the deep learning framework, they provide a systematic methodology for training SNNs using such AI accelerators. We demonstrated this with a highly efficient in-memory computing concept based on nanoscale phase-change memory devices (Figure 1d), where the spiking nature of SNNs leads to further energy savings [3]. Furthermore, we showed that SNNs are robust to operation with low-precision synaptic weights down to 4-bits of precision and can also cope with hardware imperfections such as noise and drift.
Neural circuits in the human brain exhibit an additional set of functionalities observed on the neural dynamics level, where adaptive spiking thresholds play an important role as well as on the architectural level, where lateral inhibition between neurons is a commonly observed theme. Another advantage of the SNUs is that the developed framework can easily be extended to incorporate such neural functionalities. For example, we demonstrated that the LISNU variant, that implements the lateral inhibition, achieves digit classification performance that is on par with the original SNU while significantly reducing the required number of spikes.
Our work on SNU has bridged the ANN and the SNN worlds by incorporating biologically inspired neural dynamics into deep learning, enabling to benefit from both worlds. SNU allows SNNs to take direct advantage of recent deep learning advances, which enable easy scaling up and training deep SNNs to a high degree of accuracy. From the ANN perspective, SNU implements novel temporal dynamics for machine learning applications with fewer parameters and potentially higher performance than the state-of-the-art units. Finally, hardware accelerators can also benefit from SNUs, as they allow for a highly efficient implementation and
unlock the potential of neuromorphic hardware by training deep SNNs to high accuracy. Our team is continuing the work towards developing the biologically inspired deep learning paradigm by also exploring ways of enhancing the learning algorithms with neuroscientific insights [4]. Link: [L1] https://kwz.me/h5b References: [1] W. Maass: “Networks of spiking neurons: The third generation of neural network models,” Neural Networks, vol. 10, no. 9, pp. 1659–1671, 1997. [2] S. Woźniak, et al.: “Deep learning incorporating biologically inspired neural dynamics and in-memory computing,” Nat Mach Intell, vol. 2, no. 6, pp. 325–336, Jun. 2020. [3] A. Pantazi, et al.: “All-memristive neuromorphic computing with level-tuned neurons,” Nanotechnology, vol. 27, no. 35, p. 355205, 2016. [4] T. Bohnstingl, et al.:, “Online Spatio-Temporal Learning in Deep Neural Networks,” https://arxiv.org/abs/2007.12723 Please contact: Stanisław Woźniak, IBM Research – Europe, Switzerland stw@zurich.ibm.com
Effective and Efficient Spiking Recurrent Neural Networks by Bojian Yin (CWI), Federico Corradi (IMEC Holst Centre) and Sander Bohté (CWI) Although inspired by biological brains, the neural networks that are the foundation of modern artificial intelligence (AI) use exponentially more energy than their counterparts in nature, and many local “edge” applications are energy constrained. New learning frameworks and more detailed, spiking neural models can be used to train high-performance spiking neural networks (SNNs) for significant and complex tasks, like speech recognition and EGC-analysis, where the spiking neurons communicate only sparingly. Theoretically, these networks outperform the energy efficiency of comparable classical artificial neural networks by two or three orders of magnitude. Modern deep neural networks have only vague similarities with the functioning of biological neurons in animals. As illustrated in Figure 1 (left), many details of biological neurons are abstracted in this model of neural computation, such as the spatial extent of real neurons, the variable nature and effect of real connections ERCIM NEWS 125 April 2021
formed by synapses, and the means of communication: real neurons communicate not with analogue values, but with isomorphic pulses, or spikes, and they do so only sparingly. As far back as 1997, Maass argued that mathematically, networks of spiking
neurons could be constructed that would be at least as powerful as similar sized networks of standard analogue artificial neurons [1]. Additionally, the sparse nature of spike-based communication is likely key to the energy-efficiency of biological neuronal networks, an increasingly desirable property as artifi9
