Spiking representation learning for associative memories
2406.03054
0
0
📉
Abstract
Networks of interconnected neurons communicating through spiking signals offer the bedrock of neural computations. Our brains spiking neural networks have the computational capacity to achieve complex pattern recognition and cognitive functions effortlessly. However, solving real-world problems with artificial spiking neural networks (SNNs) has proved to be difficult for a variety of reasons. Crucially, scaling SNNs to large networks and processing large-scale real-world datasets have been challenging, especially when compared to their non-spiking deep learning counterparts. The critical operation that is needed of SNNs is the ability to learn distributed representations from data and use these representations for perceptual, cognitive and memory operations. In this work, we introduce a novel SNN that performs unsupervised representation learning and associative memory operations leveraging Hebbian synaptic and activity-dependent structural plasticity coupled with neuron-units modelled as Poisson spike generators with sparse firing (~1 Hz mean and ~100 Hz maximum firing rate). Crucially, the architecture of our model derives from the neocortical columnar organization and combines feedforward projections for learning hidden representations and recurrent projections for forming associative memories. We evaluated the model on properties relevant for attractor-based associative memories such as pattern completion, perceptual rivalry, distortion resistance, and prototype extraction.
Create account to get full access
Overview
- Spiking neural networks (SNNs) are modeled after the brain's neural networks, with interconnected neurons communicating through spiking signals.
- SNNs have the potential to achieve complex pattern recognition and cognitive functions, but solving real-world problems with them has proven difficult.
- Scaling SNNs to large networks and processing large-scale datasets has been challenging, especially compared to non-spiking deep learning models.
- The critical operation needed for SNNs is the ability to learn distributed representations from data and use these representations for perceptual, cognitive, and memory operations.
Plain English Explanation
The human brain is an amazing biological machine, with networks of interconnected neurons communicating through electrical signals called "spikes." These spiking neural networks (SNNs) have the remarkable ability to recognize patterns and perform complex cognitive tasks with ease, far surpassing the capabilities of traditional computer programs.
However, creating artificial SNN that can tackle real-world problems has proven to be a significant challenge. One of the biggest hurdles is that it's difficult to scale these networks to handle large amounts of data, especially when compared to the more common "non-spiking" deep learning models.
The key to making SNNs useful is to give them the ability to learn useful representations from data and then use those representations to perform various tasks, like perception, cognition, and memory. Imagine a young child learning to recognize different types of animals - they start by noticing simple features like shapes and colors, and gradually build up a more complex understanding of what makes a dog, a cat, or a bird unique. This is the kind of capability that researchers are trying to replicate in artificial SNNs.
Technical Explanation
In this work, the researchers introduce a novel SNN architecture that performs unsupervised representation learning and associative memory operations. The architecture is inspired by the columnar organization of the neocortex and combines feedforward projections for learning hidden representations with recurrent projections for forming associative memories.
The key components of the model are:
- Neurons modeled as Poisson spike generators with sparse firing (around 1 Hz on average, up to 100 Hz maximum)
- Hebbian synaptic plasticity and activity-dependent structural plasticity to learn distributed representations
- Associative memory operations leveraging the recurrent connections
The researchers evaluate the model's performance on properties relevant for attractor-based associative memories, such as pattern completion, perceptual rivalry, distortion resistance, and prototype extraction.
Critical Analysis
The researchers acknowledge that scaling SNNs to large networks and processing large-scale real-world datasets remains a significant challenge, even with their novel architecture. There are still many open questions and areas for further research, such as improving the efficiency of deep SNN construction and exploring more biologically plausible learning rules.
Additionally, the performance evaluation in the paper is relatively limited, focusing mainly on associative memory tasks. It would be valuable to see how the model fares on a wider range of real-world problems to better understand its strengths and limitations.
Overall, the research represents an important step forward in the development of more capable and practical SNNs, but there is still much work to be done before these models can truly rival the performance of traditional deep learning approaches on complex, large-scale tasks.
Conclusion
This paper introduces a novel spiking neural network architecture that combines unsupervised representation learning with associative memory operations, drawing inspiration from the structure and function of the human neocortex. While scaling SNNs to handle large-scale data and real-world problems remains a significant challenge, this work demonstrates the potential of these biologically-inspired models to achieve complex cognitive feats. As the field of spiking neural networks continues to evolve, we can expect to see more breakthroughs that bring us closer to realizing the full computational power of the brain.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
🤷
Unsupervised representation learning with Hebbian synaptic and structural plasticity in brain-like feedforward neural networks
Naresh Ravichandran, Anders Lansner, Pawel Herman
0
0
Neural networks that can capture key principles underlying brain computation offer exciting new opportunities for developing artificial intelligence and brain-like computing algorithms. Such networks remain biologically plausible while leveraging localized forms of synaptic learning rules and modular network architecture found in the neocortex. Compared to backprop-driven deep learning approches, they provide more suitable models for deploying on neuromorphic hardware and have greater potential for scalability on large-scale computing clusters. The development of such brain-like neural networks depends on having a learning procedure that can build effective internal representations from data. In this work, we introduce and evaluate a brain-like neural network model capable of unsupervised representation learning. It builds on the Bayesian Confidence Propagation Neural Network (BCPNN), which has earlier been implemented as abstract as well as biophyscially detailed recurrent attractor neural networks explaining various cortical associative memory phenomena. Here we developed a feedforward BCPNN model to perform representation learning by incorporating a range of brain-like attributes derived from neocortical circuits such as cortical columns, divisive normalization, Hebbian synaptic plasticity, structural plasticity, sparse activity, and sparse patchy connectivity. The model was tested on a diverse set of popular machine learning benchmarks: grayscale images (MNIST, Fashion-MNIST), RGB natural images (SVHN, CIFAR-10), QSAR (MUV, HIV), and malware detection (EMBER). The performance of the model when using a linear classifier to predict the class labels fared competitively with conventional multi-layer perceptrons and other state-of-the-art brain-like neural networks.
6/10/2024
🧠
Evolutionary Spiking Neural Networks: A Survey
Shuaijie Shen, Rui Zhang, Chao Wang, Renzhuo Huang, Aiersi Tuerhong, Qinghai Guo, Zhichao Lu, Jianguo Zhang, Luziwei Leng
0
0
Spiking neural networks (SNNs) are gaining increasing attention as potential computationally efficient alternatives to traditional artificial neural networks(ANNs). However, the unique information propagation mechanisms and the complexity of SNN neuron models pose challenges for adopting traditional methods developed for ANNs to SNNs. These challenges include both weight learning and architecture design. While surrogate gradient learning has shown some success in addressing the former challenge, the latter remains relatively unexplored. Recently, a novel paradigm utilizing evolutionary computation methods has emerged to tackle these challenges. This approach has resulted in the development of a variety of energy-efficient and high-performance SNNs across a wide range of machine learning benchmarks. In this paper, we present a survey of these works and initiate discussions on potential challenges ahead.
6/19/2024
🧠
Stochastic Spiking Neural Networks with First-to-Spike Coding
Yi Jiang, Sen Lu, Abhronil Sengupta
0
0
Spiking Neural Networks (SNNs), recognized as the third generation of neural networks, are known for their bio-plausibility and energy efficiency, especially when implemented on neuromorphic hardware. However, the majority of existing studies on SNNs have concentrated on deterministic neurons with rate coding, a method that incurs substantial computational overhead due to lengthy information integration times and fails to fully harness the brain's probabilistic inference capabilities and temporal dynamics. In this work, we explore the merger of novel computing and information encoding schemes in SNN architectures where we integrate stochastic spiking neuron models with temporal coding techniques. Through extensive benchmarking with other deterministic SNNs and rate-based coding, we investigate the tradeoffs of our proposal in terms of accuracy, inference latency, spiking sparsity, energy consumption, and robustness. Our work is the first to extend the scalability of direct training approaches of stochastic SNNs with temporal encoding to VGG architectures and beyond-MNIST datasets.
5/29/2024
🧠
Spike-based computation using classical recurrent neural networks
Florent De Geeter (Montefiore Institute, University of Li`ege, Li`ege, Belgium), Damien Ernst (Montefiore Institute, University of Li`ege, Li`ege, Belgium, LTCI, T'el'ecom Paris, Institut Polytechnique de Paris, France), Guillaume Drion (Montefiore Institute, University of Li`ege, Li`ege, Belgium)
0
0
Spiking neural networks are a type of artificial neural networks in which communication between neurons is only made of events, also called spikes. This property allows neural networks to make asynchronous and sparse computations and therefore drastically decrease energy consumption when run on specialised hardware. However, training such networks is known to be difficult, mainly due to the non-differentiability of the spike activation, which prevents the use of classical backpropagation. This is because state-of-the-art spiking neural networks are usually derived from biologically-inspired neuron models, to which are applied machine learning methods for training. Nowadays, research about spiking neural networks focuses on the design of training algorithms whose goal is to obtain networks that compete with their non-spiking version on specific tasks. In this paper, we attempt the symmetrical approach: we modify the dynamics of a well-known, easily trainable type of recurrent neural network to make it event-based. This new RNN cell, called the Spiking Recurrent Cell, therefore communicates using events, i.e. spikes, while being completely differentiable. Vanilla backpropagation can thus be used to train any network made of such RNN cell. We show that this new network can achieve performance comparable to other types of spiking networks in the MNIST benchmark and its variants, the Fashion-MNIST and the Neuromorphic-MNIST. Moreover, we show that this new cell makes the training of deep spiking networks achievable.
5/7/2024