Hardware-aware training of models with synaptic delays for digital event-driven neuromorphic processors
2404.10597
0
0
Abstract
Configurable synaptic delays are a basic feature in many neuromorphic neural network hardware accelerators. However, they have been rarely used in model implementations, despite their promising impact on performance and efficiency in tasks that exhibit complex (temporal) dynamics, as it has been unclear how to optimize them. In this work, we propose a framework to train and deploy, in digital neuromorphic hardware, highly performing spiking neural network models (SNNs) where apart from the synaptic weights, the per-synapse delays are also co-optimized. Leveraging spike-based back-propagation-through-time, the training accounts for both platform constraints, such as synaptic weight precision and the total number of parameters per core, as a function of the network size. In addition, a delay pruning technique is used to reduce memory footprint with a low cost in performance. We evaluate trained models in two neuromorphic digital hardware platforms: Intel Loihi and Imec Seneca. Loihi offers synaptic delay support using the so-called Ring-Buffer hardware structure. Seneca does not provide native hardware support for synaptic delays. A second contribution of this paper is therefore a novel area- and memory-efficient hardware structure for acceleration of synaptic delays, which we have integrated in Seneca. The evaluated benchmark involves several models for solving the SHD (Spiking Heidelberg Digits) classification task, where minimal accuracy degradation during the transition from software to hardware is demonstrated. To our knowledge, this is the first work showcasing how to train and deploy hardware-aware models parameterized with synaptic delays, on multicore neuromorphic hardware accelerators.
Create account to get full access
Overview
- This paper presents a hardware-aware approach for training spiking neural network models with synaptic delays for digital event-driven neuromorphic processors like the Loihi.
- The authors demonstrate how incorporating synaptic delays during training can improve the performance of spiking neural networks on temporal signal analysis tasks, such as the Spiking Heidelberg Digits dataset.
- The proposed training framework leverages the SpikeExplorer tool to optimize the spiking neural network architecture for the target neuromorphic hardware.
Plain English Explanation
Neuromorphic processors, like the Loihi, are specialized computer chips that are designed to work more like the human brain than traditional processors. They use "spiking" neural networks, which send information in the form of rapid pulses or "spikes" rather than continuous values.
One challenge in using spiking neural networks is that the time it takes for a signal to travel between neurons (called "synaptic delays") can have a big impact on the network's performance. This paper presents a way to train spiking neural network models that takes these delays into account.
The key idea is to incorporate the synaptic delays during the training process, rather than just adding them in after the model is trained. This allows the network to learn to use the timing information more effectively. The authors demonstrate that this approach can improve the performance of spiking neural networks on tasks that involve analyzing temporal signals, such as the Spiking Heidelberg Digits dataset.
To make sure the trained models work well on actual neuromorphic hardware, the researchers also use a tool called SpikeExplorer to optimize the network architecture for the target processor, in this case the Loihi.
Technical Explanation
The paper presents a hardware-aware training framework for spiking neural networks with synaptic delays, targeting digital event-driven neuromorphic processors like the Loihi. The key contributions are:
-
Synaptic Delay Incorporation: The authors incorporate synaptic delays directly into the training process, allowing the spiking neural network to learn to leverage temporal information more effectively.
-
Hardware-Aware Optimization: The researchers use the SpikeExplorer tool to optimize the spiking neural network architecture for the target neuromorphic hardware, in this case the Loihi processor.
-
Temporal Signal Analysis Evaluation: The authors evaluate their approach on the Spiking Heidelberg Digits dataset, demonstrating improved performance compared to baseline spiking neural network models.
The training framework is based on the SpikeGrad algorithm, which enables gradient-based optimization of spiking neural networks. The authors extend this approach to incorporate synaptic delays and leverage the SpikeExplorer tool for hardware-aware architecture search and optimization.
Critical Analysis
The paper presents a promising approach for training spiking neural networks with synaptic delays for digital event-driven neuromorphic processors. The authors demonstrate the benefits of this hardware-aware training framework on a temporal signal analysis task, showing improved performance compared to baseline models.
However, the paper does not address the scalability of the proposed approach to larger and more complex spiking neural network architectures. The SpikeExplorer tool used for hardware-aware optimization may face challenges as the network size and complexity increases.
Additionally, the paper focuses on a specific neuromorphic processor, the Loihi, and it would be valuable to see how the training framework performs on other neuromorphic hardware platforms, as they may have different constraints and requirements.
Further research could also explore the generalization of the proposed approach to a wider range of temporal signal analysis tasks and datasets, as well as its potential for other applications beyond classification, such as time-series forecasting or control tasks.
Conclusion
This paper presents a hardware-aware training framework for spiking neural networks with synaptic delays, targeting digital event-driven neuromorphic processors like the Loihi. By incorporating synaptic delays into the training process and leveraging the SpikeExplorer tool for hardware-aware optimization, the authors demonstrate improved performance on temporal signal analysis tasks, such as the Spiking Heidelberg Digits dataset.
This work contributes to the ongoing efforts to develop efficient and effective spiking neural network models for neuromorphic hardware, which have the potential to enable low-power, event-driven artificial intelligence at the edge. Further research is needed to explore the scalability, generalization, and broader applications of this hardware-aware training approach.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
Temporal Spiking Neural Networks with Synaptic Delay for Graph Reasoning
Mingqing Xiao, Yixin Zhu, Di He, Zhouchen Lin
0
0
Spiking neural networks (SNNs) are investigated as biologically inspired models of neural computation, distinguished by their computational capability and energy efficiency due to precise spiking times and sparse spikes with event-driven computation. A significant question is how SNNs can emulate human-like graph-based reasoning of concepts and relations, especially leveraging the temporal domain optimally. This paper reveals that SNNs, when amalgamated with synaptic delay and temporal coding, are proficient in executing (knowledge) graph reasoning. It is elucidated that spiking time can function as an additional dimension to encode relation properties via a neural-generalized path formulation. Empirical results highlight the efficacy of temporal delay in relation processing and showcase exemplary performance in diverse graph reasoning tasks. The spiking model is theoretically estimated to achieve $20times$ energy savings compared to non-spiking counterparts, deepening insights into the capabilities and potential of biologically inspired SNNs for efficient reasoning. The code is available at https://github.com/pkuxmq/GRSNN.
5/28/2024
📶
DelGrad: Exact gradients in spiking networks for learning transmission delays and weights
Julian Goltz, Jimmy Weber, Laura Kriener, Peter Lake, Melika Payvand, Mihai A. Petrovici
0
0
Spiking neural networks (SNNs) inherently rely on the timing of signals for representing and processing information. Transmission delays play an important role in shaping these temporal characteristics. Recent work has demonstrated the substantial advantages of learning these delays along with synaptic weights, both in terms of accuracy and memory efficiency. However, these approaches suffer from drawbacks in terms of precision and efficiency, as they operate in discrete time and with approximate gradients, while also requiring membrane potential recordings for calculating parameter updates. To alleviate these issues, we propose an analytical approach for calculating exact loss gradients with respect to both synaptic weights and delays in an event-based fashion. The inclusion of delays emerges naturally within our proposed formalism, enriching the model's search space with a temporal dimension. Our algorithm is purely based on the timing of individual spikes and does not require access to other variables such as membrane potentials. We explicitly compare the impact on accuracy and parameter efficiency of different types of delays - axonal, dendritic and synaptic. Furthermore, while previous work on learnable delays in SNNs has been mostly confined to software simulations, we demonstrate the functionality and benefits of our approach on the BrainScaleS-2 neuromorphic platform.
5/1/2024
SpikePipe: Accelerated Training of Spiking Neural Networks via Inter-Layer Pipelining and Multiprocessor Scheduling
Sai Sanjeet, Bibhu Datta Sahoo, Keshab K. Parhi
0
0
Spiking Neural Networks (SNNs) have gained popularity due to their high energy efficiency. Prior works have proposed various methods for training SNNs, including backpropagation-based methods. Training SNNs is computationally expensive compared to their conventional counterparts and would benefit from multiprocessor hardware acceleration. This is the first paper to propose inter-layer pipelining to accelerate training in SNNs using systolic array-based processors and multiprocessor scheduling. The impact of training using delayed gradients is observed using three networks training on different datasets, showing no degradation for small networks and < 10% degradation for large networks. The mapping of various training tasks of the SNN onto systolic arrays is formulated, and the proposed scheduling method is evaluated on the three networks. The results are compared against standard pipelining algorithms. The results show that the proposed method achieves an average speedup of 1.6X compared to standard pipelining algorithms, with an upwards of 2X improvement in some cases. The incurred communication overhead due to the proposed method is less than 0.5% of the total required communication of training.
6/12/2024
Ultra-low-power Image Classification on Neuromorphic Hardware
Gregor Lenz, Garrick Orchard, Sadique Sheik
0
0
Spiking neural networks (SNNs) promise ultra-low-power applications by exploiting temporal and spatial sparsity. The number of binary activations, called spikes, is proportional to the power consumed when executed on neuromorphic hardware. Training such SNNs using backpropagation through time for vision tasks that rely mainly on spatial features is computationally costly. Training a stateless artificial neural network (ANN) to then convert the weights to an SNN is a straightforward alternative when it comes to image recognition datasets. Most conversion methods rely on rate coding in the SNN to represent ANN activation, which uses enormous amounts of spikes and, therefore, energy to encode information. Recently, temporal conversion methods have shown promising results requiring significantly fewer spikes per neuron, but sometimes complex neuron models. We propose a temporal ANN-to-SNN conversion method, which we call Quartz, that is based on the time to first spike (TTFS). Quartz achieves high classification accuracy and can be easily implemented on neuromorphic hardware while using the least amount of synaptic operations and memory accesses. It incurs a cost of two additional synapses per neuron compared to previous temporal conversion methods, which are readily available on neuromorphic hardware. We benchmark Quartz on MNIST, CIFAR10, and ImageNet in simulation to show the benefits of our method and follow up with an implementation on Loihi, a neuromorphic chip by Intel. We provide evidence that temporal coding has advantages in terms of power consumption, throughput, and latency for similar classification accuracy. Our code and models are publicly available.
6/26/2024