Towards Efficient Deep Spiking Neural Networks Construction with Spiking Activity based Pruning
2406.01072
0
0
Abstract
The emergence of deep and large-scale spiking neural networks (SNNs) exhibiting high performance across diverse complex datasets has led to a need for compressing network models due to the presence of a significant number of redundant structural units, aiming to more effectively leverage their low-power consumption and biological interpretability advantages. Currently, most model compression techniques for SNNs are based on unstructured pruning of individual connections, which requires specific hardware support. Hence, we propose a structured pruning approach based on the activity levels of convolutional kernels named Spiking Channel Activity-based (SCA) network pruning framework. Inspired by synaptic plasticity mechanisms, our method dynamically adjusts the network's structure by pruning and regenerating convolutional kernels during training, enhancing the model's adaptation to the current target task. While maintaining model performance, this approach refines the network architecture, ultimately reducing computational load and accelerating the inference process. This indicates that structured dynamic sparse learning methods can better facilitate the application of deep SNNs in low-power and high-efficiency scenarios.
Create account to get full access
Overview
- This paper proposes a novel method called "Spiking Activity based Pruning" (SAP) to efficiently construct deep spiking neural networks (SNNs).
- The key idea is to prune the weights of an SNN based on the spiking activity of individual neurons, which can significantly reduce the network size and inference time without compromising accuracy.
- The authors demonstrate the effectiveness of SAP on various SNN benchmarks, showing it can achieve comparable or better performance to state-of-the-art methods with much smaller network sizes.
Plain English Explanation
The paper introduces a new way to build efficient deep spiking neural networks, which are a type of AI model that works more like the human brain compared to traditional neural networks. Spiking neural networks use "spikes" (rapid changes in electrical signals) to transmit information, similar to how neurons in the brain communicate.
The main challenge with spiking neural networks is that they can be large and computationally expensive. The researchers developed a technique called "Spiking Activity based Pruning" (SAP) to address this. SAP looks at which neurons in the network are most active during a task and removes the unimportant connections (weights) between them. This shrinks the size of the network without losing too much accuracy.
The authors tested SAP on several standard benchmarks for spiking neural networks and found it could match or even outperform other state-of-the-art methods, while using much smaller and faster networks. This is an important advance, as efficient spiking neural networks could enable new applications of brain-inspired AI in areas like rapid deployment of DNNs on edge computing or lightweight, efficient spiking neural networks for IoT devices.
Technical Explanation
The paper proposes a novel method called "Spiking Activity based Pruning" (SAP) to efficiently construct deep spiking neural networks (SNNs). The key idea is to prune the weights of an SNN based on the spiking activity of individual neurons, which can significantly reduce the network size and inference time without compromising accuracy.
Specifically, the authors first train an SNN using a direct training method like direct training of high-performance deep spiking neural networks. They then analyze the spiking activity of each neuron during inference on a validation dataset. Neurons with low spiking activity are considered less important, and their outgoing weights are pruned. The pruned network is then fine-tuned to recover any potential accuracy loss.
The authors evaluate SAP on several SNN benchmarks, including image classification on the CIFAR-10 and ImageNet datasets, as well as event-based action recognition on the DvsGesture dataset. They compare SAP to other pruning and quantization methods for SNNs, and show that it can achieve comparable or better performance with much smaller network sizes. For example, on CIFAR-10, SAP can reduce the network size by up to 90% while maintaining the same accuracy as the unpruned model.
The authors also analyze the relationship between weight sparsity and activity sparsity in SNNs, and demonstrate that SAP can capture both types of sparsity effectively.
Critical Analysis
The paper presents a well-designed and thorough study on efficient SNN construction using spiking activity-based pruning. The authors clearly articulate the motivation and novelty of their approach, and provide extensive experimental validation on multiple benchmarks.
One potential limitation is that the authors only consider feedforward SNNs, and do not explore the applicability of SAP to more complex SNN architectures, such as those with recurrent connections. Additionally, the paper does not investigate the robustness of the pruned SNNs to noisy or adversarial inputs, which could be an important consideration for real-world deployment.
The authors also do not compare SAP to other SNN compression techniques, such as stochastic spiking neural networks or structured pruning. Exploring the trade-offs between these different approaches could provide further insights into the strengths and weaknesses of SAP.
Overall, the paper makes a valuable contribution to the growing field of efficient spiking neural networks, and the proposed SAP method appears to be a promising technique for constructing compact and high-performing SNNs. Further research building on these findings could lead to more powerful and practical deployments of brain-inspired AI.
Conclusion
This paper introduces a novel method called "Spiking Activity based Pruning" (SAP) to efficiently construct deep spiking neural networks (SNNs). SAP prunes the weights of an SNN based on the spiking activity of individual neurons, significantly reducing the network size and inference time without compromising accuracy.
The authors demonstrate the effectiveness of SAP on various SNN benchmarks, showing it can achieve comparable or better performance to state-of-the-art methods with much smaller network sizes. This is an important advance, as efficient spiking neural networks could enable new applications of brain-inspired AI in areas like rapid deployment on edge devices and lightweight, efficient SNNs for IoT.
While the paper focuses on feedforward SNNs, further research could explore the applicability of SAP to more complex SNN architectures and investigate the robustness of the pruned SNNs. Comparing SAP to other SNN compression techniques could also provide additional insights. Overall, this work represents a valuable contribution to the field of efficient spiking neural networks and could pave the way for more practical deployments of this promising AI technology.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
š¬
Weight Sparsity Complements Activity Sparsity in Neuromorphic Language Models
Rishav Mukherji, Mark Schone, Khaleelulla Khan Nazeer, Christian Mayr, David Kappel, Anand Subramoney
0
0
Activity and parameter sparsity are two standard methods of making neural networks computationally more efficient. Event-based architectures such as spiking neural networks (SNNs) naturally exhibit activity sparsity, and many methods exist to sparsify their connectivity by pruning weights. While the effect of weight pruning on feed-forward SNNs has been previously studied for computer vision tasks, the effects of pruning for complex sequence tasks like language modeling are less well studied since SNNs have traditionally struggled to achieve meaningful performance on these tasks. Using a recently published SNN-like architecture that works well on small-scale language modeling, we study the effects of weight pruning when combined with activity sparsity. Specifically, we study the trade-off between the multiplicative efficiency gains the combination affords and its effect on task performance for language modeling. To dissect the effects of the two sparsities, we conduct a comparative analysis between densely activated models and sparsely activated event-based models across varying degrees of connectivity sparsity. We demonstrate that sparse activity and sparse connectivity complement each other without a proportional drop in task performance for an event-based neural network trained on the Penn Treebank and WikiText-2 language modeling datasets. Our results suggest sparsely connected event-based neural networks are promising candidates for effective and efficient sequence modeling.
5/2/2024
š
Spiking representation learning for associative memories
Naresh Ravichandran, Anders Lansner, Pawel Herman
0
0
Networks of interconnected neurons communicating through spiking signals offer the bedrock of neural computations. Our brains spiking neural networks have the computational capacity to achieve complex pattern recognition and cognitive functions effortlessly. However, solving real-world problems with artificial spiking neural networks (SNNs) has proved to be difficult for a variety of reasons. Crucially, scaling SNNs to large networks and processing large-scale real-world datasets have been challenging, especially when compared to their non-spiking deep learning counterparts. The critical operation that is needed of SNNs is the ability to learn distributed representations from data and use these representations for perceptual, cognitive and memory operations. In this work, we introduce a novel SNN that performs unsupervised representation learning and associative memory operations leveraging Hebbian synaptic and activity-dependent structural plasticity coupled with neuron-units modelled as Poisson spike generators with sparse firing (~1 Hz mean and ~100 Hz maximum firing rate). Crucially, the architecture of our model derives from the neocortical columnar organization and combines feedforward projections for learning hidden representations and recurrent projections for forming associative memories. We evaluated the model on properties relevant for attractor-based associative memories such as pattern completion, perceptual rivalry, distortion resistance, and prototype extraction.
6/6/2024
On Reducing Activity with Distillation and Regularization for Energy Efficient Spiking Neural Networks
Thomas Louis, Benoit Miramond, Alain Pegatoquet, Adrien Girard
0
0
Interest in spiking neural networks (SNNs) has been growing steadily, promising an energy-efficient alternative to formal neural networks (FNNs), commonly known as artificial neural networks (ANNs). Despite increasing interest, especially for Edge applications, these event-driven neural networks suffered from their difficulty to be trained compared to FNNs. To alleviate this problem, a number of innovative methods have been developed to provide performance more or less equivalent to that of FNNs. However, the spiking activity of a network during inference is usually not considered. While SNNs may usually have performance comparable to that of FNNs, it is often at the cost of an increase of the network's activity, thus limiting the benefit of using them as a more energy-efficient solution. In this paper, we propose to leverage Knowledge Distillation (KD) for SNNs training with surrogate gradient descent in order to optimize the trade-off between performance and spiking activity. Then, after understanding why KD led to an increase in sparsity, we also explored Activations regularization and proposed a novel method with Logits Regularization. These approaches, validated on several datasets, clearly show a reduction in network spiking activity (-26.73% on GSC and -14.32% on CIFAR-10) while preserving accuracy.
6/27/2024
Spiking Convolutional Neural Networks for Text Classification
Changze Lv, Jianhan Xu, Xiaoqing Zheng
0
0
Spiking neural networks (SNNs) offer a promising pathway to implement deep neural networks (DNNs) in a more energy-efficient manner since their neurons are sparsely activated and inferences are event-driven. However, there have been very few works that have demonstrated the efficacy of SNNs in language tasks partially because it is non-trivial to represent words in the forms of spikes and to deal with variable-length texts by SNNs. This work presents a conversion + fine-tuning two-step method for training SNNs for text classification and proposes a simple but effective way to encode pre-trained word embeddings as spike trains. We show empirically that after fine-tuning with surrogate gradients, the converted SNNs achieve comparable results to their DNN counterparts with much less energy consumption across multiple datasets for both English and Chinese. We also show that such SNNs are more robust to adversarial attacks than DNNs.
6/28/2024