Going Deeper in Spiking Neural Networks: VGG and Residual Architectures

Abstract

Over the past few years, Spiking Neural Networks (SNNs) have become popular as a possible pathway to enable low-power event-driven neuromorphic hardware. However, their application in machine learning have largely been limited to very shallow neural network architectures for simple problems. In this paper, we propose a novel algorithmic technique for generating an SNN with a deep architecture, and demonstrate its effectiveness on complex visual recognition problems such as CIFAR-10 and ImageNet. Our technique applies to both VGG and Residual network architectures, with significantly better accuracy than the state-of-the-art. Finally, we present analysis of the sparse event-driven computations to demonstrate reduced hardware overhead when operating in the spiking domain.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 27

1. 1.

   Akopyan F. Sawada J. Cassidy A. Alvarez-Icaza R. Arthur J. Merolla P. (2015 ). TrueNorth: design and tool flow of a 65 mw 1 million neuron programmable neurosynaptic chip. IEEE Trans. Comp. Aided Design Integr. Circ. Syst. 34, 1537–1557.

   Not in our library10.1109/tcad.2015.2474396

2. 2.

   Cao Y. Chen Y. Khosla D. (2015 ). Spiking deep convolutional neural networks for energy-efficient object recognition. Int. J. Comp. Vision 113, 54–66.

   Not in our library10.1007/s11263-014-0788-3

3. 3.

   Chen Z. Johnson M. Wei L. Roy W. (1998 ). Estimation of standby leakage power in CMOS circuit considering accurate modeling of transistor stacks, in Proceedings, International Symposium on Low Power Electronics and Design, 1998 (Monterey, CA: IEEE), 239–244.

   Not in our library

4. 4.
   Unsupervised learning of digit recognition using spike-timing-dependent plasticity · 2015

   No notice
5. 5.

   Diehl P. U. Neil D. Binas J. Cook M. Liu S.-C. Pfeiffer M. (2015 ). Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing, in 2015 International Joint Conference on Neural Networks (IJCNN) (Killarney: IEEE), 1–8.

   Not in our library

6. 6.

   Esser S. K. Merolla P. A. Arthur J. V. Cassidy A. S. Appuswamy R. Andreopoulos A.. (2016 ). Convolutional networks for fast, energy-efficient neuromorphic computing. Proc. Natl. Acad. Sci. U.S.A. 113, 11441–11446.

   Not in our library10.1073/pnas.1604850113

7. 7.

   Farabet C. Paz R. Pérez-Carrasco J. Zamarreño-Ramos C. Linares-Barranco A. LeCun Y.. (2012 ). Comparison between frame-constrained fix-pixel-value and frame-free spiking-dynamic-pixel ConvNets for visual processing. Front. Neurosci. 6:32.

   Not in our library10.3389/fnins.2012.00032

8. 8.

   Han S. Mao H. Dally W. J. (2015a ). Deep compression: compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149.

   Not in our library

9. 9.

   Han S. Pool J. Tran J. Dally W. (2015b ). Learning both weights and connections for efficient neural network, in Advances in Neural Information Processing Systems (Montreal, QC), 1135–1143.

   Not in our library

10. 10.

    Hardt M. Ma T. (2016 ). Identity matters in deep learning. arXiv preprint arXiv:1611.04231.

    Not in our library

11. 11.

    He K. Zhang X. Ren S. Sun J. (2016a ). Deep residual learning for image recognition, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Las Vegas, NV), 770–778.

    Not in our library

12. 12.

    He K. Zhang X. Ren S. Sun J. (2016b ). Identity mappings in deep residual networks, in European Conference on Computer Vision (Springer), 630–645.

    Not in our library

13. 13.

    Hunsberger E. Eliasmith C. (2016 ). Training spiking deep networks for neuromorphic hardware. arXiv preprint arXiv:1611.05141.

    Not in our library

14. 14.

    Ioffe S. Szegedy C. (2015 ). Batch normalization: accelerating deep network training by reducing internal covariate shift, in International Conference on Machine Learning (Lille), 448–456.

    Not in our library

15. 15.

    Izhikevich E. M. (2003 ). Simple model of spiking neurons. IEEE Trans. Neural Netw. 14, 1569–1572.

    Not in our library10.1109/tnn.2003.820440

16. 16.

    Krizhevsky A. Hinton G. (2009 ). Learning multiple layers of features from tiny images. Available online at: https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf

    Not in our library

17. 17.

    Krizhevsky A. Sutskever I. Hinton G. E. (2012 ). ImageNet classification with deep convolutional neural networks, in Advances in Neural Information Processing Systems (Lake Tahoe, NV), 1097–1105.

    Not in our library

18. 18.

    LeCun Y. Bottou L. Bengio Y. Haffner P. (1998 ). Gradient-based learning applied to document recognition. Proceed. IEEE 86, 2278–2324.

    Not in our library10.1109/5.726791

19. 19.

    Merolla P. A. Arthur J. V. Alvarez-Icaza R. Cassidy A. S. Sawada J. Akopyan F.. (2014 ). A million spiking-neuron integrated circuit with a scalable communication network and interface. Science 345, 668–673.

    Not in our library10.1126/science.1254642

20. 20.

    Pérez-Carrasco J. A. Zhao B. Serrano C. Acha B. Serrano-Gotarredona T. Chen S.. (2013 ). Mapping from frame-driven to frame-free event-driven vision systems by low-Rate rate coding and coincidence processing–application to feedforward convNets. IEEE Trans. Pattern Anal. Mach. Intelligence 35, 2706–2719.

    Not in our library10.1109/tpami.2013.71

21. 21.

    Posch C. Matolin D. Wohlgenannt R. (2011 ). A QVGA 143 dB dynamic range frame-free PWM image sensor with lossless pixel-level video compression and time-domain CDS. IEEE J. Solid-State Circ. 46, 259–275.

    Not in our library10.1109/jssc.2010.2085952

22. 22.

    Posch C. Serrano-Gotarredona T. Linares-Barranco B. Delbruck T. (2014 ). Retinomorphic event-based vision sensors: bioinspired cameras with spiking output. Proc. IEEE 102, 1470–1484.

    Not in our library10.1109/jproc.2014.2346153

23. 23.
    Conversion of Continuous-Valued Deep Networks to Efficient Event-Driven Networks for Image Classification · 2017

    No notice
24. 24.

    Russakovsky O. Deng J. Su H. Krause J. Satheesh S. Ma S. (2015 ). ImageNet large scale visual recognition challenge. Int. J. Comp. Visi. 115, 211–252.

    Not in our library10.1007/s11263-015-0816-y

25. 25.

    Simonyan K. Zisserman A. (2014 ). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.

    Not in our library

26. 26.

    Srivastava N. Hinton G. E. Krizhevsky A. Sutskever I. Salakhutdinov R. (2014 ). Dropout: A simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958. Available online at: http://dl.acm.org/citation.cfm?id=2627435.2670313

    Not in our library

27. 27.

    Zhao B. Ding R. Chen S. Linares-Barranco B. Tang H. (2015 ). Feedforward categorization on AER motion events using cortex-like features in a spiking neural network. IEEE Trans. Neural Netw. Learn. Syst. 26, 1963–1978.

    Not in our library10.1109/tnnls.2014.2362542

Cited by papers we hold 4

• Spiking Neural Networks and Their Applications: A Review · 2022
  No notice
• A Meta Learning-Based Approach for Zero-Shot Co-Training · 2021
  Not checked
• Toward Scalable, Efficient, and Accurate Deep Spiking Neural Networks With Backward Residual Connections, Stochastic Softmax, and Hybridization · 2020
  Not checked
• Deep Learning With Spiking Neurons: Opportunities and Challenges · 2018
  No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.