Spatio-Temporal Backpropagation for Training High-Performance Spiking Neural Networks

Abstract

Spiking neural networks (SNNs) are promising in ascertaining brain-like behaviors since spikes are capable of encoding spatio-temporal information. Recent schemes, e.g., pre-training from artificial neural networks (ANNs) or direct training based on backpropagation (BP), make the high-performance supervised training of SNNs possible. However, these methods primarily fasten more attention on its spatial domain information, and the dynamics in temporal domain are attached less significance. Consequently, this might lead to the performance bottleneck, and scores of training techniques shall be additionally required. Another underlying problem is that the spike activity is naturally non-differentiable, raising more difficulties in supervised training of SNNs. In this paper, we propose a spatio-temporal backpropagation (STBP) algorithm for training high-performance SNNs. In order to solve the non-differentiable problem of SNNs, an approximated derivative for spike activity is proposed, being appropriate for gradient descent training. The STBP algorithm combines the layer-by-layer spatial domain (SD) and the timing-dependent temporal domain (TD), and does not require any additional complicated skill. We evaluate this method through adopting both the fully connected and convolutional architecture on the static MNIST dataset, a custom object detection dataset, and the dynamic N-MNIST dataset. Results bespeak that our approach achieves the best accuracy compared with existing state-of-the-art algorithms on spiking networks. This work provides a new perspective to investigate the high-performance SNNs for future brain-like computing paradigm with rich spatio-temporal dynamics.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 43

1. 1.

   Allen J. N. Abdel-Aty-Zohdy H. S. Ewing R. L. (2009 ). Cognitive processing using spiking neural networks, in IEEE 2009 National Aerospace and Electronics Conference (Dayton, OH), 56–64.

   Not in our library

2. 2.

   Bengio Y. Mesnard T. Fischer A. Zhang S. Wu Y. (2015 ). An objective function for stdp. Comput. Sci. preprint arXiv.

   Not in our library

3. 3.

   Benjamin B. V. Gao P. Mcquinn E. Choudhary S. Chandrasekaran A. R. Bussat J. M.. (2014 ). Neurogrid: a mixed-analog-digital multichip system for large-scale neural simulations. Proc. IEEE 102, 699–716. 7466396

   Not in our library10.1109/jproc.2014.2313565

4. 4.

   Bohte S. M. Kok J. N. Poutr J. A. L. (2000 ). Spikeprop: backpropagation for networks of spiking neurons, in Esann 2000, European Symposium on Artificial Neural Networks, Bruges, Belgium, April 26-28, 2000, Proceedings (Bruges), 419–424.

   Not in our library

5. 5.

   Chung J. Gulcehre C. Cho K. Bengio Y. (2015 ). Gated feedback recurrent neural networks. Comput. Sci. 2067–2075. preprint arXiv.16623689

   Not in our library

6. 6.

   Cohen G. K. Orchard G. Leng S. H. Tapson J. Benosman R. B. Schaik A. V. (2016 ). Skimming digits: neuromorphic classification of spike-encoded images. Front. Neurosci. 10:184.

   Not in our library10.3389/fnins.2016.00184

7. 7.

   Davis K. H. Biddulph R. Balashek S. (1952 ). Automatic recognition of spoken digits. J. Acoust. Soc. Am. 24:637. 3226132

   Not in our library10.1121/1.1906946

8. 8.
   Unsupervised learning of digit recognition using spike-timing-dependent plasticity · 2015

   No notice
9. 9.

   Diehl P. U. Neil D. Binas J. Cook M. (2015 ). Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing in International Joint Conference on Neural Networks (Killarney), 1–8.

   Not in our library

10. 10.

    Esser S. K. Merolla P. A. Arthur J. V. Cassidy A. S. Appuswamy R. Andreopoulos A.. (2016 ). Convolutional networks for fast, energy-efficient neuromorphic computing. Proc. Natl. Acad. Sci. U.S.A. 113, 11441–11446.

    Not in our library10.1073/pnas.1604850113

11. 11.

    Furber S. B. Galluppi F. Temple S. Plana L. A. (2014 ). The spinnaker project. Proc. IEEE 102, 652–665.

    Not in our library10.1109/jproc.2014.2304638

12. 12.

    Garofolo J. S. Lamel L. F. Fisher W. M. Fiscus J. G. Pallett D. S. Dahlgren N. L. (1993 ). DARPA TIMIT Acoustic-Phonetic Continous Speech Corpus CD-ROM. NIST Speech Disc 1-1.1. NASA STI/Recon Technical Report n 93.

    Not in our library

13. 13.

    Gers F. A. Schmidhuber J. Cummins F. (1999 ). Learning to forget: continual prediction with lstm. Neural Comput. 12, 2451–2571.

    Not in our library10.1162/089976600300015015

14. 14.

    Gtig R. Sompolinsky H. (2006 ). The tempotron: a neuron that learns spike timing-based decisions. Nat. Neurosci. 9:420–428.

    Not in our library10.1038/nn1643

15. 15.

    Hochreiter S. Schmidhuber J. (1997 ). Long short-term memory. Neural Comput. 9, 1735–1780.

    Not in our library10.1162/neco.1997.9.8.1735

16. 16.

    Hunsberger E. Eliasmith C. (2015 ). Spiking deep networks with lif neurons. Comput. Sci. preprint arXiv.

    Not in our library

17. 17.

    Hwu T. Isbell J. Oros N. Krichmar J. (2016 ). A self-driving robot using deep convolutional neural networks on neuromorphic hardware. arXiv.org.25540240

    Not in our library

18. 18.

    Kasabov N. Capecci E. (2015 ). Spiking neural network methodology for modelling, classification and understanding of eeg spatio-temporal data measuring cognitive processes. Infor. Sci. 294, 565–575.

    Not in our library10.1016/j.ins.2014.06.028

19. 19.

    Kheradpisheh S. R. Ganjtabesh M. Masquelier T. (2016 ). Bio-inspired unsupervised learning of visual features leads to robust invariant object recognition. Neurocomputing 205, 382–392.

    Not in our library10.1016/j.neucom.2016.04.029

20. 20.

    Kingma D. Ba J. (2015 ). Adam: A method for stochastic optimization, in International Conference on Learning Representations (ICLR).24045873

    Not in our library

21. 21.

    Lecun Y. Bottou L. Bengio Y. Haffner P. (1998 ). Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278–2324. 25791431

    Not in our library10.1109/5.726791

22. 22.
    Training Deep Spiking Neural Networks Using Backpropagation · 2016

    No notice
23. 23.

    Lichtsteiner P. Posch C. Delbruck T. (2007 ). A 128x128 120db 15us latency asynchronous temporal contrast vision sensor. IEEE J. Solid State Circ. 43, 566–576. 27987525

    Not in our library10.1109/jssc.2007.914337

24. 24.

    Mckennoch S. Liu D. Bushnell L. G. (2006 ). Fast modifications of the spikeprop algorithm, in IEEE International Joint Conference on Neural Network Proceedings (Vancouver, BC), 3970–3977.

    Not in our library

25. 25.

    Merolla P. A. Arthur J. V. Alvarezicaza R. Cassidy A. S. Sawada J. Akopyan F.. (2014 ). Artificial brains. A million spiking-neuron integrated circuit with a scalable communication network and interface. Science 345, 668–673.

    Not in our library10.1126/science.1254642

26. 26.

    Mozafari M. Kheradpisheh S. R. Masquelier T. Nowzaridalini A. Ganjtabesh M. (2017 ). First-spike based visual categorization using reward-modulated stdp. preprint arXiv.

    Not in our library10.1109/tnnls.2018.2826721

27. 27.
    Event-Driven Contrastive Divergence for Spiking Neuromorphic Systems · 2014

    Not checked
28. 28.

    Neil D. Liu S. C. (2016 ). Effective sensor fusion with event-based sensors and deep network architectures, in IEEE International Symposium on Circuits and Systems, ed René Levesque O. (Montréal, QC).

    Not in our library

29. 29.

    Neil D. Pfeiffer M. Liu S. C. (2016 ). Phased lstm: accelerating recurrent network training for long or event-based sequences. arXiv.org.10517633

    Not in our library

30. 30.

    O'Connor P. Welling M. (2016 ). Deep spiking networks. arXiv.org.25255458

    Not in our library

31. 31.
    Converting Static Image Datasets to Spiking Neuromorphic Datasets Using Saccades · 2015

    No notice
32. 32.
    Real-Time Classification and Sensor Fusion with a Spiking Deep Belief Network · 2013

    Not checked
33. 33.

    Ponulak F. (2005 ). ReSuMe-New Supervised Learning Method for Spiking Neural Networks[J]. Institute of Control and Information Engineering, Poznan University of Technology.

    Not in our library

34. 34.

    Ponulak F. Kasiski A. (2010 ). Supervised learning in spiking neural networks with resume: sequence learning, classification, and spike shifting. Neural Comput. 22, 467–510.

    Not in our library10.1162/neco.2009.11-08-901

35. 35.

    Querlioz D. Bichler O. Dollfus P. Gamrat C. (2013 ). Immunity to device variations in a spiking neural network with memristive nanodevices. IEEE Trans. Nanotechnol. 12, 288–295.

    Not in our library10.1109/tnano.2013.2250995

36. 36.

    Schrauwen B. Campenhout J. V. (2004 ). Extending spikeprop. arXiv.org 1, 471–475.

    Not in our library

37. 37.

    Simard P. Y. Steinkraus D. Platt J. C. (2003 ). Best practices for convolutional neural networks applied to visual document analysis, in International Conference on Document Analysis and Recognition (Edinburgh), 958.

    Not in our library

38. 38.

    Tavanaei A. Maida A. S. (2017 ). Bio-inspired spiking convolutional neural network using layer-wise sparse coding and stdp learning. preprint arXiv.

    Not in our library

39. 39.

    Urbanczik R. Senn W. (2009 ). A gradient learning rule for the tempotron. Neural Comput. 21, 340–352.

    Not in our library10.1162/neco.2008.09-07-605

40. 40.

    Vincent P. Larochelle H. Bengio Y. Manzagol P. A. (2008 ). Extracting and composing robust features with denoising autoencoders, in International Conference on Machine Learning (Helsinki), 1096–1103.

    Not in our library

Cited by papers we hold 1

• Deep Learning With Spiking Neurons: Opportunities and Challenges · 2018
  No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.