Converting Static Image Datasets to Spiking Neuromorphic Datasets Using Saccades

Abstract

Creating datasets for Neuromorphic Vision is a challenging task. A lack of available recordings from Neuromorphic Vision sensors means that data must typically be recorded specifically for dataset creation rather than collecting and labeling existing data. The task is further complicated by a desire to simultaneously provide traditional frame-based recordings to allow for direct comparison with traditional Computer Vision algorithms. Here we propose a method for converting existing Computer Vision static image datasets into Neuromorphic Vision datasets using an actuated pan-tilt camera platform. Moving the sensor rather than the scene or image is a more biologically realistic approach to sensing and eliminates timing artifacts introduced by monitor updates when simulating motion on a computer monitor. We present conversion of two popular image datasets (MNIST and Caltech101) which have played important roles in the development of Computer Vision, and we provide performance metrics on these datasets using spike-based recognition algorithms. This work contributes datasets for future use in the field, as well as results from spike-based algorithms against which future works can compare. Furthermore, by converting datasets already popular in Computer Vision, we enable more direct comparison with frame-based approaches.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 20

1. 1.

   Engbert R. (2006 ). Microsaccades: a microcosm for research on oculomotor control, attention, and visual perception. Prog. Brain Res. 154, 177–192.

   Not in our library10.1016/s0079-6123(06)54009-9

2. 2.

   Fei-Fei L. Fergus R. Perona P. (2007 ). Learning generative visual models from few training examples: an incremental Bayesian approach tested on 101 object categories. Comput. Vis. Image Underst. 106, 59–70.

   Not in our library10.1016/j.cviu.2005.09.012

3. 3.

   He K. Zhang X. Ren S. Sun J. (2015 ). Spatial pyramid pooling in deep convolutional networks for visual recognition, in Pattern Analysis and Machine Intelligence, IEEE Transactions on, Vol. 37, (IEEE), 1904–1916. Available online at: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=arnumber=7005506isnumber=7175116

   Not in our library10.1109/tpami.2015.2389824

4. 4.

   Kotsiantis S. Kanellopoulos D. Pintelas P. (2006 ). Handling imbalanced datasets: a review. GESTS Int. Trans. Comput. Sci. Eng. 30, 25–36.

   Not in our library

5. 5.

   Lecun Y. Bottou L. Bengio Y. Haffner P. (1998 ). Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278–2324.

   Not in our library10.1109/5.726791

6. 6.

   Lichtsteiner P. Posch C. Delbruck T. (2008 ). A 128x128 120 dB 15 us latency asynchronous temporal contrast vision sensor. Solid State Circ. IEEE J. 43, 566–576.

   Not in our library10.1109/jssc.2007.914337

7. 7.
   Unsupervised Learning of Visual Features through Spike Timing Dependent Plasticity · 2007

   No notice
8. 8.
   Real-Time Classification and Sensor Fusion with a Spiking Deep Belief Network · 2013

   Not checked
9. 9.

   Orchard G. Meyer C. Etienne-Cummings R. Posch C. Thakor N. Benosman R. (2015 ). Hfirst: a temporal approach to object recognition. Pattern Anal. Mach. Intell. IEEE Trans. 37, 2028–2040.

   Not in our library10.1109/tpami.2015.2392947

10. 10.

    Pérez-Carrasco J. A. Zhao B. Serrano C. Acha B. N. Serrano-Gotarredona T. Chen S.. (2013 ). Mapping from frame-driven to frame-free event-driven vision systems by low-rate rate coding and coincidence processing–application to feedforward ConvNets. IEEE Trans. Pattern Anal. Mach. Intell. 35, 2706–2719.

    Not in our library10.1109/tpami.2013.71

11. 11.

    Posch C. Matolin D. Wohlgenannt R. (2011 ). A QVGA 143 dB dynamic range frame-free PWM image sensor With lossless pixel-level video compression and time-domain CDS. Solid State Circ. IEEE J. 46, 259–275.

    Not in our library10.1109/jssc.2010.2085952

12. 12.

    Posch C. Serrano-Gotarredona T. Linares-Barranco B. Delbruck T. (2014 ). Retinomorphic event-based vision sensors: bioinspired cameras with spiking output. Proc. IEEE 102, 1470–1484.

    Not in our library10.1109/jproc.2014.2346153

13. 13.

    Serrano-Gotarredona T. Linares-Barranco B. (2013 ). A 128, 128 1.5% contrast sensitivity 0.9% FPN 3 μs latency 4 mW asynchronous frame-free dynamic vision sensor using transimpedance preamplifiers. IEEE J. Solid State Circ. 48, 827–838.

    Not in our library10.1109/jssc.2012.2230553

14. 14.

    Serre T. Wolf L. Bileschi S. Riesenhuber M. Poggio T. (2007 ). Robust object recognition with cortex-like mechanisms. IEEE Trans. Pattern Anal. Mach. Intell. 29, 411–426.

    Not in our library10.1109/tpami.2007.56

15. 15.
    Benchmarking Neuromorphic Vision: Lessons Learnt from Computer Vision · 2015

    Not checked
16. 16.

    Tapson J. C. Cohen G. K. Afshar S. Stiefel K. M. Buskila Y. Wang R. M.. (2013 ). Synthesis of neural networks for spatio-temporal spike pattern recognition and processing. Front. Neurosci. 7:153.

    Not in our library10.3389/fnins.2013.00153

17. 17.

    Thomee B. Shamma D. A. Friedland G. Elizalde B. Ni K. Poland D.. (2015 ). The new data and new challenges in multimedia research. arXiv:1503.01817. 26427668

    Not in our library

18. 18.

    Torralba A. Efros A. A. (2011 ). Unbiased look at dataset bias, in Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on (Providence, RI: IEEE), 1521–1528.

    Not in our library

19. 19.

    Wan L. Zeiler M. Zhang S. Cun Y. L. Fergus R. (2013 ). Regularization of neural networks using dropconnect, in Proceedings of the 30th International Conference on Machine Learning (ICML-13) (Atlanta, GA), 1058–1066.

    Not in our library

20. 20.

    Zhao B. Ding R. Chen S. Linares-Barranco B. Tang H. (2014 ). Feedforward categorization on AER motion events using cortex-Like features in a spiking neural network. IEEE Trans. Neural Netw. Learn. Syst. 26, 1963–1978.

    Not in our library10.1109/tnnls.2014.2362542

Cited by papers we hold 5

• Spiking Neural Networks and Their Applications: A Review · 2022
  No notice
• Neural heterogeneity promotes robust learning · 2021
  No notice
• Deep Learning With Spiking Neurons: Opportunities and Challenges · 2018
  No notice
• Spatio-Temporal Backpropagation for Training High-Performance Spiking Neural Networks · 2018
  No notice
• Training Deep Spiking Neural Networks Using Backpropagation · 2016
  No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.