Acceleration of Deep Neural Network Training with Resistive Cross-Point Devices: Design Considerations

Abstract

In recent years, deep neural networks (DNN) have demonstrated significant business impact in large scale analysis and classification tasks such as speech recognition, visual object detection, pattern extraction, etc. Training of large DNNs, however, is universally considered as time consuming and computationally intensive task that demands datacenter-scale computational resources recruited for many days. Here we propose a concept of resistive processing unit (RPU) devices that can potentially accelerate DNN training by orders of magnitude while using much less power. The proposed RPU device can store and update the weight values locally thus minimizing data movement during training and allowing to fully exploit the locality and the parallelism of the training algorithm. We evaluate the effect of various RPU device features/non-idealities and system parameters on performance in order to derive the device and system level specifications for implementation of an accelerator chip for DNN training in a realistic CMOS-compatible technology. For large DNNs with about 1 billion weights this massively parallel RPU architecture can achieve acceleration factors of 30, 000 × compared to state-of-the-art microprocessors while providing power efficiency of 84, 000 GigaOps∕s∕W. Problems that currently require days of training on a datacenter-size cluster with thousands of machines can be addressed within hours on a single RPU accelerator. A system consisting of a cluster of RPU accelerators will be able to tackle Big Data problems with trillions of parameters that is impossible to address today like, for example, natural speech recognition and translation between all world languages, real-time analytics on large streams of business and scientific data, integration, and analysis of multimodal sensory data flows from a massive number of IoT (Internet of Things) sensors.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 44

1. 1.

   Alaghi A. Hayes J. (2013 ). Survey of stochastic computing. ACM Trans. Embed. Comput. Syst. 12, 1–19.

   Not in our library10.1145/2465787.2465794 (opens in a new tab)

2. 2.

   Arima Y. Mashiko K. Okada K. Yamada T. Maeda A. Notani H. (1991 ). A 336-neuron, 28 K-synapse, self-learning neural network chip with branch-neuron-unit architecture. IEEE J. Solid State Circuits 26, 1637–1644.

   Not in our library

3. 3.

   Bi G. Q. Poo M. M. (1998 ). Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type. J. Neurosci. 18, 10464–10472.

   Not in our library10.1523/jneurosci.18-24-10464.1998 (opens in a new tab)

4. 4.

   Burr G. W. Shelby R. M. di Nolfo C. Jang J. W. Shenoy R. S. Narayanan P. (2014 ). Experimental demonstration and tolerancing of a large-scale neural network (165,000 synapses), using phase-change memory as the synaptic weight element, in 2014 IEEE International, Electron Devices Meeting (IEDM) (San Francisco, CA).

   Not in our library

5. 5.

   Burr G. W. Narayanan P. Shelby R. M. Sidler S. Boybat I. di Nolfo C. (2015 ). Large-scale neural networks implemented with nonvolatile memory as the synaptic weight element: comparative performance analysis (accuracy, speed, and power), in IEDM (International Electron Devices Meeting) (Washington, DC).

   Not in our library

6. 6.

   Chen G. Anders M. A. Kaul H. Satpathy S. K. Mathew S. K. Hsu S. K. (2014 ). A 340 mV-to-0.9 V 20.2 Tb/s source-synchronous hybrid packet/circuit-switched 16 × 16 network-on-chip in 22 nm tri-gate CMOS, in 2014 IEEE International Soild-State Circuits Conference Digest of techincal Papers (ISSCC) (San Francisco, CA), 276–277.

   Not in our library

7. 7.

   Chen Y. Luo T. Liu S. Zhang S. He L. Wang J.. (2014 ). DaDianNao: a machine-learning supercomputer, in 2014 47th Annual IEEE/ACM International Symposium on Microarchitecture (Cambridge, UK), 609–622.

   Not in our library

8. 8.

   Chua L. O. (1971 ). Memristor - the missing circuit element. IEEE Trans. Circuit Theory 18, 507–519.

   Not in our library10.1109/tct.1971.1083337 (opens in a new tab)

9. 9.

   Coates A. Huval B. Wang T. Wu D. J. Ng A. Y. Catanzaro B. (2013 ). Deep Learning with COTS HPC Systems. Atlanta, GA: ICML.

   Not in our library

10. 10.

    Gaines B. R. (1967 ). Stochastic computing, in Proceedings of the AFIPS Spring Joint Computer Conference (Atlantic City, NJ), 149–156.

    Not in our library

11. 11.

    Gokhale V. Jin J. Dundar A. Martini B. Culurciello E. (2014 ). A 240 g-ops/s mobile coprocessor for deep neural networks, in IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (Columbus, OH), 696–701.

    Not in our library

12. 12.

    Gupta S. Agrawal A. Gopalakrishnan K. Narayanan P. (2015 ). Deep learning with limited numerical precision. arXiv:1502.02551 [cs.LG].

    Not in our library

13. 13.

    Hinton G. Deng L. Yu D. Dahl G. E. Mohamed A. Jaitly N. (2012 ). Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Process. Mag. 29, 82–97.

    Not in our library10.1109/msp.2012.2205597 (opens in a new tab)

14. 14.

    Indiveri G. Linares-Barranco B. Legenstein R. Deligeorgis G. Prodromakis T. (2013 ). Integration of nanoscale memristor synapses in neuromorphic computing architectures. Nanotechnology 24:384010.

    Not in our library10.1088/0957-4484/24/38/384010 (opens in a new tab)

15. 15.

    Jackson B. L. Rajendran B. Corrado G. S. Breitwisch M. Burr G. W. Cheek R. (2013 ). Nanoscale electronic synapses using phase change devices. ACM J. Emerg. Technol. Comput. Syst. 9, 1–20.

    Not in our library10.1145/2463585.2463588 (opens in a new tab)

16. 16.

    Jensen K. Gaudet V. C. Levine P. M. (2013 ). Noise analysis and measurement of integrator-based sensor interface circuits for fluorescence detection in lab-on-a-chip applications, in International Conference on Noise and Fluctuation (Montpellier).

    Not in our library

17. 17.

    Jo S. H. Chang T. Ebong I. Bhadviya B. B. Mazumder P. Lu W. (2010 ). Nanoscale memristor device as synapse in neuromorphic systems. Nano Lett. 10, 1297–1301.

    Not in our library10.1021/nl904092h (opens in a new tab)

18. 18.

    Jonsson B. E. (2011a ). An empirical approach to finding energy efficient ADC architectures, in 2011 International Workshop on ADC Modelling, Testing and Data Converter Analysis and Design and IEEE 2011 ADC Forum (Orvieto), 132–137.

    Not in our library

19. 19.

    Jonsson B. E. (2011b ). Area Efficiency of ADC Architectures, in 2011 20th European Conference on Circuit Theory and Design (ECCTD) (Linkoping), 560–563.

    Not in our library

20. 20.

    Krizhevsky A. Sutskever I. Hinton G. E. (2012 ). Imagenet classification with deep convolutional neural networks, in Neural Information Processing Systems (Lake Tahoe, NV), 1097–1105.

    Not in our library

21. 21.

    Kuzum D. Yu S. Wong H. S. (2013 ). Synaptic electronics: materials, devices and applications. Nanotechnology 24:382001.

    Not in our library10.1088/0957-4484/24/38/382001 (opens in a new tab)

22. 22.

    Le Q. V. Ranzato M. A Monga R. Devin M. Chen K. Corrado G. S. (2012 ). Building high-level features using large scale unsupervised learning, in International Conference on Machine Learning (Edinburg).

    Not in our library

23. 23.
    Deep learning · 2015

    No notice
24. 24.

    LeCun Y. Bottou L. Bengio Y. Haffner P. (1998 ). Gradient-based learning applied to document recognition. Proce. IEEE 86, 2278–2324.

    Not in our library10.1109/5.726791 (opens in a new tab)

25. 25.

    Lehmann C. Viredaz M. Blayo F. A. (1993 ). generic systolic array building block for neural networks with on-chip learning. IEEE Trans. Neural Netw. 4, 400–407.

    Not in our library10.1109/72.217181 (opens in a new tab)

26. 26.

    Li B. Wang Y. Wang Y. Chen Y. Yang H. (2014 ). Training itself: mixed-signal training acceleration for memristor-based neural network in 19th Asia and South Pacific Design Automation Conference (ASP-DAC) (Suntec City).

    Not in our library

27. 27.

    Merkel C. Kudithipudi D. (2014 ). A stochastic learning algorithm for neuromemristive systems, in 27th IEEE International System-on-Chip Conference (SOCC), (Las Vegas, NV), 359–366.

    Not in our library

28. 28.

    NVIDIA (2012 ). NVIDIA's next generation CUDA compute architecture: Kepler GK110. Whitepaper.

    Not in our library

29. 29.

    O'Connor P. Welling M. (2016 ). Deep spiking networks. arXiv:1602.08323 [cs.NE].

    Not in our library

30. 30.

    Poppelbaum W. J. Afuso C. Esch J. W. (1967 ). Stochastic computing elements and systems, in Proceedings of the AFIPS Fall Joint Computer Conference (Anaheim, CA), 635–644.

    Not in our library

31. 31.

    Prezioso M. Merrikh-Bayat F. Hoskins B. D. Adam G. C. Likharev K. K. Strukov D. B. (2015 ). Training and operation of an integrated neuromorphic network based on metal-oxide memristors. Nature 521, 61–64.

    Not in our library10.1038/nature14441 (opens in a new tab)

32. 32.

    Rumelhart D. E. Hinton G. E. Willims R. J. (1986 ). Learning representations by back-propagating errors. Nature 323, 533–536.

    Not in our library

33. 33.

    Saighi S. Mayr C. G. Serrano-Gotarredona T. Schmidt H. Lecerf G. Tomas J.. (2015 ). Plasticity in memristive devices for spiking neural networks. Front. Neurosci. 9:51.

    Not in our library10.3389/fnins.2015.00051 (opens in a new tab)

34. 34.

    Seo J. Lin B. Kim M. Chen P. Kadetotad D. Xu Z. (2015 ). On-chip sparse learning acceleration with CMOS and resistive synaptic devices. IEEE Trans. Nanotechnol. 14, 969–979.

    Not in our library10.1109/tnano.2015.2478861 (opens in a new tab)

35. 35.

    Simonyan K. Zisserman A. (2015 ). Very Deep Convolutional Networks for Large-Scale Image Recognition. San Diego, CA: ICLR.

    Not in our library

36. 36.

    Soudry D. Di Castro D. Gal A. Kolodny A. Kvatinsky S. (2015 ). Memristor-based multilayer neural networks with online gradient descent training. IEEE Trans. Neural Netw. Learn. Syst. 26, 2408–2421.

    Not in our library10.1109/tnnls.2014.2383395 (opens in a new tab)

37. 37.

    Steinbuch K. (1961 ). Die lernmatrix. Kybernetik 1, 36–45.

    Not in our library10.1007/bf00290182 (opens in a new tab)

38. 38.

    Strukov D. Snider G. Stewart D. Williams R. (2008 ). The missing memristor found. Nature 453, 80–83.

    Not in our library10.1038/nature06932 (opens in a new tab)

39. 39.

    Stuecheli J. (2013 ). Next Generation POWER microprocessor, in Hot Chips Conference (Palo Alto, CA).

    Not in our library

40. 40.

    Szegedy C. Liu W. Jia Y. Sermanet P. Reed S. Anguelov D. (2015 ). Going Deeper with Convolutions. Boston, MA: CVPR.

    Not in our library

Cited by papers we hold 6

• Hardware implementation of memristor-based artificial neural networks · 2024
  No notice
• READ-Optimized 28nm HKMG Multibit FeFET Synapses for Inference-Engine Applications · 2022
  Not checked
• Mixed-Precision Deep Learning Based on Computational Memory · 2020
  Not checked
• Neuromorphic computing with multi-memristive synapses · 2018
  No notice
• Efficient and self-adaptive in-situ learning in multilayer memristor neural networks · 2018
  No notice
• Face classification using electronic synapses · 2017
  No notice

Questions about this paper

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.