Eligibility Traces and Plasticity on Behavioral Time Scales: Experimental Support of NeoHebbian Three-Factor Learning Rules

Abstract

Most elementary behaviors such as moving the arm to grasp an object or walking into the next room to explore a museum evolve on the time scale of seconds; in contrast, neuronal action potentials occur on the time scale of a few milliseconds. Learning rules of the brain must therefore bridge the gap between these two different time scales. Modern theories of synaptic plasticity have postulated that the co-activation of pre- and postsynaptic neurons sets a flag at the synapse, called an eligibility trace, that leads to a weight change only if an additional factor is present while the flag is set. This third factor, signaling reward, punishment, surprise, or novelty, could be implemented by the phasic activity of neuromodulators or specific neuronal inputs signaling special events. While the theoretical framework has been developed over the last decades, experimental evidence in support of eligibility traces on the time scale of seconds has been collected only during the last few years. Here we review, in the context of three-factor rules of synaptic plasticity, four key experiments that support the role of synaptic eligibility traces in combination with a third factor as a biological implementation of neoHebbian three-factor learning rules.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 159

1. 1.

   Arleo A. Gerstner W. (2000 ). Spatial cognition and neuro-mimetic navigation: a model of hippocampal place cell activity. Biol. Cybern. 83, 287–299.

   Not in our library10.1007/s004220000171 (opens in a new tab)

2. 2.

   Artola A. Singer W. (1993 ). Long-term depression of excitatory synaptic transmission and its relationship to long-term potentiation. Trends Neurosci. 16, 480–487.

   Not in our library10.1016/0166-2236(93)90081-v (opens in a new tab)

3. 3.

   Bailey C. H. Giustetto M. Huang Y.-Y. Hawkins R. D. Kandel E. R. (2000 ). Is heterosynaptic modulation essential for stabilizing hebbian plasiticity and memory. Nat. Rev. Neurosci. 1, 11–20.

   Not in our library10.1038/35036191 (opens in a new tab)

4. 4.

   Barrett A. B. Billings G. O. Morris R. G. van Rossum M. C. (2009 ). State based model of long-term potentiation and synaptic tagging and capture. PLoS Comput. Biol. 5:e1000259.

   Not in our library10.1371/journal.pcbi.1000259 (opens in a new tab)

5. 5.

   Bartlett P. L. Baxter J. (1999 ). Hebbian Synaptic Modification in Spiking Neurons That Learn. Technical report, Australian National University.

   Not in our library

6. 6.

   Barto A. (1985 ). Learning by statistical cooperation of self-interested neuron-like computing elements. Hum. Neurobiol. 4, 229–256.

   Not in our library

7. 7.

   Barto A. Sutton R. Anderson C. (1983 ). Neuronlike adaptive elements that can solve difficult learning and control problems. IEEE Trans. Syst. Man Cybern. 13, 835–846.

   Not in our library

8. 8.

   Benna M. K. Fusi S. (2016 ). Computational principles of synaptic memory consolidation. Nat. Neurosci. 19, 1697–1706.

   Not in our library10.1038/nn.4401 (opens in a new tab)

9. 9.

   Bienenstock E. L. Cooper L. N. Munroe P. W. (1982 ). Theory of the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex. J. Neurosci. 2, 32–48.

   Not in our library10.1523/jneurosci.02-01-00032.1982 (opens in a new tab)

10. 10.

    Bittner K. C. Grienberger C. Vaidya S. P. Milstein A. D. Macklin J. J. Suh J. (2015 ). Conjunctive input processing drives feature selectivity in hippocampal CA1 neurons. Nat. Neurosci. 357, 1133–1142.

    Not in our library10.1038/nn.4062 (opens in a new tab)

11. 11.

    Bittner K. C. Milstein A. D. Grienberger C. Romani S. Magee J. C. (2017 ). Behavioral time scale synaptic plasticity underlies CA1 place fields. Nature 357, 1033–1036.

    Not in our library10.1126/science.aan3846 (opens in a new tab)

12. 12.

    Bliss T. V. Lømo T (1973 ). Long-lasting potentation of synaptic transmission in the dendate area of anaesthetized rabbit following stimulation of the perforant path. J. Physiol. 232, 351–356.

    Not in our library10.1113/jphysiol.1973.sp010273 (opens in a new tab)

13. 13.

    Black J. Belluzzi J. D. Stein L. (1985 ). Reinforcement delay of one second severely impairs acquisition of brain self-stimulation. Brain Res. 359, 113–119.

    Not in our library10.1016/0006-8993(85)91418-0 (opens in a new tab)

14. 14.

    Bliss T. V. Collingridge G. L. (1993 ). A synaptic model of memory: long-term potentiation in the hippocampus. Nature 361, 31–39.

    Not in our library10.1038/361031a0 (opens in a new tab)

15. 15.

    Bosch M. Castro J. Saneyoshi T. Matsuno H. Sur M. Hayashi Y. (2014 ). Structural and molecular remodeling of dendritic spine substructures during long-term potentation. Neuron 82, 444–459.

    Not in our library10.1016/j.neuron.2014.03.021 (opens in a new tab)

16. 16.

    Brader J. M. Senn W. Fusi S. (2007 ). Learning real-world stimuli in a neural network with spike-driven synaptic dynamics. Neural Comput. 19, 2881–2912.

    Not in our library10.1162/neco.2007.19.11.2881 (opens in a new tab)

17. 17.

    Brea J. Gaál A. T. Urbancik R. Senn W. (2016 ). Prospective coding by spiking neurons. PLoS Comput. Biol. 12:e1005003.

    Not in our library10.1371/journal.pcbi.1005003 (opens in a new tab)

18. 18.

    Brea J. Senn W. Pfister J.-P. (2013 ). Matching recall and storage in sequence learning with spiking neural networks. J. Neurosci. 33, 9565–9575.

    Not in our library10.1523/jneurosci.4098-12.2013 (opens in a new tab)

19. 19.

    Brown M. A. Sharp P. E. (1995 ). Simulation of spatial learning in the Morris water maze by a neural network model of the hippocampal-formation and nucleus accumbens. Hippocampus 5, 171–188.

    Not in our library10.1002/hipo.450050304 (opens in a new tab)

20. 20.

    Brzosko Z. Schultz W. Paulsen O. (2015 ). Retroactive modulation of spike timing-dependent plasticity by dopamine. eLife 4:e09685.

    Not in our library10.7554/elife.09685 (opens in a new tab)

21. 21.

    Brzosko Z. Zannone S. Schultz W. Clopath C. Paulsen O. (2017 ). Sequential neuromodulation of hebbian plasticity offers mechanism for effective reward-based navigation. eLife 6:e27756.

    Not in our library10.7554/elife.27756 (opens in a new tab)

22. 22.

    Buonomano D. V. Maass W. (2009 ). State-dependent computations: spatiotemporal processing in cortical networks. Nat. Rev. Neurosci. 10, 113–125.

    Not in our library10.1038/nrn2558 (opens in a new tab)

23. 23.

    Cassenaer S. Laurent G. (2012 ). Conditional modulation of spike-timing-dependent plasticity for olfactory learning. Nature 482, 47–52.

    Not in our library10.1038/nature10776 (opens in a new tab)

24. 24.

    Clopath C. Büsing L. Vasilaki E. Gerstner W. (2010 ). Connectivity reflects coding: a model of voltage-based spike-timing-dependent-plasticity with homeostasis. Nat. Neurosci. 13, 344–352.

    Not in our library10.1038/nn.2479 (opens in a new tab)

25. 25.

    Clopath C. Ziegler L. Vasilaki E. Büsing L. Gerstner W. (2008 ). Tag-trigger-consolidation: a model of early and late long-term-potentiation and depression. PLoS Comput. Biol. 4:e1000248.

    Not in our library10.1371/journal.pcbi.1000248 (opens in a new tab)

26. 26.

    Crick F. (1984 ). Neurobiology-memory and molecular turnover. Nature 312:101

    Not in our library10.1038/312101a0 (opens in a new tab)

27. 27.

    Crow T. (1968 ). Cortical synapses and reinforcement: a hypothesis. Nature 219, 736–737.

    Not in our library10.1038/219736a0 (opens in a new tab)

28. 28.

    Daw N. D. Niv Y. Dayan P. (2005 ). Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8, 1704–1711.

    Not in our library10.1038/nn1560 (opens in a new tab)

29. 29.

    Deger M. Helias M. Rotter S. Diesmann M. (2012 ). Spike-timing dependence of structural plasticity explains cooperative synapse formation in the neocortex. PLoS Comput. Biol. 8:e1002689.

    Not in our library10.1371/journal.pcbi.1002689 (opens in a new tab)

30. 30.

    Deger M. Seeholzer A. Gerstner W. (2018 ). Multicontact co-operativity in spike-timing-dependent structural plasticity stabilizes networks. Cereb. Cortex 28, 1396–1415.

    Not in our library10.1093/cercor/bhx339 (opens in a new tab)

31. 31.

    Doya K. (2000a ). Complementary roles of basal ganglia and cerebellum in learning and motor control. Curr. Opin. Neurobiol. 10, 732–739.

    Not in our library10.1016/s0959-4388(00)00153-7 (opens in a new tab)

32. 32.

    Doya K. (2000b ). Temporal difference learning in continuous time and space. Neural Comput. 12, 219–245.

    Not in our library10.1162/089976600300015961 (opens in a new tab)

33. 33.

    Faraji M. Preuschoff K. Gerstner W. (2018 ). Balancing new against old information: the role of puzzlement surprise in learning. Neural Comput. 30, 34–83.

    Not in our library10.1162/neco_a_01025 (opens in a new tab)

34. 34.

    Fauth M. Wörgötter F. Tetzlaff C. (2015 ). The formation of multi-synaptic connections by the interaction of synaptic and structural plasticity and their functional consequences. PLoS Comput. Biol. 11:e1004031.

    Not in our library10.1371/journal.pcbi.1004031 (opens in a new tab)

35. 35.

    Fiorillo C. D. Yun S. R. Song M. R. (2013 ). Diversity and homogeneity in responses of midbrain dopamine neurons. J. Neurosci. 33, 4693–4709.

    Not in our library10.1523/jneurosci.3886-12.2013 (opens in a new tab)

36. 36.

    Florian R. V. (2007 ). Reinforcement learning through modulation of spike-timing-dependent synaptic plasticity. Neural Comput. 19, 1468–1502.

    Not in our library10.1162/neco.2007.19.6.1468 (opens in a new tab)

37. 37.

    Foster D. J. Morris R. G. Dayan P. (2000 ). Models of hippocampally dependent navigation using the temporal difference learning rule. Hippocampus 10, 1–16.

    Not in our library10.1002/(sici)1098-1063(2000)10:1<1::aid-hipo1>3.0.co;2-1 (opens in a new tab)

38. 38.
    Neuromodulated Spike-Timing-Dependent Plasticity, and Theory of Three-Factor Learning Rules · 2016

    No notice
39. 39.

    Frémaux N. Sprekeler H. Gerstner W. (2010 ). Functional requirements for reward-modulated spike-timing-dependent plasticity,. J. Neurosci. 40, 13326–13337.

    Not in our library10.1523/jneurosci.6249-09.2010 (opens in a new tab)

40. 40.

    Frémaux N. Sprekeler H. Gerstner W. (2013 ). Reinforcement learning using continuous time actor-critic framework with spiking neurons. PLoS Comput. Biol. 9:e1003024.

    Not in our library10.1371/journal.pcbi.1003024 (opens in a new tab)

41. 41.

    Frey U. Morris R. (1997 ). Synaptic tagging and long-term potentiation. Nature 385, 533–536.

    Not in our library10.1038/385533a0 (opens in a new tab)

42. 42.

    Friston K. (2010 ). The free-energy principle: a unified brain theory? Nat. Rev. Neurosci. 11, 127–138.

    Not in our library10.1038/nrn2787 (opens in a new tab)

43. 43.
    Active inference and learning · 2016

    No notice
44. 44.

    Froemke R. C. Dan Y. (2002 ). Spike-timing dependent plasticity induced by natural spike trains. Nature 416, 433–438.

    Not in our library10.1038/416433a (opens in a new tab)

45. 45.

    Fusi S. Drew P. J. Abbott L. F. (2005 ). Cascade models of synaptically stored memories. Neuron 45, 599–611.

    Not in our library10.1016/j.neuron.2005.02.001 (opens in a new tab)

46. 46.

    Gerstner W. Kempter R. van Hemmen J. L. Wagner H. (1996 ). A neuronal learning rule for sub-millisecond temporal coding. Nature 383, 76–78.

    Not in our library10.1038/383076a0 (opens in a new tab)

47. 47.

    Gjorjieva J. Clopath C. Audet J. Pfister J. P. (2011 ). A triplet spike-timing dependent plasticity model generalizes the Bienenstock-Cooper-Munro rule to higher-order spatiotemporal correlations. Proc. Natl. Sci. Acad. U.S.A. 108, 19383–19388.

    Not in our library10.1073/pnas.1105933108 (opens in a new tab)

48. 48.
    STDP in a bistable synapse model based on CaMKII and associated signaling pathways. · 2007

    Not checked
49. 49.

    Greensmith E. Bartlett P. Baxter J. (2004 ). Variance reduction techniques for gradient estimates in reinforcement learning. J. Machine Learn. Res/ 5, 1471–1530.

    Not in our library

50. 50.

    Gu Q. (2002 ). Neuromodulatory transmitter systems in the cortex and their role in cortical plasticity. Neuroscience 111, 815–835.

    Not in our library10.1016/s0306-4522(02)00026-x (opens in a new tab)

51. 51.
    Towards deep learning with segregated dendrites · 2017

    No notice
52. 52.

    Haber S. N. Kim K.-S. Mailly P. Calzavara R. (2006 ). Reward-related cortical inputs define a large striatal region in primates that interface with associative cortical connections, providing a substrate for incentive-based learning. J. Neurosci. 26, 8368–8376.

    Not in our library10.1523/jneurosci.0271-06.2006 (opens in a new tab)

53. 53.

    Hasselmo M. (2006 ). The role of acetylcholine in learning and memory. Curr. Opin. Neurobiol. 16, 710–715.

    Not in our library10.1016/j.conb.2006.09.002 (opens in a new tab)

54. 54.

    He K. Huertas M. Hong S. Z. Tie X. Hell J. W. Shouval H.. (2015 ). Distinct eligibility traces for LTP and LTD in cortical synapses. Neuron 88, 528–538.

    Not in our library10.1016/j.neuron.2015.09.037 (opens in a new tab)

55. 55.

    Hebb D. O. (1949 ). The Organization of Behavior. New York, NY: Wiley.

    Not in our library

56. 56.
    Structural plasticity controlled by calcium based correlation detection · 2008

    Not checked
57. 57.

    Hess E. H. Polt J. M. (1960 ). Pupil size as related to interest value of visual stimuli. Science 132, 349–350.

    Not in our library10.1126/science.132.3423.349 (opens in a new tab)

58. 58.

    Hodgkin A. L. Huxley A. F. (1952 ). A quantitative description of membrane current and its application to conduction and excitation in nerve. J Physiol 117, 500–544.

    Not in our library10.1113/jphysiol.1952.sp004764 (opens in a new tab)

59. 59.

    Hopfield J. J. (1982 ). Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. U.S.A. 79, 2554–2558.

    Not in our library10.1073/pnas.79.8.2554 (opens in a new tab)

60. 60.

    Huganir R. L. Nicoll R. A. (2013 ). AMPARs and synaptic plasticity: the last 25 years. Neuron 80, 704–717.

    Not in our library10.1016/j.neuron.2013.10.025 (opens in a new tab)

61. 61.

    Itti L. Baldi P. (2009 ). Bayesian surprise attracts human attention. Vis. Res. 49, 1295–1306.

    Not in our library10.1016/j.visres.2008.09.007 (opens in a new tab)

62. 62.

    Izhikevich E. (2007 ). Solving the distal reward problem through linkage of STDP and dopamine signaling. Cereb. Cortex 17, 2443–2452.

    Not in our library10.1093/cercor/bhl152 (opens in a new tab)

63. 63.

    Izhikevich E. M. Desai N. S. (2003 ). Relating STDP to BCM. Neural Comput. 15, 1511–1523.

    Not in our library10.1162/089976603321891783 (opens in a new tab)

64. 64.

    Jaeger H. Haas H. (2004 ). Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication. Science 304, 78–80.

    Not in our library10.1126/science.1091277 (opens in a new tab)

65. 65.

    Kalman R. (1960 ). A new approach to linear filtering and prediction problems. J. Basic Eng. 82, 35–45.

    Not in our library10.1115/1.3662552 (opens in a new tab)

66. 66.

    Kempter R. Gerstner W. van Hemmen J. L. (1999 ). Hebbian learning and spiking neurons. Phys. Rev. E 59, 4498–4514.

    Not in our library10.1103/physreve.59.4498 (opens in a new tab)

67. 67.

    Klopf A. (1972 ). Brain Function and Adaptive Systems-a Heterostatic Theory. Air Force Cambridge research laboratories, special Reports Vol. 133, 1–70.

    Not in our library

68. 68.
    A model–based approach totrial–by–trial P300 amplitude fluctuations · 2013

    Not checked
69. 69.

    Kolossa A. Kopp B. Finscheidt T. (2015 ). A computational analysis of the neural bases of bayesian inference. NeuroImage 106, 222–237.

    Not in our library10.1016/j.neuroimage.2014.11.007 (opens in a new tab)

70. 70.
    A Learning Theory for Reward-Modulated Spike-Timing-Dependent Plasticity with Application to Biofeedback · 2008

    No notice
71. 71.

    Levy W. B. Stewart O. (1983 ). Temporal contiguity requirements for long-term associative potentiation/depression in hippocampus. Neurosci, 8, 791–797.

    Not in our library10.1016/0306-4522(83)90010-6 (opens in a new tab)

72. 72.
    Random synaptic feedback weights support error backpropagation for deep learning · 2016

    No notice
73. 73.

    Lisman J. (1985 ). A mechanism for memory storage insensitive to molecular turnover: a bistable autophosphorylating kinase. Proc. Natl. Acad. Sci. U.S.A. 82, 3055–3057.

    Not in our library10.1073/pnas.82.9.3055 (opens in a new tab)

74. 74.

    Lisman J. (1989 ). A mechanism for Hebb and anti-Hebb processes underlying learning and memory. Proc. Natl. Acad. Sci. U.S.A. 86, 9574–9578.

    Not in our library10.1073/pnas.86.23.9574 (opens in a new tab)

75. 75.

    Lisman J. (2003 ). Long-term potentiation: outstanding questions and attempted synthesis. Phil. Trans. R. Soc. Lond B Biol. Sci. 358, 829–842.

    Not in our library10.1098/rstb.2002.1242 (opens in a new tab)

76. 76.

    Lisman J. (2017 ). Glutamatergic synapses are structurally and biochemically complex because of multiple plasticity processes: long-term potentiation, long-term depression, short-term potentiation and scaling. Phil. Trans. Roy. Soc. B 372:20160260.

    Not in our library10.1098/rstb.2016.0260 (opens in a new tab)

77. 77.

    Lisman J. Grace A. A. Duzel E. (2011 ). A neoHebbian framework for episodic memory; role of dopamine-dependent late LTP. Trends Neurosci. 34, 536–547.

    Not in our library10.1016/j.tins.2011.07.006 (opens in a new tab)

78. 78.

    Little D. J. Sommer F. T. (2013 ). Learning and exploration in action-perception loops. Front. Neural Circ. 7:37.

    Not in our library10.3389/fncir.2013.00037 (opens in a new tab)

79. 79.

    Ljunberg T. Apicella P. Schultz W. (1992 ). Responses of monkey dopamine neurons during learning of behavioral interactions. J. Neurophysiol. 67, 145–163.

    Not in our library10.1152/jn.1992.67.1.145 (opens in a new tab)

80. 80.

    Loewenstein Y. Seung H. (2006 ). Operant matching is a generic outcome of synaptic plasticity based on the covariance between reward and neural activity. Proc. Natl. Acad. Sci. U.S.A. 103, 15224–15229.

    Not in our library10.1073/pnas.0505220103 (opens in a new tab)

81. 81.

    Löwel S. Singer W. (1992 ). Selection of intrinsic horizontal connections in the visual cortex by correlated neuronal activity. Science 255, 209–212.

    Not in our library10.1126/science.1372754 (opens in a new tab)

82. 82.

    Maass W. Natschläger T. Markram H. (2002 ). Real-time computing without stable states: a new framework for neural computation based on perturbations. Neural Comput. 14, 2531–2560.

    Not in our library10.1162/089976602760407955 (opens in a new tab)

83. 83.

    Markram H. Lübke J. Frotscher M. Sakmann B. (1997 ). Regulation of synaptic efficacy by coincidence of postysnaptic AP and EPSP. Science 275, 213–215.

    Not in our library10.1126/science.275.5297.213 (opens in a new tab)

84. 84.

    Martin S. J. Grimwood P. D. Morris R. G. (2000 ). Synaptic plasticity and memory: an evaluation of the hypothesis. Ann. Rev. Neurosci. 23, 649–711.

    Not in our library10.1146/annurev.neuro.23.1.649 (opens in a new tab)

85. 85.

    Mathys C. Daunizeau J. Friston K. J. Stephan K. E. (2011 ). A bayesian foundation for individual learning under uncertainty. Front. Hum. Neurosci. 5:39.

    Not in our library10.3389/fnhum.2011.00039 (opens in a new tab)

86. 86.
    Uncertainty in perception and the Hierarchical Gaussian Filter · 2014

    No notice
87. 87.

    Matsuzaki M. Ellis-Davies G. C. Nemoto T. Iione M. Miyashita Y. Kasai H. (2001 ). Dendritic spine geometry is critical for AMPA receptor expression in hippocampal CA1 pyramidal neurons. Nat. Neurosci. 4, 1086–1092.

    Not in our library10.1038/nn736 (opens in a new tab)

88. 88.
    Opposite initialization to novel cues in dopamine signaling in ventral and posterior striatum in mice · 2017

    No notice
89. 89.

    Meyniel F. Maheu M. Dehaene S. (2016 ). Human inferences about sequences: a minimal transition probability model. PLoS Comput. Biol. 12:e1005260.

    Not in our library10.1371/journal.pcbi.1005260 (opens in a new tab)

90. 90.

    Middleton F. A. Strick P. (2000 ). Basal ganglia and cerebellar loops: motor and cognitive circuits. Brain Res. Rev. 31, 236–250.

    Not in our library10.1016/s0165-0173(99)00040-5 (opens in a new tab)

91. 91.

    Miller K. D. MacKay D. J. C. (1994 ). The role of constraints in hebbian learning. Neural Comput. 6, 100–126.

    Not in our library10.1162/neco.1994.6.1.100 (opens in a new tab)

92. 92.

    Mink J. (1996 ). The basal ganglia: focused selection and inhibition of competing motor programs. Progr. Neurobiol. 50, 381–425.

    Not in our library10.1016/s0301-0082(96)00042-1 (opens in a new tab)

93. 93.

    Mnih V. Badia A. Mirza M. Graves A. Harley T. Lillicrap T. (2016 ). Asynchronous methods for deep reinforcement learning, in Proceedings of the 33rd International Conference on Machine Learning, eds Balcan M. F. Weinberger K. Q. (New York, NY), 1928–1937.

    Not in our library

94. 94.

    Moncada D. Viola H. (2007 ). Induction of long-term memory by exposure to novelty requires protein synthesis: Evidence for a behavioral tagging. J. Neurosci. 27, 7476–7481.

    Not in our library10.1523/jneurosci.1083-07.2007 (opens in a new tab)

95. 95.

    Mongillo G. Barak O. Tsodyks M. (2008 ). Synaptic theory of working memory. Science 319, 1543–1546.

    Not in our library10.1126/science.1150769 (opens in a new tab)

96. 96.

    Nassar M. R. Wilson R. C. Heasly B. Gold J. I. (2010 ). An approximately bayesian delta-rule model explains the dynamics of belief updating in a changing environment. J. Neurosci. 30, 12366–12378.

    Not in our library10.1523/jneurosci.0822-10.2010 (opens in a new tab)

97. 97.

    Ngezahayo A. Schachner M. Artola A. (2000 ). Synaptic activation modulates the induction of bidirectional synaptic changes in adult mouse hippocampus. J. Neurosci. 20, 2451–2458.

    Not in our library10.1523/jneurosci.20-07-02451.2000 (opens in a new tab)

98. 98.

    Oja E. (1982 ). A simplified neuron model as a principal component analyzer. J. Math. Biol. 15, 267–273.

    Not in our library10.1007/bf00275687 (opens in a new tab)

99. 99.

    O'Keefe J. Nadel L. (1978 ). The Hippocampus as a Cognitive Map, Vol. 3 Oxford: Clarendon Press Oxford.

    Not in our library

100. 100.

     Okouchi H. (2009 ). Response acquisition by humans with delayed reinforcement. J. Exp. Anal. Behav. 91, 377–390.

     Not in our library10.1901/jeab.2009.91-377 (opens in a new tab)

101. 101.

     Oudeyer P. Kaplan F. Hafner V. (2007 ). Intrinsic motivation systems for autonomous mental development. IEEE Trans. Evol. Comput. 11, 265–286.

     Not in our library10.1109/tevc.2006.890271 (opens in a new tab)

102. 102.

     Pavlov I. P. (1927 ). Conditioned Reflexes: An Investigation of the Physiological Activity of the Cerebral Cortex. Oxford, UK: Oxford University Press; Humpfrey Milford.

     Not in our library10.5214/ans.0972-7531.1017309 (opens in a new tab)

103. 103.

     Pawlak V. Wickens J. Kirkwood A. Kerr J. (2010 ). Timing is not everything: neuromodulation opens the STDP gate. Front. Synaptic Neurosci. 2:146

     Not in our library10.3389/fnsyn.2010.00146 (opens in a new tab)

104. 104.

     Pfister J.-P. Toyoizumi T. Barber D. Gerstner W. (2006 ). Optimal spike-timing-dependent plasticity for precise action potential firing in supervised learning. Neural Comput. 18, 1318–1348.

     Not in our library10.1162/neco.2006.18.6.1318 (opens in a new tab)

105. 105.

     Poort J. Khan A. G. Pachitariu M. Nemri A. Orsolic I. Krupic J.. (2015 ). Learning enhances sensory and multiple non-sensory representations in primary visual cortex. Neuron 86, 1478–1490.

     Not in our library10.1016/j.neuron.2015.05.037 (opens in a new tab)

106. 106.

     Redgrave P. Gurney K. (2006 ). The short-latency dopamine signal: a role in discovering novel actions? Nat. Rev. Neurosci. 7, 967–975.

     Not in our library10.1038/nrn2022 (opens in a new tab)

107. 107.

     Redondo R. L. Morris R. G. (2011 ). Making memories last: the synaptic tagging and capture hypothesis. Nat. Rev. Neurosci. 12, 17–30.

     Not in our library10.1038/nrn2963 (opens in a new tab)

108. 108.

     Reymann K. G. Frey J. U. (2007 ). The late maintenance of hippocampal LTP: requirements, phases, synaptic tagging, late associativity and implications. Neuropharmacology 52, 24–40.

     Not in our library10.1016/j.neuropharm.2006.07.026 (opens in a new tab)

109. 109.

     Reynolds J. N. Wickens J. R. (2002 ). Dopamine-dependent plasticity of corticostriatal synapses. Neural Netw. 15, 507–521.

     Not in our library10.1016/s0893-6080(02)00045-x (opens in a new tab)

110. 110.

     Rezende D. Gerstner W. (2014 ). Stochastic variational learning in recurrent spiking networks. Front. Comput. Neurosci. 8:38

     Not in our library10.3389/fncom.2014.00038 (opens in a new tab)

111. 111.

     Roelfsema P. R. Holtmaat A. (2018 ). Control of synaptic plasticity in deep cortical networks. Nat. Rev. Neurosci. 19, 166–180.

     Not in our library10.1038/nrn.2018.6 (opens in a new tab)

112. 112.

     Roelfsema P. R. van Ooyen A. (2005 ). Attention-gated reinforcement learning of internal representations for classification. Neural Comput. 17, 2176–2214.

     Not in our library10.1162/0899766054615699 (opens in a new tab)

113. 113.

     Roelfsema P. R. van Ooyen A. Watanabe T. (2010 ). Perceptual learning rules based on reinforcers and attention. Trends Cogn. Sci. 14, 64–71.

     Not in our library10.1016/j.tics.2009.11.005 (opens in a new tab)

114. 114.

     Roeper J. (2013 ). Dissecting the diversity of midbrain dopamine neurons. Trends Neurosci. 36, 336–342.

     Not in our library10.1016/j.tins.2013.03.003 (opens in a new tab)

115. 115.
     How attention can create synaptic tags for the learning of working memories in sequential tasks. · 2015

     Not checked
116. 116.

     Rubin J. E. Gerkin R. C. Bi G.-Q. Chow C. C. (2005 ). Calcium time course as a signal for spike-timing-dependent plasticity. J. Neurophysiol. 93, 2600–2613.

     Not in our library10.1152/jn.00803.2004 (opens in a new tab)

117. 117.

     Rubin J. Lee D. D. Sompolinsky H. (2001 ). Equilibrium properties of temporally asymmetric Hebbian plasticity. Phys. Rev. Lett. 86, 364–367.

     Not in our library10.1103/physrevlett.86.364 (opens in a new tab)

118. 118.

     Schmidhuber J. (1991 ). Curious model-building control systems, in Proceedings of the International Joint Conference on Neural Networks, Vol. 2, (Singapore: IEEE press), 1458–1463.

     Not in our library

119. 119.

     Schmidhuber J. (2006 ). Developmental robotics, optimal artificial curiosity, creativity, music, and the fine arts. Connect. Sci. 18, 173–187.

     Not in our library10.1080/09540090600768658 (opens in a new tab)

120. 120.

     Schmidhuber J. (2010 ). Formal theory of creativity, fun, and intrinsic motivation (1990–2010). IEEE Trans. Auton. Mental Dev. 2, 230–247.

     Not in our library10.1109/tamd.2010.2056368 (opens in a new tab)

121. 121.

     Schoups A. Vogels R. Quian N. Orban G. (2001 ). Practising orientation identification improves orientation coding in V1 neurons. Nature 412, 549–553.

     Not in our library10.1038/35087601 (opens in a new tab)

122. 122.

     Schultz W. (1998 ). Predictive reward signal of dopamine neurons. J. Neurophysiol. 80, 1–27.

     Not in our library10.1152/jn.1998.80.1.1 (opens in a new tab)

123. 123.

     Schultz W. (2002 ). Getting formal with dopamine and reward. Neuron 36, 241–263.

     Not in our library10.1016/s0896-6273(02)00967-4 (opens in a new tab)

124. 124.

     Schultz W. Dayan P. Montague P. R. (1997 ). A neural substrate for prediction and reward. Science 275, 1593–1599.

     Not in our library10.1126/science.275.5306.1593 (opens in a new tab)

125. 125.

     Schultz W. Dickinson A. (2000 ). Neuronal coding of prediction errors. Ann. Rev. Neurosci. 23, 472–500.

     Not in our library10.1146/annurev.neuro.23.1.473 (opens in a new tab)

126. 126.

     Senn W. Tsodyks M. Markram H. (2001 ). An algorithm for modifying neurotransmitter release probability based on pre- and postsynaptic spike timing. Neural Comput. 13, 35–67.

     Not in our library10.1162/089976601300014628 (opens in a new tab)

127. 127.

     Seo M. Lee E. Averbeck B. (2012 ). Action selection and action value in frontal-striatal circuits. Neuron 74, 947–960.

     Not in our library10.1016/j.neuron.2012.03.037 (opens in a new tab)

128. 128.

     Shannon C. (1948 ). A mathematical theory of communication. Bell Syst. Techn. J. 27, 37–423.

     Not in our library10.1002/j.1538-7305.1948.tb01338.x (opens in a new tab)

129. 129.

     Shatz C. (1992 ). The developing brain. Sci. Am. 267, 60–67.

     Not in our library10.1038/scientificamerican0992-60 (opens in a new tab)

130. 130.

     Shindou T. Shindou M. Watanabe S. Wickens J. (2018 ). A silent eligibility trace enables dopamine-dependent synaptic plasticity for reinforcement learning in the mouse striatum. Eur. J. Neurosci. [Epub ahead of print].

     Not in our library10.1111/ejn.13921 (opens in a new tab)

131. 131.

     Shouval H. Z. Bear M. F. Cooper L. N. (2002 ). A unified model of NMDA receptor dependent bidirectional synaptic plasticity. Proc. Natl. Acad. Sci. U.S.A. 99, 10831–10836.

     Not in our library10.1073/pnas.152343099 (opens in a new tab)

132. 132.

     Shuler M. Bear M. (2006 ). Reward timing in the primary visual cortex. Science 311, 1606–1609.

     Not in our library10.1126/science.1123513 (opens in a new tab)

133. 133.

     Singh S. Barto A. Chentanez N. (2004 ). Intrinsically motivated reinforcement learning. Adv. Neural Inform. Proc. Syst. 17, 1281–1288.

     Not in our library

134. 134.

     Singh S. Sutton R. (1996 ). Reinforcement learning with replacing eligibility traces. Mach. Learn. 22, 123–158.

     Not in our library10.1007/bf00114726 (opens in a new tab)

135. 135.

     Sjöström P. J. Turrigiano G. G. Nelson S. B. (2001 ). Rate, timing, and cooperativity jointly determine cortical synaptic plasticity. Neuron 32, 1149–1164.

     Not in our library10.1016/s0896-6273(01)00542-6 (opens in a new tab)

136. 136.

     Song S. Miller K. Abbott L. (2000 ). Competitive Hebbian learning through spike-time-dependent synaptic plasticity. Nat. Neurosci. 3, 919–926.

     Not in our library10.1038/78829 (opens in a new tab)

137. 137.

     Squires K. C. Wickens C. Squires N. K. Donchin E. (1976 ). The effect of stimulus sequence on the waveform of the cortical event-related potential. Science 193, 1141–1146.

     Not in our library10.1126/science.959831 (opens in a new tab)

138. 138.

     Storck J. Hochreiter S. Schmidhuber J. (1995 ). Reinforcement-driven information acquisition in non-deterministic environments, in Proceedings of ICANN'95, Vol.2 (Paris: EC2-CIE), 159–164.

     Not in our library

139. 139.

     Sun Y. Gomez F. Schmidhuber J. (2011 ). Planning to be surprised: optimal Bayesian exploration in dynamic environments, in Artificial General Intelligence, Lecture Notes in Computer Science, Vol. 6830, eds Schmidhuber J. Thorisson K. R. Looks M. (Springer), 41–51.

     Not in our library

140. 140.

     Suri R. E. Schultz W. (1999 ). A neural network with dopamine-like reinforcement signal that learns a spatial delayed response task. Neuroscience 91, 871–890.

     Not in our library10.1016/s0306-4522(98)00697-6 (opens in a new tab)

141. 141.

     Susillo D. Abbott L. F. (2009 ). Generating coherent patterns of activity from chaotic neural networks. Neuron 63, 544–557.

     Not in our library10.1016/j.neuron.2009.07.018 (opens in a new tab)

142. 142.

     Sutton R. Barto A. (1998 ). Reinforcement Learning. Cambridge: MIT Press.

     Not in our library

143. 143.

     Sutton R. Barto A. (2018 ). Reinforcement Learning: an introduction (2nd Edn.) Cambridge, MA: MIT Press.

     Not in our library

144. 144.

     Sutton R. S. Barto A. G. (1981 ). Towards a modern theory of adaptive networks: expectation and prediction. Psychol. Rev. 88, 135–171.

     Not in our library10.1037/0033-295x.88.2.135 (opens in a new tab)

145. 145.

     Takeuchi T. Duszkiewicz A. J. Sonneborn A. Spooner P. A. Yamasaki M. Watanabe M.. (2016 ). Locus coeruleus and dopaminergic consolidation of everyday memory. Nature 537, 357–362.

     Not in our library10.1038/nature19325 (opens in a new tab)

146. 146.

     Thorndike E. (1911 ). Animal Intelligence. Darien, CT: Hafner.

     Not in our library

147. 147.

     Urbanczik R. Senn W. (2009 ). Reinforcement learning in populations of spiking neurons. Nat. Neurosci. 12, 250–252.

     Not in our library10.1038/nn.2264 (opens in a new tab)

148. 148.

     Urbanczik R. Senn W. (2014 ). Learning by the dendritic prediction of somatic spiking. Neuron 81, 521–528.

     Not in our library10.1016/j.neuron.2013.11.030 (opens in a new tab)

149. 149.

     van Rossum M. C. W. Bi G. Q. Turrigiano G. G. (2000 ). Stable Hebbian learning from spike timing-dependent plasticity. J. Neurosci. 20, 8812–8821.

     Not in our library10.1523/jneurosci.20-23-08812.2000 (opens in a new tab)

150. 150.

     Vasilaki E. Frémaux N. Urbanczik R. Senn W. Gerstner W. (2009 ). Spike-based reinforcement learning in continuous state and action space: When policy gradient methods fail. PLoS Comput.Biol. 5:e1000586.

     Not in our library10.1371/journal.pcbi.1000586 (opens in a new tab)

151. 151.

     Wang X.-J. Kennedy H. (2016 ). Brain structure and dynamics across scales: in search of rules. Curr. Opin. Neurobiol. 37, 92–98.

     Not in our library10.1016/j.conb.2015.12.010 (opens in a new tab)

152. 152.

     Williams R. (1992 ). Simple statistical gradient-following methods for connectionist reinforcement learning. Mach. Learn. 8, 229–256.

     Not in our library10.1007/bf00992696 (opens in a new tab)

153. 153.

     Willshaw D. J. Bunemann O. P. Longuet-Higgins H. C. (1969 ). Non-holographic associative memory. Nature 222, 960–962.

     Not in our library10.1038/222960a0 (opens in a new tab)

154. 154.

     Xie X. Seung H. S. (2004 ). Learning in neural networks by reinforcement of irregular spiking. Phys. Rev. E 69:41909.

     Not in our library10.1103/physreve.69.041909 (opens in a new tab)

155. 155.

     Yagishita S. Hayashi-Takagi A. Ellis-Davies G. C. Urakubo H. Ishii S. Kasai H. (2014 ). A critical time window for dopamine actions on the structural plasticity of dendritic spines. Science 345, 1616–1620.

     Not in our library10.1126/science.1255514 (opens in a new tab)

156. 156.

     Yu A. J. Dayan P. (2005 ). Uncertainty, neuromodulation, and attention. Neuron 46, 681–692.

     Not in our library10.1016/j.neuron.2005.04.026 (opens in a new tab)

157. 157.

     Zhang J. C. Lau P. M. Bi G. Q. (2009 ). Gain in sensitivity and loss in temporal contrast of STDP by dopaminergic modulation at hippocampal synapses. Proc. Natl. Acad. Sci. U.S.A. 106, 13028–13033.

     Not in our library10.1073/pnas.0900546106 (opens in a new tab)

158. 158.

     Zhang L. I. Tao H. W. Holt C. E. Harris W. A. Poo M. (1998 ). A critical window for cooperation and competition among developing retinotectal synapses. Nature 395, 37–44.

     Not in our library10.1038/25665 (opens in a new tab)

159. 159.

     Ziegler L. Zenke F. Kastner D. B. Gerstner W. (2015 ). Synaptic consolidation: from synapses to behavioral modeling. J. Neurosci. 35, 1319–1334.

     Not in our library10.1523/jneurosci.3989-14.2015 (opens in a new tab)

Cited by papers we hold 3

• R-STDP Spiking Neural Network Architecture for Motion Control on a Changing Friction Joint Robotic Arm · 2022
  Not checked
• Bidirectional synaptic plasticity rapidly modifies hippocampal representations · 2021
  No notice
• A solution to the learning dilemma for recurrent networks of spiking neurons · 2020
  No notice

Questions about this paper

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.