Eligibility Traces and Plasticity on Behavioral Time Scales: Experimental Support of NeoHebbian Three-Factor Learning Rules

Abstract

Most elementary behaviors such as moving the arm to grasp an object or walking into the next room to explore a museum evolve on the time scale of seconds; in contrast, neuronal action potentials occur on the time scale of a few milliseconds. Learning rules of the brain must therefore bridge the gap between these two different time scales. Modern theories of synaptic plasticity have postulated that the co-activation of pre- and postsynaptic neurons sets a flag at the synapse, called an eligibility trace, that leads to a weight change only if an additional factor is present while the flag is set. This third factor, signaling reward, punishment, surprise, or novelty, could be implemented by the phasic activity of neuromodulators or specific neuronal inputs signaling special events. While the theoretical framework has been developed over the last decades, experimental evidence in support of eligibility traces on the time scale of seconds has been collected only during the last few years. Here we review, in the context of three-factor rules of synaptic plasticity, four key experiments that support the role of synaptic eligibility traces in combination with a third factor as a biological implementation of neoHebbian three-factor learning rules.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 159

1. 1.

   Arleo A. Gerstner W. (2000 ). Spatial cognition and neuro-mimetic navigation: a model of hippocampal place cell activity. Biol. Cybern. 83, 287–299.

   Not in our library10.1007/s004220000171 (opens in a new tab)

2. 2.

   Artola A. Singer W. (1993 ). Long-term depression of excitatory synaptic transmission and its relationship to long-term potentiation. Trends Neurosci. 16, 480–487.

   Not in our library10.1016/0166-2236(93)90081-v (opens in a new tab)

3. 3.

   Bailey C. H. Giustetto M. Huang Y.-Y. Hawkins R. D. Kandel E. R. (2000 ). Is heterosynaptic modulation essential for stabilizing hebbian plasiticity and memory. Nat. Rev. Neurosci. 1, 11–20.

   Not in our library10.1038/35036191 (opens in a new tab)

4. 4.

   Barrett A. B. Billings G. O. Morris R. G. van Rossum M. C. (2009 ). State based model of long-term potentiation and synaptic tagging and capture. PLoS Comput. Biol. 5:e1000259.

   Not in our library10.1371/journal.pcbi.1000259 (opens in a new tab)

5. 5.

   Bartlett P. L. Baxter J. (1999 ). Hebbian Synaptic Modification in Spiking Neurons That Learn. Technical report, Australian National University.

   Not in our library

6. 6.

   Barto A. (1985 ). Learning by statistical cooperation of self-interested neuron-like computing elements. Hum. Neurobiol. 4, 229–256.

   Not in our library

7. 7.

   Barto A. Sutton R. Anderson C. (1983 ). Neuronlike adaptive elements that can solve difficult learning and control problems. IEEE Trans. Syst. Man Cybern. 13, 835–846.

   Not in our library

8. 8.

   Benna M. K. Fusi S. (2016 ). Computational principles of synaptic memory consolidation. Nat. Neurosci. 19, 1697–1706.

   Not in our library10.1038/nn.4401 (opens in a new tab)

9. 9.

   Bienenstock E. L. Cooper L. N. Munroe P. W. (1982 ). Theory of the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex. J. Neurosci. 2, 32–48.

   Not in our library10.1523/jneurosci.02-01-00032.1982 (opens in a new tab)

10. 10.

    Bittner K. C. Grienberger C. Vaidya S. P. Milstein A. D. Macklin J. J. Suh J. (2015 ). Conjunctive input processing drives feature selectivity in hippocampal CA1 neurons. Nat. Neurosci. 357, 1133–1142.

    Not in our library10.1038/nn.4062 (opens in a new tab)

11. 11.

    Bittner K. C. Milstein A. D. Grienberger C. Romani S. Magee J. C. (2017 ). Behavioral time scale synaptic plasticity underlies CA1 place fields. Nature 357, 1033–1036.

    Not in our library10.1126/science.aan3846 (opens in a new tab)

12. 12.

    Bliss T. V. Lømo T (1973 ). Long-lasting potentation of synaptic transmission in the dendate area of anaesthetized rabbit following stimulation of the perforant path. J. Physiol. 232, 351–356.

    Not in our library10.1113/jphysiol.1973.sp010273 (opens in a new tab)

13. 13.

    Black J. Belluzzi J. D. Stein L. (1985 ). Reinforcement delay of one second severely impairs acquisition of brain self-stimulation. Brain Res. 359, 113–119.

    Not in our library10.1016/0006-8993(85)91418-0 (opens in a new tab)

14. 14.

    Bliss T. V. Collingridge G. L. (1993 ). A synaptic model of memory: long-term potentiation in the hippocampus. Nature 361, 31–39.

    Not in our library10.1038/361031a0 (opens in a new tab)

15. 15.

    Bosch M. Castro J. Saneyoshi T. Matsuno H. Sur M. Hayashi Y. (2014 ). Structural and molecular remodeling of dendritic spine substructures during long-term potentation. Neuron 82, 444–459.

    Not in our library10.1016/j.neuron.2014.03.021 (opens in a new tab)

16. 16.

    Brader J. M. Senn W. Fusi S. (2007 ). Learning real-world stimuli in a neural network with spike-driven synaptic dynamics. Neural Comput. 19, 2881–2912.

    Not in our library10.1162/neco.2007.19.11.2881 (opens in a new tab)

17. 17.

    Brea J. Gaál A. T. Urbancik R. Senn W. (2016 ). Prospective coding by spiking neurons. PLoS Comput. Biol. 12:e1005003.

    Not in our library10.1371/journal.pcbi.1005003 (opens in a new tab)

18. 18.

    Brea J. Senn W. Pfister J.-P. (2013 ). Matching recall and storage in sequence learning with spiking neural networks. J. Neurosci. 33, 9565–9575.

    Not in our library10.1523/jneurosci.4098-12.2013 (opens in a new tab)

19. 19.

    Brown M. A. Sharp P. E. (1995 ). Simulation of spatial learning in the Morris water maze by a neural network model of the hippocampal-formation and nucleus accumbens. Hippocampus 5, 171–188.

    Not in our library10.1002/hipo.450050304 (opens in a new tab)

20. 20.

    Brzosko Z. Schultz W. Paulsen O. (2015 ). Retroactive modulation of spike timing-dependent plasticity by dopamine. eLife 4:e09685.

    Not in our library10.7554/elife.09685 (opens in a new tab)

21. 21.

    Brzosko Z. Zannone S. Schultz W. Clopath C. Paulsen O. (2017 ). Sequential neuromodulation of hebbian plasticity offers mechanism for effective reward-based navigation. eLife 6:e27756.

    Not in our library10.7554/elife.27756 (opens in a new tab)

22. 22.

    Buonomano D. V. Maass W. (2009 ). State-dependent computations: spatiotemporal processing in cortical networks. Nat. Rev. Neurosci. 10, 113–125.

    Not in our library10.1038/nrn2558 (opens in a new tab)

23. 23.

    Cassenaer S. Laurent G. (2012 ). Conditional modulation of spike-timing-dependent plasticity for olfactory learning. Nature 482, 47–52.

    Not in our library10.1038/nature10776 (opens in a new tab)

24. 24.

    Clopath C. Büsing L. Vasilaki E. Gerstner W. (2010 ). Connectivity reflects coding: a model of voltage-based spike-timing-dependent-plasticity with homeostasis. Nat. Neurosci. 13, 344–352.

    Not in our library10.1038/nn.2479 (opens in a new tab)

25. 25.

    Clopath C. Ziegler L. Vasilaki E. Büsing L. Gerstner W. (2008 ). Tag-trigger-consolidation: a model of early and late long-term-potentiation and depression. PLoS Comput. Biol. 4:e1000248.

    Not in our library10.1371/journal.pcbi.1000248 (opens in a new tab)

26. 26.

    Crick F. (1984 ). Neurobiology-memory and molecular turnover. Nature 312:101

    Not in our library10.1038/312101a0 (opens in a new tab)

27. 27.

    Crow T. (1968 ). Cortical synapses and reinforcement: a hypothesis. Nature 219, 736–737.

    Not in our library10.1038/219736a0 (opens in a new tab)

28. 28.

    Daw N. D. Niv Y. Dayan P. (2005 ). Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8, 1704–1711.

    Not in our library10.1038/nn1560 (opens in a new tab)

29. 29.

    Deger M. Helias M. Rotter S. Diesmann M. (2012 ). Spike-timing dependence of structural plasticity explains cooperative synapse formation in the neocortex. PLoS Comput. Biol. 8:e1002689.

    Not in our library10.1371/journal.pcbi.1002689 (opens in a new tab)

30. 30.

    Deger M. Seeholzer A. Gerstner W. (2018 ). Multicontact co-operativity in spike-timing-dependent structural plasticity stabilizes networks. Cereb. Cortex 28, 1396–1415.

    Not in our library10.1093/cercor/bhx339 (opens in a new tab)

31. 31.

    Doya K. (2000a ). Complementary roles of basal ganglia and cerebellum in learning and motor control. Curr. Opin. Neurobiol. 10, 732–739.

    Not in our library10.1016/s0959-4388(00)00153-7 (opens in a new tab)

32. 32.

    Doya K. (2000b ). Temporal difference learning in continuous time and space. Neural Comput. 12, 219–245.

    Not in our library10.1162/089976600300015961 (opens in a new tab)

33. 33.

    Faraji M. Preuschoff K. Gerstner W. (2018 ). Balancing new against old information: the role of puzzlement surprise in learning. Neural Comput. 30, 34–83.

    Not in our library10.1162/neco_a_01025 (opens in a new tab)

34. 34.

    Fauth M. Wörgötter F. Tetzlaff C. (2015 ). The formation of multi-synaptic connections by the interaction of synaptic and structural plasticity and their functional consequences. PLoS Comput. Biol. 11:e1004031.

    Not in our library10.1371/journal.pcbi.1004031 (opens in a new tab)

35. 35.

    Fiorillo C. D. Yun S. R. Song M. R. (2013 ). Diversity and homogeneity in responses of midbrain dopamine neurons. J. Neurosci. 33, 4693–4709.

    Not in our library10.1523/jneurosci.3886-12.2013 (opens in a new tab)

36. 36.

    Florian R. V. (2007 ). Reinforcement learning through modulation of spike-timing-dependent synaptic plasticity. Neural Comput. 19, 1468–1502.

    Not in our library10.1162/neco.2007.19.6.1468 (opens in a new tab)

37. 37.

    Foster D. J. Morris R. G. Dayan P. (2000 ). Models of hippocampally dependent navigation using the temporal difference learning rule. Hippocampus 10, 1–16.

    Not in our library10.1002/(sici)1098-1063(2000)10:1<1::aid-hipo1>3.0.co;2-1 (opens in a new tab)

38. 38.
    Neuromodulated Spike-Timing-Dependent Plasticity, and Theory of Three-Factor Learning Rules · 2016

    No notice
39. 39.

    Frémaux N. Sprekeler H. Gerstner W. (2010 ). Functional requirements for reward-modulated spike-timing-dependent plasticity,. J. Neurosci. 40, 13326–13337.

    Not in our library10.1523/jneurosci.6249-09.2010 (opens in a new tab)

40. 40.

    Frémaux N. Sprekeler H. Gerstner W. (2013 ). Reinforcement learning using continuous time actor-critic framework with spiking neurons. PLoS Comput. Biol. 9:e1003024.

    Not in our library10.1371/journal.pcbi.1003024 (opens in a new tab)

Cited by papers we hold 3

• R-STDP Spiking Neural Network Architecture for Motion Control on a Changing Friction Joint Robotic Arm · 2022
  Not checked
• Bidirectional synaptic plasticity rapidly modifies hippocampal representations · 2021
  No notice
• A solution to the learning dilemma for recurrent networks of spiking neurons · 2020
  No notice

Questions about this paper

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.