Memory trace replay: the shaping of memory consolidation by neuromodulation

1What determines which memories are retained and which are lost is an absorbing topic for scientists and nonscientists alike, yet the mechanisms underlying the persistence of some pieces of information and the forgetting of others remain to be identified. Well-established theories propose that memories are encoded during wake behavior, with information being represented in the coordinated activity of subsets of neurons forming cell assemblies,,,. However newly encoded memories are typically fragile and, because they may decay, require additional maintenance processes. At the network level, one such process is the off-line reactivation of assembly firing patterns observed during active behavior. This process is best illustrated by the location-specific firing of principal cells,,, in the hippocampus. These place cells (see Glossary) are activated sequentially as an animal runs through an arena. Subsequently, co-active place cells representing a discrete place or sequential place cell activation representing a trajectory are reactivated or replayed during SWRs (Box 1, Box 2), which intermittently occur in slow wave sleep (SWS) (sSWRs), long periods of awake immobility (iSWRs), or brief pauses in exploration (eSWRs) (reviewed in, ).

2Sequential replay can occur in both a forward,,,, and backward,,,, direction, with the directional balance proposed to be dependent on the ongoing behavioral state of the animal. Similar to theta-phase precession, these replayed sequences are temporally compressed compared with those observed during wake behavior,,,. Consequently they have been posited to provide the appropriate temporal neural activity for the Hebbian synaptic modification occurring downstream in neocortical networks during memory consolidation,,,. Although direct support for this hypothesis is currently lacking, intact NMDA receptor (NMDAR) activity during learning and intrahippocampal synaptic transmission during consolidation are at least necessary for unimpaired memory consolidation and place cell reactivation during SWRs,. Moreover, evidence suggests the hippocampus and neocortex are actively engaged during SWRs and SWS. Cortical and hippocampal sequences, reflecting the same experiences, replay together during SWS and, during SWRs, prefrontal neurons consistently fire within tight temporal windows <100 ms after hippocampal pyramidal cells, which could plausibly drive plasticity at the level of single cell pairs.

3Concordantly, a growing body of literature has linked SWRs to learning and memory. SWR incidence during SWS is increased following training on a place-reward association task. Conversely, electrically interrupting SWRs during post-training sleep impaired spatial learning,, while interrupting SWRs during training on a spatial alternation task selectively impaired spatial working memory, but not spatial reference memory. Critically, it remains to be determined whether the ongoing hippocampal network activity during SWRs (i.e., global transient changes in interneuron and pyramidal cell activities) or specifically the reactivation or replay of place cell activity during SWRs is the more important for spatial learning and memory. Indeed, SWR activity content can be biased towards newly learnt firing patterns and predict memory performance. Moreover, recent work also showed the induction of an artificial place-preference behavior following intracranial stimulation, triggered by single-place cell activity during sleep. This further suggests that replay of place cell activity serves an important role in spatial memory, although how this applies to the coordinated neuronal ensemble activity during SWRs remains to be investigated.

4By contrast, replay, particularly during awake SWRs, has also been proposed to have functional roles other than for spatial memory consolidation; for example, in temporal credit assignment to reward locations (particularly backward replay),; formation of goal-relevant or novel environment place-related assemblies,,,; evaluation of trajectory choices for decision making on spatial working memory tasks for prospection and planning (particularly forward replay),,,,; and representation of unexplored trajectories,. Preplay of trajectories yet to be experienced has been proposed to facilitate, at least in part, the selection of subsequent place cell representations in a novel environment.

5Despite over a decade's worth of literature describing hippocampal replay during SWRs, several questions remain. How is neuronal coordination during SWRs controlled? What selects which trajectories will be replayed within a given SWR? Do these mechanisms differ for SWRs in different behavioral states, or under different neuromodulators? In this review, we provide a critique of the evidence surrounding two current theories on how replay occurs, namely by lingering place-related excitability or as a result of synaptic plasticity. Given the limitations of both models, we then outline how neuromodulatory factors are likely to influence the mechanisms underlying replay and impart selection onto which trajectories are replayed. Finally, we propose an integrated model for replay in SWRs that takes into account the behavioral state of the animal and the underlying neuromodulatory tone.

6Replay in a forward or backward direction during awake iSWRs at the ends of linear tracks and in a reverse direction during eSWRs in an open-field environment has been proposed to occur via a residual, place-selective, spatial tuning drive,,. In this model, place cells receive subthreshold inputs as a function of the distance of the animal from the place field center of each cell,. During SWRs, pyramidal cells have a higher firing probability and their waking patterns are reactivated,,. However, the replay firing content per se is lingered in an order dictated by the subthreshold spatial inputs onto place cells at the current position of the animal (Figure 1). This effectively represents a nonassociative bias that can influence the spontaneous SWR response of hippocampal cell assemblies. Concordantly, on a linear track, there is a preference for reverse replay at the end of the track following even the first lap, while there is a preference for forward replay at the start of the track in anticipation of the run.

7In support of this model, the firing probability of place cells in eSWRs increased the closer the animal was to the place field center, suggesting that the momentary, place-related, excitatory drive directly contributes to reverse reactivation in an open-field environment. This is consistent with a large proportion of awake replays starting from the current location in a maze, where the spatial inputs would be stronger,,. Therefore, the model predicts that sequential activation of place cells in awake SWRs should not only reflect the actual path taken or future path from the current position, but also the cascades of spatially tuned activities dictated by the hippocampal map representation of the entire environment. In line with this suggestion, forward reactivation during eSWRs in a 2D open-field environment was not anticipatory to future path taken; reactivation in SWRs on a spatial alternation task were equally representative of actual and alternative past–future paths; and replay initiated from current location on a long linear track was not biased towards future and past trajectories. However, when the task is goal driven, trajectory sequences in awake SWRs strongly represent the path to the future goal location.

8Evidence that challenges this model comes from in vitro studies showing that somatic depolarization does not dramatically increase pyramidal cell spiking in SWRs, and that there was no difference in resting membrane potential between the pyramidal cells that spike during SWRs and those that do not. However, this potentially did not account for the role of synaptic inhibition, which can act to hyperpolarize the membrane (at membrane potentials above the inhibitory reversal potential) or as a shunt at the inhibitory reversal potential. Pyramidal cell spiking is dampened during SWRs by strong perisomatic inhibition,,, likely from parvalbumin positive basket cells, which are strongly active in SWRs,. The prolonged somatic depolarization that was used, would have also increased the size of this hyperpolarizing inhibition by increasing inhibitory drive, as the membrane potential was moved further from the inhibitory reversal potential, and this may explain the absence of a facilitating effect on pyramidal cell spiking. By contrast, the phasic depolarization induced by a dendritic spatial drive from excitatory synapses, in the lingering excitability model, with neurons at resting membrane potential and perisomatic inhibition acting as a shunt, may still be sufficient to depolarize pyramidal cells beyond action potential threshold.

9Nevertheless, this standalone model cannot explain how goal-directed but not random foraging and/or navigation biases trajectory sequences in awake SWRs to strongly represent the path to the future goal location. Neither does it explain how awake replay occurs in the absence of local sensory drive to place cells. For example, nonlocal forward and backward replay has been observed for trajectories that were either not experienced for more than 10 min, or which originated some distance away from the current position of the animal. Moreover, it has been observed that activity in a previous environment is remotely replayed during awake SWRs while the animal is exploring a new environment. Although the first active cell in these remote replays had a higher local firing rate outside of SWRs in the new environment than the last active cell, which is consistent with reverse replay depending on the recent firing history of cells, this model does not explain how the firing of one initiator cell drives the replay of entire ordered sequences of trajectories from another environment. Clearly, the model also does not explain forward and backward replay during sleep,, where any residual place selective drive has dissipated.

10A different model for the generation of sequential replay posits that place cells active during a given trajectory are coupled together by associative synaptic plasticity during exploration. An autoassociative network is required for this model, which may either be provided by the Cornu Ammonis (CA)-3 network alone, or by rapid interactions between CA3 and the dentate gyrus (DG), considering the potential involvement of the DG in promoting SWR activity. Once a given initiator cell in CA3 becomes active during subsequent SWRs, the entire trajectory sequence is reactivated along the path of least resistance, dictated by the internal connectivity and potentiated synapses between cells,,, (Figure 2). The effect of this can then be read out downstream in CA1 through Schaffer collateral connectivity. This can be considered as being similar to how the internal organization of CA3 has been proposed to underlie internally generated theta sequences and preplay activity, or indeed how down to up state transitions during the neocortical slow oscillation might initiate spontaneous sequential cortical activity. Since the likelihood of synaptic plasticity is increased following a repeated number of spike pairings, this model would explain how full replay sequences during iSWRs were not visible until at least one, but sometimes several, laps on a track were completed,. This model is also supported by computational work showing that CA1 pyramidal cell spiking in SWRs is dependent on the strength of their Schaffer collateral connections,.

11If synaptic plasticity is a necessary prerequisite of replay, manipulations that induce plasticity should facilitate SWR replay, while manipulations that prevent plasticity within the hippocampus should not. This prediction has received relatively little support so far, possibly due to methodological considerations, although it has been shown that sSWR-associated unit firing increases following a plasticity-inducing protocol. One approach to blocking plasticity has been to manipulate NMDARs, which are critical for the induction of long-term potentiation at Schaffer collateral and CA3 autoassociational synapses,. In one study, the NMDAR antagonist CPP was injected before the learning of new goal locations within an already familiar environment. While any synaptic plasticity engaged in encoding the environment would have likely already occurred, the specific reconfiguration of CA1 place cell representations caused by the learning phase would still be liable to perturbation. Consistently, while the mean firing response within eSWRs was not impaired under CPP, the learning-enhanced sleep reactivation of co-activity patterns observed at goal locations was prevented. It would be interesting to know whether the specific blockade of hippocampal NMDARs, rather than systemic CPP injections, has the same effect.

12In another study that seemingly challenges the synaptic plasticity model, mice with NMDAR1 knockout (KO) specifically in CA3 pyramidal cells and, therefore, with an absence of NMDAR-dependent long-term potentiation (LTP) at CA3 autoassociational synapses, were found to show stronger, less variable replay of CA1 place cell activity compared with control mice. In this experiment, the mice did not have prior exposure to the environment. During familiarization, lap-by-lap correlations in spiking activity between place cells increased, which is a measure of cell assembly formation. This increase and subsequent plateauing was still observed in the KO mice, but at a reduced rate compared with controls, not only indicating a role for CA3 synaptic plasticity, but also suggesting that an alternative, potentially plastic, compensatory mechanism, possibly via an alternative autoassociative network, such as DG-CA3, was engaged in binding place cells into coordinated assemblies in these mice. The alternative mechanisms engaged by the NMDAR1 KO mice may have led to the stronger cascading replay activity observed. Therefore, while CA3 NMDAR-dependent synaptic plasticity may not be necessary for the expression of replay per se, these studies suggest that NMDAR activity and, as a by product, hippocampal synaptic plasticity, are critically involved in dynamically configuring the hippocampal network into a state that can subsequently bias the SWR activity content. In line with this, while the blockade of NMDARs during learning of new goal locations impaired the sleep reactivation of new place cell representations, it unexpectedly promoted that of old representations.

13This model is supported by evidence suggesting that reactivation in SWRs is expressed as a function of potentially plasticity-inducing experience. For example, co-activation during awake and sleep SWRs is stronger for cell pairs with overlapping place fields,, which is a requirement that is critical for the induction of long-term potentiation at Schaffer collateral synapses. Indeed, reactivation in iSWRs improves with experience during exploration, in a manner dependent on the repetitiveness of the task and, therefore, the likelihood of place cells to be co-active. Consistently, reactivation during sleep was found to be dependent on the number of times place cells fired together in short windows (<50 ms) during exploration, that is, windows compatible with spike timing-dependent plasticity. Although, since asymmetrical cross-correlations during exploration between cell pairs were not required for reactivation in SWRs, it is debatable whether spike timing-dependent plasticity per se is the plasticity mechanism utilized by such a model. By contrast, another carefully designed study found no relation between awake replay and experience. Poorly and extensively experienced trajectories were replayed in similar proportions, never-experienced shortcut sequences were observed during SWRs, and replay was more representative of a scenario independent of experience.

14This latter observation casts doubt on whether this model alone can sufficiently explain all observable replay phenomena. It is difficult to reconcile a plasticity mechanism that could bind ensembles of cells together in a manner that would enable backward replay preferentially during exploratory behaviors but forward replay during sleep,,. Moreover, the preferential reactivation in SWRs of novel locations or environments over familiar ones,, (although see ), the stronger reactivation on rewarded trials over unrewarded trials, and the enhanced reactivation of firing patterns surrounding reward sites, are incompatible with the above model when considered in the absence of neuromodulatory drive.

15The hippocampus receives constant inputs related to the behavioral state of the animal, including those leading to the release of neuromodulators, which dramatically transform the functional output of neural circuitry. Acetylcholine and dopamine are two such factors whose action within the hippocampus bears particular relevance when considering hippocampal processing during spatial navigation and memory tasks. Here, we propose that these neuromodulatory factors during specific behavioral epochs can explain the observed activity of place cells within SWRs that may otherwise be considered inconsistent with the lingering excitability or synaptic plasticity models when viewed in isolation. Given their different spatiotemporal profiles of release, we propose acetylcholine and dopamine to have different functional roles for the processes underlying memory consolidation. However, at times of simultaneous cholinergic and dopaminergic release, these roles likely occur concomitantly.

16Microdialysis measurements of hippocampal acetylcholine levels show variation throughout the sleep–wake cycle, with acetylcholine high during rapid eye movement (REM) sleep and active wakefulness but decreasing levels during quiet wakefulness and SWS. SWRs are generally believed to be initiated when subcortical, particularly cholinergic, drive to the hippocampus is reduced,. Accordingly, optogenetic stimulation of cholinergic medial septal neurons strongly suppressed SWRs in awake and anaesthetized animals, while muscarinic receptor activation suppressed SWRs in vitro,. This could explain why exploratory eSWRs observed during periods of high cholinergic tone occur at a reduced rate compared with awake immobility iSWRs, when the cholinergic tone is reduced. These findings suggest that firing activity during eSWRs and i/sSWRs is differentially modulated by acetylcholine.

17Within the hippocampus, acetylcholine exerts wide-ranging cellular and synaptic effects. At the pyramidal cell level, acetylcholine causes membrane depolarization, increased input resistance,,, and enhanced NMDAR currents, specifically via the inhibition of SK channels,. Since somatic depolarization facilitates the emergence of place cell spiking in previously silent CA1 pyramidal cells, during eSWRs (but to a lesser extent in iSWRs and not in sSWRs), cholinergic-mediated depolarization would be predicted to facilitate the contribution of subthreshold place-related drive and, thus, the lingering excitability model, to the reactivation of place cell activity. Indeed, place cells have reduced firing rates following pharmacological inactivation of the medial septum, or pharmacological blockade of muscarinic receptors.

18In addition, muscarinic receptor activation by endogenously released acetylcholine in vivo,, or by pharmacological manipulations in vitro,, lowers the threshold for long-term potentiation of excitatory synaptic transmission in the hippocampus. Notably, overlapping CA1 place cell activity was able to engage LTP in CA1 only if sufficient cholinergic tone was present in vitro. Therefore, during exploratory activity, acetylcholine likely has a permissive role in the coupling of place-related cell assemblies by synaptic plasticity. It is also possible that the exploration-related cholinergic tone during eSWRs promotes the ability of intra-SWR place cell activity (i.e., replay activity itself) to generate synaptic plasticity. Accordingly, an interesting prediction from this framework is that sSWR activity, while still important for memory consolidation,, may have a reduced likelihood of inducing plasticity compared with waking SWRs, because cholinergic tone declines. Surprisingly, however, while SWRs have long been posited to provide temporal windows for synaptic plasticity within the hippocampus and in downstream areas,, few studies have tested whether SWR-driven spiking can induce synaptic plasticity.

19Clearly, the impact of acetylcholine on place cell activity during exploration and SWR activity is complex and deserves further investigation by experimentation and computational modeling. Meanwhile, the available literature, outlined above, points towards a model whereby the behavioral state of the animal influences how the hippocampus engages both the lingering excitability and synaptic plasticity mechanisms to initiate replay activity, in a manner strongly shaped by the cholinergic tone (Figure 3). This new framework may go some way to explain the directional bias of replay in different behavioral states,.

20The hippocampus is also innervated by dopaminergic mesencephalic neurons from the ventral tegmental area (VTA) and substantia nigra, although this innervation is sparse, with hippocampal dopamine concentrations much lower than in other brain areas, such as the striatum. Notable recent work also suggests that hippocampal dopamine can be released from noradrenergic neurons from the locus coeruleus. VTA neurons exhibit bursting in response to reward or reward-prediction stimuli (reviewed in ) and display increased firing, with a higher propensity to fire in bursts, during exposure to novel environments. This is associated with increased dopamine release in downstream areas, including the hippocampus.

21Interestingly, place cell ensembles are more reactivated in sleep SWRs following the exploration of novel locations and/or environments,, and following reward-driven learning tasks,. During SWRs at reward locations, there is also a higher probability of pyramidal cell firing on rewarded versus unrewarded trials, and cells with place fields surrounding the reward site have an increased likelihood of firing in both these reward SWRs and subsequent sSWRs,. Moreover, sequential replay has been shown to be biased towards goal and/or reward sites, with forward and backward replay preferentially representing sequences approaching or ending at the reward site, respectively,.

22Therefore, it is tempting to make the conjecture that dopamine release during exposure to spatial novelty or rewarded outcomes biases the content of subsequent SWR activity for the purpose of memory to represent locations spanning the entire novel environment or the particular behaviorally salient location, respectively. Concordantly, a recent study has shown that burst stimulation of VTA dopaminergic neurons, during exposure to a novel environment (to further enhance novelty-increased VTA firing ), subsequently enhanced hippocampal reactivation in a D1/D5 receptor-dependent manner. Neither the general activity of CA1 pyramidal cells during the awake and sleep periods (mean firing rate, SWR-firing rate response, or preferred theta phase) nor the sSWR incidence were modified by such an intervention. These findings suggest that dopamine promotes the consolidation of new memories by the sleep reactivation of newly formed firing patterns. Along this line, it has been further shown that, during the memory retention test of a hippocampal-dependent goal-directed task on a crossword maze, CA1 place maps formed during learning were only partially reinstated and behavioral performance was degraded. However, photostimulation of VTA dopaminergic fibers in dorsal CA1 during learning enhanced SWR reactivation of newly established place cell assemblies. This was accompanied by improved reinstatement of these firing patterns in the retention test and a stable behavioral performance. These findings are consistent with other previous findings showing that midbrain dopaminergic neurons can promote hippocampal place cell dynamics related to memory processing (e.g.,,, ) and numerous findings providing additional support at the behavioral level (e.g.,,, ).

23It is unclear how dopamine may bias SWR activity through the lingering excitability model since the effects of dopamine on hippocampal pyramidal cell excitability are mixed, with some studies reporting a decrease in excitability, for example by hyperpolarizing the membrane potential and augmenting the spike after hyperpolarization, while others report an increase in excitability. However, in both novel environments, and during learning on a goal-driven spatial navigation task, hippocampal place cells remap their activity,,. The stability of these new place cell representations, but not the initial formation, is NMDAR dependent, and can be facilitated by D1/D5 receptor activation or optogenetic stimulation of VTA dopaminergic terminals in CA1. Indeed, pharmacological inhibition of VTA neurons has been shown to impair CA1 place cell stability. These results support a model where dopamine release in novel environments or during reward-driven spatial learning facilitates synaptic plasticity, which then stabilizes place cell activity. Specifically, this may occur via the permissive role of dopamine in the transition from early to late long-term potentiation, potentially through the synaptic tagging and capture hypothesis, as reviewed elsewhere,. It should also be noted that increased CA1 pyramidal cell firing and changes in CA1 interneuron firing during novel exploration could also contribute to enhanced synaptic plasticity,,,,. In agreement, novel environment exposure facilitated the induction of LTP at Schaffer collateral synapses to a weak conditioning stimulus in vivo and this facilitatory effect was abolished by D1/D5R antagonists or mimicked by D1/D5R agonists. Consequently, via the synaptic plasticity model, dopaminergic modulation likely biases replay to preferentially reactivate cell assemblies relevant to novelty or reward (Figure 4).

24How the brain informs the dopaminergic systems about spatial novelty or rewarded outcomes remains to be investigated. This likely involves a complex network of multiple brain circuits,,. For instance, it has been recently shown that VTA dopaminergic cells receive spatiocontextual inputs from the hippocampus via the lateral septum. Moreover, how dopamine release and the firing activity of dopaminergic neurons also relates to aspects of motor actions, in addition to reward prediction and spatial novelty, remains to be disentangled.

25In conclusion, we have proposed a new conceptual framework for understanding the ordered sequential activation of prior waking activity in SWRs. This likely occurs via a combination of mutually nonexclusive mechanisms, since none of these can explain the available literature in isolation. While many questions remain (Box 3), our current understanding leads us to suggest that the lingering excitability model largely dictates local replaying sequences during awake behavior, while the synaptic plasticity model contributes to subsequent nonlocal awake replays and replaying activity during future rest. Within this framework, we have proposed that cholinergic tone is important for shaping the direction of replay in different behavioral states. Meanwhile, we have identified dopaminergic release during, for example, novel environments and reward-driven spatial tasks, as being important for biasing the content of subsequently replaying trajectories, potentially to strengthen new place cell assemblies and place-reward associations. Testing this new framework experimentally and computationally will be an important step forward for our understanding of the field.