Dopamine neurons drive fear extinction learning by signaling the omission of expected aversive outcomes

1The ability to learn which stimuli predict danger is crucial for survival but it is equally important to adapt behavior when those stimuli no longer represent a threat. One classic example of this is fear extinction learning, during which the repeated presentation of a stimulus (conditioned stimulus, CS) that no longer predicts an aversive outcome (unconditioned stimulus, US) leads to a gradual decrease in learned fear responses. Many anxiety disorders, such as post-traumatic stress disorder, are characterized by impaired extinction learning and thus understanding the neural basis of fear extinction has clinical significance.

2A large body of evidence indicates that fear extinction represents new learning rather than forgetting or the erasure of the original fear memory. In order to initiate extinction learning, the absence of the expected aversive outcome must be detected and signaled to the brain regions mediating fear extinction. Decades of research on fear extinction has revealed that a distributed network of brain structures including the amygdala, medial prefrontal cortex and hippocampus mediates the acquisition, consolidation and retrieval of fear extinction memories. However, none of these structures have been shown to signal the absence of the expected aversive outcome during fear extinction. The neural substrates of such a signal that could initiate extinction learning have therefore remained elusive.

3New learning is initiated when outcomes violate expectations. Such violations are thought to cause ‘prediction error’ signals that initiate the neural processes which ultimately lead to changes in behavior. During fear extinction, the absence of the US is an unexpected event and likely generates a prediction error signal that initiates extinction learning. More specifically, the omission of the aversive US can be conceptualized as a better-than-expected outcome. It is well-established that the activity of midbrain dopamine (DA) neurons represents the degree to which outcomes are better or worse than expected. For example, many DA neurons increase their firing to rewards that are either unexpected or better than expected and this DA signal is sufficient to drive reinforcement learning. Based on our data from human studies, we have previously proposed that DA neurons could provide a prediction error-like signal during the aversive US omission to initiate fear extinction. Consistent with this, an increase in DA release has been observed in the nucleus accumbens (NAc) during fear extinction and pharmacological blockade of DA receptors in the NAc impairs fear extinction. However, the electrical activity of DA neurons during fear extinction − particularly at the time of the aversive US omission − and its relationship to extinction learning, is incompletely understood. In this study, we hypothesized that the unexpected omission of the aversive US activates DA neurons in the ventral tegmental area (VTA) and that this signal is necessary to initiate normal fear extinction learning. To test this hypothesis, we used in vivo single-unit recordings, DA neuron-specific calcium recordings and bi-directional optogenetic manipulations in behaving mice to examine and causally test the role of VTA DA neurons in fear extinction.

4We first examined whether DA neurons in the VTA are activated by the unexpected omission of the aversive US during fear extinction. We recorded the single-unit spiking activity of VTA neurons (Figure 1A–C;Figure 1—figure supplement 1) in mice (n = 11) that were trained in a fear conditioning paradigm (Figure 1D–F;Figure 1—figure supplement 2A) where a tone (CS) was paired with an aversive foot shock (US). Twenty-four hours after fear conditioning, mice received an extinction session consisting of CS presentations in the absence of the aversive US. A total of 43 (out of 90) and 40 (out of 75) VTA neurons classified as ‘putative’ DA neurons (see Materials and methods; Figure 1—figure supplement 3) were recorded during day 1 and day 2, respectively. Analysis of neuronal firing rates during the time of the US omission revealed that 25% of putative DA neurons (10 of 40) exhibited a significant increase in firing rate to the omission of the aversive US during the early extinction trials (E-Ext: average of first 10 CSs) when the US omission was unexpected (Figure 1G-I;Figure 1—figure supplement 3B; see Materials and methods for details). This was not simply a response to the CS offset since only 2.3% of neurons (1 of 43; Figure 1H;Figure 1—figure supplement 3A) showed increased firing at the end of the CS during tone habituation (Hab). On the other hand, during late extinction trials (L-Ext: average of last 10 CSs), when the US omission was no longer unexpected and animals showed significant extinction of fear responses (Figure 1F), only 7.5% of putative DA neurons (3 of 40; Figure 1G-I) showed an increase in firing to the absence of the US. Importantly, there was no difference in freezing levels during the CS and the post-CS period in E-Ext (Figure 1—figure supplement 2B) suggesting that the observed increase in DA neuron firing was not due to an increase in movement when the CS terminates.

5Furthermore, analysis of the distribution of z-scores during the US omission (Figure 2A) revealed that none of the putative DA neurons (0 of 40) showed a selective decrease in firing to the US omission (see Materials and methods for details) during either E-Ext or L-Ext. This suggests that the dominant response of putative DA neurons to US omission was excitatory. Consistent with this, we observed a significant increase in firing to the US omission at the population level when we examined the average response of all putative DA neurons during E-Ext (paired t-test, t(39) = 2.22, p = 0.03), but not L-Ext (paired t-test, t(39) = 0.20, p = 0.83) or Hab (paired t-test, t(42) = 0.71, p = 0.47; Figure 1J).

6Local circuit interactions between DA and GABA (γ-aminobutyric acid) neurons in the VTA underlie reinforcement learning. It is therefore possible that the DA and local GABA neurons also interact to drive fear extinction learning. For instance, the increased firing of putative DA neurons could be mediated by disinhibition resulting from inhibition of local GABA neurons at the time of the US omission. To test this possibility, we analyzed the activity of putative non-DA neurons – the subset of VTA neurons which likely dominantly includes GABA cells – during the time of the US omission and found that 8.5% of neurons (3 of 35) showed a selective decrease in firing to the US omission during E-Ext (Figure 1—figure supplement 3B, Figure 2B). However, this was not significantly different from the proportion of non-DA neurons (4.2%, 2 of 47 neurons) showing decreased firing at the CS offset during Hab (Figure 1—figure supplement 3B, Figure 2B; Fisher’s exact test, p = 0.64). These results therefore suggest that the excitation observed in the putative DA neurons during US omission is unlikely mediated by the activity of the local GABA neurons. Moreover, only a small proportion of putative non-DA neurons showed increased firing at the time of the US omission during E-Ext (2.8%, 1 of 35 neurons; Figure 1—figure supplement 3B, Figure 2B). This proportion was again not different from the 2.1% of neurons (1 of 47) that showed increased firing during Hab (Figure 1—figure supplement 3B, Figure 2B; Fisher’s exact test, p = 1). Together, these results suggest that the putative non-DA, likely local GABA, neurons in VTA do not change their firing at the time of the US omission and that the US omission is signaled specifically by the putative DA neurons in VTA.

7In contrast to the uniform response observed during US omission, the responses of putative DA neurons during the CS were diverse. Figure 1—figure supplement 4 shows example neurons that displayed excitation, inhibition and also biphasic response during the CS. Whereas some neurons showed sustained responses to the CS (Figure 1—figure supplement 4B), others showed a transient response at the onset of the CS (Figure 1—figure supplement 4C). In order to quantify these CS-evoked responses, we calculated the mean response to the CS for each neuron during Hab, E-Ext and L-Ext (Figure 1—figure supplement 4A). We found that 5% (2 of 40) of putative DA neurons exhibited CS-evoked excitation and 7.5% (3 of 40) CS-evoked inhibition during E-Ext. However these percentages were not significantly different from the proportion of cells that showed excitation (6.9%, 3 of 43 neurons; Fisher’s exact test, p = 1) or inhibition (0%, 0 of 43; Fisher’s exact test, p = 0.1) during Hab. Furthermore, when we examined the population activity by averaging the response of all putative DA neurons, there was no significant change in the average response during the CS (paired t-tests, Hab: t(42) = 0.58, p = 0.56, E-Ext: t(39) = 1.89, p = 0.065, L-Ext: t(39) = 1.23, p = 0.22; Figure 1J;Figure 1—figure supplement 4A). These results suggest that the average activity of putative DA neurons in the VTA did not change during the CS even though different subsets of cells showed CS-evoked excitation and inhibition. A lack of strong responses to the CS might be due to the cued fear conditioning paradigm that we used in our study. It has recently been shown that DA neurons respond strongly to the CS when a discriminative fear conditioning task is used and the strength of their response increases with increasing discrimination between the aversive and the safe CS.

8The above results show that putative DA, but not non-DA, neurons in the VTA signal the omission of the aversive US during fear extinction, specifically during the beginning of extinction learning when the US omission is unexpected. To further confirm that DA neurons signal the unexpected omission of the US, we next measured activity-dependent calcium signals selectively in DA neurons using fiber photometry. To this end, a Cre-dependent adeno-associated virus (AAV) expressing the genetically encoded calcium (Ca+2) indicator gCaMP6 was injected, and an optical fiber implanted, in the VTA of transgenic mice expressing Cre recombinase under the control of the dopamine transporter (Dat) promoter (DAT-Cre mice; Figure 3A–B and Figure 3—figure supplement 1A). In these mice, Cre expression is highly selective for dopamine neurons. Accordingly, we observed a high degree of overlap between Cre-dependent gCaMP6 expression and immunohistochemical staining against tyrosine hydroxylase (TH; Figure 3—figure supplement 1B–C). In control mice, we injected a Cre-dependent AAV expressing GFP to examine whether changes in fluorescence could be independent of neuronal activity. Recordings from gCaMP6-expressing animals revealed transient increases in fluorescence whereas such increases were absent in mice expressing GFP (Figure 3C). Furthermore, consistent with electrophysiological studies in VTA DA neurons, we also confirmed that reward delivery caused large increases in fluorescence in the gCaMP6-expressing mice (Figure 3—figure supplement 2, also see Materials and methods), indicating that this Ca+2 signal is indeed generated by DA neuron activity.

9Both gCaMP6- and GFP-expressing animals underwent the same fear conditioning protocol as in the electrophysiology experiment (Figure 3D–E) and showed comparable levels of freezing to the CS across sessions (two-way repeated measures ANOVA; main effect of group: F1,24 = 0.04, p = 0.83; group × trial interaction: F2,24 = 0.04, p = 0.96; Figure 3F and Figure 3—figure supplement 3A). During E-Ext, we observed a significant increase in the Ca+2 signal of gCaMP6 animals at the time of the US omission compared to the pre-CS baseline (p<0.01, sign-rank test) and the GFP control group (p<0.01, rank-sum test; Figure 3G). On the other hand, during L-Ext, when the US omission was no longer unexpected, the Ca+2 signal did not change during the post-CS period (p = 0.85, sign-rank test; Figure 3G). No changes in fluorescence were observed in GFP-expressing animals during either E-Ext or L-Ext (Figure 3G). Furthermore, we did not observe any change in the Ca+2 signal during Hab (gCaMP6 group, p = 0.32, sign-rank test; Figure 3G) suggesting that the increase during US omission is unlikely a response to the CS offset. Furthermore, freezing levels during the CS and the post-CS period in E-Ext were comparable (Figure 3—figure supplement 3B) ruling out the possibility that an increase in movement when CS terminates might have resulted in the observed increase in Ca+2 signal at the time of the US omission during E-Ext. Notably, these results obtained by measuring the population Ca+2 signal from DA neurons are consistent with the results of the electrophysiology recordings where we found a significant increase in the average population activity of all putative DA neurons at the time of the US omission during E-Ext (Figure 1J).

10If activation of DA neurons at the time of the unexpected US omission drives extinction learning then this Ca+2 signal during E-Ext should be larger in animals that exhibit better extinction learning. To test this, we took advantage of the variability in the extinction learning rates of individual mice and asked whether they were correlated with the Ca+2 signal at the time of the US omission during E-Ext. This revealed a significant correlation between the Ca+2 signal during E-Ext and the change in freezing from E-Ext to L-Ext (Spearman’s correlation = = 0.83, p = 0.0037; Figure 3H). A significant correlation was also observed between the Ca+2 signal during E-Ext and the freezing levels during L-Ext (Spearman’s correlation = 0.92, p = 0.0003). However, it is possible that the variation in the Ca+2 signal during E-Ext might be due to the variation in recording locations in the VTA across animals rather than reflecting the relationship with extinction learning. We reasoned that differences in recording locations would likely result in variation in reward responses. We therefore examined the correlation between the change in freezing from E-Ext to L-Ext and reward responses and did not find a significant relationship between these two variables (Spearman’s correlation = 0.32, p = 0.36; Figure 3I). These results therefore suggest that the magnitude of the Ca+2 signal during E-Ext correlated with the level of extinction learning and the variations in the magnitude of this signal were not due to differences in the recording location.

11Contrasting with the uniform increase in DA neuron activity during US omission (Figure 3G), the responses to the CS varied across animals (Figure 4A). Some animals showed increased and some decreased fluorescence during the CS in E-Ext (Figure 4A–B). Accordingly, consistent with single unit results which showed no change in CS-evoked population activity of putative DA neurons (Figure 1J; Figure 1—figure supplement 4A), the Ca+2 signal evoked by the CS was not significantly different from the baseline during E-Ext (p = 1.0, sign-rank test) or Hab (p = 0.43, sign-rank test) when we averaged the CS responses of all animals (Figure 4A).

12Overall, these findings together with our single-unit results demonstrate that DA neurons signal the unexpected omission of the aversive US during fear extinction and that the magnitude of this DA signal predicts the strength of extinction learning.

13We next asked whether the observed increase in DA neuron firing at the time of the unexpected US omission is necessary for fear extinction learning. To address this question, we optogenetically inhibited DA neurons in the VTA at the time of the US omission during fear extinction. DAT-cre mice received bilateral injections of a Cre-dependent AAV expressing either the light-activated inhibitory opsin halorhodopsin fused with enhanced yellow fluorescent protein (NpHR-eYFP) or eYFP only (eYFP control) into the VTA, as well as bilateral implantation of optical fibers above the VTA to allow for selective inhibition of VTA DA neurons (Figure 5A–C and Figure 5—figure supplement 1A). We observed a high degree of overlap between Cre-dependent NpHR-eYFP expression and immunohistochemical staining against TH (Figure 5—figure supplement 1B–C) suggesting DA neuron-selective expression. Furthermore, we confirmed that optical stimulation of NpHR inhibits DA neuron firing in awake DAT-cre mice (Figure 5—figure supplement 2).

14Mice were trained in a fear conditioning protocol (Figure 5D) consisting of 4 CS-US pairings on day 1. Twenty-four hours after fear conditioning, mice received an extinction session. In the experimental group expressing NpHR-eYFP light was delivered bilaterally to the VTA to inhibit DA neurons specifically at the end of each CS presentation, that is during the time of the US omission (Paired-NpHR, n = 7; Figure 5E). The behavior of the experimental group was compared to two control groups: one group consisted of mice expressing eYFP only which received the identical light delivery (Paired-eYFP, n = 7) and a second group consisted of mice expressing NpHR-eYFP that received light delivery to inhibit DA neurons during the intertrial intervals (ITIs; Unpaired-NpHR, n = 8; Figure 5F).

15Compared to the two control groups, the Paired-NpHR group exhibited high freezing levels to the CS throughout the extinction session, suggesting impaired extinction learning (Figure 5G). A two-way repeated measures ANOVA revealed a significant main effect of group (F2,456 = 7.03, p = 0.0052) and a significant interaction between group and trial (F48,456 = 2.19, p<0.0001). Consistent with this, there was a significant difference between the Paired-NpHR group and the Paired-eYFP (p<0.001) or Unpaired-NpHR (p<0.001) controls during L-Ext (Figure 5I). Furthermore, impaired extinction learning resulted in a weaker memory for extinction when tested the next day (extinction recall test; two-way repeated measures ANOVA; main effect of group: F2, 456 = 7.1, p = 0.005; group × trial interaction: F48, 456 = 1.49, p = 0.02; Figure 5G). Consistently, during the early extinction recall trials (E-Ext Rec: first 10 CSs) the Paired-NpHR group froze significantly more compared to the Paired-eYFP (p<0.001) or the Unpaired-NpHR (p<0.001) controls (Figure 5J). In contrast to the Paired-NpHR group, the Unpaired-NpHR group behaved comparable to the Paired-eYFP control group during both extinction and extinction recall tests (Figure 5G–J) suggesting that optical inhibition of DA neurons per se did not result in a nonspecific increase in freezing levels, and that the behavioral effect was dependent on the temporally specific inhibition of DA neurons during the time of the US omission. Furthermore, there was no difference between the groups in their fear acquisition on day 1 (two-way repeated measures ANOVA, no main effect of group, F2,57 = 0.3, p = 0.74 or no group × trial interaction, F6,57 = 0.94, p = 0.47; Figure 5G) and all groups showed comparable levels of freezing at the start of extinction (first CS; one-way ANOVA, F2,19 = 2.06, p = 0.15; Figure 5H) before any experimental manipulation took place, ruling out the possibility that differences in the strength of fear memory on day 2 between groups might have caused the observed effect.

16Importantly, freezing levels of the NpHR group at the beginning of extinction recall were comparable to freezing levels at the beginning of extinction (paired t-test comparing first CS of extinction and first CS of extinction recall, t(6) = 0.56, p = 0.59), suggesting that no significant extinction learning happened in these animals. Furthermore, we found that the extinction rate of the Paired-NpHR group during extinction recall, in the absence of optogenetic inhibition, was comparable to the extinction rate of the Paired-eYFP and Unpaired-NpHR groups during extinction (two-way repeated measures ANOVA; no main effect of group F2, 456 = 1.02, p = 0.37 and group × trial interaction: F48, 456 = 0.86, p = 0.73). This suggests that our manipulation did not have a nonspecific long-term effect on the ability of the Paired-NpHR group to exhibit extinction learning. These results also suggest that our optogenetic manipulation likely did not affect the strength of the fear memory. Taken together, these findings demonstrate that DA neuron activation by the unexpected omission of the US is necessary for fear extinction learning.

17If DA neuron firing at the time of the unexpected US omission drives fear extinction, then enhancing this DA signal should accelerate fear extinction learning. To test this, we optogenetically excited DA neurons precisely at the time of the US omission during fear extinction learning. DAT-cre mice were bilaterally injected with a Cre-dependent AAV expressing either channelrhodopsin-2 (ChR2) fused with eYFP (ChR2-eYFP) or eYFP only (eYFP control) into the VTA, and implanted bilaterally with optical fibers above VTA (Figure 6A–C; Figure 6—figure supplement 1A). There was again a high level of overlap between Cre-dependent ChR2-eYFP expression and immunohistochemical staining against TH (Figure 6—figure supplement 1B–C) suggesting DA neuron-selective expression of ChR2. Furthermore, we confirmed that optical stimulation of ChR2 induces firing of DA neurons in awake DAT-cre mice (Figure 6—figure supplement 2).

18Mice were trained in a fear conditioning protocol (Figure 6D) similar to the optogenetic inhibition experiment. The experimental group consisted of ChR2-eYFP expressing mice which received light stimulation of DA neurons specifically at the time of the US omission (Paired-ChR2, n = 7; Figure 6E). Two control groups, one expressing eYFP only that received the identical light delivery (Paired-eYFP, n = 7) and the other expressing ChR2-eYFP that received light excitation during the ITIs (Unpaired-ChR2, n = 7; Figure 6F), were used to control for nonspecific effects of light and DA neuron stimulation, respectively.

19As expected, all groups showed a gradual decrease in freezing to the CS during the extinction session. However, in the Paired-ChR2 group, freezing decreased faster than in the control groups suggesting accelerated extinction learning (Figure 6G). A two-way repeated measures ANOVA comparing freezing levels confirmed this observation by revealing a significant main effect of group (F2,432 = 4.1, p = 0.03). Comparison of freezing levels in the three groups, particularly during E-Ext, revealed a significant difference between the Paired-ChR2 group and the Paired-eYFP (p<0.05) or Unpaired-ChR2 (p<0.05) controls (Figure 6I) suggesting accelerated extinction learning. On the other hand, the two control groups behaved comparably (p>0.05). These results suggest that optical excitation of DA neurons per se did not result in a nonspecific decrease in freezing levels and that the temporally precise excitation during the US omission is necessary for the observed behavioral effect. Furthermore, there was no difference between the groups in their fear acquisition on day 1 (two-way repeated measures ANOVA, no main effect of group, F2,54 = 0.17, p = 0.84 and no group × trial interaction, F6,54 = 0.65, p = 0.68, Figure 6G) and all groups showed comparable levels of freezing during the first CS of extinction (one-way ANOVA, F2,18 = 0.81, p = 0.66, Figure 6H) before light stimulation began. Thus, excitation of DA neurons precisely at the time of the unexpected US omission is sufficient to accelerate fear extinction learning. Finally, Paired-ChR2 mice spent less time freezing to the CS compared to the control groups during the extinction recall test (two-way repeated measures ANOVA; main effect of group: F2,162 = 5.5, p = 0.013 and group × trial interaction: F18,162 = 2.52, p = 0.0011, Figure 6G) suggesting that the accelerated extinction learning resulted in a stronger extinction memory.

20An alternative possibility is that the low level of freezing in the Paired-ChR2 group during Ext Rec was due to an effect of optical stimulation of DA neurons on the fear memory. For instance, the optogenetic manipulation could result in the erasure of the fear memory by impairing the memory reconsolidation process rather than accelerating extinction learning and strengthening extinction memory. To rule this possibility out, we tested the animals on a fear renewal test on day 4 by presenting 5 CSs in the conditioning context. It is well established that extinction learning is context-dependent such that if the animals are tested in a different context than the one they are extinguished in, fear responses return, a phenomenon called fear renewal. Therefore, if low level of freezing during Ext Rec was due to impaired reconsolidation then we would expect to see impaired fear renewal in Paired-ChR2 group. However, we found that all groups showed high freezing to the CS during the renewal test and that there was no difference in the freezing levels between the groups (Figure 6—figure supplement 3; two-way repeated measures ANOVA, no significant effect of group, F2, 72 = 0.33, p = 0.72 and group × trial interaction, F8, 72 = 0.86, p = 0.55) suggesting that all three groups showed comparable levels of fear renewal. This suggests that the low level of freezing in Paired-ChR2 group during extinction recall was not due to an affect of our optogenetic manipulation on the fear memory but rather was due to enhanced extinction learning and memory formation. Taken together, these findings demonstrate that increasing DA neuron activity at the time of US omission — and thus enhancing an endogenous extinction mechanism — is sufficient to accelerate extinction learning and strengthen extinction memory.

21Here we demonstrated that DA neurons were activated by the omission of the aversive US during fear extinction, specifically during the beginning of extinction when the US omission is most unexpected. Importantly, the magnitude of this DA signal correlated with the strength of extinction learning. Furthermore, temporally specific optogenetic inhibition of DA neurons at the time of the US omission prevented extinction, demonstrating that this signal is necessary for normal fear extinction. Conversely, enhancing this DA signal using temporally-specific optogenetic excitation was sufficient to accelerate extinction learning. Together, these results identify a crucial role of DA neurons in signaling the unexpected omission of aversive outcomes and thereby driving fear extinction learning.

22Previous studies have shown that DA neurons encode a reward prediction error, or the discrepancy between expected and actual rewards, which acts as a teaching signal for reinforcement learning. Specifically, presentation of unexpected or better than expected rewards induces increased firing in DA neurons. Our results suggest that DA neurons might also signal a better than expected outcome during fear extinction. We found that a subpopulation of putative DA neurons in the VTA increased their firing selectively at the time of the US omission during fear extinction. This US omission-responsive firing was observed specifically during the early trials of extinction when the absence of the US was unexpected, and was significantly reduced during the late stages of extinction when the US omission was no longer unexpected and animals showed significant extinction of fear responses. Importantly, these responses were not observed in putative non-DA neurons. This suggests that DA neurons encode a prediction error-like signal during fear extinction learning. Ca+2 recordings selectively in DA neurons further confirmed these results and revealed that this DA signal correlated with the strength of extinction learning. Recent studies have shown that DA neurons not only encode reward prediction errors but also signal prediction errors to gate fear learning and to drive threat avoidance. Interestingly, omission of aversive stimuli during fear extinction in fruit flies is encoded by the DA system that mediates reward, but not aversive, learning. Whether the DA signal during fear extinction in mammals is also similar to prediction error signals for reward and mediated by the brain’s reward circuitry will be important questions for future studies.

23Consistent with our results, previous studies using partial reinforcement paradigms have shown that DA neurons exhibit increased firing to the unexpected omission of aversive stimuli (;; but see ). However, responses to aversive US omission in these studies were much smaller compared to what we observed. There are several differences between these studies and ours that might account for this. The aversive US used in our study is a painful footshock whereas an air-puff was used in previous studies. Furthermore, it has been shown that the valence of the testing context influences DA neuron responses to the omission of the aversive US. Matsumoto and colleagues (2016) found a significant increase in DA neuron responses to the omission of aversive airpuff in a low reward, but not high reward, context. In our study, the animals were only fear conditioned and no reward learning happened prior to or during fear conditioning. Therefore, the context was exclusively aversive. In addition, the CS-US contingency during fear conditioning was higher in our study than in partial reinforcement tasks used in previous studies. The CS predicted the shock with 100% probability at the end of fear conditioning and therefore, the omission of the US was fully unexpected at the beginning of fear extinction in our study. On the other hand, in the studies using partial reinforcement tasks the CS-US contingency was 25–90% and the omission of the US was performed intermittently. The US omission was therefore arguably less unexpected. Together, these factors might therefore account for the differences in DA neuron responses between these studies and ours. Overall, our findings suggest that DA neurons not only signal better than expected rewards but also better than expected outcomes more generally, such as the omission of an aversive event.

24Detecting the discrepancy between expected and actual outcomes is critical for new learning. During fear extinction, the omission of the aversive US is an unexpected outcome which initiates new learning about the CS, specifically that it no longer predicts danger. However, how this learning is initiated at the neuronal level has remained unknown. Here, by using bidirectional optogenetic manipulations, we demonstrated that the DA signal during the omission of the aversive US drives normal fear extinction learning: inhibiting this DA signal prevented, while enhancing it accelerated, normal extinction learning. Our results are consistent with previous findings establishing the causal role of DA neurons in reinforcement learning and further extend their role to safety learning. Consistent with our findings, a recent study inhibited VTA DA neurons during US omission in rats and also found reduced fear extinction learning. Our study replicates this finding in mice and further extends it by showing that enhancing DA neuron activity at the time of the US omission is also sufficient to accelerate normal extinction learning. Consistent with our results, it has been shown that enhancement of DA signaling by L-DOPA administration during extinction was sufficient to initiate fear extinction learning in a mouse model of impaired extinction learning. Our results therefore suggest that enhancement of DA signaling during extinction could be a potential strategy for the treatment of anxiety disorders.

25Fear extinction is mediated by a network of brain structures consisting mainly of the amygdala and the infra-limbic (IL) subregion of the medial prefrontal cortex. Although neuronal activation has been observed in these structures during different stages of fear extinction, they occur during the CS and later than the DA signal we observed in our study. In the basolateral amygdala, a subpopulation of neurons termed 'extinction neurons' increases their firing to the CS during extinction learning. However, these neurons become CS responsive late in the extinction session right before the animals show a decrease in fear responses suggesting that these neurons likely mediate inhibition of fear responses. In the IL, a structure necessary for consolidation of extinction memories, increased firing to the CS is observed during extinction recall. In contrast, the DA signal that we demonstrated in our study occurs at the early trials of the extinction session, supporting our conclusion that this signal initiates extinction learning.

26Plasticity in the amygdala and IL underlie acquisition and consolidation of fear extinction memories. Furthermore, DA signaling in the amygdala and IL has been shown to play an important role in fear extinction. Interestingly, optogenetic inhibition of DA neurons during US omission has been found to prevent extinction-related plasticity in the amygdala and IL. However, it is unclear how this DA signal induces plasticity in the amygdala and IL to underlie extinction memory. This can be mediated through direct DA projections to these structures or indirectly through a multi-synaptic circuit mechanism. The first step in addressing this issue will be to identify the projection target of these DA neurons that signal the omission of the US during fear extinction.

27Recent studies have shown that midbrain DA neurons form functionally distinct and mostly non-overlapping subpopulations based on their projection targets. Therefore, an important question is which subpopulation of VTA DA neurons generates the response to the US omission. One possible candidate is the subpopulation projecting to the NAc. Supporting this possibility, an increase in DA release around the time of the CS offset during fear extinction has been observed in the NAc. In particular, this was observed only during the early trials of extinction, consistent with our results. DA signaling in the NAc has also been shown to be important for relief learning and avoidance behavior. Whether fear extinction learning and relief and avoidance learning share related mechanisms and involve overlapping subpopulations of DA neurons projecting to NAc is not known and will be an important question for future research. Notably, pharmacological blockade of DA receptors in the NAc have been found to impair fear extinction learning. Furthermore, we have observed that fear extinction learning in humans is accompanied by a prediction error-like activation in the ventral striatum. However, at odds with these findings, inhibition of DA terminals in NAc or DA neurons projecting to the medial shell of NAc during US omission did not affect fear extinction learning, although it did impair consolidation of extinction memory, in a recent study. Our single unit results demonstrate that a small subpopulation of DA neurons mediate this DA signal to drive extinction learning. Therefore, it is possible that this subpopulation of DA neurons projects to a specific subregion of NAc that was not targeted by. In addition to NAc, other possible candidates include the DA neurons that project to the amygdala and/or IL. Identifying which projection-defined subpopulation of DA neurons signals the omission of the US to initiate fear extinction learning will be an important question for future research.

28In conclusion, our study identifies a prediction error-like signal encoded by DA neurons that is necessary to initiate fear extinction learning. Furthermore, we found that enhancing this DA signal is sufficient to accelerate extinction learning and strengthen extinction memory consolidation. Deficits in fear extinction learning are thought to underlie anxiety disorders. Our study therefore has therapeutic implications for anxiety disorders by identifying DA neuron activity as a potential target for novel treatments.

29All procedures were conducted in accordance with the guidelines of the German Animal Protection Act and were approved by the local authorities (Regierungsprasidium Darmstadt; protocol number 1038). Male C57BL/6N mice (Charles River), aged 3 months at the start of experiments, were used in the in vivo electrophysiology experiment. Male heterozygous DAT-Cre mice (; backcrossed with C57BL/6N) aged 3–6 months at the start of experiments were used in the photometry and optogenetics experiments. All experimental groups were matched for age. All mice were individually housed on a 12 hr light/dark cycle. All experiments were performed during the light cycle.

30AAV5-EF1a-DIO-hChR2(H134R)-eYFP, AAV5-EF1a-DIO-eNpHR3.0-eYFP, AAV5-EF1a-DIO- eYFP and AAV5-CAG-Flex-GFP were produced and packaged by the University of North Carolina Vector Core. AAV5-CAG-Flex-GCaMP6f-WPRE-SV40 and AAV5-CAG-Flex- GCaMP6s-WPRE-SV40 were produced and packaged by the University of Pennsylvania Vector Core.

31Animals were anesthetized using isoflurane (1–2%) and placed in a stereotaxic frame. At the onset of anesthesia, all animals received subcutaneous injections of carprofen (4 mg/kg) and dexamethasone (2 mg/kg). The animal’s temperature was maintained for the duration of the surgical procedure using a heating blanket. Anesthesia levels were monitored throughout the surgery and the concentration of isoflurane adjusted so that the breathing rate never fell below 1 Hz. After exposing the skull surface, craniotomies were made overlying the VTA (3.2 mm posterior to bregma and 0.5 mm lateral to the midline).

32For in vivo single-unit recordings, we used a moveable bundle of 5–8 stereotrodes made by twisting together two 0.0005 inch tungsten wires (M219350, California Fine Wire). The stereotrode bundle was attached to a custom-made microdrive that made it possible to advance the electrodes along the dorsoventral axis. On the day of implantation, the stereotrodes were gold-plated to reduce the impedance to 0.2–0.3 MΩ at 1 kHz. The stereotrode bundle was inserted through the craniotomy above the VTA to a depth of 3.9–4.0 mm below bregma. All electrode wires were connected to an electrode interface board (EIB-16; Neuralynx) for relaying electrophysiological signals to the data acquisition system. The microdrive was anchored to the skull using skull screws and dental cement (Paladur).

33For the photometry experiments, DAT-cre mice were injected unilaterally with 1 μl of AAV5-CAG-Flex-GCaMP6f-WPRE-SV40 (final titer 2.7 × 1012 pp per ml) or AAV5-CAG-Flex- GCaMP6s-WPRE-SV40 (final titer 6.4 × 1012 pp per ml) or AAV5-CAG-Flex-GFP (final titer 4.5 × 1012 pp per ml). Viruses were injected in the VTA (3.2 mm posterior to bregma, 0.5 mm lateral to the midline and 4.5 mm ventral to bregma) at 50 nl/min using a 10 μl syringe with a 33-gauge needle controlled by an injection pump. The needle was left in place for an additional 10–15 min before slowly being withdrawn. Following infusion of the virus, an optical fiber (400 μm core diameter, 0.48 NA, Doric Lenses) was slowly inserted through the same craniotomy into the VTA at a depth of 4.1–4.3 mm below bregma. The optical fiber was then anchored to the skull using skull screws and dental cement (Paladur).

34For optogenetic experiments, DAT-cre mice were injected bilaterally in the VTA with 1 μl of AAV5-EF1a-DIO-hChR2(H134R)-eYFP (final titer 4.3 × 1012 pp per ml), AAV5-EF1a-DIO- eNpHR3.0-eYFP (final titer 4 × 1012 pp per ml) or AAV5-EF1a-DIO-eYFP (final titer 4.4 × 1012 pp per ml) per hemisphere using the coordinates described above. Optical fibers (200 μm core diameter, 0.22 NA, Thorlabs) were implanted bilaterally above the VTA to a depth of 3.9–4.0 mm below bregma as described above.

35Fear conditioning and extinction took place in two different contexts (A and B). Context A consisted of a square chamber with an electrical grid floor (Med Associates) used to deliver footshock. Context B consisted of a white teflon cylindrical chamber with bedding material on the floor. The chambers were located inside a sound attenuating box and were cleaned with 50% ethanol and 1% acetic acid before and after each session. All mice were habituated to contexts A and B for 10–15 min each in a counterbalanced fashion. For electrophysiology and photometry experiments, mice received a tone habituation session followed by fear conditioning on day 1. Tone habituation started following a 2 min baseline period in context A and consisted of 10 presentations of the CS (4 kHz tone, 75 dB, 10 s) with a random intertrial interval (ITI) of 40–120 s. Fear conditioning consisted of five pairings of the CS with a US (1 s footshock, 0.55 mA, ITI: 40–120 s). The onset of the US coincided with the offset of the CS. On day 2, mice received an extinction session consisting of 25–30 and 30 presentations of the CS alone in context B in the electrophysiology and photometry experiments, respectively. For optogenetic experiments, on day 1 mice underwent fear conditioning consisting of four pairings of a CS (4 kHz tone, 75 dB, 30 s) with a US (1 s footshock) with a random ITI of 40–120 s in context A. The US intensity was 0.45 mA and 0.5 mA in the optogenetic inhibition and excitation experiments, respectively. On day 2, mice received an extinction session consisting of 25 presentations of the CS alone in context B. Twenty-four hours later, mice received an extinction recall test in context B. The extinction recall test consisted of 25 and 10 presentations of the CS alone in the optogenetic inhibition and excitation experiments, respectively.

36The behavior of mice was recorded to video and scored by an experienced observer blind to the experimental condition. Behavioral freezing, defined as the absence of all bodily movements except breathing-related movement, was used as the measure of fear. Animals that showed low conditioned fear (<50% freezing) at the start of the extinction session (First CS of extinction) were excluded from the study. This criterion led to the exclusion of 1 mouse from photometry and one mouse from optogenetics experiments.

37Following one week of recovery from surgery, animals were habituated to handling and moving with the wire tether connecting the microdrive to the recording system. Before fear conditioning began, stereotrodes were slowly advanced until single-units with low baseline firing rate (<10 Hz) were observed, to increase the probability of recording putative DA neurons during the task. The spike waveforms tended to change from day 1 (tone habituation and conditioning) to day 2 (extinction) even when electrodes were not moved or in case cells were lost overnight, we advanced (40–80 μm) the microdrive to find new cells on day 2. We therefore treated the cell populations recorded on these two days as independent. On each day of the fear conditioning task, single-units were first recorded for 5–10 min while animals were in their homecage to assess their baseline firing rates.

38Neural data were acquired using a 16-channel headstage (HS-18, Neuralynx) that was connected to the electrode interface board and relayed the signals to a Digital Lynx SX (Neuralynx) data acquisition system. To extract putative spikes, neural signals were bandpass filtered between 600 and 6000 Hz, and waveforms that passed the threshold (50–60 μV) were digitized at 30 kHz. One stereotrode channel that did not have any apparent units was used as reference. We also recorded the neural signals bandpass filtered between 1 and 6000 Hz to analyze the spike waveforms. In order to verify recording locations, current (50 mA, 10 s) was passed through one of the stereotrode channels to produce a lesion in the recording site at the end of the experiment.

39Spike waveforms were sorted offline into single-unit clusters using SpikeSort3D (Neuralynx). Only well-isolated single-units that displayed a clear refractory period (>1 ms) were included in subsequent analysis, performed using scripts custom-written in MATLAB (MathWorks). The VTA contains different cell types, including DA neurons and gamma-aminobutyric acid (GABA) neurons. Midbrain DA neurons typically exhibit baseline firing rates below 10 Hz in awake animals and long duration action potentials. We therefore classified neurons whose baseline firing rate (measured in the home cage) were below 10 Hz and exhibited long duration action potentials (peak-to-peak duration >450 µs) as putative DA neurons (Figure 1—figure supplement 3). Spike waveforms were analyzed using the neural signals bandpass filtered between 1 and 6000 Hz. To analyze firing rates during the task, we constructed peri-stimulus time histograms (PSTHs) aligned to the onset of the CS (−5 s to +15 s) using 1 s bins. PSTHs were calculated separately for early extinction trials (E-Ext: average of first 10 CSs), late extinction trials (L-Ext: average of last 10 CSs) and tone habituation (Hab: average of 10 CSs). These PSTHs were then normalized with a z-score transformation by subtracting the baseline firing rate (5 s pre-CS period) from each individual 1 s bin and then dividing this difference by the standard deviation of the baseline. A neuron was classified as US omission excited if it met two criteria: 1) z-score was greater than 2 at the time of the US omission (during the 1 s bin following the offset of the CS); 2) the z-score had to increase by at least two from the last bin of the CS to the US omission bin to ensure that the increase in firing was specific to the omission of the US and not a result of sustained increase in firing to the CS. Conversely, a neuron was classified as US omission inhibited if (1) z-score was smaller than −2 at the time of the US omission (during the 1 s bin following the offset of the CS); (2) the z-score had to decrease by at least two from the last bin of the CS to the US omission bin to ensure that the decrease in firing was specific to the omission of the US and not a result of sustained decrease in firing to the CS. To quantify responses to the CS, we averaged the z-scores during the entire CS period (10 s) for Hab, E-Ext and L-Ext trials and obtained an average z-score for each neuron. The neurons were classified as excited or inhibited by the CS if they showed average z-scores greater than two or smaller than −2, respectively.

40Animals were injected with viral vectors and implanted with an optical fiber in the VTA, as described above. After a waiting period of 3–4 weeks to allow for surgical recovery and virus expression, animals were connected to a 400 μm patch cord (Doric Lenses). Fluorescence was measured by delivering 465 nm excitation light through the patch cord and separating the emission light at 525 nm with a beamsplitter (Fluorescence MiniCube FMC5, Doric Lenses). The emission light was collected using a Femtowatt Silicon Photoreceiver (Model # 2151, Newport). The voltage output of the photoreceiver was then digitized at 2 kHz (Digital Lynx SX, Neuralynx). Using the same approach we also measured the isosbestic (activity-independent) fluorescence of gCaMP6 at 430 nm using 405 nm excitation light. After animals were habituated to handling and being connected to the patch cord, they underwent the fear conditioning protocol. At the end of fear conditioning experiments, animals were tested on an operant conditioning task (Figure 3—figure supplement 2). To this end, animals were placed in an operant chamber containing a water delivery port. Each nosepoke into the delivery port that followed the previous nosepoke by a variable inter-trial interval (3–5 s) triggered the delivery of liquid reward (10% sucrose solution) with a 50% probability. This task was used to verify that reward delivery caused large increases in fluorescence in gCaMP6-expressing mice, as expected based on previous electrophysiology and fiber photometry studies, suggesting that this Ca+2 signal is indeed generated by DA neuron activity.

41The voltage output of the photoreceiver, representing fluctuations in fluorescence, was first low-pass filtered at 4 Hz and then downsampled to 10 Hz. The change in fluorescence evoked by the CS (dF/F) was then calculated by subtracting from each trace the baseline fluorescence (average during the 5 s before CS onset) and dividing it by the baseline fluorescence. dF/F traces were then averaged separately for each animal for early extinction trials (E-Ext: average of first 10 CSs), late extinction trials (L-Ext: average of last 10 CSs) and tone habituation (Hab: average of 10 CSs). To examine responses to US omission we further averaged dF/F values in the 5 s following CS offset. Similar results were obtained from animals expressing gCaMP6f (n = 5) and gCaMP6s (n = 5), therefore the data was pooled. To quantify responses to the CS, we averaged the dF/F values during the CS for each animal for Hab, E-Ext and L-Ext. To quantify responses to reward (Figure 3—figure supplement 2), average dF/F was calculated separately for rewarded and unrewarded nosepokes using the baseline fluorescence 3 s before noseport entry.

42For bilateral optogenetic manipulations during behavior, the implanted optical fibers (200 μm core diameter, 0.22 NA, Thorlabs) were connected to 200 μm patch cords (Thorlabs) with zirconia sleeves and the patch cords were connected to a light splitting rotary joint (FRJ 1 × 2 i, Doric Lenses) that was connected to a laser with a 200 μm patch cord (Thorlabs). For mice expressing the light-activated inhibitory opsin halorhodopsin (NpHR-eYFP) and their eYFP controls, yellow light was delivered from a DPSS 594 nm laser (Omicron). Laser power at the tip of the optic fiber was 10–15 mW. To inhibit DA neurons around the time of the US omission during the extinction session on day 2, the laser was turned on from 1 s before to 2 s after the CS offset. The laser was then turned off gradually using a 1 s ramp to avoid rebound excitation (; Figure 5—figure supplement 2). For mice in the Unpaired-NpHR control group, the light was delivered the same way during the ITIs. For mice expressing the light activated excitatory opsin channelrhodopsin-2 (ChR2) and their eYFP controls, blue light pulses were delivered from a 473 nm laser (LuXx473, Omicron). The laser power at the tip of the optic fiber was 5–10 mW. To phasically activate DA neurons at the time of the US omission, 5 ms light pulses were delivered at 20 Hz for 1 s beginning at the offset of the CS. The mice in the Unpaired-ChR2 group received delivery of the same light pulses during the ITIs.

43Using the surgical procedures described above DAT-cre mice were injected in the VTA with 1 μl of AAV5-EF1a-DIO-hChR2(H134R)-eYFP (final titer 4.3 × 1012 pp per ml) or AAV5-EF1a-DIO- eNpHR3.0-eYFP (final titer 4 × 1012 pp per ml) per hemisphere using the coordinates described above. Two to three weeks after virus injection, animals were anesthetized and placed in a stereotaxic frame with the skull exposed. A stainless-steel head post (Luigs and Neumann) was then cemented to the exposed skull. The area of the skull overlying VTA was left free of cement but covered with a silicon elastomer (Kwik-Sil, World Precision Instruments). Skull screws were inserted over the frontal cortex and cerebellum to serve as reference and ground, respectively, and to provide anchoring support for the cement. Following recovery from surgery, animals were handled and habituated to being head-fixed, by inserting the head post into a matching head post holder (Luigs and Neumann). Following 2–3 days of habituation to being head-fixed, animals underwent another surgery to prepare a craniotomy over the VTA. Animals were anesthetized with isoflurane and placed in a stereotaxic frame. The Kwik-Sil was removed from the skull and a small craniotomy was then made in the skull over the VTA and then sealed with Kwik-Sil. The following day, the animals were head-fixed, the Kwik-Sil removed, and a 32-channel optrode (silicon probe with an optical fiber; A1 × 32-Edge-5mm-20–177-OA32, NeuroNexus) was lowered into the VTA using a micromanipulator (SM-8, Luigs and Neumann). The optrode was then advanced to a depth of ~4700 μm below the brain surface to span the dorsal-ventral extend of the VTA. Following final placement of the optrode and a brief waiting period (∼15 min), neural activity was recorded while laser pulses (yellow light: 10–15 mW, blue light: 5–10 mW) were delivered through a patch cord (Neuronexus) connected to the optic fiber. Electrophysiological signals were filtered between 1 and 6000 Hz, digitized at 30 kHz using a digitizing headstage (RHD2132 Amplifier Board, Intan Technologies), and acquired using a USB interface board (RHD2000, Intan Technologies). Once the recording was over, optrode was removed from the brain and then was penetrated to another location in the VTA. Multiple penetrations and recordings were performed during a session. Before the final penetration, the silicone probe was coated with a fluorescent dye (DiI, Life Technologies) to assist with the identification of the recording location.

44At the end of the experiments, mice were deeply anesthetized with sodium pentobarbital and were transcardially perfused with 4% paraformaldehyde and 15% picric acid in phosphate-buffered saline (PBS). Brains were removed, post-fixed overnight and coronal brain slices (60 µm) were sectioned using a vibratome (VT1000S, Leica). Standard immunohistochemical procedures were performed on free-floating brain slices. Briefly, sections were rinsed with PBS and then incubated in a blocking solution (10% horse serum, 0.5% Triton X-100% and 0.2% BSA in PBS) for 1 hr at room temperature. Slices were then incubated in a carrier solution (1% horse serum, 0.5% Triton X-100% and 0.2% BSA in PBS) containing the primary antibody overnight at room temperature. The next day, the sections were washed in PBS and then incubated in the same carrier solution containing the secondary antibody overnight at room temperature. The following primary antibodies were used: polyclonal rabbit anti-tyrosine hydroxylase (TH, catalog # 657012, 1:1000, Calbiochem), monoclonal mouse anti-TH (catalog # MAB318, 1:1000, Millipore), polyclonal rabbit anti-GFP (catalog # A11122, 1:1000, Life Technology). The following secondary antibodies were used: Alexa Fluor 568 goat anti-rabbit (catalog # A11011, 1:1000, Thermo Fisher Scientific, Invitrogen), Alexa Fluor 568 goat anti-mouse (catalog # A11004, Thermo Fisher Scientific, Invitrogen), and Alexa Fluor 488 goat anti-rabbit (catalog # A11008, 1:1000, Thermo Fisher Scientific, Invitrogen). Finally, sections were washed with PBS, mounted on slides and coverslipped with a 4’,6-diamidin-2-phenylindol (DAPI) containing medium (VECTASHIELD, Vector Laboratories) or incubated for 10 min in 0.1 M PBS containing 0.02% DAPI (catalog # D1306, Molecular Probes, Invitrogen), washed for 10 min in PBS, mounted on slides and coverslipped. Animals with incorrect electrode/optical fiber placements were excluded from data analysis. A total of four mice were excluded from the study due to incorrect placement of electrodes/optical fibers. We excluded one animal from the electrophysiology experiment because the electrodes were placed outside (posterior to) the VTA. Three animals were excluded from the fiber photometry experiment. In one animal the optical fiber was placed too dorsal and in the other two animals too ventral in the VTA. In these animals, the neuronal activity dependent fluorescence signal was undetectable or too low. No animals were excluded from optogenetic experiments.

45Data were statistically analyzed using GraphPad Prism (GraphPad Software) and MATLAB (Mathworks). All statistical tests were two-tailed and had an α level of 0.05. All error bars show s.e.m. All ANOVAs were followed by Bonferroni post hoc tests if significant main or interaction effects were detected. No statistical methods were used to predetermine sample size, but our sample sizes were similar to those generally used in the fear conditioning field. Animals were randomly assigned to experimental groups before the start of each experiment after ensuring that all experimental groups were matched for age. All results were obtained using groups of mice that were run in several cohorts.