Multiple signals in anterior cingulate cortex

1Current Opinion in Neurobiology 2016, 37:36–43This review comes from a themed issue on Neurobiology of cognitive behaviorEdited by Alla Karpova and Roozbeh KianiFor a complete overview see the Issue and the EditorialAvailable online 8th January 2016http://dx.doi.org/10.1016/j.conb.2015.12.0070959-4388/© 2016 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).

2Current Opinion in Neurobiology 2016, 37:36–43

3This review comes from a themed issue on Neurobiology of cognitive behavior

4Edited by Alla Karpova and Roozbeh Kiani

5For a complete overview see the Issue and the Editorial

6Available online 8th January 2016

7http://dx.doi.org/10.1016/j.conb.2015.12.007

80959-4388/© 2016 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).

9When humans and other animals take a course of action they usually do so because they believe the benefits of doing so will outweigh the costs. There is an evolving understanding of the mechanisms underlying evaluation of one well-defined choice against another that have been linked to ventromedial, orbital prefrontal, and intraparietal sulcal cortex. There are also, however, times when animals decide whether it is worth acting at all or evaluate whether it is worth continuing to engage in the current behaviour or to explore alternatives. This distinct pattern of decision-making is linked to ACC; ACC manipulations affect the ability of animals to initiate any action at all, weigh up the costs and benefits of actions, switch between actions as their values change, or explore alternative choices. A series of recent studies have demonstrated the presence of activity changes in ACC that correspond to the types of signals that would be needed to guide such behaviour; these signals encode the values of actions, the average value of alternative courses of action in the environment (‘search value’) as opposed to the current or default course of action, exploration and evaluation of hypotheses about the best course of action to take, and reflect updating of decision-makers’ beliefs and internal models of their environments. Not only are such signals found in ACC but they are weak or absent in regions such as orbitofrontal and ventromedial prefrontal cortex that carry other value signals.

10In addition, however, ACC has also been linked to ‘conflict monitoring’ — the process of detecting when two competing choices might be made during a difficult task. Detecting response conflict and task difficulty is important if mistakes are to be averted. Recently it has been argued that ACC activity interpreted as reflecting value signals has been confounded with difficulty and so it has been argued that such ACC activity is more parsimoniously interpreted as simply reflecting task difficulty. Here we review evidence, first, that value signals and, second, model update signals can be separated from any effect difficulty exerts on ACC activity.

11For example, a recent study investigated how people decide whether to explore a set of alternative choices or stick with the opportunity to make a ‘default’ choice. The value of exploring was encoded by a ‘search value’ signal in ACC indexing the average value of the set of alternative choices that might be taken. In addition to search value, ACC activity was also influenced, in a negative fashion, by engage value (the value of the default option) and costs incurred by searching. This pattern of positive and negative modulations is suggestive of a comparison process taking place within ACC that could inform decisions about whether or not to explore, or ‘forage’ amongst, the alternatives.

12Figure 1a, however, summarizes how difficulty might be confounded with the difference between search and engage value — a quantity sometimes referred to as the ‘relative value of foraging’ or RVF. The probability of behavioural change — searching as opposed to ‘engaging’ with the current default — is plotted on the ordinate as a function of RVF. A confound between RVF and difficulty arises if subjects are biased to take the default. Even if the experiment examines decisions equally on either side of the objective indifference point — the point at which searching and engaging objectively have the same value — it is still possible that the sampling is unequal with respect to the subjective or empirical indifference point — the point at which a given participant has no preference between the options. The confound arises because decisions close to the subjective indifference point are the most difficult to take [for example, they are associated with long reaction times (RTs)]. If participants are very biased to nearly always take the default option then RVF and difficulty both increase together across much of the decision space.

13Experiments addressing this criticism must contain certain obvious features. First, a broad and evenly distributed range of search and engage value must be tested. However, at the same time, it is crucial that decisions are not trivially easy and that some value comparison occurs on each trial. Second, it is imperative that participants make decisions that really are guided by option values and do not always simply engage with the default option. One way of ensuring this is simply to provide adequate task training and instruction prior to scanning. If this is done then subjects make rational value-guided decisions and therefore subjective and objective indifference points are close (Figure 1c) and difficulty/value confounds disappear; many decisions are examined in decision space to the right of the subjective indifference point where RVF and difficulty are not positively correlated. Third, when analysing the data, rather than examining the neural correlates of the aggregate of decision variables — RVF — it is advisable to focus on the component values that determine RVF: search value, engage value, and costs. These component values are more easily dissociated from difficulty. Employing these principles Kolling and colleagues reduced the shared variance between search value, engage value, and difficulty to 2% so that the neural correlates of each could be separately identified (Figure 2). Now it is clear that ACC activity reflects search value shortly followed by engage value although towards the end of the decision period some variance in ACC activity is accounted for by difficulty and RT. Parallels can be drawn with recordings made in other brain areas concerned with value-guided decision making such as the intraparietal sulcus; initially activity in intraparietal neurons reflects saccade value but then it transitions to reflect action related factors.

14Such a pattern of results suggests ACC is a neural network in which decisions to explore or not are taken; activity is affected by a search value signal (apparent throughout much of the trial period) but that the network takes longer to make decisions using this signal and others when they are difficult (and therefore some variance in dACC activity at the end of the decision period is accounted for by difficulty and RT). Biophysically plausible neural networks have been proposed in which pools of neurons are active in proportion to the evidence favouring particular choices. If the representation of search value in ACC takes this form then the network activity should reflect both search value and difficulty. In fact, the prediction is that the impact of search value should scale with difficulty. Although it might be difficult to assess such a precise hypothesis with fMRI such considerations suggest that conducting experiments with decisions involving extremely high search values may be unwise; when decision difficulty is very low the network may resolve the decision and enter an attractor state so quickly that it will be difficult to see any effect of search value. In other words, exclusive sensitivity to search value, and not difficulty too, is not a prediction for a search value sensitive decision circuit but instead sensitivity to both search value and difficulty is expected. In the future, careful neurophysiological measurements will be essential for testing such potential mechanisms at the neuronal level, disentangling how aggregate measures such as the BOLD signal are derived from actual neural network operations.

15Furthermore, ACC is sometimes co-activated with adjacent medial frontal brain areas and so an important consideration when drawing conclusions about ACC is to ensure that neural activity that is recorded really is drawn from ACC rather than adjacent medial frontal areas. After controlling for difficulty, search value effects are most prominent in ACC itself (Figure 3a) but task difficulty effects lie in more dorsal areas in or anterior to the pre-supplementary motor area (Figure 3b).

16Humans and other animals should change from the behaviour they are currently engaged in and explore alternative courses of action not just when they have a sense of the value of those alternatives but also when they realise the environment is changing. ACC activity is also prominent when events suggest that a decision-maker's internal model of their environment should be updated. By definition, surprising events are ones that were not predicted by the decision-maker's current model of their environment. They are, therefore, frequently the events that indicate the need for model updating. At the same time, however, surprising events are often events to which responses are made more slowly (longer RTs) and with greater difficulty because the response, or the stimulus eliciting it, was unanticipated. Does ACC activity at the time of model updating simply reflect task difficulty — the difficulty of responding when internal model updating occurs? Or does it activate when internal models have to change?

17A recent study tested exactly this distinction (Figure 4) while controlling for response confounds and showed that model update signals can indeed be dissociated from task difficulty effects. Human participants made saccades to targets (coloured dots) that, on each trial, appeared on a circular perimeter surrounding a fixation point. The dots’ locations were usually predictable because they were similar over runs of 10–20 trials but two types of unexpected event occurred. On model update trials (Figure 4a) the dot appeared in an unexpected location and its new colour indicated that future dots were likely to appear nearby on the circle's periphery. However, on surprise only trials (Figure 4b), dots appearing in white in a surprising location indicated one-off events and no need for participants to update their internal model of where future dots would appear. The difficulty of responding on any trial reflects the surprise associated with a particular stimulus value, α, and is characterized in Information Theory by its Shannon information IS(α):where p(α|prior) is the prior probability that the observation α would be made, given the brain's internal model just before the data point was observed. Therefore the Shannon information captures how unexpected or unlikely a particular observation is, given the internal model and is directly related to the difficulty of the trial. In contrast, updating of the internal model is captured by the Kullback–Leibler divergence (DKL) between the posterior and the prior:where p(α|prior) is the probability that the observation α would be made, given the model just before α was observed, and p(α|post) is the same quantity, given the updated model just after α was observed. DKL is the probability-weighted average change in Shannon information across all possible stimuli as a consequence of updating the model.

18Although RTs increased on both model update and surprise trials ACC was preferentially engaged on model update trials. Moreover ACC activity covaried with the model updating parameter, DKL, but not surprise IS (Figure 4e and f), despite IS's relationship to difficulty. Although parietal regions were active as a function of response selection difficulty as indexed by RT, ACC was not (Figure 4h). Other studies similarly suggest ACC is activated when there is a need to update the task model even in the absence of any response selection difficulty (because no response is required at all). Model updating-related activity in ACC is, therefore, linked to behavioural flexibility and change and not simply response selection difficulty. This role of the ACC may underlie its activation during proactive control and error correction. It is possible that ACC activity in other experiments may have a similar role.

19In summary, ACC carries multiple signals. ACC activity reflects both search value and the updating of internal models of the environment. In both cases, and in other reports, ACC is linked to behavioural change, invigoration of new responses, novel response strategies, and exploration. We have conceptualized search value as the average value of choices that might be taken in an environment but it could take many other forms depending on context. We and others have argued that some of these signals may have arisen in the context of the foraging choices that animals make as they decide to leave one foraging patch to explore another. Advantages of this approach are that it situates ACC function within the context of a behaviour for which there has been substantial evolutionary pressure and it suggests ways of optimal modelling of both behaviour and neural activity. Similar processes are likely to underlie human behaviours such as task switching. Such a perspective holds great promise for making novel predictions about behaviour and neural mechanisms in a principled fashion.

20Two regions within ACC, dorsal ACC (dACC) and perigenual ACC (pgACC), carry related signals. Both areas are found in humans and macaques; each area has a distinctive pattern of interaction with wider brain circuits that is similar across species. Similar areas are also present in rodents and again they mediate related aspects of behaviour. Indeed, when a decision-maker has updated its internal model or is about to pursue an alternative course of action then it may be necessary to exert careful control over which actions are selected next. However, the same is true even when one manages to resist the attractions of an alternative course of action or when attention has lapsed or errors have been made. In all these situations it is necessary to exert greater cognitive control and this may be brought about by interactions between ACC and lateral prefrontal cortex.

21We have no conflicts of interest.

22Papers of particular interest, published within the period of review, have been highlighted as:• of special interest•• of outstanding interest

23• of special interest

24•• of outstanding interest