Premembering Experience: A Hierarchy of Time-Scales for Proactive Attention

1Memory’s most compelling illusion is that it represents the past. However, it is clear from an ecological perspective that memory is all about the future. The purpose of memory is to learn about the environment to anticipate future demands—not just putting back the pieces of the past for recollection (remembering experience) but deriving possibilities based on the past to guide future adaptive behavior (premembering experience). This notion has old roots, but, surprisingly, the fundamental operations through which memories guide adaptive behavior lack an established theoretical framework, and their mechanisms have yet to capture the in-depth and systematic investigation they deserve. We propose that memory, over a broad hierarchy of timescales, supplies the essential elements for selective attention to guide perception and performance flexibly and adaptively. We introduce the term “premembering” to capture this prospective and dynamic role of memory. Our construct complements proposals for how memories can be used to inform other cognitive functions, such as adaptive control, decision-making, and imagining future situations (Box 1).We focus on how memories guide selective attention. The role memory plays in guiding behavior has also been considered within other cognitive domains.“Cognitive control” refers to the collection of mechanisms that set and adjust our goals and that monitor and regulate performance according to competing demands. Thus, the selective attention mechanisms we review are subordinate to cognitive control, prioritizing and selecting putative targets and overcoming irrelevant distraction within a particular goal setting. Memories are increasingly recognized to play an important role in influencing the degree of top-down control exerted on a given trial. They include both short-term traces between successive trials as well as intermediate memory traces that develop over task performance.“Decision-making” refers to the process of choosing one of a set of alternatives to produce a beneficial outcome. Choices are made based on expectation of rewards developed through previous experience. Current models of decision-making mostly rely on reinforcement learning. Recent computational studies suggest that different types of memory traces may work together to optimize reinforcement learning, including those resulting from slow incremental implicit learning over trials as well as episodic traces uniquely linked to individual experiences.“Decision-making” is closely related to selective attention, which can be involved in prioritizing information for guiding choice behavior. Expectations play an essential role in both sets of processes. However, interestingly, they each stress a different consequence of prior knowledge. In decision-making, predictions based on priors are mainly used to attenuate the processing of what can be anticipated. In selective attention, predictions are mainly used to enhance the processing of anticipated task-relevant information. These two phenomena nicely illustrate the flexibility with which memory-related traces can be used to guide adaptive performance. The specific consequence of prior knowledge will be heavily dependent on the purpose of the task.“Episodic future thinking” involves drawing on previous experiences to imagine oneself in future situations. The construct is useful in different types of situations, such as navigation, planning to implement intentions, understanding others’ mental states, and simulating future events. Neuropsychological, developmental, and brain imaging studies have revealed substantial overlap between the neural system supporting episodic future thinking and episodic recollection, suggesting that the LTM traces available for recollection can also be used prospectively and flexibly to build novel plausible scenarios and run simulations. Episodic future thinking differs from our construct of premembering in being a specifically deliberative process based on LTM traces available to awareness to inform behavior in the future. Premembering is a broader construct, considering the influences of memories of different types and timescales on ongoing or imminent behavior.

2We focus on how memories guide selective attention. The role memory plays in guiding behavior has also been considered within other cognitive domains.“Cognitive control” refers to the collection of mechanisms that set and adjust our goals and that monitor and regulate performance according to competing demands. Thus, the selective attention mechanisms we review are subordinate to cognitive control, prioritizing and selecting putative targets and overcoming irrelevant distraction within a particular goal setting. Memories are increasingly recognized to play an important role in influencing the degree of top-down control exerted on a given trial. They include both short-term traces between successive trials as well as intermediate memory traces that develop over task performance.“Decision-making” refers to the process of choosing one of a set of alternatives to produce a beneficial outcome. Choices are made based on expectation of rewards developed through previous experience. Current models of decision-making mostly rely on reinforcement learning. Recent computational studies suggest that different types of memory traces may work together to optimize reinforcement learning, including those resulting from slow incremental implicit learning over trials as well as episodic traces uniquely linked to individual experiences.“Decision-making” is closely related to selective attention, which can be involved in prioritizing information for guiding choice behavior. Expectations play an essential role in both sets of processes. However, interestingly, they each stress a different consequence of prior knowledge. In decision-making, predictions based on priors are mainly used to attenuate the processing of what can be anticipated. In selective attention, predictions are mainly used to enhance the processing of anticipated task-relevant information. These two phenomena nicely illustrate the flexibility with which memory-related traces can be used to guide adaptive performance. The specific consequence of prior knowledge will be heavily dependent on the purpose of the task.“Episodic future thinking” involves drawing on previous experiences to imagine oneself in future situations. The construct is useful in different types of situations, such as navigation, planning to implement intentions, understanding others’ mental states, and simulating future events. Neuropsychological, developmental, and brain imaging studies have revealed substantial overlap between the neural system supporting episodic future thinking and episodic recollection, suggesting that the LTM traces available for recollection can also be used prospectively and flexibly to build novel plausible scenarios and run simulations. Episodic future thinking differs from our construct of premembering in being a specifically deliberative process based on LTM traces available to awareness to inform behavior in the future. Premembering is a broader construct, considering the influences of memories of different types and timescales on ongoing or imminent behavior.

3“Cognitive control” refers to the collection of mechanisms that set and adjust our goals and that monitor and regulate performance according to competing demands. Thus, the selective attention mechanisms we review are subordinate to cognitive control, prioritizing and selecting putative targets and overcoming irrelevant distraction within a particular goal setting. Memories are increasingly recognized to play an important role in influencing the degree of top-down control exerted on a given trial. They include both short-term traces between successive trials as well as intermediate memory traces that develop over task performance.

4“Decision-making” refers to the process of choosing one of a set of alternatives to produce a beneficial outcome. Choices are made based on expectation of rewards developed through previous experience. Current models of decision-making mostly rely on reinforcement learning. Recent computational studies suggest that different types of memory traces may work together to optimize reinforcement learning, including those resulting from slow incremental implicit learning over trials as well as episodic traces uniquely linked to individual experiences.

5“Decision-making” is closely related to selective attention, which can be involved in prioritizing information for guiding choice behavior. Expectations play an essential role in both sets of processes. However, interestingly, they each stress a different consequence of prior knowledge. In decision-making, predictions based on priors are mainly used to attenuate the processing of what can be anticipated. In selective attention, predictions are mainly used to enhance the processing of anticipated task-relevant information. These two phenomena nicely illustrate the flexibility with which memory-related traces can be used to guide adaptive performance. The specific consequence of prior knowledge will be heavily dependent on the purpose of the task.

6“Episodic future thinking” involves drawing on previous experiences to imagine oneself in future situations. The construct is useful in different types of situations, such as navigation, planning to implement intentions, understanding others’ mental states, and simulating future events. Neuropsychological, developmental, and brain imaging studies have revealed substantial overlap between the neural system supporting episodic future thinking and episodic recollection, suggesting that the LTM traces available for recollection can also be used prospectively and flexibly to build novel plausible scenarios and run simulations. Episodic future thinking differs from our construct of premembering in being a specifically deliberative process based on LTM traces available to awareness to inform behavior in the future. Premembering is a broader construct, considering the influences of memories of different types and timescales on ongoing or imminent behavior.

7Selective attention (hereafter called “attention”) refers to the set of functions that prioritize and select information to guide adaptive behavior. These functions modulate incoming sensory signals and influence their processing at multiple stages to inform awareness, decisions, actions, and subsequent memories. It has long been appreciated that short-term memory, or “working memory” (WM), plays an important role in forming attentional templates (e.g., ). Here we suggest that WM is part of a much larger family of heterogenous attention-guiding memory traces that span multiple timescales.

8The premembering perspective has important implications for understanding the content and formatting of memory. Rather than slavishly storing and using veridical traces, the brain flexibly selects and even distorts memory content to enhance its utility in guiding attention. Moreover, task-relevant informational content may also be stored in a neural format that is optimized for the anticipated utilization of the memory to guide performance.

9The essence of memory is the traces left by passing experience. These range from transient perturbations to engrams that last a lifetime, capturing modality-specific fragments to relational and integrated wholes and supporting unconscious states to recollective phenomena. Such traces can provide essential informational content required for prospectively prioritizing and selecting what is important. Working together, memory content and attention functions shape how the brain transforms incoming signals to guide perception, choice, action, and the formation of new memories to serve adaptive behavior in the future (Figure 1). It can be argued that these prospective properties of memory are what define its fundamental ecological purpose: to collect relevant aspects of experience to anticipate future demands and guide behavior.

10Drawing a clear line defining when present becomes past may be impossible. Nevertheless, it is clear that past traces affect perception even from their earliest moments. The very stitching of visual perception across eye movements into an apparent cohesive flow may rely on short-term memories bridging the anchoring and landing fixation contents. Furthermore, transient salient visual stimuli intrinsically capture attention and leave a brief excitatory trail, temporarily enhancing processing of stimuli that follow in their immediate wake (; Figure 2A).

11At slightly longer timeframes, WM provides a limited set of more durable traces that are independent of continuous sensory stimulation and resistant to interference and that act to guide adaptive behavior. The fundamental role WM plays in guiding attention is widely recognized and has been studied extensively. Information in WM has been considered the major source of top-down proactive attention. Even before the target stimuli appear, these memory traces influence the pattern of brain activity in a proactive fashion to facilitate the processing of signals associated with likely relevant items. These top-down anticipatory states based on WM templates are often associated with willful, volitional orienting of attention. However, it is important to note that WM content can also influence incoming information processing in an involuntary fashion (; see below).

12Moreover, beyond the scope of classic WM, the relation between successive items can also influence information processing. For example, when searching for an odd-one-out target, its identification is facilitated when the current target shares a defining feature value or location with the target in the preceding trial. Traces from preceding stimuli also alter the quality of perception. Reports of the spatial frequency and orientation of a visual item are systematically distorted by that of preceding items (e.g., ).

13Short-term memory traces can be used to enhance or suppress information processing, depending on the situation. For example, after initial facilitation of stimuli in the wake of non-informative salient stimuli, decrements in performance occur at the same location; the “inhibition of return” has been proposed to encourage sampling and exploration. Pre-exposure to irrelevant or distracting stimuli can facilitate performance in visual search tasks by helping overcome distractor interference. However, when the current target shares the identity or features of a distractor in the previous trial, performance decrements occur (negative priming;, ). Reward associations can further magnify both facilitatory and inhibitory effects between successive stimuli.

14Although most research has focused on the role of short-term traces in anticipatory attention, the memory traces that guide our perception stretch further back in time. Most of the knowledge that guides the interpretation of sensory signals and sets up expectations about the occurrence and context of relevant events is stored in our long-term memory (LTM; ). However, somewhere between short-term traces that guide performance between successive events and remote long-term memories that guide performance between distant episodes lives a rich variety of memory traces reflecting regularities learned and utilized within a task setting. An elegant study by showed the gradual transition from WM to a longer-term trace when search targets were held constant for a number of trials (Figure 2B). For early trials, they showed a robust event-related potential associated with the location and number of items in the WM template (contralateral delay activity [CDA]; ). As trials progressed, the CDA progressively diminished, implying a gradual handover to a longer-term, intermediate memory store.

15Increasing numbers of experimental approaches are beginning to explore how different types of intermediate traces guide attention within task settings. “Probability cueing” refers to performance benefits resulting from the higher prevalence of a task-relevant item within a given location. The frequent appearance of a target at a given location facilitates its identification, whereas the frequent appearance of an irrelevant stimulus at a location attenuates its distracting effect. Memory traces linked to probability cueing are long lasting and largely acquired implicitly. A related but different phenomenon is “selection history” (e.g., ), which refers to facilitation in identifying stimuli previously selected as targets independent of their probability.

16“Reward history,” referring to the likelihood or value of a reward typically associated with given target stimuli within a task, has strong and persistent effects on performance. Reward manipulations often come bundled with manipulations of other sources of attention modulation, such as probability cueing or selection history. In complex search environments, reward history combines with other factors to guide effective performance (e.g., ). In tasks that carefully control for other sources of influence, stimuli with features that have high reward associations act as potent distractors, even when they are otherwise inconspicuous and task irrelevant.

17Learning and utilization of more complex associations also occur within task settings. “Contextual cueing” shows that pick-up of repeated spatial configurations of stimuli within visual search arrays facilitates target identification. This type of memory guides search behavior, resulting in fewer fixations and faster search times. Neurophysiological markers of target detection are also enhanced. Learning of statistical regularities over time similarly leads to improvements in behavioral performance. Interestingly, the presence of statistical regularities within stimulus streams spontaneously captures attention, leading to facilitation of target identification in structured relative to unstructured streams. Both contextual cueing and statistical learning are thought to involve hippocampus-related memory systems, even though they are learned largely implicitly. A recent neurophysiological study of sequence learning in a serial response task has also revealed proactive anticipation of upcoming stimuli and associated responses based on learned spatiotemporal expectations.

18Although the role of LTM in guiding attention has been less well explored than that of WM, its fundamental role in defining perception, choosing actions, imagining, and forming new memories has long been recognized (,,; Box 1). We propose that the role memory plays in guiding attention may contribute to its fundamental influence over our mental experience and behavior.

19In everyday situations, the bank of remote memory traces from previous settings and episodes arguably provides the richest source of information for guiding attention and perception. When waiting to meet a friend on a busy street, we rely on LTM to search them out based on their appearance and likely route. LTM provides knowledge about whether and how relevant events occur in particular contexts and about their likely reward outcome values. Thus, they afford rich content for flexible, proactive, and dynamic biases to guide the prioritization and selection of relevant information. An increasing variety of experimental approaches is being used to explore how LTM content is used proactively to guide adaptive behavior. Although differing in detail, the approaches share the notion that the brain uses LTM information constantly, proactively, and predictively.

20In visual search tasks using simple stimulus arrays, semantic knowledge of features commonly associated with objects facilitates target identification, overcoming usual costs associated with feature binding. On the other hand, distractors with associative links to the target disrupt the visual search by capturing attention. Furthermore, newly learned associations continue to affect performance on subsequent tasks, even when completely irrelevant.

21In tasks using complex scenes or environments, “contextual priming” studies show that previous experiences within particular contexts guide the identification and interpretation of objects within the same contexts. When searching for a specific item within a scene, LTM works alongside lower-level statistics of visual features to guide performance. Multiple types of LTM are at play. Search performance benefits from schematic probabilistic knowledge of encountering the item within a particular context as well as from its likely location relative to the layout of the scene and to other objects within a scene. In addition, specific knowledge about the location of an item within a particular scene also contributes—episodic guidance. Interestingly, incidental memories formed through active search and selection of items within a scene lead to stronger performance benefits and subsequent memories than familiarity or explicit attempts to memorize items within scenes, suggesting that effective memories fall out of natural behavioral interactions.

22Memory-guided attention tasks show how learning of object locations through repeated visual search in complex scenes or environments facilitates performance in subsequent tasks. Performance measures reveal significant improvements in detecting and discriminating target items as a result of previously learned specific contextual association between the particular item and a scene or three-dimensional environment.

23“Memory-guided orienting” tasks provide a good platform to investigate anticipatory biases based on long-term memories (; Figure 2C). Participants form object-scene associations to a high and stable level during a learning session; the consequences of these memory traces are then tested in a separate attention-orienting session. The separation helps disentangle effects of early learning from those of subsequent memory utilization. In our own research, we have used learned scenes presented in isolation as memory cues, followed by the appearance of a target stimulus to be identified or discriminated. The critical behavioral comparison is between performance depending on whether the target location is correctly predicted by previous memory experience (valid memory cue) versus incorrectly predicted (invalid memory cue) or unpredicted (neutral memory cue). Strong and reliable effects occur for both perceptual sensitivity and response times. By presenting memory cues and targets individually, it is possible to measure memory-related brain activity that biases excitability according to anticipation of the target location. fMRI reveals that memory cues engage the hippocampus as well as the dorsal frontoparietal network associated with the control of spatial attention. In addition, activity levels in visual cortical regions become spatially biased in anticipation of the target location (; Figure 2C). Electrophysiological recordings also reveal memory-related spatial anticipatory biases in the form of lateralization of alpha-band power. Anticipatory biases in the cue-target period are followed by modulation of event-related potentials related to visual processing and spatial selection of target items. Reward associations further potentiate the effects of memory-based orienting. Even a single association with a modest monetary gain or avoidance of loss during the last learning block leads to performance improvements in the subsequent attention-orienting session, even though reward is irrelevant to the orienting task. Target stimuli during the orienting task elicit larger visual P1 potentials when previously associated with reward or punishment avoidance.

24have followed a similar approach involving learning and utilizing spatial contextual associations in a change-detection task. By comparing memory-guided and visually guided attention conditions, they have highlighted a set of cortical regions and differences in hemispheric lateralization associated with LTM-based orienting.

25Thus, findings with memory-guided orienting tasks clearly illustrate the ability of the brain to use LTM content proactively to guide prioritization and suggest the involvement of limbic, memory-related circuits. In monkeys, recent studies recording activity from neurons in the hippocampus and entorhinal cortex during free viewing provide converging evidence that medial temporal areas, traditionally associated with memory, are closely related to attention. Neurons in the entorhinal cortex show strong visuospatial coding properties, signaling gaze location using multiple frames of reference. Entorhinal neurons further show a grid cell-like arrangement of viewed spatial locations, using only covert attention in the absence of any eye or other physical movement. In addition to grid cells, saccade direction cells are also present, with largely independent populations encoding the direction of previous versus future saccades. Together, such findings point to mechanisms that could support spatial attention based on LTM. Recordings in the hippocampus showed that slow-wave ripples, thought to promote plasticity, occurred during a search for object changes within scenes and that their probability increased with scene repetition and near remembered targets. Such a mechanism would be a good candidate for supporting memory-guided attention during search.

26In addition to carrying information about the location and identifying attributes of anticipated events, memory traces also carry information about their timing. Memory-based temporal expectations can therefore enable the brain to prepare for events in a dynamic, temporally structured, and efficient way. Temporal expectations rely on learning temporal regularities that occur over multiple timescales. Neural markers of proactive temporal anticipation have been observed when the timing of stimuli follows a regular temporal rhythm, probabilistic conditional probabilities, or sequences. Across episodes, long-term memories can also guide temporally structured anticipatory attention. Using a temporal variant of the memory-guided orienting task, showed improved perceptual sensitivity and response times to detect and discriminate visual targets occurring at the learned temporal interval (valid memory cue) relative to the other interval (invalid memory cue). Event-related potentials elicited by the memory cue revealed clear modulation of target anticipation according to temporal expectation. Behavioral and neural markers of validity effects correlated with one another and with the quality of learning of the temporal association.

27Recognizing that memory traces of different types and timescales proactively guide perception invites us to reconsider dominant concepts and dichotomies in the attention literature. The simple separation between “bottom-up” sources of prioritization based on physical salience and “top-down” sources based on current goals is clearly insufficient. To explore the natural kinds of memory-based attention, it is important to separate two fundamental factors: their volitional character and their source.

28Subjectively, attention can be voluntary or involuntary, and its source can be external, based on physical salience (exogenous), or internal, based on brain states linked to current goals or previous experience (endogenous). Often these two factors are conflated, with exogenous sources assumed to guide involuntary attention and endogenous sources assumed to guide voluntary attention. Instead, they should be viewed as theoretically separable. Although physically salient stimuli capture attention involuntarily, endogenous sources can guide attention both voluntarily and involuntarily. In addition to their use in goal-based voluntary attention, both short-term as well as LTM sources can also have automatic, involuntary consequences for perception. For example, short-term memory carrying goal-related information to perform one task also inadvertently influences performance in another task, sometimes causing significant interference. Memory traces acquired over a task setting—reflected in probability cuing, selection history, reward history, contextual cueing, or sequence learning—are often acquired incidentally and utilized involuntarily and without explicit knowledge. Remote, long-term memories for the presence and location of items within rich, complex scenes are more often accompanied by conscious recollection. Although it is likely that these associations are used to direct attention voluntarily, they may also orient attention involuntarily. Under most conditions, benefits in LTM-based orienting correlate with measures of explicit memory, suggesting a viable source for voluntary attention. However, dissociations have also been noted. Although explicit retrieval of object-scene memories is significantly compromised in older individuals compared with younger counterparts, the benefits of memory-based attention are spared. The relationship and the degree of interaction between these two modes of memory-based attention are important directions for future research.

29In recognizing the role that memory associations within task settings play in attention, some contemporary models of attention have proposed adding a third, “history” source to the traditional bottom-up and top-down sources in attention (e.g., ). The inclusion is an important first step toward acknowledging the fundamental role played by memory, but there are also inherent limitations with treating memory as a single and separate additional source of attention control. On one hand, it is not possible to extricate memory from the two traditional sources of attention. Brief memory traces contribute to effects of bottom-up attention, prioritizing stimuli that trail physical singletons, and WM traces are essential for goal-based top-down attention. On the other hand, amalgamating memory sources into one common history factor works against evidence suggesting the plurality of memory mechanisms at play.

30The internal, memory-based sources of attention are many. Memories of various durations and types contribute, from traces that stitch the continuity of perception to those that guide the recognition of a long-lost friend. Most likely, multiple memory systems and mechanisms are involved. For example, both contextual cueing and memory-based orienting involve LTM for the location of a target object within a given configuration in an array or scene, and in both cases, the medial-temporal areas have been implicated. However, brain imaging shows increased hippocampal activity during memory-guided orienting in scenes and decreased activity by repeated versus novel arrays in contextual cueing tasks. Electrophysiological recordings also show different patterns of modulation of potentials associated with target selection (N2pc), with attenuation by memory-guided orienting and enhancement by contextual cueing. Striking differences occur even when comparing different types of memory traces within a single task. contrasted the effects of contextual cueing (associations between specific array configurations and target locations) versus probability cueing (probabilistic learning of most likely target locations and response choices) in guiding visual search in a common task. Although contextual cueing modulated activity in the hippocampus, probability cueing modulated activity in the striatum. The effects were selectively predictive of attention benefits across successive trials and correlated with performance measures. Numerous functional dissociations have also been observed in behavioral effects of attention based on different types of memory within a task context. Although often entangled in natural contexts, probability cueing effects can be separated from single-trial priming effects, from selection history, and from goal-based, short-term-memory cueing. Likewise, reward history can be separated from selection history.

31Thus, the literature suggests a rich diversity of mnemonic influences on attention, with at least some functional dissociations. There is much empirical work ahead to reveal the natural kinds and neural mechanisms of memory-based attention to derive its principles of organization and develop useful theoretical models. Traces of different kinds and durations could act separately and largely independently; they could compete, they could become integrated within a unified predictive model (e.g., in a Bayesian framework; ), they could converge and operate through a common priority map to modulate sensory processing, or they could combine in mutually supportive ways to enhance the quality and flexibility of biasing signals. At this stage, it is premature to clump together memories into a single source of attentional control, and doing so could discourage or misguide much needed investigation in this important area.

32Premembering has implications for understanding the nature of memory itself. Consonant with a role in serving future behavior by grounding attentional selection, the very nature of memory traces is forward looking. What is encoded, maintained, and selected for retrieval is strongly influenced by what is likely to be important for future behavior.

33At the shortest timescales, the contents of memories maintained across eye fixations are strongly influenced by the location of the upcoming saccade. Even before a saccade is made, the information corresponding to its upcoming landing zone is remembered better than information at other, equidistant locations.

34Within WM research, findings increasingly highlight the prospective qualities of these traces. The contents of WM are strongly influenced by current task goals. They are selective and can even be distorted to facilitate performance based on anticipated task demands. The selectivity of WM was elegantly illustrated by a neurophysiological study in monkeys showing that neuronal activity in the prefrontal cortex (PFC) was largely dominated by the single relevant item of a multi-item array during a selective delayed-match-to-sample task. Later, responses in prefrontal neurons were shown to be most sensitive to stimulus dimensions that were relevant for making categorical discriminations imposed by the task. Although prefrontal neurons differentiated these discriminant features, they were less sensitive to other equally available, non-discriminant physical features until these became task relevant. Similar effects were reported for the parietal cortex when monkeys performed categorical discrimination tasks based on arbitrary boundaries along continuous feature dimensions.

35Selectivity in WM is also well documented in the human brain. The CDA marker of WM derived from electroencephalograms (EEGs) shows that encoding can be spatially and item selective (e.g., ). In addition, selectivity can also occur for relevant features within memory items. Using multivariate pattern analysis (MVPA) to derive population-response properties from fMRI data, found that decoding during a WM delay depended on the expected demands during recall. Participants performed a delayed matching task on colored gratings and were cued that either color or orientation was relevant to the discrimination. Patterns of activity in the visual cortex selectively maintained the task-relevant feature, consistent with a prospective memory code for guiding future behavior (Figure 3A).

36Studies in human participants have also demonstrated that selective contents in WM can be updated flexibly as predictions change about the items or features that will be relevant for task performance. Cues presented during the delay that retrospectively inform participants about the item most likely to be probed (retro-cues) result in significantly better memory performance. fMRI studies show selective maintenance of task-relevant features after informative retro-cues that predict or instruct which stimulus or attributes will be probed. Human neurophysiological studies also show flexible updating of selective content in WM during maintenance with retro-cues that indicate which item in memory is most likely to be probed (,,; Figure 3B). Such updating can also occur in the absence of external cues, when the passage of time is associated with the likely item to be probed, illustrating the flexibility of updating selective WM contents and the role of temporal expectations.

37From a functional perspective, it is not always optimal simply to maintain a veridical representation of previous input. When making some types of fine-grained delayed discriminations, it can be more advantageous to focus on neural information tuned away from the behaviorally relevant feature to maximize separability from distractor items. In such tasks, behavioral measures confirm that WM templates are adaptive distortions of the experienced information. Application of multivariate methods to derive tuning functions of fMRI voxels confirmed that WM templates in such tasks reflect distortions of the perceived stimulus, thus being adaptive rather than veridical. Interestingly, in addition to being shifted, the relevant feature information maintained in the target template can also be sharpened to increase separability from distractor stimuli in the task (,; Figure 3C).

38These studies clearly reveal the prospective face of WM: content is optimized to the expected demands of future processing. In some cases, the prospective WM representation triggered by a stimulus bears no physical resemblance to it. trained monkeys on a paired-associate task that systematically manipulated the relationship between memory items and expected probe stimuli. This revealed a subset of neurons coding for the expected stimulus during the delay period even though it was not actually presented on that trial. This also demonstrates the interaction between WM and LTM.

39WM representations also go beyond the experienced stimulus to construct motor representations when specific actions can be anticipated. Most WM tasks tend to probe the perceptual content of the memoranda and typically isolate perceptual qualities of stimuli from particular responses. In everyday life, however, the perceptual content of WM often also carries motor affordances, such as when reaching out and grasping the cup of coffee outside your field of view or repeating a telephone number to a friend. In a recent visual WM task that linked the reporting of particular stimulus orientations to a specific hand, it became clear that motor representations are derived from visually encoded stimuli and that motor information is in a similar state of readiness as visual content when participants are probed (; Figure 3D).

40The prospective attributes of LTM are much more difficult to study. Except under highly constrained experimental conditions, it is difficult to determine what aspects of experience will prove useful to guide future behavior at distant time points. Nevertheless, the prospective nature of LTM can be gleaned from a few observations. The content of LTM is selective. We better remember items that were relevant during the encoding context. We retain aspects of stimuli that are useful for adaptive behavior. Although our memories of the details in the appearance of a penny are shockingly poor, we readily remember the attributes that distinguish a penny from other coins. Similarly, our well-established difficulty with individuating and remembering faces from other cultures and races may, in part, reflect our focus on discriminant features that are informative within our given cultural context.

41When considering the prospective nature of memory, studies have typically focused on the task dependency of the representational content (as reviewed above). This information-based perspective provides an important starting point. At minimum, the content of memory provides the basic information needed for adaptive guidance of preparatory attention. However, we need to delve deeper to understand how memories proactively exert their influence in adaptive behavior. Focusing specifically on WM, we consider how mnemonic states functionally contribute to memory-dependent processing of subsequent input.

42From a functional perspective, we can distinguish the task relevance of specific types of information from their intended use (Figure 4). For example, tasks may differ in terms of which stimuli are relevant (e.g., squares versus circles in a match-to-sample paradigm), but tasks can also differ in how memories will be used. The same memory item (e.g., red square) could be required for comparing it with a previous item (e.g., color match-to-sample) or for reproducing its exact hue (e.g., continuous report). Under these circumstances, the same information is maintained but for different computational purposes.

43Theoretical simulations suggest that WM may be coded differently depending on expected future use. found that a recurrent neural network trained to perform WM exploited a range of different possible coding schemes depending on the precise nature of the memory task. Essentially, the same information content of a visual stimulus (e.g., orientation) was stored differently depending on how it would be interrogated at the end of the delay.

44The extant empirical data are also consistent with the theoretical proposal that the format of WM depends on expected use. For example, showed that delay-related activity in the PFC differs when the monkey remembers the same stimuli (natural objects) for different types of recall (match judgement versus serial order). reported similar evidence in humans. Using fMRI, they found that the patterns of neural activity coding for visual stimuli during a memory delay critically depended on how memories were to be used at the end of the trial (implement versus recall).

45If computational specificity is functionally relevant to future processing and also inherent to WM, then we might predict that the content of WM always guides attention. The evidence for this hypothesis is mixed. In a series of behavioral and neuroimaging studies, found that holding an item in WM automatically biased attention to matching items in an unrelated secondary search task. This effect persisted even when the task strongly encouraged participants to mentally separate the WM item from the search task. This obligatory link suggests that simply holding something in WM is sufficient for guiding attention. However, it turns out that the story is a bit more complicated. When there is more than one item in the WM set, it is easier to decouple from attentional selection.

46To reconcile the apparent contradiction, proposed that one item in WM is in a prioritized state and serves as an attentional template. We term this the “active” item. If only one item is in memory, then it is automatically also the active item guiding visual search. If, however, there are multiple items in WM, then only one item is functionally active, and others act as functionally “latent” items. In the absence of any particular difference in relevance among items, priority could be assigned to any one of them. On average, this dilutes the effect of automatic attentional capture because the critical item only coincides with the active one on some trials. Unless priority is under direct experimental control, it is difficult to disentangle variability in priority state across trials from a diluted effect shared across items.

47Active and latent memories can be studied more directly by experimentally manipulating the relative priority of items in WM using retro-cues. Behavioral and neural evidence confirmed that participants can flexibly toggle priority between different items in WM, resulting in performance benefits for the currently relevant item. Interestingly, latent items are not forgotten, just temporarily de-prioritized while the active item guides current behavior.

48It is tempting to think of retro-cueing simply as an internal spotlight boosting the representational content of cued relative to uncued (and potentially distracting) memoranda. However, we argue that retro-cues may also transform the format of a memorandum, preparing it for future use, such as by placing it in a prioritized state and linking its informational content to future action. Moreover, a change in priority status need not result in a cost to other information in WM. When one item is initially prioritized, the performance cost to uncued items can be overcome when there is time or opportunity to reset priorities. Thus, in contrast to the typical consequence of attention (in perception), trade-offs in information quality are not necessary when prioritizing items in WM. Instead, deficits arise mainly because item-related traces in WM are not currently optimized for readout.

49Human neurophysiology also supports the general view that prioritizing an item in WM involves a discrete state transition rather than sustained internal attention. Using magnetoencephalography (MEG), we observed that shifting priority in WM leads to a transient lateralization of posterior alpha power (; Figure 3B), indexing a punctate process of spatial selection. This contrasts with a sustained pattern of alpha lateralization, as observed during anticipatory attention in perception (; see also ). We interpret this transient response as a discrete process of setting up the most relevant item to guide future target processing.

50Recent brain imaging studies also identified different neural correlates of WM depending on current priority status. There is a quantitative difference in decodability between active and latent items. Across multiple experiments, have shown that active items more readily decoded from patterns of visual activity. Indeed, latent items often have no detectable neural trace even though they remain in WM and are available for future use when they are re-prioritized.

51These decoding studies show that latent working memories can be nearly impossible to detect in delay period activity. Nevertheless, complementary recent evidence shows that latent memories may be measured using approaches that are sensitive to changes in neural states that might not be reflected in tonic activity. The impulse response approach borrows from the logic of active sonar, in which a well-characterized impulse (“ping”) is emitted toward a hidden landscape, and the contours are inferred from distortions in the reflected signal. In the case of “neural sonar,” a neutral visual stimulus acts as a sensory ping impulse and interacts with the brain state, resulting in a state-dependent impulse response. Changes in the neural landscape can be inferred from distortions in this output response. Importantly, this approach is theoretically sensitive to any change in the functional state of the targeted system. In addition to the manifest delay activity relating to firing in neuronal assemblies, which has been the focus of most WM studies to date, it can also reveal activity-silent neural states, such as temporary changes in synaptic weights within neuronal assemblies.

52Using this approach, it has been possible to observe robust memory signals in the impulse response despite dramatic differences in activity-based decoding between active and latent items. This suggests that even functionally latent states have a robust neural signature in silent neural states. Therefore, latent WM is not necessarily just a weaker version of active WM, but, rather, it can be maintained in a qualitatively different neural state. Such a scheme makes sense when one considers that latent memories carry information that may become important down the line and can therefore be just as important to maintain as currently active memories. The important difference is that latent memories should be kept functionally dormant while the active memory is guiding behavior to avoid cross-item interference. Some recent fMRI studies provide initial support for this idea. For example, recent fMRI studies identified different brain areas and different coding schemes within the visual cortex associated with active and latent working memories.

53The classic biased-competition model of attention proposed that perceptual templates in WM bias visual processing to prioritize task-relevant items. This idea is grounded in the notion that WM is maintained via persistent activation of sensory-specific neural codes (e.g., ), resulting in an elevated baseline for subsequent processing of related input. For example, persistent delay activity associated with remembering the letter X essentially pre-activates the neural code for X, providing a head start for the sensory processing of X (or X-related) stimuli.

54Indeed, there is extensive evidence for such persistent activity associated with WM and preparatory attention. During WM delays, persistent activity occurs across many brain areas, from the visual cortex right up to the PFC. The earliest evidence for WM-related persistent activity was observed in the monkey PFC (e.g., ). WM delay activity is selective for the content of memory; specific cells are more active when their preferred (relative to non-preferred) stimulus is held in mind (e.g.,, ). At the population level, item selectivity gives rise to a population code that can be “decoded” by downstream regions. Similar evidence for item-decodable engrams have also been reported in the parietal cortex and in visual areas such as the inferior temporal cortex. Correspondingly, brain imaging studies using multivoxel pattern analysis have found stimulus-specific delay activity throughout the human visual system (e.g., ) and higher-order brain areas. This suggests that WM could be a systems-level phenomenon, with different areas contributing to distinct but complementary functions (e.g., ).

55A very similar profile of activity is observed for preparatory attention. In preparation for visual search guided by a specific object or location in WM, activity in visual areas representing the relevant object or location is elevated in anticipation of the search array. Human fMRI studies have also reported elevated levels of activity for the spatial location or identity of relevant, anticipated objects based on WM templates. Such findings have supported the influential idea that persistent activity associated with maintenance in WM provides the major neurophysiological mechanism for top-down attentional modulation by effectively biasing subsequent activation of matching sensory input. According to a simple baseline shift model, persistent activity for WM becomes preparatory activity for attention, giving relevant information an advantageous head start in competitive sensory processing.

56Despite the general appeal of elevated baseline activity linking WM and attention, the specific neurophysiological evidence is not straightforward. Although classic evidence for item-specific delay activity highlighted pre-activation of target items, the actual correspondence between selectivity during the delay and sensory-related responses is more complex. A neuron’s stimulus preference during the delay often differs from preferences during stimulus processing in the visual cortex (e.g., ) as well as the PFC. The evidence is also mixed for another core prediction: pre-stimulus activity should directly translate to a corresponding boost in the sensory signal. Although a number of studies found that trial-wise variance in item-specific delay activity correlates with behavioral performance (e.g., ), there is minimal evidence that this is achieved through a corresponding boost to target processing (e.g., ). For example, in classic studies demonstrating anticipatory sustained delay activity (e.g., ), the initial neuronal response to the target array is equivalent regardless of whether the target to be selected is effective or ineffective at driving the cell. Only later does neuronal firing settle into a strong or weak response depending on whether the target was effective or ineffective, respectively.

57A different but related idea is that WM establishes a match filter in sensory areas that effectively computes the perceptual similarity between incoming sensory signals and an internal template (,,; Figure 5). Theoretically, a match filter template does not depend on pre-activation of target templates but, rather, can be instantiated by more complex activity dynamics (e.g., ) or by short-term synaptic plasticity. This idea can be tested by examining the response to memory-matching stimuli rather than delay activity. For example, in a classic study, found an enhanced response for stimuli that matched the cued color in V4. observed a similar enhancement for matching stimuli, which they interpreted as evidence for a match filter mechanism. further demonstrated that the amplitude of match-related activity in single neurons in high-level visual areas co-varied with the amplitude of activity during encoding rather than during delay activity. This pattern was consistent with predictions from their match filter model instantiated with synaptic plasticity. Using MEG in human participants, we showed that perceptual decision-making based on templates held in WM is also consistent with a match filter mechanism.

58Although the match filter model is typically invoked to explain WM decisions (e.g.,, ), the match-related enhancement signal could also serve attentional selection, providing a relative boost to target stimuli. Moreover, the same match-related enhancement would also naturally benefit the processing of related features, such as items in the spatial location of the match object. Match-related attentional enhancement is consistent with previous evidence presented above for automatic memory-based capture (e.g., ).

59In general terms, WM can be thought of as setting the initial conditions for state-dependent processing. The initial condition at the time of target processing is determined by previous input. Differences in stimulus history cause differences in the initial condition, which, in turn, alter the response dynamics of the system during target processing. These state-dependent dynamics can perform a number of memory-related operations, such as reproduce a previous item and its temporal delay, bias the incoming signal for a match-filter response, adaptively distort processing based on off-channel tuning for optimal discrimination performance, and instantiate a temporary decision rule for flexible decision-making. Importantly, the previous input that defines the initial conditions includes recent stimulation history (e.g., specific items in WM) but also stretches back across timescales (e.g., task instructions and learning form intermediate and LTM).

60Mnemonic states could be expressed via elevated patterns of neuronal activity and/or altered patterns of synaptic weights over short and long timescales. In terms of WM specifically, understanding the relative contributions of neural states defined by activity patterns versus activity-silent traces remains an important area for investigation. Various accounts propose mappings between different functional and neural states in WM; functionally active WM is expressed as decodable activity, whereas functionally latent states are silent (e.g.,,, ). However, at least theoretically, functionally active or latent memories could be stored in either activity-dependent or silent neural states. We should not conflate the two notions of active.

61In summary, a functional perspective re-casts WM as a flexible shift in how the brain prepares to process future information rather than just maintaining a representation of past information. Considered this way, it is the functionality of the neural state that is most important and not merely the decodability of memory content. Decodability is only a minimal requirement for a memory. To understand how memories are stored for future use (recall, attention, or anything else), it is necessary to understand how mnemonic states interact with subsequent input to produce appropriate output. Recent methodological developments provide an expanding toolbox for exploring the functional properties of mnemonic states. For example, impulse perturbations provide a powerful tool for probing the functional state during memory delays. Analysis tools for characterizing context-dependent neural dynamics will also shed further light on the critical interaction between WM and subsequent processing.

62In principle, many of the mechanisms we have considered in the context of how working memories guide future processing may also apply to LTM. It will be important for future studies to determine the extent to which activity-silent LTM representations modulate information processing directly, depending on the overlap between present and past settings and goals. It will be informative to understand under what conditions long-term traces are prioritized and energized to guide adaptive behavior. In the latter context, it will also be interesting to learn whether or when activated LTM traces necessarily operate through functionally active WM states to guide attention (e.g., ).

63We have considered memory through a different temporal lens. Rather than focusing on how memories echo the past, we have considered how we use past experience to anticipate relevant events to unfold. In addition to providing evidence that traces across several timescales premember events proactively and dynamically, we argue that this basic ecological function of memory shapes the content and the format of what is stored, maintained, and accessed. Significant work lies ahead to understand how mnemonic neural substrates facilitate the processing of incoming information to guide adaptive behavior. Looking at memory from this perspective opens many interesting doors for future exploration, such as the types of memory-based modulatory mechanisms, the existence of a common or multiple memory-based priority maps to guide attention, and how memories can be selective and context dependent and yet be used to generalize to novel situations.