The Interactive Account of ventral occipitotemporal contributions to reading

1There has been considerable interest in the role of the ventral occipitotemporal cortex (vOT) during reading. Learning to read increases left vOT activation in response to written words and damage to left vOT impairs the ability to read. These and other findings have led to claims that the response properties of vOT change during reading acquisition, leading to neuronal populations that are selectively tuned to orthographic inputs. However, a significant number of studies have reported that, even after learning to read, vOT is highly responsive to non-orthographic stimuli, with a selectivity that depends on the nature of the task and the stimulus. The same vOT area also responds to orthographic and non-orthographic tactile stimuli. These diverse response properties suggest that vOT contributes to many different functions that change as it interacts with different areas. In this context, it is difficult to find a functional label that explains all vOT responses.

2To explain the heterogeneity of responses in vOT, we formalize the Interactive Account of vOT function during reading by presenting it within a predictive coding (i.e. a generative) framework. This perspective provides a parsimonious explanation of empirical findings and is based on established theoretical and neurobiological principles. Before presenting this framework, we begin with an anatomical description of vOT.

3vOT is centred on the occipitotemporal sulcus but extends medially onto the lateral crest of the fusiform gyrus and laterally onto the medial crest of the inferior temporal gyrus. In the posterior–anterior direction, vOT is located on the ventral border of the occipital and temporal lobes (Figure 1a), which lies between y = –50 and y = –60 in standard Montreal Neurological Institute (MNI) space. More posteriorly, activation is highest to visual inputs, but more anteriorly activity increases in response to familiar visual, tactile or auditory stimuli, consistent with a basal temporal language area. Given its position between visual and language areas, it is not surprising that vOT responds to a range of visual stimuli as well as the language demands of the task. The association between vOT and language processing is further supported by observations that lateralization (left versus right hemisphere dominance) in vOT correlates with language lateralization in frontal language areas.

4The Interactive Account is based on the premise that perception involves recurrent or reciprocal interactions between sensory cortices and higher order processing regions via a hierarchy of forward and backward connections (Figure 2). Within the hierarchy, the function of a region depends on its synthesis of bottom-up sensory inputs conveyed by forward connections and top-down predictions mediated by backward connections. These predictions are based on prior experience and are needed to resolve uncertainty and ambiguity about the causes of the sensory inputs on which predictions are based. The hierarchical nature of neocortical organization is reflected in the abundance of backward relative to forward connections. Because functional magnetic resonance imaging (fMRI) does not distinguish between synaptic activity induced by forward connections and that induced by backward connections, it reports their combined contribution (Figure 2), which includes prediction error.

5For reading, the sensory inputs are written words (or Braille in the tactile modality) and the predictions are based on prior association of visual or tactile inputs with phonology and semantics. In cognitive terms, vOT is therefore an interface between bottom-up sensory inputs and top-down predictions that call on non-visual stimulus attributes. Without prior knowledge the relationship between orthography and phonology, vOT activation to words will be low because phonological areas do not send backward predictions to vOT (Figure 2 and Box 1). Once phonological associations are learned, backward connections can deliver top-down predictions to vOT when the stimuli are words or word-like. In this context, top-down processing does not imply a conscious strategy; it is mandated by unconscious (hierarchical) perceptual inference. In other words, it represents the intimate association between visual inputs and higher level linguistic representations that occurs automatically and is modulated by attention and task demands. Interpreting activation in vOT therefore requires consideration of the stimulus, experience-dependent learning and context (i.e. the task requirements and the attentional demands). Likewise, interpreting the effect of damage to vOT depends on how word recognition is affected by disrupting top-down inputs from higher order regions to vOT, and from vOT to lower level visual regions (Box 2).

6Our account assumes that neuronal populations in vOT are not tuned selectively to orthographic inputs (Box 3). Instead, orthographic representations emerge from the interaction of backward and forward influences. In the forward direction, we postulate that neurons in vOT accumulate information about the elemental form of stimuli from complex receptive fields (Figure 1 and Box 3). In the backward direction, higher order conceptual and phonological knowledge predicts the pattern of activity distributed across multiple neurons within vOT. Put another way, orthographic representations are maintained by the consensual integration of visual inputs with higher level language representations. This perspective allows the same neuronal populations to contribute to different functions depending on the regions with which they interact and the predictions for which the current context calls. In this context, the neural implementation of classical cognitive functions (e.g. orthography, semantics, phonology) is in distributed patterns of activity across hierarchical levels that are not fully dissociable from one another.

7The visual information that is accumulated in vOT must be sufficiently specific to induce coherent patterns of activation in semantic and phonological areas that send top-down predictions back to vOT. For example, in McClelland and Rumelhart's Interactive Activation model of visual word recognition, partial visual information cascades forward activating incomplete phonological and semantic patterns, which in turn feed back to support consistent orthographic patterns and suppress inconsistent ones. As in connectionist models of reading, we propose that patterns of activation across vOT neurons encoding shape information are sufficient to partially activate neurons encoding semantics and phonology in higher order association regions, which provide recurrent inputs to vOT until the top-down predictions and bottom-up inputs are maximally consistent. Thus, predictions are optimized during the synthesis of bottom-up and top-down information (Figure 2).

8In cognitive terms, top-down processing typically refers to conscious, strategic and task-related effects. Automatic, non-strategic top-down processes are also recognized, particularly in computational models of reading. The ubiquity of automatic top-down effects has been demonstrated neurophysiologically in monkeys, where inactivating higher-order cortical areas (by cooling) results in changes to extra-classical receptive fields, despite the monkey being anesthetized.

9Here we make a clear distinction between strategic and non-strategic top-down influences on vOT activation. Strategic influences have been demonstrated in studies showing that vOT activation changes with task, even when the stimulus, attention and response times are controlled. In contrast, non-strategic top-down influences on vOT activation are generated automatically and unconsciously from previous experience with similar stimuli (Figure 2 and Box 1). That is, visual words automatically engage processing of their sounds and meaning, which provide predictive feedback to the bottom-up processing of visual attributes.

10A clear example of automatic (non-strategic) top-down effects on vOT activation comes from a picture-word priming experiment that found reduced vOT activation for unconsciously perceived primes that were conceptually and phonologically identical to a stimulus that was subsequently named. For example, when a visually presented written object name (e.g. LION) was preceded by a rapidly presented, masked (unconscious) picture of the same object, activation in vOT was reduced relative to when it was preceded by a picture of a different object (e.g. a chair). Similarly, masked written object names (words) reduced vOT activation for pictures of the same objects. These findings can be explained easily by automatic, top-down predictions that prime visual shape information in vOT. In essence, the brief (and unconsciously perceived) prime is sufficient to engage phonological and/or semantic processing that automatically sends predictions regarding the identity of the next stimulus (the target) back to vOT, thereby reducing prediction error and activation. The fact that priming occurs across stimulus formats (pictures/words) demonstrates that these backward projections predict all visual forms of a concept (e.g. object form and written form). The same account also explains reduced vOT activation when a word is primed by the same word in a different case (e.g. AGE–age) without postulating the need for abstract visual word form detectors.

11The effect of word–picture priming on vOT activation cannot be explained in terms of feed-forward visual processing because there is no visual similarity between the prime and the target that can serve as the basis for reduced vOT activation (e.g. through simple adaptation effects). Explanations based on strategic top-down processing are also insufficient, because participants are not aware of the primes and thus cannot use them to generate conscious expectations. The effects can nevertheless be explained by the Interactive Account in terms of automatic top-down influences that combine with bottom-up visual information to determine information processing in vOT.

12Several studies have shown activity is higher in response to pseudowords than to words in posterior parts of the occipitotemporal sulcus (y = –60 to y = –70 in MNI space) and more sensitive to words than to pseudowords in anterior parts of the occipitotemporal sulcus (y = –40 to y = –50) (for a review, see ). However, here we consider the more perplexing pattern of selectivity that occurs at the centre of vOT (y = –50 to y = –60), where activity has been reported to be greater for: i) pseudowords (e.g. GHOTS) than for consonant letter strings (e.g. GHVST); (ii) pseudowords than words (e.g. GHOST); and (iii) low versus high frequency words (GHOST versus GREEN). This combination of effects cannot be explained by a progressive increase or decrease in vOT response to familiarity (consonants < pseudowords < low frequency words < high frequency words) because responses to pseudowords are higher than those to both unfamiliar consonants and familiar words. Nor can vOT response selectivity be explained by bigram or trigram frequency, because greater activation has been reported for pseudowords than for words when bigram and trigram frequency are controlled.

13The Interactive Account explains vOT responses to different types of stimulus simply, in terms of interactions between bottom-up visual information and top-down predictions (Figure 2). During passive viewing tasks, activation increases for pseudowords relative to consonant letter strings because pseudowords are more word-like and therefore engage top-down predictions from phonological areas. By contrast, activation is greater for pseudowords than for words because, although both activate top-down predictions, there is a greater prediction error for pseudowords. That is, for a previously encountered stimulus (i.e. a word) there is a good match between predictions and the visual representations being predicted, producing minimal prediction error, whereas for unfamiliar pseudowords there is a poor match that increases prediction error and activation in vOT. Similarly, prediction error and activation will be less for high than for low frequency words because high frequency words are more familiar, which means their predictions are more efficient because they call on stronger associations between visual and linguistic codes.

14This account also explains apparent word selectivity, such as repetition suppression in vOT for words primed by an identical word but not for those where the prime differs from the target by one letter (e.g. coat–boat). Clearly, the non-identical prime activates different phonological and semantic patterns than the target word, leading to increased prediction error in vOT. In contrast, small orthographic differences between the prime and the target that result in only minor phonological and semantic changes (e.g. teacher–teach) yield minimal prediction error, resulting in reduced vOT activation.

15It is important to note that selectivity (in terms of greater activation for one stimulus relative to another) depends on numerous bottom-up and top-down processing demands that change with the task, familiarity with the stimulus, and the degree of overlap between the stimulus and other stimuli that might compete for a response (i.e. the orthographic neighbourhood effect). It is possible that selectivity can be reversed in one context relative to another. For example, during passive viewing conditions, vOT activation can be higher for words than for consonant strings because top-down predictions are activated by words that look familiar. In contrast, in attentionally demanding paradigms (e.g. the one-back task), vOT activation can be higher for consonants than for words because, in the absence of top-down support from semantics and phonology, the visual processing demands of the task are greater for consonants.

16When semantic and phonological associations are controlled by comparing written object names to pictures of the same objects, activation in vOT is typically greater for pictures than for written words, but again, it depends on the combination of the task and the bottom up visual inputs. During a non-linguistic task such as passive viewing, colour decision or a one-back task, vOT activation can be higher for words than for pictures when the physical dimensions of the visual stimuli are matched, although the location of this effect may be anterior to vOT proper. By contrast, during naming tasks, vOT activation has only been reported as greater for pictures than for words.

17Again, the task-specific reversal of stimulus selectivity can be explained by the Interactive Account in terms of a combination of forward inputs, top-down predictions and the mismatch between them (i.e. the prediction error). Activation related to forward inputs is greater for larger and more complex visual stimuli (e.g. pictures). Activation related to top-down predictions is greater for words than for pictures during non-linguistic tasks because only words have a sufficiently tight relationship with phonology to induce top-down predictions automatically. Activation related to prediction error is higher for pictures than for words during naming tasks because access to phonology is needed to name pictures and words, but the links between vOT and phonological areas are less accurate (more error-prone) for pictures. Thus, the Interactive Account provides a systematic and parsimonious explanation of a previously unexplained range of empirical data.

18In summary, we have presented an Interactive Account that is based on a generic framework for understanding brain function (Figure 2). It explains vOT activation in terms of the synthesis of visual inputs carried in the forward connections, top-down predictions conveyed by backward connections, and the mismatch between these bottom-up and top-down inputs.

19Although there are many outstanding questions (Box 4), we suggest that: (i) vOT activation to orthographic stimuli increases while individuals are learning to read because inter-regional interactions become established and top-down predictions from phonological and semantic processing areas become available; (ii) vOT activation is greater for pseudowords than for words, and for low relative to high frequency words because of increased prediction error; (iii) greater activation for pictures of objects than for their written names is the combined consequence of more complex visual features, less constrained top-down predictions and therefore increased prediction error; (iv) greater activation for written words than objects is observed when the task does not control for the top-down influence of language on written word processing; (v) damage to vOT impairs reading, object naming and perceptual processing because visual inputs are disconnected from top-down predictions from vOT; and (vi) vOT activation will be lower in developmental dyslexics, in whom top-down predictions from phonological and semantic processing areas are less automatically generated than in age-matched skilled readers.

20The automatic interactions between visual, phonological and semantic information that we argue for are a fundamental property of almost all cognitive models of visual word recognition and are necessary to explain a range of reading behaviours. Incorporating them within a neural framework obviates the need to postulate a novel form of learning-related plasticity (e.g. ‘neuronal recycling’) or reading-specific neuronal responses (e.g. ‘bigram detectors’). Instead, the Interactive Account relies on well established principles of neocortical function that are not specific to reading, but nonetheless accommodate this recently developed cultural skill.Bottom-up sensory information: external information arrives at the senses and projects to primary sensory cortices. These drive secondary, tertiary and higher order association cortices via forward connections arising primarily from superficial (layer II and III) pyramidal neurons. Within the ventral occipitotemporal cortex (vOT), the primary source of bottom-up information is visual, presumably from areas V2, V4v, and posterior parts of the lingual and fusiform gyri.Generative models: probabilistic models of how (sensory) data are caused. In machine learning, they include both bottom-up ‘recognition’ connections and top-down ‘predictive’ connections. These models learn multilayer representations by adjusting the top-down connection weights to better predict sensory input. Existing computational models of reading use implicit generative models and share many important features such as interactivity and the use of prediction errors to learn weights (e.g. through back-propagation of errors).Predictive coding: a ubiquitous estimation scheme (developed in engineering) and instantiated in hierarchical generative models of brain function. Here, cortical regions receive bottom-up input encoding features present in the environment as well as top-down predictions. These predictions attempt to reconcile sensory input with one's internal knowledge of how input is generated. Thus, the function of any region is to integrate these two sources of input dynamically into a coherent, consistent, stable pattern of activity.Prediction error: the difference between bottom-up (sensory) input and top-down predictions. Within vOT, prediction error is minimized when they agree. Any irresolvable mismatch (e.g. when processing pseudowords) elicits prediction error, which elicits an increased BOLD signal response (Figure 2).Top-down predictions: the automatic input a region receives from areas above it in the anatomical hierarchy. These connections attempt to predict the bottom-up inputs based on the context and active features. Important sources of top-down input to vOT are (deep) pyramidal cells in cortical areas that contribute to representing the sound, meaning and actions associated with a given stimulus.

21Bottom-up sensory information: external information arrives at the senses and projects to primary sensory cortices. These drive secondary, tertiary and higher order association cortices via forward connections arising primarily from superficial (layer II and III) pyramidal neurons. Within the ventral occipitotemporal cortex (vOT), the primary source of bottom-up information is visual, presumably from areas V2, V4v, and posterior parts of the lingual and fusiform gyri.Generative models: probabilistic models of how (sensory) data are caused. In machine learning, they include both bottom-up ‘recognition’ connections and top-down ‘predictive’ connections. These models learn multilayer representations by adjusting the top-down connection weights to better predict sensory input. Existing computational models of reading use implicit generative models and share many important features such as interactivity and the use of prediction errors to learn weights (e.g. through back-propagation of errors).Predictive coding: a ubiquitous estimation scheme (developed in engineering) and instantiated in hierarchical generative models of brain function. Here, cortical regions receive bottom-up input encoding features present in the environment as well as top-down predictions. These predictions attempt to reconcile sensory input with one's internal knowledge of how input is generated. Thus, the function of any region is to integrate these two sources of input dynamically into a coherent, consistent, stable pattern of activity.Prediction error: the difference between bottom-up (sensory) input and top-down predictions. Within vOT, prediction error is minimized when they agree. Any irresolvable mismatch (e.g. when processing pseudowords) elicits prediction error, which elicits an increased BOLD signal response (Figure 2).Top-down predictions: the automatic input a region receives from areas above it in the anatomical hierarchy. These connections attempt to predict the bottom-up inputs based on the context and active features. Important sources of top-down input to vOT are (deep) pyramidal cells in cortical areas that contribute to representing the sound, meaning and actions associated with a given stimulus.

22Bottom-up sensory information: external information arrives at the senses and projects to primary sensory cortices. These drive secondary, tertiary and higher order association cortices via forward connections arising primarily from superficial (layer II and III) pyramidal neurons. Within the ventral occipitotemporal cortex (vOT), the primary source of bottom-up information is visual, presumably from areas V2, V4v, and posterior parts of the lingual and fusiform gyri.

23Generative models: probabilistic models of how (sensory) data are caused. In machine learning, they include both bottom-up ‘recognition’ connections and top-down ‘predictive’ connections. These models learn multilayer representations by adjusting the top-down connection weights to better predict sensory input. Existing computational models of reading use implicit generative models and share many important features such as interactivity and the use of prediction errors to learn weights (e.g. through back-propagation of errors).

24Predictive coding: a ubiquitous estimation scheme (developed in engineering) and instantiated in hierarchical generative models of brain function. Here, cortical regions receive bottom-up input encoding features present in the environment as well as top-down predictions. These predictions attempt to reconcile sensory input with one's internal knowledge of how input is generated. Thus, the function of any region is to integrate these two sources of input dynamically into a coherent, consistent, stable pattern of activity.

25Prediction error: the difference between bottom-up (sensory) input and top-down predictions. Within vOT, prediction error is minimized when they agree. Any irresolvable mismatch (e.g. when processing pseudowords) elicits prediction error, which elicits an increased BOLD signal response (Figure 2).

26Top-down predictions: the automatic input a region receives from areas above it in the anatomical hierarchy. These connections attempt to predict the bottom-up inputs based on the context and active features. Important sources of top-down input to vOT are (deep) pyramidal cells in cortical areas that contribute to representing the sound, meaning and actions associated with a given stimulus.