Decoding Sound and Imagery Content in Early Visual Cortex

1We used fMRI in combination with multivariate pattern analysis (MVPA) to explore how complex information from audition and imagery translates to the coding space of early visual cortex in the absence of feedforward visual stimulation. Throughout our experiments, we omitted any visual stimulation by blindfolding our subjects (Figure 1). In experiment 1, subjects listened to three types of natural sounds: bird singing, traffic noise, and a talking crowd (see Figure 2). fMRI activity patterns were extracted from retinotopically mapped visual areas 1, 2, and 3 (V1, V2, and V3) (Figure 1) and fed into a multivariate pattern classifier (linear support vector machine; see Supplemental Experimental Procedures available online). The classifier successfully discriminated the three different sounds in early visual cortex, particularly in V2 and V3 (at ∼42%; see Figure 2; for results with increased statistical power, see Figure S1A). Hence, activity patterns in early visual cortex contained sufficient information from auditory stimulation to allow the content-specific discrimination of natural sounds. As expected, the classifier performed very well in auditory cortex (positive control), but not in an unrelated cortical area (motor cortex; negative control). At different eccentricities, classification was successful in peripheral and far peripheral areas, particularly in V1 and V2, but not in foveal regions, consistent with structural and functional evidence for auditory influences on early visual cortex (e.g.,).

2Sounds could have induced crossmodal top-down expectations or mental imagery, which can be conceptualized as one form of nonretinal input to early visual cortex. In experiment 2, we investigated whether sounds could be decoded in early visual cortex even when they were merely imagined and whether feedback information from real and imagined sounds elicited similar activity patterns. Here, runs with natural sound stimulation were interleaved with runs in which subjects solely imagined the sounds upon hearing the word cues “forest,” “traffic,” or “people” (Figure 2D). Subjects were instructed to engage in mental imagery of the sounds and a corresponding natural scene. Successful discrimination of imagined sounds was observed in both foveal and peripheral areas of early visual cortex (but not far periphery), in V1 and auditory cortex (Figures 2E and 2F; classification of real sounds replicated the results of experiment 1, cf. Figure S1B). Therefore, even in the absence of both visual and auditory stimulation, the contents of mental imagery could be decoded from both V1 and auditory cortex.

3Furthermore, we performed a cross-classification analysis between auditory perception and imagery, i.e., we trained the classifier on runs with sound stimulation and tested on runs with pure imagery and vice versa. Cross-classification succeeded in V1 and V2 (Figure 2G). This demonstrates that both sounds and imagery cues induced similar activity patterns in early visual cortex and that feedback information is coded consistently across imagery and auditory perception. In auditory cortex, cross-classification did not succeed, indicating that activity patterns induced by feedforward auditory stimulation are coded differently than those induced by feedback through auditory imagery.

4In experiments 3 and 4, we explored the robustness of cortical feedback to interference with orthogonal engagement of attention, working memory, and visuospatial processing. During natural sound stimulation, subjects performed an orthogonal task that was either an auditory working memory task (experiment 3) or a visuospatial imagery task (experiment 4). Again, both experiments omitted any visual stimulation. In experiment 3, subjects retained a list of five words (animals or everyday objects) in memory during the natural sound stimulation and subsequently matched it with a second word list in scrambled order (Figure 3A). Activity patterns during natural sound stimulation were again successfully decoded from early visual cortex, mainly in peripheral regions and consistently in V2 (Figures 3B and 3C). This demonstrates that simultaneous retention of orthogonal contents in working memory did not strongly affect classification.

5In experiment 4, subjects engaged in an imaginary cube-assembly task. Here, subjects mentally constructed an imaginary figure according to five assembly instructions and rotated the imaginary figure 90° clockwise while hearing the natural sound. Subsequently, they matched the rotated figure held in memory with a second list of instructions. Although the classifier failed to discriminate the three natural sounds in most of early visual cortex, residual above-chance classification remained in the far periphery of V2 (Figures 3E and 3F) despite the orthogonal engagement of attentionally demanding active visuospatial processing.

6We performed a whole-brain searchlight analysis to identify other areas that contain information from real and imagined sound content and may mediate information feedback to early visual cortex. Unsurprisingly, sounds could be decoded in a large part of bilateral superior temporal sulcus mostly belonging to auditory cortex (Figure 4). In experiments 1 and 2, real and imagined sounds could be decoded in parts of the precuneus and in posterior superior temporal sulcus (pSTS) (see overlapping regions in Figure 4). Sounds and, to a lesser extent, imagined sounds were successfully classified in a network of frontal regions, including superior and middle frontal sulci.

7Given previous controversial evidence of whether mental imagery elicits positive activity in early visual cortex, we performed a univariate generalized linear model analysis to see whether our decoding results were based on positive or negative activation profiles. Even at very liberal thresholds (p < 0.05 uncorrected; Figure S3), listening to sounds in the absence of visual stimulation elicited no positive activation in early visual areas but instead elicited a weak deactivation, consistent with previous findings (e.g.,) and in contrast to classical findings for visual mental imagery. Imagery (experiment 2) elicited no positive activity but exhibited weak deactivations in both early visual and auditory cortices. In experiments 3 and 4, the secondary tasks activated early visual areas consistent with an engagement of object attention.

8In experiment 5, we were interested in the specificity of the information that is fed back to early visual cortex. We hypothesized two possibilities: (1) sounds trigger a unique picture-like representation that reinstates the same activity patterns in early visual cortex as a real image does and thus allows successful decoding, and (2) higher-level abstract or categorical information is fed down to early visual cortex causing the differential activity patterns. The purpose of such information transfer could be to provide categorical expectations as proposed by models of predictive coding (e.g.,). We presented subjects with three different sound exemplars (6 s each) for each of the categories “human” and “inanimate.” The crucial experimental manipulation here was that two sound exemplars in each category could induce similar pictorial representations (different snapshots of a similar environment: “people 1” and “people 2” and “traffic 1” and “traffic 2”), whereas the third could induce a very different image due to a different feature (“playing kids” and “starting airplane”).

9Classification of exemplars of the “human” versus the “inanimate” category was successful in several early visual areas for eight out of nine exemplar combinations (Figure 3G; Table S1), replicating in part the results of experiment 1 and demonstrating decoding of sounds of the categories “human” and “inanimate” with different sound exemplars and shorter stimulus presentation times.

10Crucially, cross-classification succeeded in V2 and V3 in two out of three combinations, i.e., training the classifier for the pair “traffic 1” versus “people 1” lead to successful classification of “traffic 2” versus “people 2,” and training the classifier for the pair “traffic 2” versus “people 2” lead to successful classification of “airplane” versus “kids” (Figure 3H; Table S1). That is, the information contained in these activity patterns is generalizable across different sound exemplars within a category, demonstrating that sounds trigger shared categorical information transfer to early visual cortex rather than a fine-grained pictorial representation.

11Our series of five fMRI experiments provides converging evidence for consistent abstract information feedback from nonretinal sources to human early visual cortex.

12We show that category-specific information from audition and imagery can be decoded from early visual cortex activity. The fact that our classifier could predict which sound was heard or imagined means that our results go beyond previous studies demonstrating an overall activity increase in early visual cortex in response to auditory stimulation or visual mental imagery. Our study shows that sound stimulation and associated imagery generate shared and meaningful information feedback to early visual cortex, carrying abstract and possibly semantic information.

13Previous studies focused on the decoding of visual mental imagery and the consistency of activity patterns across visual mental imagery and visual perception. Mostly, decoding of object categories worked in higher visual areas such as lateral occipital complex or ventral temporal cortex and to some extent in extrastriate cortex, but not in V1. Our study is the first to show that inducing multisensory imagery allows decoding of complex mental imagery content in V1. Furthermore, whereas previous studies reported successful cross-classification between imagery and active visual perception, our cross-classification analysis demonstrates a consistency of activity patterns in early visual areas across imagery and auditory perception. This is converging evidence that nonretinal feedback is consistent with respect to its semantic content, no matter its exact source.

14Our results also show that this feedback is robust to mild interference with low attentional and working memory load (experiment 3) and to some extent even to interference with a visuospatially and attentionally highly demanding task (experiment 4).

15The whole-brain searchlight analysis identified higher-level multisensory brain areas such as pSTS and precuneus possibly mediating the information feedback from sounds and imagery to early visual areas. The precuneus has been identified as an area responding to both visual and auditory stimuli and possibly serving as an audiovisual convergence area. pSTS is implicated in audiovisual integration and has been shown to feed down information to primary visual and auditory cortices. In the context of our findings, we suggest that the content-specific information from sounds, when they are heard and/or imagined, is relayed from auditory cortex to early visual cortex via pSTS and precuneus, eliciting differential activity patterns in both of these regions. Apart from the route via multisensory areas, there is evidence for multisensory integration on the subcortical level and for direct anatomical connections between early auditory and early visual areas, mostly reaching peripheral regions, consistent with both our eccentricity and searchlight results. Also, hippocampal projections to peripheral early visual regions have been demonstrated in the context of boundary extension for scene processing. However, whether these pathways play a causal role in inducing differential activity patterns remains to be investigated.

16The successful classification in experiments 1 and 2 was driven by differential patterns of deactivation rather than activation, and, thus, our results are unlikely to be caused by the same neural mechanisms as those suggested in earlier studies on visual mental imagery. This also means that our results were not caused by an unspecific attention effect or a simple reactivation of early visual cortex due to pictorial visual mental imagery. The univariate activity profile also showed that classification was not driven by one sound eliciting more attention-related activity than another sound (Figure S3).

17The results of experiment 5 suggest that the information that is fed down to early visual cortex is not only content specific but also category specific, i.e., related to the information shared by sound exemplars of the same category. This suggests that information feedback is unlikely to be caused by an exact pictorial representation but instead contains abstract and possibly semantic information. The findings of experiment 5 furthermore demonstrate that the successful decoding in experiment 1 was not specific to the first sound exemplars we used and could not be caused by differential low-level acoustic features of the sounds (e.g., frequency distribution).

18Note that despite relatively low classification accuracies, our series of experiments replicated the successful decoding of sounds in early visual areas several times, demonstrating proof of principle and the robustness of our results across different subject and stimulus samples.

19Previous fMRI studies using MVPA have provided evidence for nonfeedforward input to early visual cortex. For example, activity patterns in nonstimulated parts of early visual cortex contain content-specific information from the surrounding visual context, from objects presented in the periphery, and from visual stimuli solely held in working memory rather than being actively perceived. Moreover, higher visual areas project back to V1 the associated color of grayscale objects or the predicted motion path of an apparent motion illusion. Our results provide further novel evidence that early visual cortex receives category-specific feedback from auditory, multisensory, memory, or imagery areas in the absence of any actual visual stimulation. Furthermore, many studies of top-down or multisensory influences on sensory regions, such as the decoding of sound-implying visual images in auditory cortex, the decoding of touch-implying visual images in somatosensory cortex, the recruitment of early visual cortex in blindfolded subjects by touch, or the decoding of memory traces in early visual cortex, could have been caused or accompanied by a form of mental imagery. Our study has explored the role of mental imagery in depth and has demonstrated that, in terms of reactivation of early visual cortex by a pictorial representation similar to actual visual perception, a simplistic mental imagery account falls short of explaining our results entirely.

20Why should category-specific information be fed down all the way to early visual areas? One interpretation is that the brain provides priors fitting to the best prediction, and these priors can be transmitted between different sensory modalities. Within the framework of predictive coding, early sensory areas are “prepared” with a predictive model for the external incoming information through cortical feedback from higher cognitive areas, the hippocampus, and other sensory modalities. In the present case, early visual cortex may anticipate certain visual information due to real or imagined auditory information. That is, auditory stimulation or imagery triggers a predictive model reaching early visual areas via feedback connections from higher multisensory or imagery areas and evoking content-specific activity patterns. Our results demonstrate that the information arriving in early visual cortex is categorical and independent of its exact source. In fact, previous accounts suggested that prediction and mental imagery may involve overlapping brain mechanisms, and mental imagery might have evolved from predictive brain mechanisms. What distinguishes both from each other remains an interesting question to be investigated, both experimentally and theoretically. Omitting feedforward stimulation is a promising step in studying nonvisual input to early visual cortex; however, without feedforward stimulation, it is difficult to study the functional role of this influence in actual visual perception. Audiovisual priming studies with natural stimuli indicate a facilitatory role for visual perception.

21Our results demonstrate that abstract information from nonretinal input, induced by both complex sound stimulation and mental imagery, can be translated to the coding space of early visual cortex. The purpose of such abstract information feedback might be to provide early visual cortex with a categorical prediction for the incoming visual input.