Measuring speaker–listener neural coupling with functional near infrared spectroscopy

1Verbal communication involves the relaying of information between individuals through the use of sound patterns within a structure of language. For decades, neuroimaging technologies have been applied to study the neural mechanisms underlying the production and comprehension of language. Multiple brain areas have been identified to be involved with verbal communication using Positron Emission Tomography (PET) and functional Magnetic Resonance Imaging (fMRI), whereas the timing of auditory processing was studied with the aid of Electroencephalogram (EEG) and Magnetoencephalogram (MEG). Although important findings have been discovered using these technologies, there are two limitations in traditional neurolinguistic studies. First, these studies are mostly concerned with the cognitive process of either speech production or speech comprehension and confine the analysis to be within individual brains. Verbal communication, however, is an interactive process between speaker and listener. As pointed out by Hasson and others, a complete understanding of the cognitive processes involved cannot be achieved without examining and understanding the interaction of neural activity among individuals. Second, cognitive functions are traditionally studied in a controlled laboratory environment. While this practice helps to isolate various factors (e.g. syntactical transformations or the representation of isolated lexical items), the ecological validity of the findings is not clear until tested in a real-life context. In addition, many studies confined the auditory stimuli to short lengths, often using isolated words or sentences for experimental control. As a result, questions regarding the brain’s ability to accumulate information over longer time scales cannot be effectively investigated.

2With the recent advances in neuroimaging systems and methodology, researchers can now address which brain processes are involved in social interaction. Stephens et al. investigated the alignment (correlation) of neural activity between speaker and listener during natural verbal communication using fMRI. In the study, brain activity was recorded when a speaker was telling a real-life story and later when listeners were listening to the audio recording of the story. Listeners’ brain activity was found to be coupled with the speaker’s brain activity with a delay, although for certain brain areas, listeners were ahead of the speaker in time, possibly due to a predictive anticipatory effect. Remarkably, higher coupling was found to be associated with better understanding of the story. This neural coupling between speaker and listener was further supported by a recent EEG study in which the coordination between the brain activity of speakers and listeners was investigated with canonical correlation analysis. Additional findings in the same EEG study suggest that this speaker-listener neural coupling might not be restricted to homologous brain areas. In another study using fMRI, Lerner et al. recorded the Blood Oxygenation Level Dependent (BOLD) response of participants listening to a real-life story scrambled at the time scales of words, sentences and paragraphs. Inter-subject correlation analyses were employed to estimate the reliability of neural responses across subjects, and striking topography differences in brain activation were found at the different time scales.

3Although novel findings have been discovered using fMRI and EEG to address the aforementioned challenges, certain limitations of the two neuroimaging technologies have hindered the investigation of neural coupling during natural verbal communication. fMRI, for example, requires subjects to lie down motionlessly in a noisy scanning environment. Simultaneous scanning of multiple individuals engaged in a face-to-face communication is impractical for fMRI based setups. EEG, on the other hand, is able to provide a more naturalistic environment. However, EEG is susceptible to muscle induced artifacts during vocalization, and is therefore less suitable for studying speaker-listener interactions. Furthermore, the localization of sources from the EEG signal requires higher-density recordings and additional computation to solve the inverse problem.

4In this study, we propose using functional near-infrared spectroscopy (fNIRS) to investigate speaker-listener coupling as an effective complement to the existing studies. fNIRS is an optical brain imaging technology for monitoring the concentration changes of oxygenated hemoglobin (ΔHbO) and deoxygenated hemoglobin (ΔHbR) in the cortex. By utilizing portable and wearable sensors, fNIRS provides an imaging solution with high ecological validity for studying cortical hemodynamic changes in real-life contexts. Furthermore, fNIRS has been used in natural environments, including on mobile individuals outdoors, consistent with the neuroergonomic and mobile brain/body imaging approaches. fNIRS has been adopted to study brain-to-brain coupling during a cooperation-competition game and a finger-tapping imitation task. fNIRS has demonstrated usefulness for studying single brain activations during social interactions in a natural setting with traditional block design. At present, studying brain-to-brain coupling during natural verbal communication using fNIRS has not been demonstrated.

5The main objective of this study is to evaluate speaker-listener brain-to-brain coupling in natural everyday settings. To achieve this objective, we designed an fNIRS experiment to replicate the speaker-listener neural coupling results from a previous fMRI study. A native English speaker and two native Turkish speakers told an unrehearsed real-life story in their native language. An additional real-life English story E2 (“Pie-man”, recorded at “The Moth”, a live storytelling event in New York City) used in recent fMRI studies of natural verbal communication, was also used here. The resulting two English stories (E1 and E2) and two Turkish stories (T1 and T2, the control conditions) were played to listeners who only understand English. We hypothesized that: (1) the listeners’ brain activity will demonstrate inter-subject correlation only when listening to a story they understand; (2) the English speaker’s brain activity during production of E1 will be coupled with the activity of the listeners during comprehension. We have targeted both prefrontal and parietal cortices, as these include cognitive and higher order extralinguistic areas that are known to be involved in social information processing crucial for successful communication including, among others, the capacity to discern the beliefs, desires, and goals of others.

6A second objective of this study is to compare the fNIRS recorded in this study with the fMRI BOLD response recorded in the previous fMRI study. It has been known that fNIRS and fMRI BOLD signals, both of which are based on the neurovascular coupling phenomenon, are correlated in various cognitive tasks. However, previous studies primarily adopted block designs with simple stimulation to investigate the mean activity induced by trigger-averaging a condition over time. The relationship between fNIRS and BOLD during natural verbal communication is yet to be shown. To this end, we hypothesized that fNIRS biomarkers (ΔHbO and ΔHbR) of our listeners are correlated with the fMRI BOLD responses of the previously tested listeners during the comprehension of only the same English story E2 (which all of these listeners heard) and not for the other English story E1 and Turkish stories T1 and T2 (which the previously tested fMRI participants did not hear). The convergence of fNIRS and fMRI for comprehension of the same story serves as a further validation that fNIRS can be used to investigate brain-to-brain coupling during natural verbal communication.

7Natural communication unfolds through collective participation of speaker and listener: speakers construct grammatical sentences based on thoughts, convert these to motor plans, and execute the plans to produce vocal sounds; listeners analyze the sounds, build phonemes into words and sentences, and ultimately decode sound patterns into meaning. In our study, we observed significant speaker–listener temporal coupling only during successful verbal communication. When communication was blocked (i.e., during listening to a foreign, incomprehensible language), the synchronization was lost. As expected, this synchronization was present in a temporally shifted time course of the speaker’s brain activity relative to the moment of vocalization. Hence, listeners’ brain activity mirrors the speaker’s brain activity with a delay. These lagged responses suggest that on average, the speaker’s production-based processes precede and likely induce the mirrored activity observed in the listeners’ brains during comprehension. Furthermore, synchronization was also present between listeners with zero shift, representing temporally aligned listener-to-listener inter-subject coupling. Again, this coupling was only present during successful communication and disappeared when communication was blocked. Finally, we found strong consistencies between the evidence from fNIRS and fMRI for the presence of coupling during successful communication.

8For each story, significantly coupled optodes were identified using multilevel general linear model (GLM) and the results are shown in Fig. 1. As expected, significant results were found only for the English conditions E1 and E2 (false discovery rate [FDR] q < 0.01), indicating that neural coupling only emerges during successful communication (i.e. when subjects understand the story content). ΔHbO shows a much stronger coupling effect than ΔHbR. A contrast for listening comprehension using successful communication stories (E1 and E2) minus control conditions (T1 and T2) was further included in Fig. 1 as E1 + E2 −T1 − T2. Consistent with previous fMRI studies, superior frontal gyrus, inferior parietal and angular gyrus were robustly correlated across listeners during speech comprehension. Although the function of these regions is far from clear, they have been associated with various linguistic functions—in particular, semantic processing and comprehension.

9The neural coupling between speaker and listener may not be restricted to homologous brain areas. Previous studies also showed that the neural responses of listeners can lag behind or precede those of the speaker, facilitating comprehension and anticipation, respectively. To investigate these effects, multilevel GLM has been adopted to evaluate the coupling between all permutations of (speaker optode, listener optode) pairs with the speaker’s time course shifted with respect to those of the listeners from −20 s to 20 s in 0.5 s increments, where a positive shift represents the speaker preceding (listener lagging), and the results are shown in Fig. 2. For the English story E1, the listeners’ fNIRS signals were found to be significantly coupled with the speaker’s signal with a 5–7 s time delay, and the number of significantly coupled optodes peaked at 5 s (Fig. 2a). As expected, no temporal asymmetry has been found for the listener-listener case and alignment is coupled to the incoming auditory input (i.e. lag 0, moment of vocalization) (Fig. 2b). The speaker-listener lagged correlation replicated the speaker-listener lagged correlation observed with fMRI. Significant couplings can mainly be found between prefrontal of speaker and parietal of listeners in the medial prefrontal and left parietal areas for ΔHbO and no results was significant at FDR q < 0.01 level for ΔHbR (Fig. 2c). The relationships between significantly coupled anatomical locations in speaker and listener brains are further listed in Figure S1. These areas include many of the sensory, classic linguistic-related, and extralinguistic-related brain areas, demonstrating that areas involved in speech comprehension (listener–listener coupling) are also aligned during communication (speaker–listener coupling). No significant speaker-listener coupling was found for either of the Turkish stories (stories T1 and T2). Using identical analysis methods and statistical thresholds, we found no significant coupling between the speaker and the listeners or among the listeners during these control conditions (see Fig. 1, T1 and T2).

10To further investigate the temporal asymmetry of coupling, the average t-statistics for all significantly coupled speaker-listener optode-pairs were assessed across time shifts between the speaker and listener time series, as shown in Fig. 3 (red curve). The peak of the curve is centered at 5 seconds, which shows that, on average, listeners’ time courses lagged behind the speaker’s. In comparison, the time courses of the listeners were synchronized (with each other) at 0 sec (Fig. 3, blue curve).

11As a verification of the fNIRS approach, we reanalyzed an fMRI dataset of 17 subjects listening to story E2 (the “Pie-man” story), which was recorded and used in a previous study. To compare with the fNIRS results, we considered only voxels from the outer layer of the cortex in the neighboring regions of prefrontal and parietal sites. The coupling results estimated using the multilevel GLM model are shown in Fig. 4. Of the 994 investigated voxels, 551 showed significant listener-listener coupling (FDR q < 0.01. This result replicates published result and demonstrate a nice convergence across fNIRS and fMRI methods.

12Previous studies have shown that fNIRS and fMRI signals are highly correlated across multiple cognitive tasks. In our study, two groups of subjects, the brain activity of one group measured with fNIRS and the other with fMRI, were engaged in the same task of listening to the E2 story (“Pie-man”). We hypothesize that the BOLD and fNIRS signals share common information even though they were measured from different subjects and in different recording environments. To directly compare the signals across fMRI and fNIRS, we estimated correlations between spatially overlapping voxel-optode pairs while subjects listened to the exact same story. Widespread significant correlations can be found between BOLD and ΔHbO only when the participants were listening to the same story (i.e. E2) as shown in Fig. 5a. For control stories T1, T2 (Turkish) and E1 (English), we observed no coupling effect.

13During social interaction, the brains of individuals become coupled as those individuals send and receive signals (light, sound, etc.) through the environment, analogous to a wireless communication system. This brain-to-brain coupling relies on the stimulus-to-brain coupling which reflects the brain’s ability to be coupled with the physical world in order to represent it, veridically and dynamically. In this study, we identified the brain-to-brain coupling between a speaker telling an unrehearsed real-life story and a group of listeners listening to the story, with the aid of fNIRS. We have demonstrated for the first time that it is feasible to study neural coupling during natural verbal communication with fNIRS. While there is a growing literature using fMRI and EEG to study brain-to-brain coupling during social interaction, the application of fNIRS in the field is still rare. Cui et al. in 2012 first adopted fNIRS to investigate neural coupling between pairs of subjects playing a simple cooperation and competition game. In the game, participants were asked to press a response key after a ‘go’ signal, either in synchrony with (cooperation mode) or faster than (competition mode) their partner. An increase in neural correlation between the members of a pair was found only during cooperation and was associated with better cooperation performance. Holper et al. in 2012 investigated neural coupling between a model and an imitator during a finger tapping task. A stronger increase in neural coupling was found during the imitation condition compared with a control condition in which the imitator no longer needed to follow the model’s tapping pace. These two studies, however, involved only simple stimuli and contexts.

14Our results demonstrated that: (1) the brain activation recorded by fNIRS was coupled between a speaker telling a real-life story and listeners listening to the story (speaker-listener coupling); (2) on average, the listeners’ brain activity lagged behind that of the speaker (5-s delay); (3) the brain activity evoked by the same story was reliable across the listeners (listener-to-listener coupling); and (4) coupling was not present when listeners heard stories in a language incomprehensible to them. These findings are consistent with previous work using fMRI and EEG and demonstrate that fNIRS can be used to study brain-to-brain coupling during social interaction in a natural communicative context.

15During brain-to-brain coupling, activity in areas of prefrontal and parietal cortex previously reported to be involved in sentence comprehension were robustly correlated across subjects, as revealed in the inter-subject correlation analysis. As these are task-related (active listening) activation periods (not resting, etc.), the correlations reflect modulation of these regions by the time-varying content of the narratives, and comprise linguistic, conceptual and affective processing. As expected, and in agreement with previous work, the activity among listeners is time locked to the moment of vocalization. In contrast, the activity in the listener’s brain lagged behind the activity in the speaker’s brain. Though the sluggish hemodynamic response measures of fMRI and fNIRS have lower temporal resolution and could mask the exact temporal speaker–listener coupling, the delay (5-sec) we have identified in this study is consistent with previous work on the multiple timescales of linguistic processing. In particular, analyses integrating single-unit, ECoG, and fMRI data have uncovered a hierarchy of timescales over which natural communication unfolds in the brain, ranging from hundreds of milliseconds (single words) to a few seconds (sentences) to tens of seconds (paragraphs). These previous findings indicate that each brain area along this hierarchical pathway accumulates and integrates information at a preferred timescale (e.g., early auditory areas integrate phonemes to detect words, while later areas integrate words to form a sentence, etc.) and transmits the chunked information upstream to the next processing level. The relatively long (5-sec) lag between speaker and listener seems to match the processing timescale of natural discourse, which unfolds at the sentence level. In agreement with such an observation, the speaker-listener neural coupling emerges in areas with relatively long processing timescales.

16Another interesting observation is that significant inter-subject correlations were found primarily between prefrontal areas in the speaker and parietal areas in the listeners. This result supports a previous EEG study in which the coupling between speaker and listener was found to be mainly between different channels. In that study, listeners watched the video playback of speakers who were telling either a fairytale or the plot of their favorite movie or book. A canonical correlation analysis between the EEG of the speakers and listeners showed that coupling was mainly limited to non-homologous channels although source localization was not applied. To our best knowledge, our study presents the first fNIRS-based evidence that the brain-to-brain coupling between speaker and listener was mainly between non-homologous brain areas.

17In the current study, we further compared the fMRI and fNIRS signal time courses when two groups of subjects were listening to the same audio recording of a real-life story. The neural activation of one group was recorded with fNIRS. The neural activation of the other group (the fMRI group) was recorded with fMRI in a previous study. We first analyzed the two datasets separately with inter-subject multilevel GLM and found similar patterns of coupling between listeners in the two modalities. We then investigated the correlation between fMRI and fNIRS signals and found that ΔHbO and BOLD were significantly correlated, despite the fact that they were collected from different subjects in different recording environments, and with different techniques (fMRI vs. fNIRS). Furthermore, the fMRI voxels that were significantly correlated with fNIRS optodes were not randomly distributed but came from brain areas usually considered to be related to listening comprehension. When the fNIRS and fMRI signals corresponding to different stories were compared for control purposes, no significant correlation was found, as expected. Significant BOLD-ΔHbR correlations were found but to a much lesser extent compared to ΔHbO (Fig. 5a). One possible explanation is the superior reliability of ΔHbO across the listeners compared to ΔHbR as shown in Fig. 1.

18For many years, researchers have been interested in comparing the fNIRS and fMRI responses during cognitive tasks. Strangman et al. simultaneously recorded fNIRS and fMRI in a finger flexion/extension task. Although the authors expected the ΔHbR-BOLD correlation to be strongest due to the causal relation between the ΔHbR and BOLD signals, the results suggested stronger BOLD-ΔHbO correlations. The authors suspected this might be due to a higher signal-to-noise ratio (SNR) of ΔHbO in response to the task. Cui et al. simultaneously recorded fNIRS and fMRI for the same group of subjects during 4 tasks: finger tapping, go/no-go, judgment of line orientation and visuospatial n-back. They found that both ΔHbR and ΔHbO were correlated with BOLD, despite differences in SNR, and the type of task did not significantly affect the correlations. ΔHbO-BOLD correlations were found to be slightly but significantly higher than ΔHbR-BOLD. Noah et al. compared fMRI and fNIRS measurements in a naturalistic task in which participants played the video game Dance Dance Revolution and rested alternately for 30 second blocks, and the results suggested a high ΔHbO-BOLD correlation within the same measurement areas. All of the studies above compared the mean triggered average activity induced by averaging a condition over time. However, Ben-Yakov et al. demonstrated the shortcoming of the triggered averaging method for detecting an event-specific responses which are locked to the structure of each particular exemplar. In our study, each event in the story is unique and singular and cannot be averaged with the responses evoked by other events in the story. Our findings provide evidence that the response dynamics to a sequence of events in the story are robust and reliable, and can be detected both with fMRI and fNIRS, by measuring the reliability (correlation) of responses to the story within and between the two methods in real-life complex settings that unfolds over several minutes.

19The current study, however, is only the first step toward studying brain coupling during natural communication using fNIRS. While fNIRS has obvious disadvantages relative to fMRI, which include coarser spatial resolution and an inability to measure signal beneath the cortical surface, it also has some advantages over fMRI. The advantages of fNIRS over fMRI include: (1) lower cost; (2) easier setup; (3) higher temporal resolution, (4) greater ecological validity, which can allow face-to-face communication (in fMRI setups subjects cannot see each other); (5) greater ease of connecting two systems simultaneously to test bi-directional dialogue-based communication; and (6) ease of use in real-life clinical communicative contexts. In recent studies, fNIRS has been used in extreme settings such as in aerospace applications and with mobile participants walking outdoors. In the context of the current study, the true advantages of fNIRS can be exploited when the neural activation of two or more subjects is studied during face-to-face conversations in a natural context such as a classroom.

20Despite these promising results, our study was limited in certain aspects. First, for the fNIRS-fMRI comparison, the spatial resolution and coverage of our fNIRS system were limited, so fMRI-fNIRS correlations were estimated between all possible voxel-optode pairs within large cortical areas or the whole brain. Second, our data from fNIRS and fMRI were not recorded simultaneously and involved different participants, so any possible between-subject differences should be taken into account when interpreting the results. Future studies, preferably with concurrent fNIRS and fMRI, can validate our findings and deepen our understanding of the fNIRS-fMRI relationship for complex natural stimuli. Potential future uses of this approach would include everyday settings such as classrooms, where we could investigate communication between a teacher and students, or in business meetings, across a speaker and attendees. However, due to the nature of the hemodynamic response that fNIRS measures, rapid communication may be difficult to analyze at short timescales (e.g., at the word-level). Future studies can investigate the temporal and spatial limits and requirements of fNIRS-based speaker-listener coupling.

21In summary, our results showed that: (1) A speaker’s and listeners’ brain activity as measured with fNIRS were coupled only when the listeners understood the story; (2) listeners’ brain activity mirrored the speaker’s brain activity with a delay, (3) only during listening to the same real-life story, common brain activation patterns were evoked across listeners that were independent of the imaging technology (fNIRS or fMRI) and recording environment (quiet or noisy; sitting down or lying down); and (4) fNIRS and fMRI signals were correlated during the comprehension of the same real-life story. These results support fNIRS as a viable future tool to study brain-to-brain coupling during social interaction, in real-life and clinical settings such as a classroom, a business meeting, a political rally, or a doctor’s office.

22Three speakers (one male native English speaker, two male native Turkish speakers) and 15 native English listeners (8 females) volunteered to participate in the study and were included in the analysis. An additional six subjects participated in the study but were excluded from analysis due to technical issues during recording or the excessive motion artifact (detected both visually and using an automatic algorithm) presented in the large sensor array data that covers both prefrontal and parietal cortices. Subjects were all right-handed (mean LQ = 74.5, SD = 24.1) based on Edinburgh Handedness Inventory and ages 18–35 years. All subjects had normal or corrected-to-normal vision. Participants did not have any history of neurological/mental disorder and were not taking any medication known to affect alertness or brain activity. None of the listeners understood Turkish. The protocol used in the study was reviewed and approved by the Institutional Review Board (IRB) of the Drexel University (DU). The methods were carried out in accordance with approved guidelines and participants gave written informed consent approved by the IRB of DU.

23All participants were seated comfortably in front of a computer screen throughout the experiment. fNIRS data of the three speakers were recorded while they told an unrehearsed real-life story in their native language (either English or Turkish). Audio of the stories was recorded using a microphone. The resulting one English story (E1) and two Turkish stories (T1 and T2) were played to the listeners later. An additional real-life English story E2 (“Pie-man”, recorded at “The Moth”, a live storytelling event in New York City) used in several recent fMRI studies of natural verbal communication, was also played to the listeners.

24fNIRS data were recorded from the listeners throughout the audio playbacks. The playback sequence always began with E2 (English story, Pie-man), and order of the remaining stories (E1, T1, and T2) was counterbalanced across subjects. Before each story playback, short samples of scrambled audio were played to the subjects so they could adjust the volume of the headphones they were wearing. Before the start and after the end of the audio story playback, there was a 15-s fixation period for stabilizing the signal. Immediately after each playback, subjects were asked to write a detailed report of the story they just heard to verify if they understood the story. Figure 6, below, shows the timeline of a story session.

25Two continuous wave optical brain imaging devices were used simultaneously on each participant to record brain activity from prefrontal cortex (PFC) and parietal cortex (PL) using 40 measurement locations (optodes). Prefrontal and parietal regions were selected based on the significant areas found in the previous fMRI-based speaker-listener neural coupling study by Stephens, et al.. Anterior prefrontal cortex was recorded by a 16-optode continuous wave fNIRS system (fNIR Imager Model 1100; fNIR Devices, LLC) first described by Chance et al. and developed in our lab at Drexel University. The sensor was positioned based on the anatomical landmarks as described before in Ayaz et al.. Briefly, the center of the sensor was aligned to the midline and the bottom of the sensor was touching the participant’s eyebrow so that the center point of the sensor was approximately at Fpz according to the 10–20 international system (see Fig. 7). The sensor has a source-detector distance of 2.5 cm and the sampling rate was 2 Hz. Parietal cortex was recorded using a 24-optode continuous wave Hitachi fNIRS system (ETG 4000; Hitachi Medical Systems). Two “3 × 3” measurement patches were attached to a cap that was customized for the measurement of the parietal cortex. For each subject, the center of the two patches was placed at Pz, which was located using a measuring tape. Sensors from each patch measured the fNIRS signal of one hemisphere from 12 channels. The sensor has a source-detector distance of 3 cm and the sampling rate was 10 Hz. Figure 7 shows the complete sensor setup and optode configuration.

26Time synchronization markers (triggers) were sent from the stimulus presentation computer to both fNIRS acquisition computers for registration of audio playback start and end times on both fNIRS devices and for temporal aligning recorded data for all subjects.

27The approximate projection of the channel locations onto the cortical surface in MNI space was estimated using a virtual spatial registration approach. In this approach, the sensor patches are virtually placed on an ideal scalp and the projected Montreal Neurological Institute (MNI) coordinates on the cortical surface and the standard deviation of displacement were estimated from the magnetic resonance (MR) images of 17 individuals that were obtained from a publicly available dataset. The results are shown in Fig. 8. The optodes covered regions in the frontopolar area, orbitofrontal area, dorsolateral prefrontal cortex, primary somatosensory cortex, somatosensory association cortex, supramarginal gyrus and angular gyrus. Detailed lists of anatomical locations and Brodmann areas corresponding to each optode are included in Table S1 and Table S2, respectively.

28fNIRS raw light intensity signals were converted to changes in oxygenated hemoglobin (ΔHbO) and deoxygenated hemoglobin (ΔHbR) concentrations using the modified Beer-Lambert law. The raw signal and hemoglobin concentration changes were inspected both visually and also using the automated SMAR algorithm, which uses a coefficient-of-variation based approach to assess signal quality and reject problematic channels with bad contact or saturated raw light intensity. Two optodes, 1 and 15, were over the hairline for most participants and hence were rejected from the study. A total of 11.7% were rejected by visual inspection and SMAR. Next, the ΔHbO and ΔHbR time series for each optode and participant were band-pass filtered (0.01–0.5 Hz) and the recordings from parietal sites were down-sampled to 2 Hz to match the sampling rate of prefrontal sites. In the fNIRS literature, various filtering settings have been adopted to reduce physiological artifacts. The high cutoff of a band-pass filter, for example, ranges from 0.1–1 Hz. We chose 0.01–0.5 Hz cutoffs to reduce slow signal drift and physiological artifacts from cardiac activities but maximally preserve activities related to listening comprehension. We considered only the period from 15 to 399 seconds, with respect to story start, in the signal time courses from each audio story. The first 15 s were rejected to account for subjects’ initial period of adjustment to the listening comprehension task, and 399 s is the duration of the shortest story. Prior to subsequent analysis, the signal time courses were standardized optode-wise using a Z-score transform.

29We first evaluated the reliability of the correlation between listeners’ brain activity using an inter-subject multilevel GLM similar to the one adopted by Stephens et al. for each of the four conditions: E1, E2, T1 and T2. We expected neural coupling between listeners to emerge only for the English story conditions E1 and E2, as none of the subjects understood Turkish.

30At the individual subject level, a GLM with AR(1) (first-order autoregressive) error model was estimated using the average time course of the listeners as the independent variable and the time course of an individual listener as the dependent variable as follows:

31where is the time course of a channel from listener k, is the average time course of a channel from all the other listeners except for listener k, ρ is the autocorrelation coefficient and. AR(1) error model has been frequently adopted in event-related fNIRS analysis to model auto-correlated noise caused by low-frequency drift and physiological processes such as cardio-respiratory and blood pressure changes.

32At the group level, we tested the hypothesis using a one-tailed one-sample t-test evaluated on the slopes estimated at the individual level. We used the Benjamini–Hochberg procedure to control FDR among 80 statistical tests (2[ΔHbO/ΔHbR] × 40[optodes]) with.

33We evaluated the coupling between optode of speaker and optode of the listeners for all permutations of (. The multilevel GLM was applied as it was for listener-listener coupling except that the time course of optode of a speaker was used as independent variable and the time course of optode j of each individual listener was used as dependent variable.

34The speaker-listener coupling analysis was repeated by shifting the speaker’s time course with respect to those of the listeners from −20 s to 20 s in 0.5 s increments. At each time shift, FDR was controlled among 3200 statistical tests (2[ΔHbO/ΔHbR] × (40 × 40)[optode pairs]) with. As a comparison, listener-listener coupling was re-evaluated by shifting the average listener time series with respect to that of each individual listener.

35The fMRI dataset included 17 subjects listening to story E2 (the “Pie-man” story), which was recorded and used in a previous study. To compare with the fNIRS results, we considered only voxels from the outer layer of the cortex in the neighboring regions of prefrontal and parietal sites as shown in Fig. 8. The “neighboring regions” are defined as voxels within a radius of 2.7 standard deviations from the center of each optode projection. A total of 994 voxels were chosen in this manner. The voxel time courses were high-pass filtered at 0.01 Hz (for comparison with fNIRS signals) and trimmed to only include 15–399 seconds (with respect to story start), and the inter-subject multilevel GLM described in section 0 was employed for model analysis. FDR was controlled among the 994 voxels with.

36To estimate the correlation between BOLD and fNIRS signals, both signals were high-pass filtered at 0.01 Hz. The BOLD time courses were z-scored normalized for each voxel and then averaged across the 17 subjects. fNIRS time courses were down-sampled to Hz to match the fMRI sampling rate. The inter-subject multilevel GLM approach was then applied with the time course of optode of an fNIRS subject as independent variable and the time course of voxel averaged across fMRI subjects as dependent variable.

37The aforementioned procedure was applied first to estimate all possible correlations between channels and their corresponding voxels in the left prefrontal (8 optodes × 131 voxels), right prefrontal (8 optodes × 144 voxels), left parietal (12 optodes × 376 voxels) and right parietal (12 optodes × 343 voxels) areas. The voxels were selected as described in section 0. To correct for multiple comparisons, FDR was controlled with a threshold of 0.05.

38How to cite this article: Liu, Y. et al. Measuring speaker-listener neural coupling with functional near infrared spectroscopy. Sci. Rep. 7, 43293; doi: 10.1038/srep43293 (2017).

39Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.