Neural population coding: combining insights from microscopic and mass signals

1In complex animals, information about behaviorally important variables such as sensory signals or motor actions is carried by the activity of populations of neurons. Our understanding of neural information processing is founded on the conceptual assumption that, if two or more sensory stimuli can be discriminated, or two or more behavioral responses are different, their associated patterns of neural activity must be readily discriminable. Several key ingredients shape the capacity of a neural population code (see Glossary) to form such discriminable representations: the diversity of neural response properties, their spatial and temporal response profiles, the cross-neural correlations, and the state-dependence of cortical activity (Box 1).

2Recent progress in understanding the contribution and the interplay of each of these key ingredients arises from both technical and conceptual developments. Experimental methods now allow measuring and manipulating up to hundreds of neurons simultaneously in behaving animals and permit a direct link between population codes and behavior. Multi-scale studies combining invasive recordings with measurements of neuroimaging signals are becoming more frequent, allowing us to combine insights across methodologies. Finally, advances in computational methods for single-trial analysis of multivariate data allow us to fully exploit the avenues opened by high-density brain measurements.

3We review here how these developments facilitate the convergence of knowledge gained from invasive multi-neuron recordings, neuroimaging data, and mathematical modeling, and begin to reveal the organization and computations of neural population codes at multiple scales of organization. For simplicity, we mostly focus on the encoding of sensory variables, but the concepts are relevant for the generic encoding of any variable (for example, motor variables).

4The computational properties of population codes are usually quantified using measures of the information they carry about sensory stimuli. The most widely used quantifications of neural information are based upon Shannon information (quantifying the accuracy of discrimination among different stimuli in a set) or Fisher information (quantifying discrimination of small stimulus changes, or the accuracy of decoding an individual stimulus). Because most concepts reviewed here apply to both types of information, we only distinguish between them when necessary.

5How a neural population represents information is partly determined by the diverse selectivity of individual neurons. Individual neurons can carry information about sensory stimuli using a firing-rate code, for example by elevating their firing rate when presented with ‘preferred’ stimuli, and decreasing their rate when presented with other stimuli (Box 1). Nevertheless, even neighboring neurons may have heterogeneous stimulus selectivity. For example, different neurons may have different stimulus tuning curves exhibiting a preference for different stimulus features or exhibiting different stimulus tuning widths. The heterogeneity of stimulus tuning generally implies that individual neurons may carry complementary information to that provided by others. As a result, the ability of a heterogeneous population to discriminate among stimuli in a set should, under most conditions, increase with population size. If individual neurons in a population each prefer well-separated stimulus features (such that their tuning curves do not overlap), the range of stimulus features encoded by that population would increase with population size. If, instead, neurons in a population have diverse but partly overlapping tuning curves, the complementary information carried by different neurons would lead to a better discrimination of stimulus features in the regions where the tuning curves overlap.

6The scaling of information with population size depends on the structure of tuning preferences and of trial-to-trial response correlations, and its investigation can provide important insights. It can indicate how many neurons would be sufficient to achieve a desired level of sensory accuracy (assuming that the decoding mechanisms or later processing stages do not discard any of this information). Its extrapolation to infinite population size sets an upper bound on the information that can be achieved by a population with the considered response properties.

7The scaling of information with population size is often studied by averaging information over subpopulations that are randomly sampled from the recorded neurons. This approach typically reveals a steady increase of information with population size (Figure 1A), and led to the hypothesis that increasing population size allows the encoding of arbitrarily high amounts of information. However, recent work has shown that this steady increase may actually be an artifact resulting from random subsampling. This artifact arises because of an often neglected aspect of neural heterogeneity: only a small fraction of neurons in a given population carry significant sensory information in a specific context (Figure 1B). As a result of this heterogeneity in single-neuron properties, a small but highly informative subset of neurons is sufficient to carry essentially all the information present in the entire observed population (Figure 1A).

8The above results are part of an emerging picture that a small-dimensional subspace of the experimentally measured activity suffices to explain the population dynamics underling sensory processing and motor behavior. This picture is consistent with the observed sparseness of cortical activity (at any moment only a small fraction of neurons are active) and is compatible with studies showing that perception and actions can be driven by small groups of neurons. A sparse population code may be advantageous for metabolic efficiency and may facilitate dendritic computations requiring the separation of individual synaptic afferents.

9Studies on the auditory system suggest that those neurons contributing the most to a population code are those that respond sparsely over time, and do so with precisely timed temporal spike patterns. Importantly, the information carried by these precise spike patterns cannot be replaced by the information in the coarse-scale firing rates of other neurons in the population (Figure 1C), because information in spike patterns is complementary to that in firing rates and because the fraction of neurons carrying information by rates is limited. This implies that spike timing remains crucial even for population codes. The complementary nature of millisecond-scale spike patterns and firing rates has similarly been observed in the somatosensory and visual system. Thus, both the spatial and temporal dimensions are important for understanding neural population codes, raising questions as to how such precise timing or population sparseness can be assessed using neuroimaging.

10In higher association regions the heterogeneity of cortical neurons expresses itself as patterns of selectivity to multiple sensory and task-related variables, which can be mixed in complex, sometimes nonlinear ways. A pioneering study unveiled the advantages of nonlinear mixed selectivity. If responses were selective only to individual task or stimulus aspects, or to their linear combinations (linear mixed selectivity; Figure 1D), the dimensionality of the neural representation would be lower than the number of neurons in the population. In the example in Figure 1D–F, the neural representation of a set of stimuli of two linearly mixed neurons lies on a line. This implies that complex nonlinear operations are required for decoding the information content (Figure 1F). By contrast, a heterogeneous nonlinear mixed representation (Figure 1E) leads to a richer population representation that has higher dimensionality than its linear counterpart and that can be more easily decoded by downstream areas using linear combinations of neural activity (Figure 1F). In sum, heterogeneous nonlinear mixtures of selectivity increase the effective dimensionality of a population code and help to extract diverse information using simple linear decoding.

11This high-dimensional population representation by mixed selectivity may seem at odds with reports of small subpopulations effectively encoding primary sensory information. However, sparseness and high dimensionality of mixed nonlinear selectivity can coexist and may even combine optimally under appropriate conditions. In fact, when nonlinearly mixed neural representations are also sparse, they naturally achieve an optimal trade-off between the need to maximize the diversity of responses to different stimuli (increasing the stimulus discrimination properties of population responses), and the need to maintain high response reliability (i.e., achieving consistent responses for noisy variations of the input). Nonetheless, systematic studies comparing the sparseness of neural responses across brain regions will be necessary to clarify the extent to which neural populations take advantage of the potential benefits offered by sparse representations. Neuroimaging appears suited for this goal because it allows comparing neural representation spaces across species and sensory systems, and tracing these over time during task performance.

12The computational properties of a population code depend also on the pattern of response correlations between neurons. Traditionally, a distinction is made between signal correlations (quantifying the correlation of the trial-averaged neural responses across the different stimuli, and thus quantifying the similarity of stimulus tuning) and noise correlations (quantifying the correlation of trial-by-trial variations of response at fixed stimulus attributes). Much work on population codes has focused on correlations between the firing rates of pairs of neurons.

13When tools for simultaneous multi-neuron recordings became widely available in the 1990s, two divergent views were proposed about the impact of noise correlations between firing rates. The first holds that noise correlations of firing rates – if modulated by the stimulus – may act as a separate coding mechanism complementary to firing rates. For example, it has been proposed that the dynamic stimulus-dependence of noise correlations among visual neurons (originating from changes in gamma-band synchronization among populations) carries information about whether individual neurons respond to the same or separate objects. This view has deeply influenced the interpretation of neural mass signals measured by neuroimaging because it suggests that changes in correlation (or coherence) between the activities of distinct populations may reflect how these populations encode information. The second view is that noise correlations of firing rates may impose limits on the growth of information with population size. For example, for neurons with identical tuning curves, the presence even of weak positive noise correlations precludes the possibility to improve the accuracy of sensory information encoding by simply averaging the rate of many neurons (correlated noise can only be removed to a limited extent by averaging). This second view influenced studies on neural population codes, and motivated recent work on how neuronal networks may dynamically decrease noise correlations to reduce their potentially limiting effect on information encoding. It has also been argued that the limiting effect of correlation may partially be overcome in populations with heterogeneous tuning and correlation properties.

14The potential impact of correlations on population coding has been studied extensively using model-based approaches that quantify stimulus information under various assumptions about correlation structures or readout mechanisms. This approach has shown that correlations can influence neural population coding in more complex way than described above: depending on the precise pattern of signal and noise correlations, and the choice of readout mechanisms, correlations can increase, decrease, or leave unaffected the information carried by a population. An important insight of these theoretical studies is that the precise pattern of noise correlation matters more than noise correlation strength or population size. For example, a recent study considered the ability (quantified using Fisher information) to estimate small changes in a stimulus from population activity, and found that the type of noise correlations that limit the increase of information with population size are so-called differential correlations. Intuitively, such differential correlations correspond to correlated trial-to-trial variability of neural responses that shift the profile of population activity such that it looks like the population activity that would have been elicited by a slightly different stimulus. Thus, this correlated variability makes a noisy fluctuation of population activity look like the signal representing a different stimulus value, and thus acts as a source of noise that cannot be eliminated even by increasing the number of neurons. Whether these information-limiting correlations actually occur can be empirically investigated by computing information versus the number of neurons and investigating whether information saturates with population size. Because information depends crucially on the structure of correlations, extrapolation of its dependence on population size from real data should be performed using decoding approaches rather than making strong but difficult-to-validate assumptions about the structure and shape of correlations.

15Besides determining the amount of information carried by neural populations, noise correlations in firing rates may also be indicative of important computational functions. Noise-correlations may have a role in probabilistic codes representing sensory uncertainty by reflecting correlations in the uncertainty associated with individual stimulus variables. For example, for an ambiguous input consistent with one of two possible stimuli, the neurons representing these two stimuli would be negatively correlated because evidence for one stimulus speaks against the other. It has also been argued that positive correlations within larger populations make the code sparser by increasing periods of population quiescence, and hence concentrate information into rare periods of common activity. This latter property may strongly shape how cortical activity is seen through neuroimaging.

16Correlations between spike times can also contribute to population codes, for example by facilitating the readout of temporal response patterns in a relative latency code. Individual neurons can encode stimuli by their response latency (that is, the delay between stimulus onset and neural firing). Although information in the latency of an individual neuron may not be directly accessible to a downstream decoder (because measuring latency requires information about the precise time of stimulus onset), information in the relative latency between neurons may be robustly extracted by exploiting the fact that trial-by-trial shifts of latency are often correlated across a population. For example, latencies of different neurons may tend to be all-late or all-early because of global fluctuations in network excitability. Correlated latency shifts preserve the relative order of firing or latency differences across neurons, whereas uncorrelated shifts do not. In auditory cortex, the readout of such relative timing may be further supported by time-reference events, in other words, population response patterns that encode the stimulus time with an early, stimulus-invariant latency. Stimulus-selective neurons can then encode the stimulus identity by the time of their spikes relative to these reference events. Although these relative-latency codes are often studied at the pairwise level, recent work suggests that they may reflect a more general larger-scale organization of population activity: groups of neurons may be co-active in stereotyped sequences (termed packets ), and the relative timing and strength of each neuron within this sequence may encode specific stimulus attributes.

17Finally, correlations between spike times may also ensure the transmission of information across areas, for example by facilitating the impact on post-synaptic targets or by facilitating the selective read-out of specific combinations of afferents. All in all, this suggests that the relative timing between neurons contributes both to representing sensory information and to transmitting this information between areas.

18Local neural activity not only depends on the current sensory input but also on the current brain state. Although feed-forward inputs to local circuits provide sensory afferents, the abundance of recurrent and feedback connections, and of neuromodulatory inputs, shapes the background activity on which this sensory information is processed. For example, neuromodulatory inputs that are not directly stimulus-related (such as cholinergic or noradrenergic signals that are an essential component of the control of the animal's behavioral state) can profoundly change the excitability of sensory cortical circuits, the degree of correlation among neurons, and the gain and reliability of individual neurons. Variations in brain state account for a significant fraction of the trial-to-trial variability of population activity.

19This state-dependence must have profound – but largely unexplored – implications for how population codes operate. State-dependence may imply that populations transmit information only using codes that are robust to state fluctuations (e.g., relative latencies or relative firing-rates). Alternatively, downstream areas may extract variables indicating the current state from network activity (similarly to procedures used in data analysis ) and then use state-dependent decoders to interpret population activity. Progress in understanding state-dependent coding requires better statistical methods for single-trial analysis that can be applied to different measurements of brain activity to disentangle state-dependent and -independent aspects of a code.

20Although a complete description of population codes may require recording all neurons involved in the considered task, important empirical knowledge about population codes can be gained from measurements of neural mass signals with high temporal resolution (LFP, ECOG, EEG, MEG). These measurements lack cellular-level resolution but can be easily applied to multiple brain areas and complex tasks, and are sensitive to both supra- and sub-threshold activity. Importantly, they have the potential to capture indicators of cortical state that cannot be easily extracted from the spiking activity of a few neurons alone.

21Advances in single-trial data analysis have expanded the use of mass signals to study sensory transformations, permitting researchers to study the same questions using neuroimaging, multi-neuron recordings, and computational models. For example, recent neuroimaging studies extracted features of population activity influencing the variability of single-trial percepts, and demonstrated that visual object categories or fine details of auditory signals can be recovered using stimulus reconstruction or decoding methods.

22Additional insights can be gained from mass signals by decomposing them into specific frequency bands and separating the sensory information carried by power and phase of individual bands. Recent work, based on both invasive (LFP) and non-invasive (EEG and MEG) recordings, has individuated the phase of low-frequency activity as a particularly informative feature of mass activity. In the visual system, detailed sensory features are reflected more in the phase of low-frequency (below 12 Hz) activity than in the power (Figure 2A). Related findings were made in the auditory system, where the phase of low-frequency activity encodes speech or complex sounds (Figure 2B,C). Stepping beyond single-site analysis, some studies found that the relative timing of neural responses across different sites carries more sensory information than the activation of individual sites. These results hence suggest that the relative timing of population activity could at least be as important for sensory coding as the strength of activity of an individual population, in line with insights from studies on spiking activity that highlight the role of relative response timing.

23An important population coding principle emerging from the analysis of mass signals (both using MEG and EEG or intracranial recordings ) is that of multiplexing of sensory information: different frequency bands of population activity each carry complementary information about stimulus features. For example, nested patterns of slower (e.g., delta or theta) and faster (e.g., gamma) auditory cortical rhythmic activity encode complementary aspects of speech (Figure 2B). These results illustrate the multiple coding dimensions of population signals, and highlight the importance of understanding how specific aspects of coding in spiking activity map onto phase and power of rhythmic mass signals. Specific challenges for understanding the link between non-invasive neuroimaging and neural population codes are outlined in Box 2 and in the outstanding questions (Box 3).

24Important insights about population codes further arise from studies that simultaneously measure spiking responses and mass signals, or that perform comparative analysis on such data obtained in separate experiments.

25One key opportunity of the joint measurements arises from the complementary nature of single-neuron firing and of mass signals: mass signals capture aspects of sub-threshold activity and of intrinsically driven state-changes that cannot be measured by observing the spiking activity of a few neurons alone. Thus, low-frequency mass activity can be seen as a measure of the background state fluctuations constituting the ‘context’ that affects processing of the ‘content’ carried by sensory inputs. Examining the responses of individual neurons relative to the phase of low-frequency network rhythms can shed light on how the local circuit context affects the specific sensory content encoded in spiking activity.

26Converging studies suggest that single-neuron spike timing depends on the frequency-specific oscillatory LFP phase (Figure 3A), and that the sensory information carried by spiking activity can be better interpreted when the network context during which spikes were emitted is known (Figure 3A). A recent study in the human medial temporal lobe showed that perception-related spiking activity is locked to an early stereotyped increase in theta LFP signals, suggesting that specific interactions between network rhythms and firing of individual neurons may define temporal windows facilitating conscious perception. Complementing measurements of spiking activity with LFP recordings may allow differentiating otherwise ambiguous population states and provides a means to disentangle context-related activity fluctuations from those induced by sensory noise or other causes.

27Joint measurements of single-neuron spikes and mass signals also facilitate our understanding of the coordination of population codes across brain structures. Recent concurrent multi-site recordings of spiking activity and LFPs have shown that the spiking activity of a single neuron depends not only on the phase of the LFP oscillation at the same site but also on the dynamic patterns of the relationships (or ‘phase coupling’) between the phases of neural oscillations measured by LFP at multiple distant sites (Figure 3B). The similarity of preferences in LFP phase coupling between neurons also predicts the similarity of their firing-rate variations: neurons that prefer similar patterns of phase coupling exhibit similar changes in firing rates with time or task, whereas neurons with different phase preferences show divergent firing-rate dynamics. This provides direct evidence that large-scale functional connectivity shapes local activation patterns and controls the co-activation and coordination of anatomically dispersed but functionally integrated ensembles of neurons, thus breaking new ground in the understanding of the large-scale organization of neural population codes.

28Another approach to clarify the neural basis of the sensory information carried by different components of MEG/EEG signals is based on the use of information theoretic or stimulus-decoding methods to compare quantitatively the similarities of stimulus encoding in mass signals and multi-neuron recordings. This approach revealed (Figure 2C) that the phase (and to a much lesser extent the power) of low-frequency mass activity captures some aspects of the sensory information carried by population spiking activity. Such comparative approaches will be crucial for understanding what mass signals can tell us about the underlying neural information processing (Box 2) and open the possibility to study emerging principles such as multiplexing, mixed selectivity, or of state-dependence across scales of brain measurements.

29Understanding the properties and principles of cortical population codes requires identifying the most informative patterns within the spatio-temporal complexity of multi-neuron activity and understanding how coding properties are affected by large-scale state changes. Invasive recordings provide detailed access to the heterogeneity, temporal precision, and correlation structure of multi-neuron activity, properties which recent computational studies highlight as being crucial for shaping the information-coding properties of a population. Mass signals, on the other hand, provide more direct access to large-scale changes in network state and connectivity, other crucial properties that shape information coding and which can account for trial-by-trial variations in cognitive tasks. Combining insights from both mass signals and multi-neuron recordings is a key challenge for the future.

30In light of the reviewed properties of population codes several questions emerge about the integration of microscopic and macroscopic structure of population codes (Box 3). For example, can the mechanisms that seem crucial at the microscopic scale (such as neuronal heterogeneity, sparseness, or correlations or mixed selectivity) be observed from mass signals? In turn, how do specific patterns of activity observed at the macroscopic scale (such as patterns of phase coupling across multiple areas or global state changes) relate to the coding properties that are crucial at the microscopic scale? For sure, much is to be learned by methods, such as single-trial stimulus decoding or reconstruction methods, that allow a comparative and detailed assessment of similarity and complementarity of activity at each spatial and temporal scale.

31To enhance our understanding of population coding, future developments of methods for high-dimensional data analysis and neural modeling will be necessary as well. Recent work has provided improvements in analytical techniques for reducing high-dimensional datasets and extracting the relevant sensory representations. In addition, biophysical models of sensory representations in cortical microcircuits allow a principled and rigorous direct link between neural signals and computing architectures, facilitating our understanding of how coding principles are implemented in the neural circuits. Being able to subject different neural signals to the same scientific question and analysis routine, and being able to compute both spiking and mass signals from the same plausible neural network models, are two crucial features to further improve our understanding of neural population coding in the future.