A Tutorial for Information Theory in Neuroscience

Abstract

Understanding how neural systems integrate, encode, and compute information is central to understanding brain function. Frequently, data from neuroscience experiments are multivariate, the interactions between the variables are nonlinear, and the landscape of hypothesized or possible interactions between variables is extremely broad. Information theory is well suited to address these types of data, as it possesses multivariate analysis tools, it can be applied to many different types of data, it can capture nonlinear interactions, and it does not require assumptions about the structure of the underlying data (i.e., it is model independent). In this article, we walk through the mathematics of information theory along with common logistical problems associated with data type, data binning, data quantity requirements, bias, and significance testing. Next, we analyze models inspired by canonical neuroscience experiments to improve understanding and demonstrate the strengths of information theory analyses. To facilitate the use of information theory analyses, and an understanding of how these analyses are implemented, we also provide a free MATLAB software package that can be applied to a wide range of data from neuroscience experiments, as well as from other fields of study.

Shown for reading. Its publisher allows quoting it, not republishing it.

References 112

1. 1.

   Adriaans P (2012 ) Information In: Standford Encyclopedia of Philosophy (Zalta EN, ed.).

   Not in our library

2. 2.

   Asaad WF, Lauro PM, Perge JA, Eskandar EN (2017 ) Prefrontal neurons encode a solution to the credit-assignment problem. J Neurosci 37:6995–7007.

   Not in our library10.1523/jneurosci.3311-16.2017

3. 3.

   Bear MF, Connors BW, Paradiso MA (2007 ) Neuroscience: exploring the brain, Third Edition Baltimore, MD: Lippincott Williams and Wilkins.

   Not in our library

4. 4.

   Beer RD, Williams PL (2014 ) Information processing and dynamics in minimally cognitive agents. Cogn Sci 1–38.

   Not in our library10.1111/cogs.12142

5. 5.

   Benjamini Y, Hochberg Y (1995 ) Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Stat Soc B 57:289–300.

   Not in our library

6. 6.

   Benjamini Y, Yekutieli D (2001 ) The control of the false discovery rate in multiple testing under dependency. Ann Stat 29:1165–1188.

   Not in our library10.1214/aos/1013699998

7. 7.

   Bertschinger N, Rauh J, Olbrich E, Jost J, Ay N (2014 ) Quantifying unique information. Entropy 16:2161–2183.

   Not in our library10.3390/e16042161

8. 8.

   Bettencourt LMA, Gintautas V, Ham MI (2008 ) Identification of functional information subgraphs in complex networks. Phys Rev Lett 100:238701.

   Not in our library10.1103/physrevlett.100.238701

9. 9.

   Bialek W, Rieke F, De Ruyter van Steveninck RR, Warland D (1991 ) Reading a neural code. Science 252:1854–1857.

   Not in our library10.1126/science.2063199

10. 10.

    Bonachela JA, Hinrichsen H, Muñoz MA (2008 ) Entropy estimates of small data sets. J Phys A 41:202001.

    Not in our library10.1088/1751-8113/41/20/202001

11. 11.

    Bonifazi P, Goldin M, Picardo MA, Jorquera I, Cattani A, Bianconi G, Represa A, Ben-Ari Y, Cossart R (2009 ) GABAergic hub neurons orchestrate synchrony in developing hippocampal networks. Science 326:1419–1424.

    Not in our library10.1126/science.1175509

12. 12.

    Borst A, Theunissen FE (1999 ) Information theory and neural coding. Nat Neurosci 2:947–957.

    Not in our library10.1038/14731

13. 13.

    Bossomaier T, Barnett L, Harre M, Lizier JT (2016 ) An introduction to transfer entropy. New York: Springer International.

    Not in our library

14. 14.

    Brenner N, Strong SP, Koberle R, Bialek W, de R, van Steveninck RR (2000 ) Synergy in a neural code. Neur Comput 12:1531–1532.

    Not in our library10.1162/089976600300015259

15. 15.

    Bullmore E, Sporns O (2009 ) Complex brain networks: graph theoretical analysis of structural and function systems. Nat Rev Neurosci 10:186–198.

    Not in our library10.1038/nrn2618

16. 16.

    Butts DA (2003 ) How much information is associated with a particular stimulus? Netw Comput Neur Syst 14:177–187.

    Not in our library10.1088/0954-898x_14_2_301

17. 17.

    Butts DA, Weng C, Jin J, Yeh C, Lesica NA, Alonso J, Stanley GB (2007 ) Temporal precision in the neural code and the timescales of natural vision. Nat Lett 449:92–95.

    Not in our library10.1038/nature06105

18. 18.

    Castellucci V, Pinsker H, Kupfermann I, Kandel ER (1970 ) Neuronal mechanisms of habituation and dishabituation of the gill-withdrawal reflex in aplysia. Science 167:1745–1748.

    Not in our library10.1126/science.167.3926.1745

19. 19.

    Cover TM, Thomas JA (2006 ) Elements of Information Theory, 2nd Edition New York: Wiley-Interscience.

    Not in our library

20. 20.

    Cunningham JP, Yu BM (2014 ) Dimensionality reduction for large-scale neural recordings. Nat Neurosci 17:1500–1509.

    Not in our library10.1038/nn.3776

21. 21.

    Daw CS, Finney CEA, Tracy ER (2003 ) A review of symbolic analysis of experimental data. Rev Scientif Instr 74:915–930.

    Not in our library10.1063/1.1531823

22. 22.

    DeWeese MR, Meister M (1999 ) How to measure the information gained from one symbol. Netw Comput Neur Syst 10:325–340.

    Not in our library10.1088/0954-898x_10_4_303

23. 23.

    Dimitrov AG, Lazar AA, Victor JD (2011 ) Information theory in neuroscience. J Comput Neurosci 30:1–5.

    Not in our library10.1007/s10827-011-0314-3

24. 24.

    Ding M, Chen Y, Bressler SL (2006 ) Granger causality: basic theory and application to neuroscience In: Handbook of Time Series Analysis: Recent Theoretical Developments and Applications (Schelter B, Winterhalder M, Timmer J, eds), p 437 Weinheim, Germany: Wiley-VCH.

    Not in our library

25. 25.

    Dit-Contributors (2018 ) Dit: Discrete information theory. Available at https://dit.readthedocs.io/en/latest/.

    Not in our library

26. 26.

    Finn C, Lizier JT (2018 ) Pointwise information decomposition using the specificity and ambiguity lattices. arXiv doi:10.3390/ecea-4-05024.

    Not in our library10.3390/e20040297

27. 27.

    Friston KJ (2011 ) Functional and effective connectivity: a review. Brain Connect 1:13–36.

    Not in our library10.1089/brain.2011.0008

28. 28.

    Friston KJ, Harrison L, Penny W (2003 ) Dynamic causal modelling. Neuroimage 19:1273–1302.

    Not in our library10.1016/s1053-8119(03)00202-7

29. 29.

    Georgopoulos AP, Kalaska JF, Caminiti R, Massey JT (1982 ) On the relations between the direction of two-dimensional arm movements and cell discharge in primate motor cortex. J Neurosci 2:1527–1537.

    Not in our library10.1523/jneurosci.02-11-01527.1982

30. 30.

    Goldberg DH, Victor JD, Gardner EP, Gardner D (2009 ) Spike train analysis toolkit: enabling wider application of information-theoretic techniques in neurophysiology. Neuroinformatics 7:165–178.

    Not in our library10.1007/s12021-009-9049-y

31. 31.

    Gomez-Herraro G, Wu W, Rutanen K, Soriano MC, Pipa G, Vicente R (2015 ) Assessing coupling dynamics from an ensemble of time series. Entropy 17:1958–1970.

    Not in our library10.3390/e17041958

32. 32.

    Granger CWJ (1969 ) Investigating causal relations by econometric models and cross-spectral methods. Econometrica 37:424–438.

    Not in our library10.2307/1912791

33. 33.

    Griffith V, Chong EKP, James RG, Ellison CJ, Crutchfield JP (2014 ) Intersection information based on common randomness. Entropy 16:1985–2000.

    Not in our library10.3390/e16041985

34. 34.

    Harder M, Salge C, Polani D (2013 ) Bivariate measure of redundant information. Phys Rev E 87:012130.

    Not in our library10.1103/physreve.87.012130

35. 35.

    Honey CJ, Kotter R, Breakspear M, Sporns O (2007 ) Network structure of cerebral cortex shapes functional connectivity on multiple time scales. Proc Natl Acad Sci U S A 104:10240–10245.

    Not in our library10.1073/pnas.0701519104

36. 36.

    Hramov AE, Koronovskii AA, Makarov VA, Pavlov AN, Sitnikova E (2015 ) Wavelets in neuroscience. Berlin: Springer.

    Not in our library

37. 37.

    Ignac TM, Skupin A, Sakhanenko NA, Galas DJ (2014 ) Discovering pair-wise genetic interactions: an information theory-based approach. PLOS One 9:e92310.

    Not in our library10.1371/journal.pone.0092310

38. 38.

    Ince RAA (2017 ) Measuring multivariate redundant information with pointwise common change in surprisal. Entropy 19:318–344.

    Not in our library10.3390/e19070318

39. 39.

    Ince RAA, Petersen RS, Swan DC, Panzeri S (2009 ) Python for information theoretic analysis of neural data. Front Neuroinform 3:4.

    Not in our library10.3389/neuro.11.004.2009

40. 40.

    Ince RAA, Mazzoni A, Petersen RS, Panzeri S (2010 ) Open source tools for the information theoretic analysis of neural data. Front Neurosci 4:62–70.

    Not in our library10.3389/neuro.01.011.2010

41. 41.

    Ince RAA, Rijsbergen NJ, Thut G, Rousselet GA, Gross J, Panzeri S, Schyns PG (2015 ) Tracing the flow of perceptual features in an algorithmic brain network. Sci Reports 5:17681.

    Not in our library10.1038/srep17681

42. 42.
    Extending Transfer Entropy Improves Identification of Effective Connectivity in a Spiking Cortical Network Model · 2011

    No notice
43. 43.

    Ito S, Yeh FC, Hiolski E, Rydygier P, Gunning DE, Hottowy P, Timme N, Litke AM, Beggs JM (2014 ) Large-scale, high-resolution multielectrode-array recording depicts functional network differences of cortical and hippocampal cultures. PloS One 9:e105324.

    Not in our library10.1371/journal.pone.0105324

44. 44.

    Izhikevich EM (2003 ) Simple model of spiking neurons. IEEE Trans Neur Netw 14:1569–1572.

    Not in our library10.1109/tnn.2003.820440

45. 45.

    Izhikevich EM (2007 ) Dynamical Systems in Neuroscience. Cambridge, MA: MIT Press.

    Not in our library

46. 46.
    Multivariate Dependence beyond Shannon Information · 2017

    Not checked
47. 47.

    Jeong J, Gore JC, Peterson BS (2001 ) Mutual information analysis of the EEG in patients with Alzheimer’s disease. Clin Neurophysiol 112:827–835.

    Not in our library10.1016/s1388-2457(01)00513-2

48. 48.

    Koller D, Friedman N (2009 ) Probabilistic Graphical Models: Principles and Techniques. Cambridge, MA: MIT Press.

    Not in our library

49. 49.

    Kraskov A, Stoegbauer H, Grassberger P (2004 ) Estimating mutual information. Phys Rev E 69:066138.

    Not in our library10.1103/physreve.69.066138

50. 50.

    Kruschke JK (2015 ) Doing Bayesian Data Analysis. London: Elsevier.

    Not in our library

51. 51.

    Kuffler SW (1953 ) Discharge patterns and functional organization of mammalian retina. J Neurophysiol 16:37–68.

    Not in our library10.1152/jn.1953.16.1.37

52. 52.
    TRENTOOL: A Matlab open source toolbox to analyse information flow in time series data with transfer entropy · 2011

    No notice
53. 53.

    Lizier JT (2014 ) JIDT: an information-theoretic toolkit for studying the dynamics of complex systems. Front Robotics AI 1:11.

    Not in our library10.3389/frobt.2014.00011

54. 54.

    Lizier JT, Prokopenko M, Zomaya AY (2008 ) Local information transfer as a spatiotemporal filter for complex systems. Phys Rev E 77:026110.

    Not in our library10.1103/physreve.77.026110

55. 55.

    Lizier JT, Heinzle J, Horstmann A, Haynes J, Prokopenko M (2011 ) Multivariate information-theoretic measures reveal directed information structure and task relevant changes in fMRI connectivity. J Comput Neurosci 30:85–107.

    Not in our library10.1007/s10827-010-0271-2

56. 56.
    A toolbox for the fast information analysis of multiple-site LFP, EEG and spike train recordings · 2009

    No notice
57. 57.

    McShane BB, Gal D, Gelman A, Robert C, Tackett JL (2017 ) Abandon statistical significance. arXiv 1709.07588.

    Not in our library

58. 58.
    MuTE: a MATLAB toolbox to compare established and novel estimators of the multivariate transfer entropy. · 2014

    Not checked
59. 59.

    Moore DG, Valentini G, Walker SI, Levin M (2017 ) Inform: a toolkit for information-theoretic analysis of complex systems. In 2017 IEEE Symposium Series on Computational Intelligence (SSCI).

    Not in our library

60. 60.

    Myung IJ (2003 ) Tutorial on maximum likelihood estimation. J Math Psychol 47:90–100.

    Not in our library10.1016/s0022-2496(02)00028-7

61. 61.

    Nakhnikian A, Rebec GV, Grasse LM, Dwiel LL, Shimono M, Beggs JM (2014 ) Behavior modulates effective connectivity between cortex and striatum. PloS One 9:e89443.

    Not in our library10.1371/journal.pone.0089443

62. 62.

    Nemenman I, Shafee F, Bialek W (2002 ) Entropy and inference, revisited. Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press 471–478.

    Not in our library

63. 63.

    Nemenman I, Bialek W, De Ruyter van Steveninck RR (2004 ) Entropy and information in neural spike trains: progress on the sampling problem. Phys Rev E 69:056111.

    Not in our library10.1103/physreve.69.056111

64. 64.

    Nigam S, Shimono M, Ito S, Yeh FC, Timme N, Myroshnychenko M, Lapish CC, Tosi Z, Hottowy P, Smith WC, Masmanidis SC, Litke AM, Sporns O, Beggs JM (2016 ) Rich-club organization in effective connectivity among cortical neurons. J Neurosci 36:670–684.

    Not in our library10.1523/jneurosci.2177-15.2016

65. 65.

    O’Keefe J, Dostrovsky J (1971 ) The hippocampus as a spatial map. Brain Res 34:171–175.

    Not in our library10.1016/0006-8993(71)90358-1

66. 66.

    Paninski L (2003 ) Estimation of entropy and mutual information. Neur Comput 15:1191–1253.

    Not in our library10.1162/089976603321780272

67. 67.

    Panzeri S, Treves A (1996 ) Analytical estimates of limited sampling biases in different information measures. Netw Comput Neur Syst 7:87–107.

    Not in our library10.1080/0954898x.1996.11978656

68. 68.

    Panzeri S, Schultz SR (2001 ) A unified approach to the study of temporal, correlational, and rate coding. Neur Comput 13:1311–1349.

    Not in our library10.1162/08997660152002870

69. 69.

    Panzeri S, Senatore R, Montemurro MA, Petersen RS (2007 ) Correcting for the sampling bias problem in spike train information measures. J Neurophysiol 98:1064–1072.

    Not in our library10.1152/jn.00559.2007

70. 70.

    Panzeri S, Petersen RS, Schultz SR, Lebedev M, Diamon ME (2001 ) The role of spike timing in the coding of stimulus location in rat somatosensory cortex. Neuron 29:769.

    Not in our library10.1016/s0896-6273(01)00251-3

71. 71.

    Pastore VP, Poli D, Godjoski A, Martinoia S, Massobrio P (2016 ) ToolConnect: a functional connectivity toolbox for in vitro networks. Front Neuroinform 10:13.

    Not in our library10.3389/fninf.2016.00013

72. 72.

    Pica G, Piasini E, Chicharro D, Panzeri S (2017 ) Invariant components of synergy, redundancy, and unique information among three variables. arXiv 1706.08921.

    Not in our library

73. 73.
    Quantifying Synergistic Information Using Intermediate Stochastic Variables † · 2017

    Not checked
74. 74.

    Quinn CJ, Coleman TP, Kiyavash N, Hatsopoulos N (2011 ) Estimating the directed information to infer causal relationships in ensemble neural spike train recordings. J Comput Neurosci 30:17–44.

    Not in our library10.1007/s10827-010-0247-2

75. 75.

    Quiroga RQ, Panzeri S (2009 ) Extracting information from neuronal populations: information theory and decoding approaches. Nat Rev Neurosci 10:173–185.

    Not in our library10.1038/nrn2578

76. 76.

    Quiroga RQ, Panzeri S, eds (2013 ) Principles of Neural Coding. Boca Raton, FL: CRC Press.

    Not in our library

77. 77.
    Minimum Sample Size for Reliable Causal Inference Using Transfer Entropy · 2017

    Not checked
78. 78.

    www.nicholastimme.com (2018 ) Redacted for double-blind review.

    Not in our library

79. 79.

    Rieke F, Warland D, de Ruyter van Steveninck RR, Bialek W (1997 ) Spikes: Exploring the Neural Code. Cambridge, MA: MIT Press.

    Not in our library

80. 80.

    Rolston JD, Wagenaar DA, Potter SM (2007 ) Precisely timed spatiotemporal patterns of neural activity in dissociated cortical cultures. Neuroscience 148:294–303.

    Not in our library10.1016/j.neuroscience.2007.05.025

81. 81.

    Schneidman E, Bialek W, Berry MJ, II (2003a ) Synergy, redundancy, and independence in population codes. J Neurosci 23:11539–11553.

    Not in our library10.1523/jneurosci.23-37-11539.2003

82. 82.

    Schneidman E, Still S, Berry MJ, II, Bialek W (2003b ) Network information and connected correlations. Phys Rev Lett 91:238701.

    Not in our library10.1103/physrevlett.91.238701

83. 83.

    Schölvinck ML, Leopold DA, Brookes MJ, Khader PH (2013 ) The contribution of electrophysiology to functional connectivity mapping. NeuroImage 80:297–306.

    Not in our library10.1016/j.neuroimage.2013.04.010

84. 84.

    Schreiber T (2000 ) Measuring information transfer. Phys Rev Lett 85:461–464.

    Not in our library10.1103/physrevlett.85.461

85. 85.

    Shannon CE (1948 ) A mathematical theory of communication. Bell Sys Tech J 27:379–423.

    Not in our library10.1002/j.1538-7305.1948.tb01338.x

86. 86.

    Smouse PE, Whitehead MR, Peakall R (2015 ) An informational diversity framework, illustrated with sexually deceptive orchids in early stages of speciation. Mol Ecol Res 15:1375–1384.

    Not in our library10.1111/1755-0998.12422

87. 87.

    Sporns O (2007 ) Brain connectivity. Scholarpedia 2:4695.

    Not in our library10.4249/scholarpedia.4695

88. 88.

    Sporns O (2013 ) Structure and function of complex brain networks. Dialog Clin Neurosci 15:247–262.

    Not in our library10.31887/dcns.2013.15.3/osporns

89. 89.

    Staniek M, Lehnertz K (2008 ) Symbolic transfer entropy. Phys Rev Lett 100:158101.

    Not in our library10.1103/physrevlett.100.158101

90. 90.

    Stanley GB (2013 ) Reading and writing the neural code. Nat Neurosci 16:259–263.

    Not in our library10.1038/nn.3330

91. 91.

    Stone JV (2018 ) Information theory: a tutorial introduction. arXiv 1802.05968.

    Not in our library

92. 92.

    Szabo Z (2014 ) Information theoretical estimators toolbox. J Mach Learn Res 15:283–287.

    Not in our library

93. 93.

    Timme N, Alford W, Flecker B, Beggs JM (2014a ) Synergy, redundancy, and multivariate information measures: an experimentalist’s perspective. J Comput Neurosci 36:119–140.

    Not in our library10.1007/s10827-013-0458-4

94. 94.
    Multiplex networks of cortical and hippocampal neurons revealed at different timescales. · 2014

    Not checked
95. 95.

    Timme NM, Ito S, Myroshnychenko M, Nigam S, Shimono M, Yeh FC, Hottowy P, Litke A, Beggs JM (2016 ) High-degree neurons feed cortical computations. PLoS Comput Biol 12:e1004858.

    Not in our library10.1371/journal.pcbi.1004858

96. 96.

    Timme NM, Lapish C (2018 ) GitHub - nmtimme/Neuroscience-Information-Theory-Toolbox: A MATLAB toolbox for performing information theory analyses of neuroscience data. https://github.com/nmtimme/Neuroscience-Information-Theory-Toolbox

    Not in our library

97. 97.

    Treves A, Panzeri S (1995 ) The upward bias in measures of information derived from limited data samples. Neur Comput 7:399–407.

    Not in our library10.1162/neco.1995.7.2.399

98. 98.

    Van Rullen R, Thorpe SJ (2001 ) Rate coding versus temporal order coding: what the retinal ganglion cells tell the visual cortex. Neur Comput 13:1255–1283.

    Not in our library10.1162/08997660152002852

99. 99.

    Vicente R, Wibral M, Lindner M, Pipa G (2011 ) Transfer entropy: a model-free measure of effective connectivity for the neurosciences. J Comput Neurosci 30:45–67.

    Not in our library10.1007/s10827-010-0262-3

100. 100.

     Victor JD (2002 ) Binless strategies for estimation of information from neural data. Phys Rev E 66:051903.

     Not in our library10.1103/physreve.66.051903

101. 101.

     Victor JD (2006 ) Approaches to information-theoretic analysis of neural activity. Biol Theory 1:302–316.

     Not in our library10.1162/biot.2006.1.3.302

102. 102.

     Victor JD, Purpura KP (1996 ) Nature and precision of temporal coding in visual cortex: a metric-space analysis. J Neurophysiol 76:1310–1326.

     Not in our library10.1152/jn.1996.76.2.1310

103. 103.

     Vinga S (2013 ) Information theory applications for biological sequence analysis. Brief Bioinfo 15:376–389.

     Not in our library10.1093/bib/bbt068

104. 104.
     Bits from Brains for Biologically-Inspired Computing · 2015

     Not checked
105. 105.

     Wibral M, R Vicente, Lizier JT, eds (2014b ) Directed information measures in neuroscience. Berlin: Springer.

     Not in our library

106. 106.
     Measuring Information-Transfer Delays · 2013

     No notice
107. 107.

     Williams PL, Beer RD (2010 ) Nonnegative decomposition of multivariate information. arXiv 1004.2515.

     Not in our library

108. 108.

     Williams PL, Beer RD (2011 ) Generalized measures of information transfer. arXiv 1102.1507.

     Not in our library

109. 109.
     Efficient transfer entropy analysis of non-stationary neural time series. · 2014

     Not checked
110. 110.

     Wollstadt P, Sellers KK, Rudelt L, Priesemann V, Hutt A, Fröhlich F, Wibral M (2017 ) Breakdown of local information processing may underlie isoflurane anesthesia effects. PLoS Comput Biol 13:e1005511.

     Not in our library10.1371/journal.pcbi.1005511

111. 111.

     Wolpert DH, Wolf DR (1995 ) Estimating functions of probability distributions from a finite set of samples. Phys Rev E 52:6841.

     Not in our library10.1103/physreve.52.6841

112. 112.

     Zar JH (2010 ) Biostatistical Analysis, 5th Edition Upper Saddle River, NJ: Prentice Hall, Inc.

     Not in our library

Cited by papers we hold 1

• PLD3 affects axonal spheroids and network defects in Alzheimer’s disease · 2022
  No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.