Learnable latent embeddings for joint behavioural and neural analysis

Abstract

Mapping behavioural actions to neural activity is a fundamental goal of neuroscience. As our ability to record large neural and behavioural data increases, there is growing interest in modelling neural dynamics during adaptive behaviours to probe neural representations 1–3 . In particular, although neural latent embeddings can reveal underlying correlates of behaviour, we lack nonlinear techniques that can explicitly and flexibly leverage joint behaviour and neural data to uncover neural dynamics 3–5 . Here, we fill this gap with a new encoding method, CEBRA, that jointly uses behavioural and neural data in a (supervised) hypothesis- or (self-supervised) discovery-driven manner to produce both consistent and high-performance latent spaces. We show that consistency can be used as a metric for uncovering meaningful differences, and the inferred latents can be used for decoding. We validate its accuracy and demonstrate our tool’s utility for both calcium and electrophysiology datasets, across sensory and motor tasks and in simple or complex behaviours across species. It allows leverage of single- and multi-session datasets for hypothesis testing or can be used label free. Lastly, we show that CEBRA can be used for the mapping of space, uncovering complex kinematic features, for the production of consistent latent spaces across two-photon and Neuropixels data, and can provide rapid, high-accuracy decoding of natural videos from visual cortex.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 62

1. 1.

   Urai AE, Doiron B, Leifer AM, Churchland AK. Large-scale neural recordings call for new insights to link brain and behavior. Nat. Neurosci. 2021;25:11-19.

   Not in our library10.1038/s41593-021-00980-9 (opens in a new tab)

2. 2.

   Krakauer JW, Ghazanfar AA, Gomez-Marin A, MacIver MA, Poeppel D. Neuroscience needs behavior: correcting a reductionist bias. Neuron. 2017;93:480-490.

   Not in our library10.1016/j.neuron.2016.12.041 (opens in a new tab)

3. 3.

   Jazayeri M, Ostojic S. Interpreting neural computations by examining intrinsic and embedding dimensionality of neural activity. Curr. Opin. Neurobiol. 2021;70:113-120.

   Not in our library10.1016/j.conb.2021.08.002 (opens in a new tab)

4. 4.

   Humphries, M. D. Strong and weak principles of neural dimension reduction. Neuron. Behav. Data Anal. Theoryhttps://nbdt.scholasticahq.com/article/24619 (2020).

   Not in our library

5. 5.

   Zhou, D., & Wei, X. Learning identifiable and interpretable latent models of high-dimensional neural activity using pi-VAE. Adv. Neural Inf. Process. Syst.https://proceedings.neurips.cc//paper/2020/file/510f2318f324cf07fce24c3a4b89c771-Paper.pdf (2020).

   Not in our library

6. 6.

   Vargas-Irwin CE, et al. Decoding complete reach and grasp actions from local primary motor cortex populations. J. Neurosci. 2010;30:9659-9669.

   Not in our library10.1523/jneurosci.5443-09.2010 (opens in a new tab)

7. 7.

   Okorokova, E. V., Goodman, J. M., Hatsopoulos, N. G. & Bensmaia, S. J. Decoding hand kinematics from population responses in sensorimotor cortex during grasping. J. Neural Eng. 17, 046035 (2020).

   Not in our library10.1088/1741-2552/ab95ea (opens in a new tab)

8. 8.

   Yu BM, et al. Gaussian-process factor analysis for low-dimensional single-trial analysis of neural population activity. J. Neurophysiol. 2008;102:614-635.

   Not in our library10.1152/jn.90941.2008 (opens in a new tab)

9. 9.

   Churchland M, et al. Neural population dynamics during reaching. Nature. 2012;487:51-56.

   Not in our library10.1038/nature11129 (opens in a new tab)

10. 10.
    Cortical population activity within a preserved neural manifold underlies multiple motor behaviors · 2018

    No notice
11. 11.

    McInnes L, Healy J, Saul N, Großberger L. UMAP: Uniform Manifold Approximation and Projection for dimension reduction. J. Open Source Softw. 2018;3:861.

    Not in our library10.21105/joss.00861 (opens in a new tab)

12. 12.

    Maaten LV, Postma EO, Herik JV. Dimensionality reduction: a comparative review. J. Mach. Learn. Res. 2009;10:13.

    Not in our library

13. 13.

    Roeder, G., Metz, L. & Kingma, D. P. On linear identifiability of learned representations. Proc. Mach. Learn. Res.139, 9030–9039 (2021).

    Not in our library

14. 14.

    Hyvärinen, A., Sasaki, H. & Turner, R. E. Nonlinear ICA using auxiliary variables and generalized contrastive learning. Proc. Mach. Learn. Res.89, 859–868 (2019).

    Not in our library

15. 15.

    Sani OG, Abbaspourazad H, Wong YT, Pesaran B, Shanechi MM. Modeling behaviorally relevant neural dynamics enabled by preferential subspace identification. Nat. Neurosci. 2020;24:140-149.

    Not in our library10.1038/s41593-020-00733-0 (opens in a new tab)

16. 16.

    Klindt, D. A. et al. Towards nonlinear disentanglement in natural data with temporal sparse coding. International Conference on Learning Representationshttps://openreview.net/forum?id=EbIDjBynYJ8 (2021).

    Not in our library

17. 17.

    Pandarinath C, et al. Inferring single-trial neural population dynamics using sequential auto-encoders. Nat. Methods. 2017;15:805-815.

    Not in our library10.1038/s41592-018-0109-9 (opens in a new tab)

18. 18.

    Prince, L. Y., Bakhtiari, S., Gillon, C. J., & Richards, B. A. Parallel inference of hierarchical latent dynamics in two-photon calcium imaging of neuronal populations. Preprint at https://www.biorxiv.org/content/10.1101/2021.03.05.434105v1 (2021).

    Not in our library

19. 19.

    Gutmann MU, Hyvärinen A. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. J. Mach. Learn. Res. 2012;13:307-361.

    Not in our library

20. 20.

    Oord, A. V., Li, Y. & Vinyals, O. Representation learning with contrastive predictive coding. Preprint at 10.48550/arXiv.1807.03748 (2018).

    Not in our library

21. 21.

    Khosla P, et al. Supervised contrastive learning. Adv. Neural Inf. Process. Syst. 2020;33:18661-18673.

    Not in our library

22. 22.

    Chen, T., Kornblith, S., Norouzi, M. & Hinton, G. E. A simple framework for contrastive learning of visual representations. Proc. Mach. Learn. Res.119, 1597–1607 (2020).

    Not in our library

23. 23.

    Hälvä H, et al. Disentangling identifiable features from noisy data with structured nonlinear ICA. Adv. Neural Inf. Process. Syst. 2021;34:1624-1633.

    Not in our library

24. 24.

    Zimmermann, R. S., Sharma, Y., Schneider, S., Bethge, M. & Brendel, W. Contrastive learning inverts the data generating process. Proc. Mach. Learn. Res.139, 12979–12990 (2021).

    Not in our library

25. 25.

    Keshtkaran MR, et al. A large-scale neural network training framework for generalized estimation of single-trial population dynamics. Nat. Methods. 2022;19:1572-1577.

    Not in our library10.1038/s41592-022-01675-0 (opens in a new tab)

26. 26.

    Grosmark AD, Buzsáki G. Diversity in neural firing dynamics supports both rigid and learned hippocampal sequences. Science. 2016;351:1440-1443.

    Not in our library10.1126/science.aad1935 (opens in a new tab)

27. 27.

    Huxter JR, Burgess N, O’Keefe J. Independent rate and temporal coding in hippocampal pyramidal cells. Nature. 2003;425:828-832.

    Not in our library10.1038/nature02058 (opens in a new tab)

28. 28.

    Moser EI, Kropff E, Moser M. Place cells, grid cells, and the brain’s spatial representation system. Annu. Rev. Neurosci. 2008;31:69-89.

    Not in our library10.1146/annurev.neuro.31.061307.090723 (opens in a new tab)

29. 29.

    Dombeck DA, Harvey CD, Tian L, Looger LL, Tank DW. Functional imaging of hippocampal place cells at cellular resolution during virtual navigation. Nat. Neurosci. 2010;13:1433-1440.

    Not in our library10.1038/nn.2648 (opens in a new tab)

30. 30.

    Curto C. What can topology tell us about the neural code? Bull. Am. Math. Soc. 2016;54:63-78.

    Not in our library10.1090/bull/1554 (opens in a new tab)

31. 31.

    Chaudhuri R, Gerçek B, Pandey B, Peyrache A, Fiete IR. The intrinsic attractor manifold and population dynamics of a canonical cognitive circuit across waking and sleep. Nat. Neurosci. 2019;22:1512-1520.

    Not in our library10.1038/s41593-019-0460-x (opens in a new tab)

32. 32.

    Silva VD, Morozov D, Vejdemo-Johansson M. Persistent cohomology and circular coordinates. Discrete Comput. Geom. 2009;45:737-759.

    Not in our library10.1007/s00454-011-9344-x (opens in a new tab)

33. 33.
    Toroidal topology of population activity in grid cells · 2022

    No notice
34. 34.

    Chowdhury RH, Glaser JI, Miller LE. Area 2 of primary somatosensory cortex encodes kinematics of the whole arm. eLife. 2019;9:e48198.

    Not in our library10.7554/elife.48198 (opens in a new tab)

35. 35.

    Prud’homme MJ, Kalaska JF. Proprioceptive activity in primate primary somatosensory cortex during active arm reaching movements. J. Neurophysiol. 1994;72:2280-2301.

    Not in our library10.1152/jn.1994.72.5.2280 (opens in a new tab)

36. 36.

    London BM, Miller LE. Responses of somatosensory area 2 neurons to actively and passively generated limb movements. J. Neurophysiol. 2013;109:1505-1513.

    Not in our library10.1152/jn.00372.2012 (opens in a new tab)

37. 37.

    Pei, F. et al. Neural Latents Benchmark ‘21: Evaluating latent variable models of neural population activity. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarkshttps://openreview.net/forum?id=KVMS3fl4Rsv (2021).

    Not in our library

38. 38.
    Community-based benchmarking improves spike rate inference from two-photon calcium imaging data · 2018

    No notice
39. 39.

    Hafting T, Fyhn M, Molden S, Moser M, Moser EI. Microstructure of a spatial map in the entorhinal cortex. Nature. 2005;436:801-806.

    Not in our library10.1038/nature03721 (opens in a new tab)

40. 40.

    Schultz W, Dayan P, Montague PR. A neural substrate of prediction and reward. Science. 1997;275:1593-1599.

    Not in our library10.1126/science.275.5306.1593 (opens in a new tab)

41. 41.

    Cohen JY, Haesler S, Vong L, Lowell BB, Uchida N. Neuron-type specific signals for reward and punishment in the ventral tegmental area. Nature. 2011;482:85-88.

    Not in our library10.1038/nature10754 (opens in a new tab)

42. 42.
    Dopamine neurons projecting to the posterior striatum form an anatomically distinct subclass · 2015

    No notice
43. 43.

    Hubel DH, Wiesel TN. Ferrier lecture – functional architecture of macaque monkey visual cortex. Proc. R. Soc. Lond. B Biol. Sci. 1977;198:1-59.

    Not in our library10.1098/rspb.1977.0085 (opens in a new tab)

44. 44.

    Niell CM, Stryker MP, Keck WM. Highly selective receptive fields in mouse visual cortex. J. Neurosci. 2008;28:7520-7536.

    Not in our library10.1523/jneurosci.0623-08.2008 (opens in a new tab)

45. 45.
    Spatial clustering of tuning in mouse primary visual cortex · 2016

    No notice
46. 46.

    de Vries SE, et al. A large-scale standardized physiological survey reveals functional organization of the mouse visual cortex. Nat. Neurosci. 2019;23:138-151.

    Not in our library10.1038/s41593-019-0550-9 (opens in a new tab)

47. 47.

    Siegle JH, et al. Survey of spiking in the mouse visual system reveals functional hierarchy. Nature. 2021;592:86-92.

    Not in our library10.1038/s41586-020-03171-x (opens in a new tab)

48. 48.

    Caron, M. et al. Emerging properties in self-supervised vision transformers. IEEE/CVF International Conference on Computer Vision 9630–9640 (2021).

    Not in our library

49. 49.

    Esfahany, K., Siergiej, I., Zhao, Y. & Park, I. M. Organization of neural population code in mouse visual system. eNeuro5, ENEURO.0414-17.2018 (2018).

    Not in our library10.1523/eneuro.0414-17.2018 (opens in a new tab)

50. 50.

    Jin M, Glickfeld LL. Mouse higher visual areas provide both distributed and specialized contributions to visually guided behaviors. Curr. Biol. 2020;30:4682-4692.

    Not in our library10.1016/j.cub.2020.09.015 (opens in a new tab)

51. 51.

    Dinh, L., Sohl-Dickstein, J. N. & Bengio, S. Density estimation using Real NVP. International Conference on Learning Representationshttps://openreview.net/pdf?id=HkpbnH9lx (2017).

    Not in our library

52. 52.

    Deitch D, Rubin A, Ziv Y. Representational drift in the mouse visual cortex. Curr. Biol. 2021;31:4327-4339.

    Not in our library10.1016/j.cub.2021.07.062 (opens in a new tab)

53. 53.

    Wang, T. & Isola, P. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. Proc. Mach. Learn. Res.119, 9929–9939 (2020).

    Not in our library

54. 54.

    Hendrycks, D. & Gimpel, K. Gaussian error linear units (GELUs). Preprint at 10.48550/arXiv.1606.08415 (2016).

    Not in our library

55. 55.

    Paszke, A. et al. PyTorch: an imperative style, high-performance deep learning library. Neural Inf. Process. Syst. 32, 8024–8035 (2019).

    Not in our library

56. 56.

    Walt SV, Colbert SC, Varoquaux G. The NumPy array: a structure for efficient numerical computation. Comput. Sci. Eng. 2011;13:22-30.

    Not in our library10.1109/mcse.2011.37 (opens in a new tab)

57. 57.

    Pedregosa F, et al. Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 2011;12:2825-2830.

    Not in our library

58. 58.

    Policar, P. G., Stražar, M. & Zupan, B. openTSNE: a modular Python library for t-SNE dimensionality reduction and embedding. Preprint at bioRxiv10.1101/731877 (2019).

    Not in our library

59. 59.

    Kobak D, Linderman GC. Initialization is critical for preserving global data structure in both t-SNE and UMAP. Nat. Biotechnol. 2021;39:156-157.

    Not in our library10.1038/s41587-020-00809-z (opens in a new tab)

60. 60.

    Schönemann PH. A generalized solution of the orthogonal procrustes problem. Psychometrika. 1966;31:1-10.

    Not in our library10.1007/bf02289451 (opens in a new tab)

61. 61.

    Tralie CJ, Saul N, Bar-On R. Ripser.py: a lean persistent homology library for Python. J. Open Source Softw. 2018;3:925.

    Not in our library10.21105/joss.00925 (opens in a new tab)

62. 62.

    Tralie, C. J., Mease, T. & Perea, J. Dreimac: dimension reduction with Eilenberg–Maclane coordinates. GitHub https://github.com/ctralie/DREiMac/tree/cdd6d02ba53c3597a931db9da478fd198d6ed00f (2018).

    Not in our library

Cited by papers we hold 1

• Preserved neural dynamics across animals performing similar behaviour · 2023
  No notice

Questions about this paper

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.