Identifying behavioral structure from deep variational embeddings of animal motion

Abstract

Quantification and detection of the hierarchical organization of behavior is a major challenge in neuroscience. Recent advances in markerless pose estimation enable the visualization of high-dimensional spatiotemporal behavioral dynamics of animal motion. However, robust and reliable technical approaches are needed to uncover underlying structure in these data and to segment behavior into discrete hierarchically organized motifs. Here, we present an unsupervised probabilistic deep learning framework that identifies behavioral structure from deep variational embeddings of animal motion (VAME). By using a mouse model of beta amyloidosis as a use case, we show that VAME not only identifies discrete behavioral motifs, but also captures a hierarchical representation of the motif’s usage. The approach allows for the grouping of motifs into communities and the detection of differences in community-specific motif usage of individual mouse cohorts that were undetectable by human visual observation. Thus, we present a robust approach for the segmentation of animal motion that is applicable to a wide range of experimental setups, models and conditions without requiring supervised or a-priori human interference.

References 60

1. 1.

   Krakauer JW, Ghazanfar AA, Gomez-Marin A, MacIver MA, Poeppel D. Neuroscience needs behavior: correcting a reductionist bias. Neuron. 2017;93:480-490.

   Not in our library10.1016/j.neuron.2016.12.041

2. 2.

   Nilsson, S. R. O. et al. Simple behavioral analysis (simba) – an open source toolkit for computer classification of complex social behaviors in experimental animals. Preprint at https://www.biorxiv.org/content/10.1101/2020.04.19.049452v2 (2020).

   Not in our library

3. 3.
   The Mouse Action Recognition System (MARS) software pipeline for automated analysis of social behaviors in mice · 2021

   No notice
4. 4.
   DeepEthogram, a machine learning pipeline for supervised behavior classification from raw pixels · 2021

   No notice
5. 5.

   Gomez-Marin A, Paton JJ, Kampff AR, Costa RM, Mainen ZF. Big behavioral data: psychology, ethology and the foundations of neuroscience. Nat. Neurosci. 2014;17:1455-1462.

   Not in our library10.1038/nn.3812

6. 6.

   Anderson DJ, Perona P. Toward a science of computational ethology. Neuron. 2014;84:18-31.

   Not in our library10.1016/j.neuron.2014.09.005

7. 7.

   Wiltschko AB, et al. Mapping sub-second structure in mouse behavior. Neuron. 2015;88:1121-1135.

   Not in our library10.1016/j.neuron.2015.11.031

8. 8.
   Measuring behavior across scales · 2018

   No notice
9. 9.

   Brown AEX, de Bivort B. Ethology as a physical science. Nat. Phys. 2018;14:653-657.

   Not in our library10.1038/s41567-018-0093-0

10. 10.

    Datta SR, Anderson DJ, Branson K, Perona P, Leifer A. Computational neuroethology: a call to action. Neuron. 2019;104:11-24.

    Not in our library10.1016/j.neuron.2019.09.038

11. 11.
    Mapping the stereotyped behaviour of freely moving fruit flies · 2014

    No notice
12. 12.
    B-SOiD, an open-source unsupervised algorithm for identification and fast prediction of behaviors · 2021

    No notice
13. 13.

    Mathis A, et al. DeepLabCut: markerless pose estimation of user-defined body parts with deep learning. Nat. Neurosci. 2018;21:1281-1289.

    Not in our library10.1038/s41593-018-0209-y

14. 14.
    SLEAP: A deep learning system for multi-animal pose tracking · 2022

    Corrected
15. 15.
    DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning · 2019

    No notice
16. 16.

    Shi C, et al. Learning disentangled behavior embeddings. Neural Inf. Process. Syst. 2021;34:22562-22573.

    Not in our library

17. 17.

    Kingma, D. P. & Welling, M. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, ICLR (2014).

    Not in our library

18. 18.

    Rezende, D. J., Mohamed, S. & Wierstra, D. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st International Conference on Machine Learning, vol. 31, 1278–1286 (2014).

    Not in our library

19. 19.

    Bengio Y, Courville A, Vincent P. Representation learning: a review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 2013;35:1798-1828.

    Not in our library10.1109/tpami.2013.50

20. 20.

    Kuehne H, Richard A, Gall J. A hybrid rnn-hmm approach for weakly supervised temporal action segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 2020;42:765-779.

    Not in our library10.1109/tpami.2018.2884469

21. 21.

    Chung J, et al. A recurrent latent variable model for sequential data. Adv. Neural Inf. Process. Syst. 2015;28:2980-2988.

    Not in our library

22. 22.

    Chen, X. et al. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems, vol. 29, 2172–2180 (2016).

    Not in our library

23. 23.

    Higgins, I. et al. beta-vae: Learning basic visual concepts with a constrained variational framework. In 5th International Conference on Learning Representations, ICLR (2017).

    Not in our library

24. 24.

    Jiang, Z., Zheng, Y., Tan, H., Tang, B. & Zhou, H. Variational deep embedding: an unsupervised and generative approach to clustering. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, vol. 26, 1965–1972 (2017).

    Not in our library

25. 25.

    Pereira, J. & Silveira, M. Learning representations from healthcare time series data for unsupervised anomaly detection. In 2019 IEEE International Conference on Big Data and Smart Computing (BigComp), 1–7 (2019).

    Not in our library

26. 26.

    Ma, Q., Zheng, J., Li, S. & Cottrell, G. W. Learning representations for time series clustering. In Advances in Neural Information Processing Systems, vol. 32, 3776–3786 (2019).

    Not in our library

27. 27.

    Jankowsky JL, et al. Mutant presenilins specifically elevate the levels of the 42 residue beta-amyloid peptide in vivo: evidence for augmentation of a 42-specific gamma secretase. Hum. Mol. Genet. 2004;13:159-170.

    Not in our library10.1093/hmg/ddh019

28. 28.

    Schuster M, Paliwal KK. Bidirectional recurrent neural networks. IEEE Trans. Signal Process. 1997;45:2673-2681.

    Not in our library10.1109/78.650093

29. 29.

    Cho, K. et al. Learning phrase representations using rnn encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, 1724–1734 (2014).

    Not in our library

30. 30.

    Srivastava, N., Mansimov, E. & Salakhudinov, R. Unsupervised learning of video representations using lstms. In Proceedings of the 32nd International Conference on Machine Learning, vol. 37, 843–852 (2015).

    Not in our library

31. 31.

    Rabiner L, Juang B-H. An introduction to hidden Markov models. IEEE ASSP Mag. 1986;3:4-16.

    Not in our library10.1109/massp.1986.1165342

32. 32.

    Huang H, et al. Characterization of AD-like phenotype in aged APPSwe/PS1dE9 mice. Age (Dordr., Neth.). 2016;38:303-322.

    Not in our library10.1007/s11357-016-9929-7

33. 33.

    Onos KD, et al. Enhancing face validity of mouse models of Alzheimer’s disease with natural genetic variation. PLoS Genet. 2019;15:e1008155.

    Not in our library10.1371/journal.pgen.1008155

34. 34.

    Lalonde R, Kim HD, Fukuchi K. Exploratory activity, anxiety, and motor coordination in bigenic APPswe + PS1/DeltaE9 mice. Neurosci. Lett. 2004;369:156-161.

    Not in our library10.1016/j.neulet.2004.07.069

35. 35.

    Janus C, Flores AY, Xu G, Borchelt DR. Behavioral abnormalities in APPSwe/PS1dE9 mouse model of AD-like pathology: comparative analysis across multiple behavioral domains. Neurobiol. Aging. 2015;36:2519-2532.

    Not in our library10.1016/j.neurobiolaging.2015.05.010

36. 36.

    Giovannetti, E. A. et al. Restoring memory by optogenetic synchronization of hippocampal oscillations in an alzheimer’s disease mouse model. Preprint at https://www.biorxiv.org/content/10.1101/363820v1 (2018).

    Not in our library

37. 37.
    Comprehensive behavioral characterization of an APP/PS-1 double knock-in mouse model of Alzheimer's disease · 2013

    No notice
38. 38.

    Biallosterski BT, et al. Changes in voiding behavior in a mouse model of alzheimer’s disease. Front. Aging Neurosci. 2015;7:160.

    Not in our library10.3389/fnagi.2015.00160

39. 39.

    Chaudhuri R, Gerçek B, Pandey B, Peyrache A, Fiete I. The intrinsic attractor manifold and population dynamics of a canonical cognitive circuit across waking and sleep. Nat. Neurosci. 2019;22:1512-1520.

    Not in our library10.1038/s41593-019-0460-x

40. 40.
    Revealing neural correlates of behavior without behavioral measurements · 2019

    No notice

Cited by papers we hold 3

• Keypoint-MoSeq: parsing behavior by linking point tracking to pose dynamics · 2024
  No notice
• The Mouse Action Recognition System (MARS) software pipeline for automated analysis of social behaviors in mice · 2021
  No notice
• B-SOiD, an open-source unsupervised algorithm for identification and fast prediction of behaviors · 2021
  No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.