Real-time, low-latency closed-loop feedback using markerless posture tracking

Abstract

The ability to control a behavioral task or stimulate neural activity based on animal behavior in real-time is an important tool for experimental neuroscientists. Ideally, such tools are noninvasive, low-latency, and provide interfaces to trigger external hardware based on posture. Recent advances in pose estimation with deep learning allows researchers to train deep neural networks to accurately quantify a wide variety of animal behaviors. Here, we provide a new <monospace>DeepLabCut-Live!</monospace> package that achieves low-latency real-time pose estimation (within 15 ms, >100 FPS), with an additional forward-prediction module that achieves zero-latency feedback, and a dynamic-cropping mode that allows for higher inference speeds. We also provide three options for using this tool with ease: (1) a stand-alone GUI (called <monospace>DLC-Live! GUI</monospace>), and integration into (2) <monospace>Bonsai,</monospace> and (3) <monospace>AutoPilot</monospace>. Lastly, we benchmarked performance on a wide range of systems so that experimentalists can easily decide what hardware is required for their needs.

References 54

1. 1.

   Abadi M, Barham P, Chen J, Davis A, Dean J, Devin M, Ghemawat S, Irving G, Isard M, Chen Z. Tensorflow: a system for large-scale machine learning. 2016;265-283.

   Not in our library

2. 2.

   Alted F, Haenel V, Team BD. Github. 2020.

   Not in our library

3. 3.

   Andriluka M, Iqbal U, Insafutdinov E, Pishchulin L, Milan A, Gall J, Schiele B. Posetrack: a benchmark for human pose estimation and tracking. 2018;5167-5176.

   Not in our library10.1109/cvpr.2018.00542

4. 4.
   Automated markerless pose estimation in freely moving macaques with OpenMonkeyStudio · 2020

   No notice
5. 5.

   Bazarevsky V, Kartynnik Y, Vakunov A, Raveendran K, Grundmann M. Blazeface: sub-millisecond neural face detection on mobile gpus. arXiv. 2019.

   Not in our library

6. 6.

   Bradski G. Dr. Dobb’s Journal of Software Tools. 2000.

   Not in our library

7. 7.

   Cao Z, Simon T, Wei S, Sheikh Y. Realtime multi-person 2d pose estimation using part affinity fields. 2017;1302-1310.

   Not in our library10.1109/cvpr.2017.143

8. 8.

   Chen L, Ai H, Chen R, Zhuang Z, Liu S. Cross-view tracking for multi-human 3d pose estimation at over 100 fps. 2020;3276-3285.

   Not in our library10.1109/cvpr42600.2020.00334

9. 9.

   Chettih SN, Harvey CD. Single-neuron perturbations reveal feature-specific competition in V1. Nature. 2019;567:334-340.

   Not in our library10.1038/s41586-019-0997-6

10. 10.

    Datta SR, Anderson DJ, Branson K, Perona P, Leifer A. Computational neuroethology: a call to action. Neuron. 2019;104:11-24.

    Not in our library10.1016/j.neuron.2019.09.038

11. 11.

    Findley TM, Wyrick DG, Cramer JL, Brown MA, Holcomb B, Attey R, Yeh D, Monasevitch E, Nouboussi N, Cullen I, Songco J, King JF, Ahmadian Y, Smear MC. Sniff-synchronized. gradient-guided olfactory search by freely moving mice. bioRxiv. 2020.

    Not in our library10.1101/2020.04.29.069252

12. 12.

    Forys BJ, Xiao D, Gupta P, Murphy TH. Real-Time selective markerless tracking of forepaws of head fixed mice using deep neural networks. Eneuro. 2020;7:ENEURO.0096-20.2020.

    Not in our library10.1523/eneuro.0096-20.2020

13. 13.
    DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning · 2019

    No notice
14. 14.

    Hill P, Jain A, Hill M, Zamirai B, Hsu CH, Laurenzano M, Mahlke SA, Tang L, Mars J. DeftNN: addressing bottlenecks for dnn execution on gpus via synapse vector elimination and near-compute data fission. 2017;786-799.

    Not in our library10.1145/3123939.3123970

15. 15.

    Insafutdinov E, Pishchulin L, Andres B, Andriluka M, Schiele B. DeeperCut: a deeper, stronger, and faster multi-person pose estimation model. 2016;34-50.

    Not in our library10.1007/978-3-319-46466-4_3

16. 16.

    Jazayeri M, Afraz A. Navigating the neural space in search of the neural code. Neuron. 2017;93:1003-1014.

    Not in our library10.1016/j.neuron.2017.02.019

17. 17.

    Jung AB, Wada K, Crall J, Tanaka S, Graving J, Reinders C, Yadav S, Banerjee J, Vecsei G, Kraft A, Rui Z, Borovec J, Vallentin C, Zhydenko S, Pfeiffer K, Cook B, Fernández I, De Rainville FM, Weng CH, Ayala-Acevedo A. Github. 2020.

    Not in our library

18. 18.

    Kane AG. Software Heritage. 2020.

    Not in our library

19. 19.

    Kane G, Mathis MW. Zenodo. 2019.

    Not in our library10.5281/zenodo.3360725

20. 20.

    Kim CK, Adhikari A, Deisseroth K. Integration of optogenetics with complementary methodologies in systems neuroscience. Nature Reviews Neuroscience. 2017;18:222-235.

    Not in our library10.1038/nrn.2017.15

21. 21.

    Krakauer JW, Ghazanfar AA, Gomez-Marin A, MacIver MA, Poeppel D. Neuroscience needs behavior: correcting a reductionist Bias. Neuron. 2017;93:480-490.

    Not in our library10.1016/j.neuron.2016.12.041

22. 22.

    Kreiss S, Bertoni L, Alahi A. Pifpaf: composite fields for human pose estimation. 2019;11977-11986.

    Not in our library10.1109/cvpr.2019.01225

23. 23.

    Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL. Microsoft coco: common objects in context. 2014;740-755.

    Not in our library

24. 24.
    Bonsai: an event-based framework for processing and controlling data streams · 2015

    No notice
25. 25.

    Lopes G, Farrell K, Horrocks EAB, Lee CY, Morimoto MM, Muzzu T, Papanikolaou A, Rodrigues FR, Wheatcroft T, Zucca S, Solomon SG, Saleem AB. BonVision – an open-source software to create and control visual environments. bioRxiv. 2020.

    Not in our library10.1101/2020.03.09.983775

26. 26.

    Mathis MW, Mathis A, Uchida N. Somatosensory cortex plays an essential role in forelimb motor adaptation in mice. Neuron. 2017;93:1493-1503.

    Not in our library10.1016/j.neuron.2017.02.049

27. 27.

    Mathis A, Mamidanna P, Cury KM, Abe T, Murthy VN, Mathis MW, Bethge M. Zenodo. 2018a.

    Not in our library10.5281/zenodo.4008504

28. 28.

    Mathis A, Mamidanna P, Cury KM, Abe T, Murthy VN, Mathis MW, Bethge M. DeepLabCut: markerless pose estimation of user-defined body parts with deep learning. Nature Neuroscience. 2018b;21:1281-1289.

    Not in our library10.1038/s41593-018-0209-y

29. 29.

    Mathis A, Biasi T, Schneider S, Yüksekgönül M, Rogers B, Bethge M, Mathis MW. Pretraining boosts out-of-domain robustness for pose estimation. arXiv. 2020a.

    Not in our library

30. 30.

    Mathis A, Schneider S, Lauer J, Mathis MW. A primer on motion capture with deep learning: principles, pitfalls, and perspectives. Neuron. 2020b;108:44-65.

    Not in our library10.1016/j.neuron.2020.09.017

31. 31.

    Mathis MW, Mathis A. Deep learning tools for the measurement of animal behavior in neuroscience. Current Opinion in Neurobiology. 2020;60:1-11.

    Not in our library10.1016/j.conb.2019.10.008

32. 32.

    Mathis A, Warren RA. On the inference speed and video-compression robustness of deeplabcut. bioRxiv. 2018.

    Not in our library10.1101/457242

33. 33.

    McKerns MM, Strand L, Sullivan T, Fang A, Aivazis MAG. Building a framework for predictive science, CoRR. arXiv. 2012.

    Not in our library

34. 34.

    McKinney W, van der Walt S, Millman J. Data structures for statistical computing in Python. 2010;56-61.

    Not in our library

35. 35.

    Nath T, Mathis A, Chen AC, Patel A, Bethge M, Mathis MW. Using DeepLabCut for 3D markerless pose estimation across species and behaviors. Nature Protocols. 2019;14:2152-2176.

    Not in our library10.1038/s41596-019-0176-0

36. 36.

    Newell A, Yang K, Deng J. Stacked hourglass networks for human pose estimation. 2016;483-499.

    Not in our library

37. 37.

    Ning G, Pei J, Huang H. Lighttrack: a generic framework for online top-down human pose tracking. 2020;1034-1035.

    Not in our library10.1109/cvprw50498.2020.00525

38. 38.

    Pereira TD, Aldarondo DE, Willmore L, Kislin M, Wang SS, Murthy M, Shaevitz JW. Fast animal pose estimation using deep neural networks. Nature Methods. 2019;16:117-125.

    Not in our library10.1038/s41592-018-0234-5

39. 39.

    Privitera M, Ferrari KD, von Ziegler LM, Sturman O, Duss SN, Floriou-Servou A, Germain PL, Vermeiren Y, Wyss MT, De Deyn PP, Weber B, Bohacek J. A complete pupillometry toolbox for real-time monitoring of locus coeruleus activity in rodents. Nature Protocols. 2020;15:2301-2320.

    Not in our library10.1038/s41596-020-0324-6

40. 40.

    Sahasrabuddhe K, Khan AA, Singh AP, Stern TM, Ng Y, Tadić A, Orel P, LaReau C, Pouzzner D, Nishimura K, Boergens KM, Shivakumar S, Hopper MS, Kerr B, Hanna MES, Edgington RJ, McNamara I, Fell D, Gao P. The argo: a 65,536 channel recording system for high density neural recording in vivo. bioRxiv. 2020.

    Not in our library10.1101/2020.07.17.209403

Cited by papers we hold 4

• Facemap: a framework for modeling neural activity based on orofacial tracking · 2023
  No notice
• SLEAP: A deep learning system for multi-animal pose tracking · 2022
  Corrected
• Multi-animal pose estimation, identification and tracking with DeepLabCut · 2022
  No notice
• Anipose: A toolkit for robust markerless 3D pose estimation · 2021
  No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.