Multi-camera real-time three-dimensional tracking of multiple flying animals

Abstract

Automated tracking of animal movement allows analyses that would not otherwise be possible by providing great quantities of data. The additional capability of tracking in real time—with minimal latency—opens up the experimental possibility of manipulating sensory feedback, thus allowing detailed explorations of the neural basis for control of behaviour. Here, we describe a system capable of tracking the three-dimensional position and body orientation of animals such as flies and birds. The system operates with less than 40 ms latency and can track multiple animals simultaneously. To achieve these results, a multi-target tracking algorithm was developed based on the extended Kalman filter and the nearest neighbour standard filter data association algorithm. In one implementation, an 11-camera system is capable of tracking three flies simultaneously at 60 frames per second using a gigabit network of nine standard Intel Pentium 4 and Core 2 Duo computers. This manuscript presents the rationale and details of the algorithms employed and shows three implementations of the system. An experiment was performed using the tracking system to measure the effect of visual contrast on the flight speed of Drosophila melanogaster. At low contrasts, speed is more variable and faster on average than at high contrasts. Thus, the system is already a useful tool to study the neurobiology and behaviour of freely flying animals. If combined with other techniques, such as ‘virtual reality’-type computer graphics or genetic manipulation, the tracking system would offer a powerful new way to investigate the biology of flying animals.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 64

1. 1.

   Srinivasan M. V. Zhang S. W. Lehrer M. Collett T. S. 1996 Honeybee navigation en route to the goal: visual flight control and odometry. J. Exp. Biol. 199, 237–244

   Not in our library10.1242/jeb.199.1.237

2. 2.

   Schilstra C. Hateren J. H. 1999 Blowfly flight and optic flow. I. Thorax kinematics and flight dynamics. J. Exp. Biol. 202, 1481–1490

   Not in our library10.1242/jeb.202.11.1481

3. 3.

   Tammero L. F. Dickinson M. H. 2002 The influence of visual landscape on the free flight behavior of the fruit fly Drosophila melanogaster. J. Exp. Biol. 205, 327–343

   Not in our library10.1242/jeb.205.3.327

4. 4.

   Kern R. van Hateren J. H. Michaelis C. Lindemann J. P. Egelhaaf M. 2005 Function of a fly motion-sensitive neuron matches eye movements during free flight. PLoS Biol. 3, 1130–1138 (doi:10.1371/journal.pbio.0030171)

   Not in our library10.1371/journal.pbio.0030171

5. 5.

   Land M. F. Collett T. S. 1974 Chasing behavior of houseflies (Fannia canicularis): a description and analysis. J. Comp. Physiol. 89, 331–357 (doi:10.1007/BF00695351)

   Not in our library10.1007/bf00695351

6. 6.

   Buelthoff H. Poggio T. Wehrhahn C. 1980 3-D analysis of the flight trajectories of flies (Drosophila melanogaster). Z. Nat. 35c, 811–815

   Not in our library

7. 7.

   Wehrhahn C. Poggio T. Bülthoff H. 1982 Tracking and chasing in houseflies (musca). Biol. Cybernet. 45, 123–130 (doi:10.1007/BF00335239)

   Not in our library10.1007/bf00335239

8. 8.

   Wagner H. 1986 Flight performance and visual control of the flight of the free-flying housefly (Musca domestica). II. Pursuit of targets. Phil. Trans. R. Soc. Lond. B 312, 553–579 (doi:10.1098/rstb.1986.0018)

   Not in our library10.1098/rstb.1986.0018

9. 9.

   Mizutani A. Chahl J. S. Srinivasan M. V. 2003 Insect behaviour: motion camouflage in dragonflies. Nature 423, 604 (doi:10.1038/423604a)

   Not in our library10.1038/423604a

10. 10.

    Frye M. A. Tarsitano M. Dickinson M. H. 2003 Odor localization requires visual feedback during free flight in Drosophila melanogaster. J. Exp. Biol. 206, 843–855 (doi:10.1242/jeb.00175)

    Not in our library10.1242/jeb.00175

11. 11.

    Budick S. A. Dickinson M. H. 2006 Free-flight responses of Drosophila melanogaster to attractive odors. J. Exp. Biol. 209, 3001–3017 (doi:10.1242/jeb.02305)

    Not in our library10.1242/jeb.02305

12. 12.

    David C. T. 1978 Relationship between body angle and flight speed in free-flying Drosophila. Physiol. Entomol. 3, 191–195 (doi:10.1111/j.1365-3032.1978.tb00148.x)

    Not in our library10.1111/j.1365-3032.1978.tb00148.x

13. 13.

    Fry S. N. Sayaman R. Dickinson M. H. 2003 The aerodynamics of free-flight maneuvers in Drosophila. Science 300, 495–498 (doi:10.1126/science.1081944)

    Not in our library10.1126/science.1081944

14. 14.

    Collett T. S. Land M. F. 1975 Visual control of flight behaviour in the hoverfly Syritta pipiens L. J. Comp. Physiol. A Neuroethol. Sens. Neural Behav. Physiol. 99, 1–66 (doi:10.1007/BF01464710)

    Not in our library10.1007/bf01464710

15. 15.

    Collett T. S. Land M. F. 1978 How hoverflies compute interception courses. J. Comp. Physiol. A Neuroethol. Sens. Neural Behav. Physiol. 125, 191–204 (doi:10.1007/BF00656597)

    Not in our library10.1007/bf00656597

16. 16.

    Wehrhahn C. 1979 Sex-specific differences in the chasing behaviour of houseflies (musca). Biol. Cybernet. 32, 239–241 (doi:10.1007/BF00337647)

    Not in our library10.1007/bf00337647

17. 17.

    Dahmen H.-J. Zeil J. 1984 Recording and reconstructing three-dimensional trajectories: a versatile method for the field biologist. Proc. R. Soc. Lond. B 222, 107–113 (doi:10.1098/rspb.1984.0051)

    Not in our library10.1098/rspb.1984.0051

18. 18.

    Srinivasan M. V. Zhang S. W. Chahl J. S. Barth E. Venkatesh S. 2000 How honeybees make grazing landings on flat surfaces. Biol. Cybernet. 83, 171–183 (doi:10.1007/s004220000162)

    Not in our library10.1007/s004220000162

19. 19.

    Hedrick T. L. Biewener A. A. 2007 Low speed maneuvering flight of the rose-breasted cockatoo (Eolophus roseicapillus). I. Kinematic and neuromuscular control of turning. J. Exp. Biol. 210, 1897–1911 (doi:10.1242/jeb.002055)

    Not in our library10.1242/jeb.002055

20. 20.

    Tian X. Diaz J. I. Middleton K. Galvao R. Israeli E. Roemer A. Sullivan A. Song A. Swartz S. Breuer K. 2006 Direct measurements of the kinematics and dynamics of bat flight. Bioinspiration & Biomimetics 1, S10–S18 (doi:10.1088/1748-3182/1/4/S02)

    Not in our library10.1088/1748-3182/1/4/s02

21. 21.

    Khan Z. Balch T. Dellaert F. 2005 MCMC-based particle filtering for tracking a variable number of interacting targets. IEEE Trans. Pattern Anal. Mach. Intell. 27, 2005

    Not in our library10.1109/tpami.2005.223

22. 22.

    Khan Z. Balch T. Dellaert F. 2006 MCMC data association and sparse factorization updating for real time multitarget tracking with merged and multiple measurements. IEEE Trans. Pattern Anal. Mach. Intell. 28, 1960–1972 (doi:10.1109/TPAMI.2006.247)

    Not in our library10.1109/tpami.2006.247

23. 23.

    Branson K. Robie A. Bender J. Perona P. Dickinson M. H. 2009 High-throughput ethomics in large groups of Drosophila. Nat. Methods 6, 451–457 (doi:10.1038/nmeth.1328)

    Not in our library10.1038/nmeth.1328

24. 24.

    Klein D. J. 2008 Coordinated control and estimation for multi-agent systems: theory and practice, ch. 6. Tracking Multiple Fish Robots Using Underwater Cameras, PhD thesis, University of Washington

    Not in our library

25. 25.

    Qu W. Schonfeld D. Mohamed M. 2007 Distributed Bayesian multiple-target tracking in crowded environments using multiple collaborative cameras. EURASIP J. Adv. Signal Process. Article ID 38373. (doi:10.1155/2007/38373)

    Not in our library10.1155/2007/38373

26. 26.

    Qu W. Schonfeld D. Mohamed M. 2007 Real-time interactively distributed multi-object tracking using a magnetic-inertia potential model. IEEE Trans. Multimedia (TMM) 9, 511–519 (doi:10.1109/TMM.2006.886266)

    Not in our library10.1109/tmm.2006.886266

27. 27.

    Ballerini M. 2008 Empirical investigation of starling flocks: a benchmark study in collective animal behaviour. Anim. Behav. 76, 201–215 (doi:10.1016/j.anbehav.2008.02.004)

    Not in our library10.1016/j.anbehav.2008.02.004

28. 28.

    Cavagna A. Cimarelli A. Giardina I. Orlandi A. Parisi G. Procaccini A. Santagati R. Stefanini F. 2008 New statistical tools for analyzing the structure of animal groups. Math. Biosci. 214, 32–37 (doi:10.1016/j.mbs.2008.05.006)

    Not in our library10.1016/j.mbs.2008.05.006

29. 29.

    Cavagna A. Giardina I. Orlandi A. Parisi G. Procaccini A. 2008 The starflag handbook on collective animal behaviour: 2. Three-dimensional analysis. Anim. Behav. 76, 237–248 (doi:10.1016/j.anbehav.2008.02.003)

    Not in our library10.1016/j.anbehav.2008.02.003

30. 30.

    Cavagna A. Giardina I. Orlandi A. Parisi G. Procaccini A. Viale M. Zdravkovic V. 2008 The starflag handbook on collective animal behaviour: 1. Empirical methods. Anim. Behav. 76, 217–236 (doi:10.1016/j.anbehav.2008.02.002)

    Not in our library10.1016/j.anbehav.2008.02.002

31. 31.

    Wu H. Zhao Q. Zou D. Chen Y. 2009 Acquiring 3d motion trajectories of large numbers of swarming animals. IEEE Int. Conf. on Computer Vision (ICCV) Workshop on Video Oriented Object and Event Classification.

    Not in our library

32. 32.

    Zou D. Zhao Q. Wu H. Chen Y. 2009 Reconstructing 3d motion trajectories of particle swarms by global correspondence selection. In Int. Conf. on Computer Vision (ICCV 09). Workshop on Video Oriented Object and Event Classification, pp. 1578–1585

    Not in our library

33. 33.

    Bomphrey R. J. Walker S. M. Taylor G. K. 2009 The typical flight performance of blowflies: measuring the normal performance envelope of Calliphora vicina using a novel corner-cube arena. PLoS ONE 4, e7852 (doi:10.1371/journal.pone.0007852)

    Not in our library10.1371/journal.pone.0007852

34. 34.

    Marden J. H. Wolf M. R. Weber K. E. 1997 Aerial performance of Drosophila melanogaster from populations selected for upwind flight ability. J. Exp. Biol. 200, 2747–2755

    Not in our library10.1242/jeb.200.21.2747

35. 35.

    Fry S. N. Muller P. Baumann H. J. Straw A. D. Bichsel M. Robert D. 2004 Context-dependent stimulus presentation to freely moving animals in 3d. J. Neurosci. Methods 135, 149–157 (doi:10.1016/j.jneumeth.2003.12.012)

    Not in our library10.1016/j.jneumeth.2003.12.012

36. 36.

    Fry S. N. Rohrseitz N. Straw A. D. Dickinson M. H. 2008 Trackfly-virtual reality for a behavioral system analysis in free-flying fruit flies. J. Neurosci. Methods 171, 110–117 (doi:10.1016/j.jneumeth.2008.02.016)

    Not in our library10.1016/j.jneumeth.2008.02.016

37. 37.

    Grover D. Tower J. Tavare S. 2008 O fly, where art thou? J. R. Soc. Interface 5, 1181–1191 (doi:10.1098/rsif.2007.1333)

    Not in our library10.1098/rsif.2007.1333

38. 38.

    Maimon G. Straw A. D. Dickinson M. H. 2008 A simple vision-based algorithm for decision making in flying Drosophila. Curr. Biol. 18, 464–470 (doi:10.1016/j.cub.2008.03.050)

    Not in our library10.1016/j.cub.2008.03.050

39. 39.

    Piccardi M. 2004 Background subtraction techniques: a review. In Proc. of IEEE SMC 2004 Int. Conf. on Systems, Man and Cybernetics. The Hague, The Netherlands, October 2004.

    Not in our library

40. 40.

    Bar-Shalom Y. Fortmann T. E. 1988 Tracking and data association. New York, NY: Academic Press

    Not in our library

Cited by papers we hold 3

• Understanding how animal groups achieve coordinated movement · 2016
  No notice
• The Dilemmas of the Gourmet Fly: The Molecular and Neuronal Mechanisms of Feeding and Nutrient Decision Making in Drosophila · 2013
  No notice
• Deciphering Interactions in Moving Animal Groups · 2012
  No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.