Multi-camera real-time three-dimensional tracking of multiple flying animals

Abstract

Automated tracking of animal movement allows analyses that would not otherwise be possible by providing great quantities of data. The additional capability of tracking in real time—with minimal latency—opens up the experimental possibility of manipulating sensory feedback, thus allowing detailed explorations of the neural basis for control of behaviour. Here, we describe a system capable of tracking the three-dimensional position and body orientation of animals such as flies and birds. The system operates with less than 40 ms latency and can track multiple animals simultaneously. To achieve these results, a multi-target tracking algorithm was developed based on the extended Kalman filter and the nearest neighbour standard filter data association algorithm. In one implementation, an 11-camera system is capable of tracking three flies simultaneously at 60 frames per second using a gigabit network of nine standard Intel Pentium 4 and Core 2 Duo computers. This manuscript presents the rationale and details of the algorithms employed and shows three implementations of the system. An experiment was performed using the tracking system to measure the effect of visual contrast on the flight speed of Drosophila melanogaster. At low contrasts, speed is more variable and faster on average than at high contrasts. Thus, the system is already a useful tool to study the neurobiology and behaviour of freely flying animals. If combined with other techniques, such as ‘virtual reality’-type computer graphics or genetic manipulation, the tracking system would offer a powerful new way to investigate the biology of flying animals.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 64

1. 1.

   Srinivasan M. V. Zhang S. W. Lehrer M. Collett T. S. 1996 Honeybee navigation en route to the goal: visual flight control and odometry. J. Exp. Biol. 199, 237–244

   Not in our library10.1242/jeb.199.1.237

2. 2.

   Schilstra C. Hateren J. H. 1999 Blowfly flight and optic flow. I. Thorax kinematics and flight dynamics. J. Exp. Biol. 202, 1481–1490

   Not in our library10.1242/jeb.202.11.1481

3. 3.

   Tammero L. F. Dickinson M. H. 2002 The influence of visual landscape on the free flight behavior of the fruit fly Drosophila melanogaster. J. Exp. Biol. 205, 327–343

   Not in our library10.1242/jeb.205.3.327

4. 4.

   Kern R. van Hateren J. H. Michaelis C. Lindemann J. P. Egelhaaf M. 2005 Function of a fly motion-sensitive neuron matches eye movements during free flight. PLoS Biol. 3, 1130–1138 (doi:10.1371/journal.pbio.0030171)

   Not in our library10.1371/journal.pbio.0030171

5. 5.

   Land M. F. Collett T. S. 1974 Chasing behavior of houseflies (Fannia canicularis): a description and analysis. J. Comp. Physiol. 89, 331–357 (doi:10.1007/BF00695351)

   Not in our library10.1007/bf00695351

6. 6.

   Buelthoff H. Poggio T. Wehrhahn C. 1980 3-D analysis of the flight trajectories of flies (Drosophila melanogaster). Z. Nat. 35c, 811–815

   Not in our library

7. 7.

   Wehrhahn C. Poggio T. Bülthoff H. 1982 Tracking and chasing in houseflies (musca). Biol. Cybernet. 45, 123–130 (doi:10.1007/BF00335239)

   Not in our library10.1007/bf00335239

8. 8.

   Wagner H. 1986 Flight performance and visual control of the flight of the free-flying housefly (Musca domestica). II. Pursuit of targets. Phil. Trans. R. Soc. Lond. B 312, 553–579 (doi:10.1098/rstb.1986.0018)

   Not in our library10.1098/rstb.1986.0018

9. 9.

   Mizutani A. Chahl J. S. Srinivasan M. V. 2003 Insect behaviour: motion camouflage in dragonflies. Nature 423, 604 (doi:10.1038/423604a)

   Not in our library10.1038/423604a

10. 10.

    Frye M. A. Tarsitano M. Dickinson M. H. 2003 Odor localization requires visual feedback during free flight in Drosophila melanogaster. J. Exp. Biol. 206, 843–855 (doi:10.1242/jeb.00175)

    Not in our library10.1242/jeb.00175

11. 11.

    Budick S. A. Dickinson M. H. 2006 Free-flight responses of Drosophila melanogaster to attractive odors. J. Exp. Biol. 209, 3001–3017 (doi:10.1242/jeb.02305)

    Not in our library10.1242/jeb.02305

12. 12.

    David C. T. 1978 Relationship between body angle and flight speed in free-flying Drosophila. Physiol. Entomol. 3, 191–195 (doi:10.1111/j.1365-3032.1978.tb00148.x)

    Not in our library10.1111/j.1365-3032.1978.tb00148.x

13. 13.

    Fry S. N. Sayaman R. Dickinson M. H. 2003 The aerodynamics of free-flight maneuvers in Drosophila. Science 300, 495–498 (doi:10.1126/science.1081944)

    Not in our library10.1126/science.1081944

14. 14.

    Collett T. S. Land M. F. 1975 Visual control of flight behaviour in the hoverfly Syritta pipiens L. J. Comp. Physiol. A Neuroethol. Sens. Neural Behav. Physiol. 99, 1–66 (doi:10.1007/BF01464710)

    Not in our library10.1007/bf01464710

15. 15.

    Collett T. S. Land M. F. 1978 How hoverflies compute interception courses. J. Comp. Physiol. A Neuroethol. Sens. Neural Behav. Physiol. 125, 191–204 (doi:10.1007/BF00656597)

    Not in our library10.1007/bf00656597

16. 16.

    Wehrhahn C. 1979 Sex-specific differences in the chasing behaviour of houseflies (musca). Biol. Cybernet. 32, 239–241 (doi:10.1007/BF00337647)

    Not in our library10.1007/bf00337647

17. 17.

    Dahmen H.-J. Zeil J. 1984 Recording and reconstructing three-dimensional trajectories: a versatile method for the field biologist. Proc. R. Soc. Lond. B 222, 107–113 (doi:10.1098/rspb.1984.0051)

    Not in our library10.1098/rspb.1984.0051

18. 18.

    Srinivasan M. V. Zhang S. W. Chahl J. S. Barth E. Venkatesh S. 2000 How honeybees make grazing landings on flat surfaces. Biol. Cybernet. 83, 171–183 (doi:10.1007/s004220000162)

    Not in our library10.1007/s004220000162

19. 19.

    Hedrick T. L. Biewener A. A. 2007 Low speed maneuvering flight of the rose-breasted cockatoo (Eolophus roseicapillus). I. Kinematic and neuromuscular control of turning. J. Exp. Biol. 210, 1897–1911 (doi:10.1242/jeb.002055)

    Not in our library10.1242/jeb.002055

20. 20.

    Tian X. Diaz J. I. Middleton K. Galvao R. Israeli E. Roemer A. Sullivan A. Song A. Swartz S. Breuer K. 2006 Direct measurements of the kinematics and dynamics of bat flight. Bioinspiration & Biomimetics 1, S10–S18 (doi:10.1088/1748-3182/1/4/S02)

    Not in our library10.1088/1748-3182/1/4/s02

21. 21.

    Khan Z. Balch T. Dellaert F. 2005 MCMC-based particle filtering for tracking a variable number of interacting targets. IEEE Trans. Pattern Anal. Mach. Intell. 27, 2005

    Not in our library10.1109/tpami.2005.223

22. 22.

    Khan Z. Balch T. Dellaert F. 2006 MCMC data association and sparse factorization updating for real time multitarget tracking with merged and multiple measurements. IEEE Trans. Pattern Anal. Mach. Intell. 28, 1960–1972 (doi:10.1109/TPAMI.2006.247)

    Not in our library10.1109/tpami.2006.247

23. 23.

    Branson K. Robie A. Bender J. Perona P. Dickinson M. H. 2009 High-throughput ethomics in large groups of Drosophila. Nat. Methods 6, 451–457 (doi:10.1038/nmeth.1328)

    Not in our library10.1038/nmeth.1328

24. 24.

    Klein D. J. 2008 Coordinated control and estimation for multi-agent systems: theory and practice, ch. 6. Tracking Multiple Fish Robots Using Underwater Cameras, PhD thesis, University of Washington

    Not in our library

25. 25.

    Qu W. Schonfeld D. Mohamed M. 2007 Distributed Bayesian multiple-target tracking in crowded environments using multiple collaborative cameras. EURASIP J. Adv. Signal Process. Article ID 38373. (doi:10.1155/2007/38373)

    Not in our library10.1155/2007/38373

26. 26.

    Qu W. Schonfeld D. Mohamed M. 2007 Real-time interactively distributed multi-object tracking using a magnetic-inertia potential model. IEEE Trans. Multimedia (TMM) 9, 511–519 (doi:10.1109/TMM.2006.886266)

    Not in our library10.1109/tmm.2006.886266

27. 27.

    Ballerini M. 2008 Empirical investigation of starling flocks: a benchmark study in collective animal behaviour. Anim. Behav. 76, 201–215 (doi:10.1016/j.anbehav.2008.02.004)

    Not in our library10.1016/j.anbehav.2008.02.004

28. 28.

    Cavagna A. Cimarelli A. Giardina I. Orlandi A. Parisi G. Procaccini A. Santagati R. Stefanini F. 2008 New statistical tools for analyzing the structure of animal groups. Math. Biosci. 214, 32–37 (doi:10.1016/j.mbs.2008.05.006)

    Not in our library10.1016/j.mbs.2008.05.006

29. 29.

    Cavagna A. Giardina I. Orlandi A. Parisi G. Procaccini A. 2008 The starflag handbook on collective animal behaviour: 2. Three-dimensional analysis. Anim. Behav. 76, 237–248 (doi:10.1016/j.anbehav.2008.02.003)

    Not in our library10.1016/j.anbehav.2008.02.003

30. 30.

    Cavagna A. Giardina I. Orlandi A. Parisi G. Procaccini A. Viale M. Zdravkovic V. 2008 The starflag handbook on collective animal behaviour: 1. Empirical methods. Anim. Behav. 76, 217–236 (doi:10.1016/j.anbehav.2008.02.002)

    Not in our library10.1016/j.anbehav.2008.02.002

31. 31.

    Wu H. Zhao Q. Zou D. Chen Y. 2009 Acquiring 3d motion trajectories of large numbers of swarming animals. IEEE Int. Conf. on Computer Vision (ICCV) Workshop on Video Oriented Object and Event Classification.

    Not in our library

32. 32.

    Zou D. Zhao Q. Wu H. Chen Y. 2009 Reconstructing 3d motion trajectories of particle swarms by global correspondence selection. In Int. Conf. on Computer Vision (ICCV 09). Workshop on Video Oriented Object and Event Classification, pp. 1578–1585

    Not in our library

33. 33.

    Bomphrey R. J. Walker S. M. Taylor G. K. 2009 The typical flight performance of blowflies: measuring the normal performance envelope of Calliphora vicina using a novel corner-cube arena. PLoS ONE 4, e7852 (doi:10.1371/journal.pone.0007852)

    Not in our library10.1371/journal.pone.0007852

34. 34.

    Marden J. H. Wolf M. R. Weber K. E. 1997 Aerial performance of Drosophila melanogaster from populations selected for upwind flight ability. J. Exp. Biol. 200, 2747–2755

    Not in our library10.1242/jeb.200.21.2747

35. 35.

    Fry S. N. Muller P. Baumann H. J. Straw A. D. Bichsel M. Robert D. 2004 Context-dependent stimulus presentation to freely moving animals in 3d. J. Neurosci. Methods 135, 149–157 (doi:10.1016/j.jneumeth.2003.12.012)

    Not in our library10.1016/j.jneumeth.2003.12.012

36. 36.

    Fry S. N. Rohrseitz N. Straw A. D. Dickinson M. H. 2008 Trackfly-virtual reality for a behavioral system analysis in free-flying fruit flies. J. Neurosci. Methods 171, 110–117 (doi:10.1016/j.jneumeth.2008.02.016)

    Not in our library10.1016/j.jneumeth.2008.02.016

37. 37.

    Grover D. Tower J. Tavare S. 2008 O fly, where art thou? J. R. Soc. Interface 5, 1181–1191 (doi:10.1098/rsif.2007.1333)

    Not in our library10.1098/rsif.2007.1333

38. 38.

    Maimon G. Straw A. D. Dickinson M. H. 2008 A simple vision-based algorithm for decision making in flying Drosophila. Curr. Biol. 18, 464–470 (doi:10.1016/j.cub.2008.03.050)

    Not in our library10.1016/j.cub.2008.03.050

39. 39.

    Piccardi M. 2004 Background subtraction techniques: a review. In Proc. of IEEE SMC 2004 Int. Conf. on Systems, Man and Cybernetics. The Hague, The Netherlands, October 2004.

    Not in our library

40. 40.

    Bar-Shalom Y. Fortmann T. E. 1988 Tracking and data association. New York, NY: Academic Press

    Not in our library

41. 41.

    Abdel-Aziz Y. I. Karara H. M. 1971 Direct linear transformation from comparator coordinates into object space coordinates. Proc. American Society of Photogrammetry Symp. on Close-Range Photogrammetry, Urbana, IL, 26–29 January 1971, pp. 1–18 Falls Church, VA: American Society of Photogrammetry

    Not in our library

42. 42.

    Svoboda T. Martinec D. Pajdla T. 2005 A convenient multi-camera self-calibration for virtual environments. PRESENCE: Teleoperators and Virtual Environments 14, 407–422 (doi:10.1162/105474605774785325)

    Not in our library10.1162/105474605774785325

43. 43.

    Sturm P. Triggs B. 1996 A factorization based algorithm for multi-image projective structure and motion. In Proc. 4th Eur. Conf. on Computer Vision, April 1996, Cambridge, UK, pp. 709–720 Springer

    Not in our library

44. 44.

    Hartley R. I. Zisserman A. 2003 Multiple view geometry in computer vision, 2nd edn Cambridge, UK: Cambridge University Press

    Not in our library

45. 45.

    Bouguet J. 2010 Camera calibration toolbox. See http://www.vision.caltech.edu/bouguetj/calib_doc.

    Not in our library

46. 46.

    Prescott B. McLean G. F. 1997 Line-based correction of radial lens distortion. Graph. Models Image Process. 59, 39–47 (doi:10.1006/gmip.1996.0407)

    Not in our library10.1006/gmip.1996.0407

47. 47.

    Straw A. D. Dickinson M. H. 2009 Motmot, an open-source toolkit for realtime video acquisition and analysis. Source Code Biol. Med. 4, 5, 1–20 (doi:10.1186/1751-0473-4-5)

    Not in our library10.1186/1751-0473-4-5

48. 48.

    Correll K. Barendt N. Branicky M. 2005 Design considerations for software-only implementations of the IEEE 1588 precision time protocol. Proc. Conf. on IEEE-1588 Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems, NIST and IEEE, Winterthur, Switzerland, 10–12 October 1975.

    Not in our library

49. 49.

    Kennedy J. S. 1939 Visual responses of flying mosquitoes. Proc. Zool. Soc. Lond. 109, 221–242

    Not in our library

50. 50.

    Neumann T. R. 2002 Modeling insect compound eyes: space-variant spherical vision. In Proc. of the 2nd Int. Workshop on Biologically Motivated Computer Vision (eds Bülthoff H. H. Lee S.-W. Poggio T. Wallraven C. ), LNCS 2525, pp. 360–367 Berlin, Germany: Springer

    Not in our library

51. 51.

    Dickson W. B. Straw A. D. Dickinson M. H. 2008 Integrative model of drosophila flight. AIAA J. 46, 2150–2165 (doi:10.2514/1.29862)

    Not in our library10.2514/1.29862

52. 52.

    Strauss R. Schuster S. Götz K. G. 1997 Processing of artificial visual feedback in the walking fruit fly Drosophila melanogaster. J. Exp. Biol. 200, 1281–1296

    Not in our library10.1242/jeb.200.9.1281

53. 53.

    Straw A. D. 2008 Vision egg: an open-source library for realtime visual stimulus generation. Front. Neuroinformatics 2, 1–10 (doi:10.3389/neuro.11.004.2008)

    Not in our library10.3389/neuro.11.004.2008

54. 54.

    Stamps J. Buechner M. Alexander K. Davis J. Zuniga N. 2005 Genotypic differences in space use and movement patterns in Drosophila melanogaster. Anim. Behav. 70, 609–618 (doi:10.1016/j.anbehav.2004.11.018)

    Not in our library10.1016/j.anbehav.2004.11.018

55. 55.

    Dvorak D. Srinivasan M. V. French A. S. 1980 The contrast sensitivity of fly movement-detecting neurons. Vision Res. 20, 397–407 (doi:10.1016/0042-6989(80)90030-9)

    Not in our library10.1016/0042-6989(80)90030-9

56. 56.

    Srinivasan M. V. Dvorak D. R. 1980 Spatial processing of visual information in the movement-detecting pathway of the fly: characteristics and functional significance. J. Comp. Physiol. A 140, 1–23 (doi:10.1007/BF00613743)

    Not in our library10.1007/bf00613743

57. 57.

    Egelhaaf M. Borst A. 1989 Transient and steady-state response properties of movement detectors. J. Opt. Soc. Am. A 6, 116–127 (doi:10.1364/JOSAA.6.000116)

    Not in our library10.1364/josaa.6.000116

58. 58.

    Hassenstein B. Reichardt W. 1956 Systemtheoretische analyse der zeit-, reihenfolgen- und vorzeichenauswertung bei der bewegungsperzeption des rüsselkafers Chlorophanus. Z. Nat. 11b, 513–524

    Not in our library

59. 59.

    David C. T. 1982 Compensation for height in the control of groundspeed by drosophila in a new, barbers pole wind-tunnel. J. Comp. Physiol. 147, 485–493 (doi:10.1007/BF00612014)

    Not in our library10.1007/bf00612014

60. 60.

    Srinivasan M. V. Lehrer M. Kirchner W. H. Zhang S. W. 1991 Range perception through apparent image speed in freely flying honeybees. Vis. Neurosci. 6, 519–535 (doi:10.1017/S095252380000136X)

    Not in our library10.1017/s095252380000136x

61. 61.

    Srinivasan M. V. Zhang S. W. Chandrashekara K. 1993 Evidence for two distinct movement detecting mechanisms inc insect vision. Naturwissenschaften 80, 38–41 (doi:10.1007/BF01139758)

    Not in our library10.1007/bf01139758

62. 62.

    Si A. Srinivasan M. V. Zhang S. 2003 Honeybee navigation: properties of the visually driven ‘odometer’. J. Exp. Biol. 206, 1265–1273 (doi:10.1242/jeb.00236)

    Not in our library10.1242/jeb.00236

63. 63.

    Baird E. Srinivasan M. V. Zhang S. Cowling A. 2005 Visual control of flight speed in honeybees. J. Exp. Biol. 208, 3895–3905 (doi:10.1242/jeb.01818)

    Not in our library10.1242/jeb.01818

64. 64.

    Harris R. A. O'Carroll D. C. Laughlin S. B. 2000 Contrast gain reduction in fly motion adaptation. Neuron 28, 595–606 (doi:10.1016/S0896-6273(00)00136-7)

    Not in our library10.1016/s0896-6273(00)00136-7

Cited by papers we hold 3

• Understanding how animal groups achieve coordinated movement · 2016
  No notice
• The Dilemmas of the Gourmet Fly: The Molecular and Neuronal Mechanisms of Feeding and Nutrient Decision Making in Drosophila · 2013
  No notice
• Deciphering Interactions in Moving Animal Groups · 2012
  No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.