Why rankings of biomedical image analysis competitions should be interpreted with care

Abstract

International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 66

1. 1.

   Ayache N, Duncan J. 20th anniversary of the medical image analysis journal (MedIA). Med. Image Anal. 2016;33:1-3.

   Not in our library10.1016/j.media.2016.07.004

2. 2.

   Chen, W. Li, W. Dong, X. Pei, J. A review of biological image analysis. Curr. Bioinform. 13, 337–343 (2018).

   Not in our library

3. 3.

   Price K. Anything you can do, I can do better (no you can’t). Comput. Gr. Image Process. 1986;36:387-391.

   Not in our library10.1016/0734-189x(86)90083-6

4. 4.

   West J, et al. Comparison and evaluation of retrospective intermodality brain image registration techniques. J. Comput. Assist. Tomogr. 1997;21:554-568.

   Not in our library10.1097/00004728-199707000-00007

5. 5.

   Müller H, Rosset A, Vallée JP, Terrier F, Geissbuhler A. A reference data set for the evaluation of medical image retrieval systems. Comput. Med. Imaging Graph. 2004;28:295-305.

   Not in our library10.1016/j.compmedimag.2004.04.005

6. 6.

   ImageCLEF/LifeCLEF – Multimedia Retrieval in CLEF. 2004. http://www.imageclef.org/. Accessed 20 Feb 2018

   Not in our library

7. 7.

   Kalpathy-Cramer J, et al. Evaluating performance of biomedical image retrieval systems – an overview of the medical image retrieval task at ImageCLEF 2004-2013. Comput. Med. Imaging Graph. 2015;39:55-61.

   Not in our library10.1016/j.compmedimag.2014.03.004

8. 8.

   Cleverdon CW. The aslib cranfield research project on the comparative efficiency of indexing systems. Aslib Proc. 1960;12:421-431.

   Not in our library10.1108/eb049778

9. 9.

   Heimann T, et al. Comparison and evaluation of methods for liver segmentation from CT datasets. IEEE Trans. Med. Imaging. 2009;28:1251-1265.

   Not in our library10.1109/tmi.2009.2013851

10. 10.

    Chenouard N, et al. Objective comparison of particle tracking methods. Nat. Methods. 2014;11:281-289.

    Not in our library10.1038/nmeth.2808

11. 11.

    Sage Daniel, et al. Quantitative evaluation of software packages for single-molecule localization microscopy. Nat. Methods. 2015;12:717-724.

    Not in our library10.1038/nmeth.3442

12. 12.

    Menze BH, et al. The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Trans. Med. Imaging. 2015;34:1993-2024.

    Not in our library10.1109/tmi.2014.2377694

13. 13.

    Ulman V, et al. An objective comparison of cell-tracking algorithms. Nat. Methods. 2017;14:1141.

    Not in our library10.1038/nmeth.4473

14. 14.
    The challenge of mapping the human connectome based on diffusion tractography · 2017

    Corrected
15. 15.

    Setio AAA, et al. Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the LUNA16 challenge. Med. Image Anal. 2017;42:1-13.

    Not in our library10.1016/j.media.2017.06.015

16. 16.

    Zheng G, et al. Evaluation and comparison of 3D intervertebral disc localization and segmentation methods for 3D T2 MR data: a grand challenge. Med. Image Anal. 2017;35:327-344.

    Not in our library10.1016/j.media.2016.08.005

17. 17.

    Wang CW, et al. A benchmark for comparison of dental radiography analysis algorithms. Med. Image Anal. 2016;31:63-76.

    Not in our library10.1016/j.media.2016.02.004

18. 18.

    Bernal J, et al. Comparative validation of polyp detection methods in video colonoscopy: results from the MICCAI 2015 Endoscopic Vision Challenge. IEEE Trans. Med. Imaging. 2017;36:1231-1249.

    Not in our library10.1109/tmi.2017.2664042

19. 19.

    Sirinukunwattana K, et al. Gland segmentation in colon histology images: The glas challenge contest. Med. Image Anal. 2017;35:489-502.

    Not in our library10.1016/j.media.2016.08.008

20. 20.

    Maier O, et al. ISLES 2015-A public evaluation benchmark for ischemic stroke lesion segmentation from multispectral MRI. Med. Image Anal. 2017;35:250-269.

    Not in our library10.1016/j.media.2016.07.009

21. 21.

    Carass A, et al. Longitudinal multiple sclerosis lesion segmentation: resource and challenge. NeuroImage. 2017;148:77-102.

    Not in our library10.1016/j.neuroimage.2016.12.064

22. 22.

    Wang CW, et al. Evaluation and comparison of anatomical landmark detection methods for cephalometric x-ray images: a grand challenge. IEEE Trans. Med. Imaging. 2015;34:1890-1900.

    Not in our library10.1109/tmi.2015.2412951

23. 23.

    Bernard O, et al. Standardized evaluation system for left ventricular segmentation algorithms in 3D echocardiography. IEEE Trans. Med. Imaging. 2016;35:967-977.

    Not in our library10.1109/tmi.2015.2503890

24. 24.

    Bron EE, et al. Standardized evaluation of algorithms for computer-aided diagnosis of dementia based on structural MRI: the CADDementia challenge. NeuroImage. 2015;111:562-579.

    Not in our library10.1016/j.neuroimage.2015.01.048

25. 25.

    Jimenez-del-Toro O, et al. Cloud-based evaluation of anatomical structure segmentation and landmark detection algorithms: VISCERAL anatomy benchmarks. IEEE Trans. Med. Imaging. 2016;35:2459-2475.

    Not in our library10.1109/tmi.2016.2578680

26. 26.

    Hogeweg L, et al. Clavicle segmentation in chest radiographs. Med. Image Anal. 2012;16:1490-1502.

    Not in our library10.1016/j.media.2012.06.009

27. 27.

    Tobon-Gomez C, et al. Benchmark for algorithms segmenting the left atrium from 3D CT and MRI datasets. IEEE Trans. Med. Imaging. 2015;34:1460-1473.

    Not in our library10.1109/tmi.2015.2398818

28. 28.

    Rueda S, et al. Evaluation and comparison of current fetal ultrasound image segmentation methods for biometric measurements: a grand challenge. IEEE Trans. Med. Imaging. 2014;33:797-813.

    Not in our library10.1109/tmi.2013.2276943

29. 29.

    Karim R, et al. Evaluation of state-of-the-art segmentation algorithms for left ventricle infarct from late Gadolinium enhancement MR images. Med. Image Anal. 2016;30:95-107.

    Not in our library10.1016/j.media.2016.01.004

30. 30.

    Kirişli HA, et al. Standardized evaluation framework for evaluating coronary artery stenosis detection, stenosis quantification and lumen segmentation algorithms in computed tomography angiography. Med. Image Anal. 2013;17:859-876.

    Not in our library10.1016/j.media.2013.05.007

31. 31.

    Küffner R, et al. Crowdsourced analysis of clinical trial data to predict amyotrophic lateral sclerosis progression. Nat. Biotechnol. 2015;33:51.

    Not in our library10.1038/nbt.3051

32. 32.

    Daducci A, et al. Quantitative comparison of reconstruction methods for intra-voxel fiber recovery from diffusion MRI. IEEE Trans. Med. Imaging. 2014;33:384-399.

    Not in our library10.1109/tmi.2013.2285500

33. 33.

    Išgum I, et al. Evaluation of automatic neonatal brain segmentation algorithms: the NeoBrainS12 challenge. Med. Image Anal. 2015;20:135-151.

    Not in our library10.1016/j.media.2014.11.001

34. 34.

    Foggia P, Percannella G, Soda P, Vento M. Benchmarking HEp-2 cells classification methods. IEEE Trans. Med. Imaging. 2013;32:1878-1889.

    Not in our library10.1109/tmi.2013.2268163

35. 35.

    Litjens G, et al. Evaluation of prostate segmentation algorithms for MRI: the PROMISE12 challenge. Med. Image Anal. 2014;18:359-373.

    Not in our library10.1016/j.media.2013.12.002

36. 36.

    Petitjean C, et al. Right ventricle segmentation from cardiac MRI: a collation study. Med. Image Anal. 2015;19:187-202.

    Not in our library10.1016/j.media.2014.10.004

37. 37.

    Rudyanto RD, et al. Comparing algorithms for automated vessel segmentation in computed tomography scans of the lung: the VESSEL12 study. Med. Image Anal. 2014;18:1217-1232.

    Not in our library10.1016/j.media.2014.07.003

38. 38.

    Tobon-Gomez C, et al. Benchmarking framework for myocardial tracking and deformation algorithms: an open access database. Med. Image Anal. 2013;17:632-648.

    Not in our library10.1016/j.media.2013.03.008

39. 39.

    Murphy K, et al. Evaluation of registration methods on thoracic CT: the EMPIRE10 challenge. IEEE Trans. Med. Imaging. 2011;30:1901-1920.

    Not in our library10.1109/tmi.2011.2158349

40. 40.

    Van Ginneken B, et al. Comparing and combining algorithms for computer-aided detection of pulmonary nodules in computed tomography scans: the ANODE09 study. Med. Image Anal. 2010;14:707-722.

    Not in our library10.1016/j.media.2010.05.005

41. 41.

    Lo P, et al. Extraction of airways from CT (EXACT'09). IEEE Trans. Med. Imaging. 2012;31:2093-2107.

    Not in our library10.1109/tmi.2012.2209674

42. 42.

    Niemeijer M, et al. Retinopathy online challenge: automatic detection of microaneurysms in digital color fundus photographs. IEEE Trans. Med. Imaging. 2010;29:185-195.

    Not in our library10.1109/tmi.2009.2033909

43. 43.

    Hameeteman K, et al. Evaluation framework for carotid bifurcation lumen segmentation and stenosis grading. Med. Image Anal. 2011;15:477-488.

    Not in our library10.1016/j.media.2011.02.004

44. 44.

    Schaap M, et al. Standardized evaluation methodology and reference database for evaluating coronary artery centerline extraction algorithms. Med. Image Anal. 2009;13:701-714.

    Not in our library10.1016/j.media.2009.06.003

45. 45.

    Kaggle Inc. The Home of Data Science & Machine Learning. https://www.kaggle.com/. Accessed 20 Feb 2018 (2010).

    Not in our library

46. 46.

    Tassey, G., Rowe, B. R., Wood, D. W., Link, A. N. & Simoni, D. A. Economic impact assessment of NIST’s text retrieval conference (TREC) program. Technical Report 0211875, RTI International (2010).

    Not in our library

47. 47.

    Tsikrika, T., Herrera, A. G. S. de & Müller, H. Assessing the scholarly impact of ImageCLEF. In Multilingual and Multimodal Information Access Evaluation 95–106 (Springer, Berlin, Heidelberg, 2011).

    Not in our library

48. 48.

    Russakovsky O, et al. ImageNET large scale visual recognition challenge. Int. J. Comput. Vis. 2015;115:211-252.

    Not in our library10.1007/s11263-015-0816-y

49. 49.

    Grünberg, K. et al. Annotating Medical Image Data. in Cloud-Based Benchmarking of Med. Image Anal. 45–67 (Springer, Cham, 2017).

    Not in our library

50. 50.

    Dice LR. Measures of the amount of ecologic association between species. Ecology. 1945;26:297-302.

    Not in our library10.2307/1932409

51. 51.

    Huttenlocher DP, Klanderman GA, Rucklidge WJ. Comparing images using the Hausdorff distance. IEEE Trans. Pattern Anal. Mach. Intell. 1993;15:850-863.

    Not in our library10.1109/34.232073

52. 52.

    Dubuisson, M.-P. & Anil K. J. A modified Hausdorff distance for object matching. In Proc. IEEE Int. Conf. Pattern Recognit.566–568 (IEEE, Jerusalem, 1994).

    Not in our library

53. 53.

    Kendall MG. A new measure of rank correlation. Biometrika. 1938;30:81-93.

    Not in our library10.1093/biomet/30.1-2.81

54. 54.

    Sculley, D., Snoek, J., Rahimi, A., & Wiltschko, A. Winner’s curse? On pace, progress, and empirical rigor. in Proc. Int. Conf. Mach. Learn. Workshop (2018).

    Not in our library

55. 55.

    Barnes, D., Wilkerson, T., & Stephan, M. Contributing to the development of grand challenges in maths education. in Proc. Int. Congress on Math. Educ. 703–704 (Springer, Cham, 2017).

    Not in our library

56. 56.

    NCTM Research Committee.. Grand challenges and opportunities in mathematics education research. J. Res. Math. Educ. 2017;46:134-146.

    Not in our library10.5951/jresematheduc.46.2.0134

57. 57.

    Dream Challenges. DREAM Challenges. http://dreamchallenges.org/. Accessed16 July 2018 (2006)

    Not in our library

58. 58.

    Lipton, Z. C. & Steinhardt, J. Troubling trends in machine learning scholarship. Preprint at https://arxiv.org/abs/1807.03341 (2018).

    Not in our library

59. 59.
    A manifesto for reproducible science · 2017

    No notice
60. 60.
    Why Most Published Research Findings Are False · 2005

    Corrected
61. 61.

    Armstrong, T. G., Moffat, A., Webber, W. & Zobel, J. Improvements that don’t add up: ad-hoc retrieval results since 1998. in Proc. 18th ACM conference on Information and knowledge management. 601–610 (ACM, New York, 2009).

    Not in our library

62. 62.

    Blanco, R. & Zaragoza, H. Beware of relatively large but meaningless improvements. Tech. Rep., Yahoo! Research YL-2011-001 (2011).

    Not in our library

63. 63.

    Boutros PC, Margolin AA, Stuart JM, Califano A, Stolovitzky G. Toward better benchmarking: challenge-based methods assessment in cancer genomics. Genome Biol. 2014;15:462.

    Not in our library10.1186/s13059-014-0462-7

64. 64.

    Jannin P, Grova C, Maurer CR. Model for defining and reporting reference-based validation protocols in medical image processing. Int. J. CARS. 2006;1:63-73.

    Not in our library10.1007/s11548-006-0044-6

65. 65.

    Langville, A. N. & Carl D. Meyer. Who’s #1? The Science of Rating and Ranking. (Princeton University Press, Princeton, New Jersey, 2012).

    Not in our library

66. 66.

    Maier-Hein, L. et al. Is the winner really the best? A critical analysis of common research practice in biomedical image analysis competitions (Version 1.0.0) [Data set]. Zenodo. 10.5281/zenodo.1453313 (2018).

    Not in our library

Cited by papers we hold 9

• Artificial intelligence for diagnosis and Gleason grading of prostate cancer: the PANDA challenge · 2022
  No notice
• Recommendations for machine learning benchmarks in neuroimaging · 2022
  Not checked
• The Medical Segmentation Decathlon · 2022
  No notice
• Machine learning for medical imaging: methodological failures and recommendations for the future · 2022
  No notice
• Surgical data science – from concepts toward clinical translation · 2022
  No notice
• Benchmarking atlas-level data integration in single-cell genomics · 2021
  No notice
• Introducing Biomedisa as an open-source online platform for biomedical image segmentation · 2020
  Corrected
• Causality matters in medical imaging · 2020
  No notice
• Gender imbalance in medical imaging datasets produces biased classifiers for computer-aided diagnosis · 2020
  No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.