Why rankings of biomedical image analysis competitions should be interpreted with care

Abstract

International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 66

1. 1.

   Ayache N, Duncan J. 20th anniversary of the medical image analysis journal (MedIA). Med. Image Anal. 2016;33:1-3.

   Not in our library10.1016/j.media.2016.07.004

2. 2.

   Chen, W. Li, W. Dong, X. Pei, J. A review of biological image analysis. Curr. Bioinform. 13, 337–343 (2018).

   Not in our library

3. 3.

   Price K. Anything you can do, I can do better (no you can’t). Comput. Gr. Image Process. 1986;36:387-391.

   Not in our library10.1016/0734-189x(86)90083-6

4. 4.

   West J, et al. Comparison and evaluation of retrospective intermodality brain image registration techniques. J. Comput. Assist. Tomogr. 1997;21:554-568.

   Not in our library10.1097/00004728-199707000-00007

5. 5.

   Müller H, Rosset A, Vallée JP, Terrier F, Geissbuhler A. A reference data set for the evaluation of medical image retrieval systems. Comput. Med. Imaging Graph. 2004;28:295-305.

   Not in our library10.1016/j.compmedimag.2004.04.005

6. 6.

   ImageCLEF/LifeCLEF – Multimedia Retrieval in CLEF. 2004. http://www.imageclef.org/. Accessed 20 Feb 2018

   Not in our library

7. 7.

   Kalpathy-Cramer J, et al. Evaluating performance of biomedical image retrieval systems – an overview of the medical image retrieval task at ImageCLEF 2004-2013. Comput. Med. Imaging Graph. 2015;39:55-61.

   Not in our library10.1016/j.compmedimag.2014.03.004

8. 8.

   Cleverdon CW. The aslib cranfield research project on the comparative efficiency of indexing systems. Aslib Proc. 1960;12:421-431.

   Not in our library10.1108/eb049778

9. 9.

   Heimann T, et al. Comparison and evaluation of methods for liver segmentation from CT datasets. IEEE Trans. Med. Imaging. 2009;28:1251-1265.

   Not in our library10.1109/tmi.2009.2013851

10. 10.

    Chenouard N, et al. Objective comparison of particle tracking methods. Nat. Methods. 2014;11:281-289.

    Not in our library10.1038/nmeth.2808

11. 11.

    Sage Daniel, et al. Quantitative evaluation of software packages for single-molecule localization microscopy. Nat. Methods. 2015;12:717-724.

    Not in our library10.1038/nmeth.3442

12. 12.

    Menze BH, et al. The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Trans. Med. Imaging. 2015;34:1993-2024.

    Not in our library10.1109/tmi.2014.2377694

13. 13.

    Ulman V, et al. An objective comparison of cell-tracking algorithms. Nat. Methods. 2017;14:1141.

    Not in our library10.1038/nmeth.4473

14. 14.
    The challenge of mapping the human connectome based on diffusion tractography · 2017

    Corrected
15. 15.

    Setio AAA, et al. Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the LUNA16 challenge. Med. Image Anal. 2017;42:1-13.

    Not in our library10.1016/j.media.2017.06.015

16. 16.

    Zheng G, et al. Evaluation and comparison of 3D intervertebral disc localization and segmentation methods for 3D T2 MR data: a grand challenge. Med. Image Anal. 2017;35:327-344.

    Not in our library10.1016/j.media.2016.08.005

17. 17.

    Wang CW, et al. A benchmark for comparison of dental radiography analysis algorithms. Med. Image Anal. 2016;31:63-76.

    Not in our library10.1016/j.media.2016.02.004

18. 18.

    Bernal J, et al. Comparative validation of polyp detection methods in video colonoscopy: results from the MICCAI 2015 Endoscopic Vision Challenge. IEEE Trans. Med. Imaging. 2017;36:1231-1249.

    Not in our library10.1109/tmi.2017.2664042

19. 19.

    Sirinukunwattana K, et al. Gland segmentation in colon histology images: The glas challenge contest. Med. Image Anal. 2017;35:489-502.

    Not in our library10.1016/j.media.2016.08.008

20. 20.

    Maier O, et al. ISLES 2015-A public evaluation benchmark for ischemic stroke lesion segmentation from multispectral MRI. Med. Image Anal. 2017;35:250-269.

    Not in our library10.1016/j.media.2016.07.009

21. 21.

    Carass A, et al. Longitudinal multiple sclerosis lesion segmentation: resource and challenge. NeuroImage. 2017;148:77-102.

    Not in our library10.1016/j.neuroimage.2016.12.064

22. 22.

    Wang CW, et al. Evaluation and comparison of anatomical landmark detection methods for cephalometric x-ray images: a grand challenge. IEEE Trans. Med. Imaging. 2015;34:1890-1900.

    Not in our library10.1109/tmi.2015.2412951

23. 23.

    Bernard O, et al. Standardized evaluation system for left ventricular segmentation algorithms in 3D echocardiography. IEEE Trans. Med. Imaging. 2016;35:967-977.

    Not in our library10.1109/tmi.2015.2503890

24. 24.

    Bron EE, et al. Standardized evaluation of algorithms for computer-aided diagnosis of dementia based on structural MRI: the CADDementia challenge. NeuroImage. 2015;111:562-579.

    Not in our library10.1016/j.neuroimage.2015.01.048

25. 25.

    Jimenez-del-Toro O, et al. Cloud-based evaluation of anatomical structure segmentation and landmark detection algorithms: VISCERAL anatomy benchmarks. IEEE Trans. Med. Imaging. 2016;35:2459-2475.

    Not in our library10.1109/tmi.2016.2578680

26. 26.

    Hogeweg L, et al. Clavicle segmentation in chest radiographs. Med. Image Anal. 2012;16:1490-1502.

    Not in our library10.1016/j.media.2012.06.009

27. 27.

    Tobon-Gomez C, et al. Benchmark for algorithms segmenting the left atrium from 3D CT and MRI datasets. IEEE Trans. Med. Imaging. 2015;34:1460-1473.

    Not in our library10.1109/tmi.2015.2398818

28. 28.

    Rueda S, et al. Evaluation and comparison of current fetal ultrasound image segmentation methods for biometric measurements: a grand challenge. IEEE Trans. Med. Imaging. 2014;33:797-813.

    Not in our library10.1109/tmi.2013.2276943

29. 29.

    Karim R, et al. Evaluation of state-of-the-art segmentation algorithms for left ventricle infarct from late Gadolinium enhancement MR images. Med. Image Anal. 2016;30:95-107.

    Not in our library10.1016/j.media.2016.01.004

30. 30.

    Kirişli HA, et al. Standardized evaluation framework for evaluating coronary artery stenosis detection, stenosis quantification and lumen segmentation algorithms in computed tomography angiography. Med. Image Anal. 2013;17:859-876.

    Not in our library10.1016/j.media.2013.05.007

31. 31.

    Küffner R, et al. Crowdsourced analysis of clinical trial data to predict amyotrophic lateral sclerosis progression. Nat. Biotechnol. 2015;33:51.

    Not in our library10.1038/nbt.3051

32. 32.

    Daducci A, et al. Quantitative comparison of reconstruction methods for intra-voxel fiber recovery from diffusion MRI. IEEE Trans. Med. Imaging. 2014;33:384-399.

    Not in our library10.1109/tmi.2013.2285500

33. 33.

    Išgum I, et al. Evaluation of automatic neonatal brain segmentation algorithms: the NeoBrainS12 challenge. Med. Image Anal. 2015;20:135-151.

    Not in our library10.1016/j.media.2014.11.001

34. 34.

    Foggia P, Percannella G, Soda P, Vento M. Benchmarking HEp-2 cells classification methods. IEEE Trans. Med. Imaging. 2013;32:1878-1889.

    Not in our library10.1109/tmi.2013.2268163

35. 35.

    Litjens G, et al. Evaluation of prostate segmentation algorithms for MRI: the PROMISE12 challenge. Med. Image Anal. 2014;18:359-373.

    Not in our library10.1016/j.media.2013.12.002

36. 36.

    Petitjean C, et al. Right ventricle segmentation from cardiac MRI: a collation study. Med. Image Anal. 2015;19:187-202.

    Not in our library10.1016/j.media.2014.10.004

37. 37.

    Rudyanto RD, et al. Comparing algorithms for automated vessel segmentation in computed tomography scans of the lung: the VESSEL12 study. Med. Image Anal. 2014;18:1217-1232.

    Not in our library10.1016/j.media.2014.07.003

38. 38.

    Tobon-Gomez C, et al. Benchmarking framework for myocardial tracking and deformation algorithms: an open access database. Med. Image Anal. 2013;17:632-648.

    Not in our library10.1016/j.media.2013.03.008

39. 39.

    Murphy K, et al. Evaluation of registration methods on thoracic CT: the EMPIRE10 challenge. IEEE Trans. Med. Imaging. 2011;30:1901-1920.

    Not in our library10.1109/tmi.2011.2158349

40. 40.

    Van Ginneken B, et al. Comparing and combining algorithms for computer-aided detection of pulmonary nodules in computed tomography scans: the ANODE09 study. Med. Image Anal. 2010;14:707-722.

    Not in our library10.1016/j.media.2010.05.005

Cited by papers we hold 9

• Artificial intelligence for diagnosis and Gleason grading of prostate cancer: the PANDA challenge · 2022
  No notice
• Recommendations for machine learning benchmarks in neuroimaging · 2022
  Not checked
• The Medical Segmentation Decathlon · 2022
  No notice
• Machine learning for medical imaging: methodological failures and recommendations for the future · 2022
  No notice
• Surgical data science – from concepts toward clinical translation · 2022
  No notice
• Benchmarking atlas-level data integration in single-cell genomics · 2021
  No notice
• Introducing Biomedisa as an open-source online platform for biomedical image segmentation · 2020
  Corrected
• Causality matters in medical imaging · 2020
  No notice
• Gender imbalance in medical imaging datasets produces biased classifiers for computer-aided diagnosis · 2020
  No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.