The Medical Segmentation Decathlon

Abstract

International challenges have become the de facto standard for comparative assessment of image analysis algorithms. Although segmentation is the most widely investigated medical image processing task, the various challenges have been organized to focus only on specific clinical tasks. We organized the Medical Segmentation Decathlon (MSD)—a biomedical image analysis challenge, in which algorithms compete in a multitude of both tasks and modalities to investigate the hypothesis that a method capable of performing well on multiple tasks will generalize well to a previously unseen task and potentially outperform a custom-designed solution. MSD results confirmed this hypothesis, moreover, MSD winner continued generalizing well to a wide range of other clinical problems for the next two years. Three main conclusions can be drawn from this study: (1) state-of-the-art image segmentation algorithms generalize well when retrained on unseen tasks; (2) consistent algorithmic performance across multiple tasks is a strong surrogate of algorithmic generalizability; (3) the training of accurate AI segmentation models is now commoditized to scientists that are not versed in AI model training.

References 45

1. 1.
   Deep learning as a tool for increased accuracy and efficiency of histopathological diagnosis · 2016

   No notice
2. 2.

   Poplin R, et al. Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning. Nat. Biomed. Eng. 2018;2:158-164.

   Not in our library10.1038/s41551-018-0195-0

3. 3.

   Stokes JM, et al. A deep learning approach to antibiotic discovery. Cell. 2020;180:688-702.

   Not in our library10.1016/j.cell.2020.01.021

4. 4.

   Ayache N, Duncan J. 20th anniversary of the medical image analysis journal (media). Med. Image Anal. 2016;33:1-3.

   Not in our library10.1016/j.media.2016.07.004

5. 5.

   Liang S, et al. Deep-learning-based detection and segmentation of organs at risk in nasopharyngeal carcinoma computed tomographic images for radiotherapy planning. Eur. Radiol. 2019;29:1961-1967.

   Not in our library10.1007/s00330-018-5748-9

6. 6.

   Assefa D, et al. Robust texture features for response monitoring of glioblastoma multiforme on-weighted and-flair mr images: A preliminary investigation in terms of identification and segmentation. Med. Phys. 2010;37:1722-1736.

   Not in our library10.1118/1.3357289

7. 7.
   Why rankings of biomedical image analysis competitions should be interpreted with care · 2018

   Corrected
8. 8.

   Isensee F, Jaeger PF, Kohl SA, Petersen J, Maier-Hein KH. nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nat. Commun. 2021;18:203-211.

   Not in our library10.1038/s41592-020-01008-z

9. 9.

   Dice LR. Measures of the amount of ecologic association between species. Ecology. 1945;26:297-302.

   Not in our library10.2307/1932409

10. 10.
    Clinically Applicable Segmentation of Head and Neck Anatomy for Radiotherapy: Deep Learning Algorithm Development and Validation Study · 2021

    No notice
11. 11.

    Ronneberger, O., Fischer, P. & Brox, T. U-net: Convolutional networks for biomedical image segmentation, In Proc. International Conference on Medical Image Computing and Computer-Assisted Intervention, 234–241 (Springer, 2015).

    Not in our library

12. 12.

    Kingma, D. P. & Ba, J. Adam: A method for stochastic optimization. arXiv Preprint at https://arxiv.org/abs/1412.6980 (2014).

    Not in our library

13. 13.

    Zhang, T. Solving large scale linear prediction problems using stochastic gradient descent algorithms, In Proc. Twenty-First International Conference on Machine Learning. 116 (Association for Computing Machinery, 2004).

    Not in our library

14. 14.

    Xia, Y. et al. 3d semi-supervised learning with uncertainty-aware multi-view co-training, In Proc. IEEE Winter Conference on Applications of Computer Vision, 3646–3655 (IEEE Computer Society, 2020).

    Not in our library

15. 15.

    He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition, In Proc. IEEE Conference on Computer Vision and Pattern Recognition. 770–778 (IEEE, 2016).

    Not in our library

16. 16.

    He X, Zhao K, Chu X. Automl: a survey of the state-of-the-art. Knowl. Based Syst. 2021;212:106622.

    Not in our library10.1016/j.knosys.2020.106622

17. 17.
    Plasma Hsp90 levels in patients with systemic sclerosis and relation to lung and skin involvement: a cross-sectional and longitudinal study · 2021

    No notice
18. 18.

    Kendall MG. A new measure of rank correlation. Biometrika. 1938;30:81-93.

    Not in our library10.1093/biomet/30.1-2.81

19. 19.

    Ma, J. Cutting-edge 3d medical image segmentation methods in 2020: Are happy families all alike? arXiv Preprint at https://arxiv.org/abs/2101.00232 (2021).

    Not in our library

20. 20.

    Reinke, A. et al. How to exploit weaknesses in biomedical challenge design and organization, In Proc. International Conference on Medical Image Computing and Computer-Assisted Intervention, 388–395 (Springer, 2018).

    Not in our library

21. 21.

    Joskowicz L, Cohen D, Caplan N, Sosna J. Inter-observer variability of manual contour delineation of structures in ct. Eur. Radiol. 2019;29:1391-1399.

    Not in our library10.1007/s00330-018-5695-5

22. 22.

    Maier-Hein L, et al. Bias: transparent reporting of biomedical image analysis challenges. Med. Image Anal. 2020;66:101796.

    Not in our library10.1016/j.media.2020.101796

23. 23.

    Reinke, A. et al. Common limitations of image processing metrics: A picture story. arXiv Preprint at https://arxiv.org/abs/2104.05642 (2021).

    Not in our library

24. 24.

    Breslow NE, Clayton DG. Approximate inference in generalized linear mixed models. J. Am. Stat. Assoc. 1993;88:9-25.

    Not in our library

25. 25.

    Codella, N.C.F et al. Skin lesion analysis toward melanoma detection: a challenge at the 2017 international symposium on biomedical imaging (ISBN), hosted by the international skin imaging collaboration (ISIC). In Proc. IEEE 15th International Symposium on Biomedical Imaging, 168–172 (ISBI 2018).

    Not in our library

26. 26.

    Ross T, et al. Comparative validation of multi-instance instrument segmentation in endoscopy: results of the ROBUST-MIS 2019 challenge. Med. Image Anal. 2021;70:101920.

    Not in our library10.1016/j.media.2020.101920

27. 27.

    Campello, V. M. et al. Multi-Centre, Multi-Vendor and Multi-Disease Cardiac Segmentation: The M&Ms Challenge. IEEE Transactions on Medical Imaging. 40, 3543–3554 (IEEE, 2021).

    Not in our library10.1109/tmi.2021.3090082

28. 28.

    Campadelli P, Casiraghi E, Esposito A. Liver segmentation from computed tomography scans: a survey and a new algorithm. Artif. Intel. Med. 2009;45:185-196.

    Not in our library10.1016/j.artmed.2008.07.020

29. 29.

    Sirinukunwattana K, et al. Gland segmentation in colon histology images: the glas challenge contest. Med. Image Anal. 2017;35:489-502.

    Not in our library10.1016/j.media.2016.08.008

30. 30.

    Re TJ, et al. Enhancing pancreatic adenocarcinoma delineation in diffusion derived intravoxel incoherent motion f-maps through automatic vessel and duct segmentation. Magn. Reson. Med. 2011;66:1327-1332.

    Not in our library10.1002/mrm.22931

31. 31.

    Bello, I. et al. Revisiting resnets: Improved training and scaling strategies. arXiv Preprint at https://arxiv.org/abs/2103.07579 (2021).

    Not in our library

32. 32.

    Elsken T, Metzen JH, Hutter F, et al. Neural architecture search: a survey. J. Mach. Learn. Res. 2019;20:1-21.

    Not in our library

33. 33.

    He, Y., Yang, D., Roth, H., Zhao, C. & Xu, D. Dints: differentiable neural network topology search for 3d medical image segmentation. CoRR abs/2103.15954. http://arxiv.org/abs/2103.15954 (2021).

    Not in our library

34. 34.
    Why rankings of biomedical image analysis competitions should be interpreted with care · 2018

    Corrected
35. 35.

    Simpson, A.L. et al. A large annotated medical image dataset for the development and evaluation of segmentation algorithms. arXiv e-printshttp://arxiv.org/abs/1902.09063 (2019).

    Not in our library

36. 36.

    Menze BH, et al. The multimodal brain tumor image segmentation benchmark (brats). IEEE Trans. Med. Imag. 2015;34:1993-2024.

    Not in our library10.1109/tmi.2014.2377694

37. 37.
    Advancing The Cancer Genome Atlas glioma MRI collections with expert segmentation labels and radiomic features · 2017

    No notice
38. 38.

    Bakas, S. et al. Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge. arXiv Preprint at https://arxiv.org/abs/1811.02629 (2018b).

    Not in our library

39. 39.

    Tobon-Gomez C, et al. Benchmark for algorithms segmenting the left atrium from 3d ct and mri datasets. IEEE Trans. Med. Imag. 2015;34:1460-1473.

    Not in our library10.1109/tmi.2015.2398818

40. 40.

    Bilic, P. et al. The Liver Tumor Segmentation Benchmark (LiTS). arXiv e-printshttp://arxiv.org/abs/1901.04056 (2019).

    Not in our library10.1016/j.media.2022.102680

Cited by papers we hold 4

• Segment anything in medical images · 2024
  No notice
• MedMNIST v2 - A large-scale lightweight benchmark for 2D and 3D biomedical image classification · 2023
  No notice
• The Cell Tracking Challenge: 10 years of objective benchmarking · 2023
  No notice
• Deep learning empowered volume delineation of whole-body organs-at-risk for accelerated radiotherapy · 2022
  No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.