Evaluation of algorithms for Multi-Modality Whole Heart Segmentation: An open-access grand challenge

Abstract

Knowledge of whole heart anatomy is a prerequisite for many clinical applications. Whole heart segmentation (WHS), which delineates substructures of the heart, can be very valuable for modeling and analysis of the anatomy and functions of the heart. However, automating this segmentation can be challenging due to the large variation of the heart shape, and different image qualities of the clinical data. To achieve this goal, an initial set of training data is generally needed for constructing priors or for training. Furthermore, it is difficult to perform comparisons between different methods, largely due to differences in the datasets and evaluation metrics used. This manuscript presents the methodologies and evaluation results for the WHS algorithms selected from the submissions to the Multi-Modality Whole Heart Segmentation (MM-WHS) challenge, in conjunction with MICCAI 2017. The challenge provided 120 three-dimensional cardiac images covering the whole heart, including 60 CT and 60 MRI volumes, all acquired in clinical environments with manual delineation. Ten algorithms for CT data and eleven algorithms for MRI data, submitted from twelve groups, have been evaluated. The results showed that the performance of CT WHS was generally better than that of MRI WHS. The segmentation of the substructures for different categories of patients could present different levels of challenge due to the difference in imaging and variations of heart shapes. The deep learning (DL)-based methods demonstrated great potential, though several of them reported poor results in the blinded evaluation. Their performance could vary greatly across different network structures and training strategies. The conventional algorithms, mainly based on multi-atlas segmentation, demonstrated good performance, though the accuracy and computational efficiency could be limited. The challenge, including provision of the annotated training data and the blinded evaluation for submitted algorithms on the test data, continues as an ongoing benchmarking resource via its homepage (www.sdspeople.fudan.edu.cn/zhuangxiahai/0/mmwhs/).

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 55

1. 1.

   Avendi M.R., Kheradvar A., Jafarkhani H. A combined deep-learning and deformable-model approach to fully automatic segmentation of the left ventricle in cardiac MRI. Med. Image Anal. 2016;30:108-119.

   Not in our library10.1016/j.media.2016.01.005

2. 2.

   Benjamini Y., Yekutieli D. The control of the false discovery rate in multiple testing under dependency. Ann. Stat. 2001;29(4):1165-1188.

   Not in our library

3. 3.

   Bernard O., Lalande A., Zotti C., Cervenansky F., Yang X., Heng P.-A., Cetin I., Lekadir K., Camara O., Ballester M.A.G. Deep learning techniques for automatic MRI cardiac multi-structures segmentation and diagnosis: Is the problem solved? IEEE Trans. Med. Imag. 2018;37(11):2514-2525.

   Not in our library10.1109/tmi.2018.2837502

4. 4.

   Cai K., Yang R., Chen H., Li L., Zhou J., Ou S., Liu F. A framework combining window width-level adjustment and Gaussian filter-based multi-resolution for automatic whole heart segmentation. Neurocomputing. 2017;220:138-150.

   Not in our library

5. 5.

   Crum W.R., Camara O., Hill D.L.G. Generalized overlap measures for evaluation and validation in medical image analysis. IEEE Trans. Med. Imag. 2006;25(11):1451-1461.

   Not in our library10.1109/tmi.2006.880587

6. 6.

   Dong C., Loy C.C., He K., Tang X. Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach.Intell. 2016;38(2):295-307.

   Not in our library10.1109/tpami.2015.2439281

7. 7.

   Galisot G., Brouard T., Ramel J.-Y. Local probabilistic atlases and a posteriori correction for the segmentation of heart images. International Workshop on Statistical Atlases and Computational Models of the Heart. 2017;207-214.

   Not in our library

8. 8.

   Goodfellow I., Pouget-Abadie J., Mirza M., Xu B., Warde-Farley D., Ozair S., Courville A., Bengio Y. Generative adversarial nets. Advances in neural information processing systems. 2014;2672-2680.

   Not in our library

9. 9.

   Heinrich M.P., Blendowski M. Multi-organ segmentation using vantage point forests and binary context features. International Conference on Medical Image Computing and Computer-Assisted Intervention. 2016;598-606.

   Not in our library

10. 10.

    Heinrich M.P., Jenkinson M., Bartlomiej W. P., Brady M., Schnabel J.A. Towards realtime multimodal fusion for image-guided interventions using self-similarities. International conference on medical image computing and computer-assisted intervention. 2013;187-194.

    Not in our library10.1007/978-3-642-40811-3_24

11. 11.

    Heinrich M.P., Jenkinson M., Brady M., Schnabel J.A. MRF-Based deformable registration and ventilation estimation of lung CT. IEEE Trans. Med. Imag. 2013;32(7):1239-1248.

    Not in our library10.1109/tmi.2013.2246577

12. 12.

    Heinrich M.P., Oster J. MRI whole heart segmentation using discrete nonlinear registration and fast non-local fusion. International Workshop on Statistical Atlases and Computational Models of the Heart. 2017;233-241.

    Not in our library

13. 13.

    Kang D., Woo J., Kuo C.C.J., Slomka P.J., Dey D., Germano G. Heart chambers and whole heart segmentation techniques: a review. J. Electron. Imag. 2012;21(1):010901.

    Not in our library

14. 14.

    Karim R., Blake L.-E., Inoue J., Tao Q., Jia S., Housden R.J., Bhagirath P., Duval J.-L., Varela M., Behar J. Algorithms for left atrial wall segmentation and thickness–evaluation on an open-source CT and MRI image database. Med. Image Anal. 2018;50:36-53.

    Not in our library10.1016/j.media.2018.08.004

15. 15.

    Karim R., Housden R.J., Balasubramaniam M., Chen Z., Perry D., Uddin A., Al-Beyatti Y., Palkhi E., Acheampong P., Obom S. Evaluation of current algorithms for segmentation of scar tissue from late gadolinium enhancement cardiovascular magnetic resonance of the left atrium: an open-access grand challenge. J. Cardiovasc. Magnetic Reson. 2013;15(1):105.

    Not in our library10.1186/1532-429x-15-105

16. 16.

    Kim J., Kwon Lee J., Mu Lee K. Deeply-recursive convolutional network for image super-resolution. Proceedings of the IEEE conference on computer vision and pattern recognition. 2016;1637-1645.

    Not in our library

17. 17.

    Mahbod A., Chowdhury M., Smedby O., Wang C. Automatic brain segmentation using artificial neural networks with shape context. Pattern Recognit. Lett. 2018;101:74-79.

    Not in our library

18. 18.

    Mendis S., Puska P., Norrving B. Global Atlas on Cardiovascular Disease Prevention and Control. 2011.

    Not in our library

19. 19.

    Milletari F., Navab N., Ahmadi S.-A. V-net: Fully convolutional neural networks for volumetric medical image segmentation. International Conference on 3D Vision. 2016;565-571.

    Not in our library

20. 20.

    Mo Y., Liu F., McIlwraith D., Yang G., Zhang J., He T., Guo Y. The deep poincaré map: a novel approach for left ventricle segmentation. International Conference on Medical Image Computing and Computer-Assisted Intervention. 2018;561-568.

    Not in our library

21. 21.

    Moghari, M.H., Pace, D.F., Akhondi-Asl, A., Powell, A.J., 2016. HVSMR 2016: MICCAI workshop on whole-heart and great vessel segmentation from 3D cardiovascular MRI in congenital heart disease. http://segchd.csail.mit.edu/index.html.

    Not in our library

22. 22.

    Mortazi A., Burt J., Bagci U. Multi-planar deep segmentation networks for cardiac substructures from MRI and CT. International Workshop on Statistical Atlases and Computational Models of the Heart. 2017;199-206.

    Not in our library

23. 23.

    Mortazi A., Karim R., Rhode K., Burt J., Bagci U. CardiacNET: segmentation of left atrium and proximal pulmonary veins from MRI using multi-view CNN. International Conference on Medical Image Computing and Computer-Assisted Intervention. 2017;377-385.

    Not in our library

24. 24.

    Ngo T.A., Lu Z., Carneiro G. Combining deep learning and level set for the automated segmentation of the left ventricle of the heart from cardiac cine magnetic resonance. Med. Image Anal. 2017;35:159-171.

    Not in our library10.1016/j.media.2016.05.009

25. 25.

    Nikolaou K., Alkadhi H., Bamberg F., Leschka S., Wintersperger B.J. MRI and CT in the diagnosis of coronary artery disease: indications and applications. InsightsImag. 2011;2(1):9-24.

    Not in our library10.1007/s13244-010-0049-0

26. 26.

    Pace D.F., Dalca A.V., Geva T., Powell A.J., Moghari M.H., Golland P. Interactive whole-heart segmentation in congenital heart disease. International Conference on Medical Image Computing and Computer-Assisted Intervention. 2015;80-88.

    Not in our library10.1007/978-3-319-24574-4_10

27. 27.

    Payer C., Štern D., Bischof H., Urschler M. Regressing heatmaps for multiple landmark localization using CNNs. International Conference on Medical Image Computing and Computer-Assisted Intervention. 2016;230-238.

    Not in our library

28. 28.

    Payer C., Štern D., Bischof H., Urschler M. Multi-label whole heart segmentation using CNNs and anatomical label configurations. International Workshop on Statistical Atlases and Computational Models of the Heart. 2017;190-198.

    Not in our library

29. 29.
    A review of heart chamber segmentation for structural and functional analysis using cardiac magnetic resonance imaging · 2016

    No notice
30. 30.

    Petitjean C., Zuluaga M.A., Bai W., Dacher J.-N., Grosgeorge D., Caudron J., Ruan S., Ayed I.B., Cardoso M.J., Chen H.-C. Right ventricle segmentation from cardiac MRI: a collation study. Med. Image Anal. 2015;19(1):187-202.

    Not in our library10.1016/j.media.2014.10.004

31. 31.

    Radau P., Lu Y., Connelly K., Paul G., Dick A., Wright G. Evaluation framework for algorithms segmenting short axis cardiac MRI. MIDAS J.-Cardiac MR Left Ventricle Segmentation Challenge. 2009;49.

    Not in our library

32. 32.

    Roberts W.T., Bax J.J., Davies L.C. Cardiac CT and CT coronary angiography: technology and application. Heart. 2008;94(6):781-792.

    Not in our library10.1136/hrt.2007.116392

33. 33.

    Ronneberger O., Fischer P., Brox T. U-net: convolutional networks for biomedical image segmentation. International Conference on Medical image computing and computer-assisted intervention. 2015;234-241.

    Not in our library

34. 34.

    Roth H.R., Lu L., Seff A., Cherry K.M., Hoffman J., Wang S., Liu J., Turkbey E., Summers R.M. A new 2.5 D representation for lymph node detection using random sets of deep convolutional neural network observations. International conference on medical image computing and computer-assisted intervention. 2014;520-527.

    Not in our library10.1007/978-3-319-10404-1_65

35. 35.

    Shen D., Wu G., Suk H.-I. Deep learning in medical image analysis. Annu. Rev. Biomed. Eng. 2017;19:221-248.

    Not in our library10.1146/annurev-bioeng-071516-044442

36. 36.

    Suinesiaputra A., Cowan B.R., Finn J.P., Fonseca C.G., Kadish A.H., Lee D.C., Medrano-Gracia P., Warfield S.K., Tao W., Young A.A. Left ventricular segmentation challenge from cardiac MRI: a collation study. International Workshop on Statistical Atlases and Computational Models of the Heart. 2011;88-97.

    Not in our library

37. 37.

    Tan L.K., McLaughlin R.A., Lim E., Abdul Aziz Y.F., Liew Y.M. Fully automated segmentation of the left ventricle in cine cardiac MRI using neural network regression. J. Magnetic Reson. Imag. 2018;48(1):140-152.

    Not in our library10.1002/jmri.25932

38. 38.

    Tobon-Gomez C., Geers A.J., Peters J., Weese J., Pinto K., Karim R., Ammar M., Daoudi A., Margeta J., Sandoval Z. Benchmark for algorithms segmenting the left atrium from 3D CT and MRI datasets. IEEE Trans. Med. Imag. 2015;34(7):1460-1473.

    Not in our library10.1109/tmi.2015.2398818

39. 39.

    Tong Q., Ning M., Si W., Liao X., Qin J. 3D deeply-supervised U-Net based whole heart segmentation. International Workshop on Statistical Atlases and Computational Models of the Heart. 2017;224-232.

    Not in our library

40. 40.

    Tran D., Bourdev L., Fergus R., Torresani L., Paluri M. Learning spatiotemporal features with 3D convolutional networks. Proceedings of the IEEE international conference on computer vision. 2015;4489-4497.

    Not in our library

41. 41.

    Wang C., Smedby O. Automatic multi-organ segmentation in non-enhanced CT datasets using hierarchical shape priors. International Conference on Pattern Recognition. 2014;3327-3332.

    Not in our library

42. 42.

    Wang C., Smedby O. Automatic whole heart segmentation using deep learning and shape context. International Workshop on Statistical Atlases and Computational Models of the Heart. 2017;242-249.

    Not in our library

43. 43.

    Xu Z., Lee C.P., Heinrich M.P., Modat M., Rueckert D., Ourselin S., Abramson R.G., Landman B.A. Evaluation of six registration methods for the human abdomen on clinically acquired CT. IEEE Trans. Biomed. Eng. 2016;63(8):1563-1572.

    Not in our library10.1109/tbme.2016.2574816

44. 44.

    Yang G., Sun C., Chen Y., Tang L., Shu H., Dillenseger J.-l. Automatic whole heart segmentation in CT images based on multi-atlas image registration. International Workshop on Statistical Atlases and Computational Models of the Heart. 2017;250-257.

    Not in our library

45. 45.

    Yang X., Bian C., Yu L., Ni D., Heng P.-A. 3D convolutional networks for fully automatic fine-grained whole heart partition. International Workshop on Statistical Atlases and Computational Models of the Heart. 2017;181-189.

    Not in our library

46. 46.

    Yang X., Bian C., Yu L., Ni D., Heng P.-A. Hybrid loss guided convolutional networks for whole heart parsing. International Workshop on Statistical Atlases and Computational Models of the Heart. 2017;215-223.

    Not in our library

47. 47.

    Yu F., Koltun V., Funkhouser T. Dilated residual networks. Proceedings of the IEEE conference on computer vision and pattern recognition. 2017;472-480.

    Not in our library

48. 48.

    Yushkevich P.A., Piven J., Hazlett H.C., Smith R.G., Ho S., Gee J.C., Gerig G. User-guided 3d active contour segmentation of anatomical structures: significantly improved efficiency and reliability. Neuroimage. 2006;31(3):1116-1128.

    Not in our library10.1016/j.neuroimage.2006.01.015

49. 49.

    Zhao, J., Xiong, Z., 2018. 2018 atrial segmentation challenge. http://atriaseg2018.cardiacatlas.org/.

    Not in our library

50. 50.

    Zhou R., Liao Z., Pan T., Milgrom S.A., Pinnix C.C., Shi A., Tang L., Yang J., Liu Y., Gomez D. Cardiac atlas development and validation for automatic segmentation of cardiac substructures. Radiother. Oncol. 2017;122(1):66-71.

    Not in our library10.1016/j.radonc.2016.11.016

51. 51.

    Zhuang X. Challenges and methodologies of fully automatic whole heart segmentation: a review. J. Healthcare Eng. 2013;4(3):371-407.

    Not in our library10.1260/2040-2295.4.3.371

52. 52.

    Zhuang X., Bai W., Song J., Zhan S., Qian X., Shi W., Lian Y., Rueckert D. Multiatlas whole heart segmentation of CT data using conditional entropy for atlas ranking and selection. Med. Phys. 2015;42(7):3822-3833.

    Not in our library10.1118/1.4921366

53. 53.

    Zhuang, X., Li, L., Xu, J., Zhou, Y., Luo, X., 2019. Multi-sequence cardiac MR segmentation challenge. https://zmiclab.github.io/mscmrseg19/.

    Not in our library10.1016/j.media.2022.102528

54. 54.

    Zhuang X., Shen J. Multi-scale patch and multi-modality atlases for whole heart segmentation of MRI. Med. Image Anal. 2016;31:77-87.

    Not in our library10.1016/j.media.2016.02.006

55. 55.

    Zuluaga M.A., Cardoso M.J., Modat M., Ourselin S. Multi-atlas propagation whole heart segmentation from MRI and CTA using a local normalised correlation coefficient criterion. International Conference on Functional Imaging and Modeling of the Heart. 2013;174-181.

    Not in our library

Cited by papers we hold 3

• SynthSeg: Segmentation of brain MRI scans of any contrast and resolution without retraining · 2023
  No notice
• Simultaneous Multi-Structure Segmentation of the Heart and Peripheral Tissues in Contrast Enhanced Cardiac Computed Tomography Angiography · 2020
  Not checked
• Deep Learning for Cardiac Image Segmentation: A Review · 2020
  No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.