Annotation-efficient deep learning for automatic medical image segmentation

Abstract

Automatic medical image segmentation plays a critical role in scientific research and medical care. Existing high-performance deep learning methods typically rely on large training datasets with high-quality manual annotations, which are difficult to obtain in many clinical applications. Here, we introduce Annotation-effIcient Deep lEarning (AIDE), an open-source framework to handle imperfect training datasets. Methodological analyses and empirical evaluations are conducted, and we demonstrate that AIDE surpasses conventional fully-supervised models by presenting better performance on open datasets possessing scarce or noisy annotations. We further test AIDE in a real-life case study for breast tumor segmentation. Three datasets containing 11,852 breast images from three medical centers are employed, and AIDE, utilizing 10% training annotations, consistently produces segmentation maps comparable to those generated by fully-supervised counterparts or provided by independent radiologists. The 10-fold enhanced efficiency in utilizing expert labels has the potential to promote a wide range of biomedical applications.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 66

1. 1.

   Hollon TC, et al. Near real-time intraoperative brain tumor diagnosis using stimulated Raman histology and deep neural networks. Nat. Med. 2020;26:52-58.

   Not in our library10.1038/s41591-019-0715-9

2. 2.

   Bai W, et al. A population-based phenome-wide association study of cardiac and aortic structure and function. Nat. Med. 2020;26:1654-1662.

   Not in our library10.1038/s41591-020-1009-y

3. 3.

   Mei X, et al. Artificial intelligence-enabled rapid diagnosis of patients with COVID-19. Nat. Med. 2020;26:1224-1228.

   Not in our library10.1038/s41591-020-0931-3

4. 4.

   Kickingereder P, et al. Automated quantitative tumour response assessment of MRI in neuro-oncology with artificial neural networks: a multicentre, retrospective study. Lancet Oncol. 2019;20:728-740.

   Not in our library10.1016/s1470-2045(19)30098-1

5. 5.

   Wang L, et al. Benchmark on automatic 6-month-old infant brain segmentation algorithms: the iSeg-2017 challenge. IEEE Trans. Med. Imaging. 2019;38:2219-2230.

   Not in our library10.1109/tmi.2019.2901712

6. 6.

   Tang H, et al. Clinically applicable deep learning framework for organs at risk delineation in CT images. Nat. Mach. Intell. 2019;1:480-491.

   Not in our library10.1038/s42256-019-0099-z

7. 7.

   Khened M, Kollerathu VA, Krishnamurthi G. Fully convolutional multi-scale residual DenseNets for cardiac segmentation and automated cardiac diagnosis using ensemble of classifiers. Med. Image Anal. 2019;51:21-45.

   Not in our library10.1016/j.media.2018.10.004

8. 8.

   Litjens G, et al. A survey on deep learning in medical image analysis. Med. Image Anal. 2017;42:60-88.

   Not in our library10.1016/j.media.2017.07.005

9. 9.

   Shen D, Wu G, Suk H-I. Deep learning in medical image analysis. Annu. Rev. Biomed. Eng. 2017;19:221-248.

   Not in our library10.1146/annurev-bioeng-071516-044442

10. 10.
    Deep Learning Techniques for Medical Image Segmentation: Achievements and Challenges · 2019

    No notice
11. 11.

    Li X, et al. H-DenseUNet: Hybrid densely connected UNet for liver and tumor segmentation from CT volumes. IEEE Trans. Med. Imaging. 2018;37:2663-2674.

    Not in our library10.1109/tmi.2018.2845918

12. 12.

    Dolz J, et al. HyperDense-Net: a hyper-densely connected CNN for multi-modal image segmentation. IEEE Trans. Med. Imaging. 2019;38:1116-1126.

    Not in our library10.1109/tmi.2018.2878669

13. 13.

    Haberl MG, et al. CDeep3M—Plug-and-Play cloud-based deep learning for image segmentation. Nat. Methods. 2018;15:677-680.

    Not in our library10.1038/s41592-018-0106-z

14. 14.
    Deep learning · 2015

    No notice
15. 15.

    Webb S. Deep learning for biology. Nature. 2018;554:555-557.

    Not in our library10.1038/d41586-018-02174-z

16. 16.

    Karimi D, Dou H, Warfield SK, Gholipour A. Deep learning with noisy labels: exploring techniques and remedies in medical image analysis. Med. Image Anal. 2020;65:101759.

    Not in our library10.1016/j.media.2020.101759

17. 17.

    Joskowicz L, Cohen D, Caplan N, Sosna J. Inter-observer variability of manual contour delineation of structures in CT. Eur. Radiol. 2019;29:1391-1399.

    Not in our library10.1007/s00330-018-5695-5

18. 18.

    Towards trustable machine learning. Nat. Biomed. Eng. 2, 709–710 (2018).

    Not in our library10.1038/s41551-018-0315-x

19. 19.

    Lutnick B, et al. An integrated iterative annotation technique for easing neural network training in medical image analysis. Nat. Mach. Intell. 2019;1:112-119.

    Not in our library10.1038/s42256-019-0018-3

20. 20.

    Esteva A, et al. A guide to deep learning in healthcare. Nat. Med. 2019;25:24-29.

    Not in our library10.1038/s41591-018-0316-z

21. 21.

    Holmberg OG, et al. Self-supervised retinal thickness prediction enables deep learning from unlabelled data to boost classification of diabetic retinopathy. Nat. Mach. Intell. 2020;2:719-726.

    Not in our library10.1038/s42256-020-00247-1

22. 22.

    Tajbakhsh N, et al. Embracing imperfect datasets: a review of deep learning solutions for medical image segmentation. Med. Image Anal. 2020;63:101693.

    Not in our library10.1016/j.media.2020.101693

23. 23.

    Blum, A. & Mitchell, T. Combining labeled and unlabeled data with co-training. In Conference on Computational Learning Theory (eds Bartlett, P. L. & Mansour, Y.) 92–100 (ACM, 1998).

    Not in our library

24. 24.

    Qiao, S., Shen, W., Zhang, Z., Wang, B. & Yuille, A. Deep co-training for semi-supervised image recognition. In European Conference on Computer Vision (ECCV) (eds Ferrari, V., Hebert, M., Sminchisescu, C. & Weiss, Y.) 135–152 (Springer, Cham, 2018).

    Not in our library

25. 25.

    Peng J, Estrada G, Pedersoli M, Desrosiers C. Deep co-training for semi-supervised image segmentation. Pattern Recogn. 2020;107:107269.

    Not in our library10.1016/j.patcog.2020.107269

26. 26.

    Xia Y, et al. Uncertainty-aware multi-view co-training for semi-supervised medical image segmentation and domain adaptation. Med. Image Anal. 2020;65:101766.

    Not in our library10.1016/j.media.2020.101766

27. 27.

    Han, B. et al. Co-teaching: robust training of deep neural networks with extremely noisy labels. In Conference on Neural Information Processing Systems (NeurIPS) (Montreal, Canada, 2018).

    Not in our library

28. 28.

    Yu, X. et al. How does disagreement help generalization against label corruption? In International Conference on Machine Learning (ICML) (Long Beach, California, USA, 2019).

    Not in our library

29. 29.

    Falk T, et al. U-Net: deep learning for cell counting, detection, and morphometry. Nat. Methods. 2019;16:67-70.

    Not in our library10.1038/s41592-018-0261-2

30. 30.

    Hazirbas, C. & Ma, L. FuseNet: incorporating depth into semantic segmentation via fusion-based CNN architecture. In Asian Conference on Computer Vision (ACCV) (Taipei, Taiwan, China, 2016).

    Not in our library

31. 31.

    Vohra, Y. et al. Data distillation: towards omni-supervised learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 4119–4128 (IEEE, 2018).

    Not in our library

32. 32.

    Lee, D.-H. Pseudo-label: the simple and efficient semi-supervised learning method for deep neural networks. ICML2013 Workshop on Challenges in Representational Learning, Vol. 3 (Atlanta, USA, 2013).

    Not in our library

33. 33.

    Kavur AE, et al. CHAOS Challenge—combined (CT-MR) healthy abdominal organ segmentation. Med. Image Anal. 2021;69:101950.

    Not in our library10.1016/j.media.2020.101950

34. 34.

    Litjens G, et al. Evaluation of prostate segmentation algorithms for MRI: the PROMISE12 challenge. Med. Image Anal. 2014;18:359-373.

    Not in our library10.1016/j.media.2013.12.002

35. 35.

    Zhang J, Saha A, Zhu Z, Mazurowski MA. Hierarchical convolutional neural networks for segmentation of breast tumors in MRI with application to radiogenomics. IEEE Trans. Med. Imaging. 2019;38:435-447.

    Not in our library10.1109/tmi.2018.2865671

36. 36.

    Qiao M, et al. Three-dimensional breast tumor segmentation on DCE-MRI with a multilabel attention-guided joint-phaselearning network. Comput. Med. Imaging Graph. 2021;90:101909.

    Not in our library10.1016/j.compmedimag.2021.101909

37. 37.

    Sun H, et al. AUNet: attention-guided dense-upsampling networks for breast mass segmentation in whole mammograms. Phys. Med. Biol. 2020;65:055005.

    Not in our library10.1088/1361-6560/ab5745

38. 38.

    Zhou Y, Huang W, Dong P, Xia Y, Wang S. D-UNet: a dimension-fusion U shape network for chronic stroke lesion segmentation. IEEE/ACM Trans. Comput. Biol. Bioinform. 2021;18:940-950.

    Not in our library10.1109/tcbb.2019.2939522

39. 39.

    Yang, H. et al. CLCI-Net: Cross-level fusion and context inference networks for lesion segmentation of chronic stroke. In International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) (eds Shen, D. et al.) Vol. 11766, 266–274 (Springer, Cham, 2019).

    Not in our library

40. 40.

    Qi, K. et al. X-Net: brain stroke lesion segmentation based on depthwise separable convolution and long-range dependencies. In International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) (eds Shen, D. et al.) Vol. 11766, 247–255 (Springer, Cham, 2019).

    Not in our library

41. 41.

    Zhang, Z., Yang, L. & Zheng, Y. Translating and segmenting multimodal medical volumes with cycle- and shape-consistency generative adversarial network. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 9242–9251 (IEEE, 2018).

    Not in our library

42. 42.

    Chen C, Dou Q, Chen H, Qin J, Heng PA. Unsupervised bidirectional cross-modality adaptation via deeply synergistic image and feature alignment for medical image segmentation. IEEE Trans. Med. Imaging. 2020;39:2494-2505.

    Not in our library10.1109/tmi.2020.2972701

43. 43.

    Wang Y, et al. Augmenting vascular disease diagnosis by vasculature-aware unsupervised learning. Nat. Mach. Intell. 2020;2:337-346.

    Not in our library10.1038/s42256-020-0188-z

44. 44.

    Nie, D., Gao, Y., Wang, L. & Shen, D. ASDNet: attention based semi-supervised deep networks for medical image segmentation. In International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) (eds Frangi, A., Schnabel, J., Davatzikos, C., Alberola-López, C. & Fichtinger, G.) Vol. 11073, 370–378 (Springer, Cham, 2018).

    Not in our library

45. 45.

    Lee H, et al. An explainable deep-learning algorithm for the detection of acute intracranial haemorrhage from small datasets. Nat. Biomed. Eng. 2019;3:173-182.

    Not in our library10.1038/s41551-018-0324-9

46. 46.

    Li, Z., Kamnitsas, K. & Glocker, B. Overfitting of neural nets under class imbalance: analysis and improvements for segmentation. In International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) (eds Shen, D. et al.) Vol. 11766, 402–410 (Springer, Cham, 2019).

    Not in our library

47. 47.

    Ren X, et al. Interleaved 3D-CNNs for joint segmentation of small-volume structures in head and neck CT images. Med. Phys. 2018;45:2063-2075.

    Not in our library10.1002/mp.12837

48. 48.

    Bai, W. et al. Self-supervised learning for cardiac MR image segmentation by anatomical position prediction. In International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) (eds Shen, D. et al.) Vol. 11765, 541–549 (Springer, Cham, 2019).

    Not in our library

49. 49.

    Zhou, Z. et al. Models genesis: generic autodidactic models for 3d medical image analysis. In International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) (eds Shen, D. et al.) Vol. 11767, 384–393 (Springer, Cham, 2019).

    Not in our library10.1007/978-3-030-32251-9_42

50. 50.

    Hardt, M. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations (ICLR) (Toulon, France, 2017).

    Not in our library

51. 51.

    Le, H. et al. Pancreatic cancer detection in whole slide images using noisy label annotations. In International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) (eds Shen, D. et al.) Vol. 11764, 541–549 (Springer, Cham, 2019).

    Not in our library

52. 52.

    Zhu, H., Shi, J. & Wu, J. Pick-and-learn: automatic quality evaluation for noisy-labeled image segmentation. In International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) (eds Shen, D. et al.) Vol. 11769, 576–584 (Springer, Cham, 2019).

    Not in our library

53. 53.

    Shu, Y., Wu, X. & Li, W. LVC-Net: Medical image segmentation with noisy label based on local visual cues. In International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) (eds Shen, D. et al.) Vol. 11769, 558–566 (Springer, Cham, 2019).

    Not in our library

54. 54.

    Bai, W. et al. Semi-supervised learning for network-based cardiac MR image segmentation. In International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) (eds Descoteaux, M., Maier-Hein, L., Franz, A.,Jannin, P., Collins, D. & Duchesne, S.) Vol. 10434, 253–260 (Springer, Cham, 2017).

    Not in our library

55. 55.

    Xia, Y. et al. 3D semi-supervised learning with uncertainty-aware multi-view co-training. In Winter Conference on Applications of Computer Vision (WACV) 3646–3655 (IEEE, 2020).

    Not in our library

56. 56.

    Arplt, D. et al. A closer look at memorization in deep networks. In International Conference on Machine Learning (ICML) (Sydney, Australia, 2017).

    Not in our library

57. 57.

    Greenspan H, Ginneken Bvan, Summers RM. Deep learning in medical imaging: overview and future promise of an exciting new technique. IEEE Trans. Med. Imaging. 2016;35:1153-1159.

    Not in our library10.1109/tmi.2016.2553401

58. 58.

    Kavur AE, et al. Comparison of semi-automatic and deep learning-based automatic methods for liver segmentation in living liver transplant donors. Diagnostic Interv. Radiol. 2020;26:11-21.

    Not in our library10.5152/dir.2019.19025

59. 59.

    Kavur, A. E., Selver, M. A., Dicle, O., Barış, M. & Gezer, N. S. CHAOS—Combined (CT-MR) Healthy Abdominal Organ Segmentation Challenge Data (Zenodo, 2019).

    Not in our library10.1016/j.media.2020.101950

60. 60.

    Bloch, N. et al. NCI-ISBI 2013 challenge: automated segmentation of prostate structures. In International Symposium on Biomedical Imaging (ISBI), (San Francisco, California, USA, 2013).

    Not in our library

61. 61.

    Hinton, G., Vinyals, O. & Dean, J. Distilling the knowledge in a neural network. In Conference on Neural Information Processing Systems (NeurIPS) (Montreal, Canada, 2015).

    Not in our library

62. 62.

    Berthelot, D. et al. MixMatch: a holistic approach to semi-supervised learning. In Conference on Neural Information Processing Systems (NeurIPS) (Vancouver, Canada, 2019).

    Not in our library

63. 63.

    Jiang, L., Zhou, Z., Leung, T., Li, L.-J. & Li, F.-F. MentorNet: learning data-driven curriculum for very deep neural networks on corrupted labels. In International Conference on Machine Learning (ICML) (Stockholm, Sweden, 2018).

    Not in our library

64. 64.

    Chen, P., Liao, B., Chen, G. & Zhang, S. Understanding and utilizing deep neural networks trained with noisy labels. In International Conference on Machine Learning (ICML) (Long Beach, California, USA, 2019).

    Not in our library

65. 65.

    Li, C. et al. Learning cross-modal deep representations for multi-modal MR image segmentation. In International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) (eds Shen, D. et al.) Vol. 11765, 57–65 (Springer, Cham, 2019).

    Not in our library

66. 66.
    Annotation-efficient deep learning for automatic medical image segmentation · 2021

    No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.