Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks

Abstract

Despite the state-of-the-art performance for medical image segmentation, deep convolutional neural networks (CNNs) have rarely provided uncertainty estimations regarding their segmentation outputs, e.g., model (epistemic) and image-based (aleatoric) uncertainties. In this work, we analyze these different types of uncertainties for CNN-based 2D and 3D medical image segmentation tasks at both pixel level and structure level. We additionally propose a test-time augmentation-based aleatoric uncertainty to analyze the effect of different transformations of the input image on the segmentation output. Test-time augmentation has been previously used to improve segmentation accuracy, yet not been formulated in a consistent mathematical framework. Hence, we also propose a theoretical formulation of test-time augmentation, where a distribution of the prediction is estimated by Monte Carlo simulation with prior distributions of parameters in an image acquisition model that involves image transformations and noise. We compare and combine our proposed aleatoric uncertainty with model uncertainty. Experiments with segmentation of fetal brains and brain tumors from 2D and 3D Magnetic Resonance Images (MRI) showed that 1) the test-time augmentation-based aleatoric uncertainty provides a better uncertainty estimation than calculating the test-time dropout-based model uncertainty alone and helps to reduce overconfident incorrect predictions, and 2) our test-time augmentation outperforms a single-prediction baseline and dropout-based multiple predictions.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 44

1. 1.
   Automated medical image segmentation techniques · 2010

   No notice
2. 2.

   Withey D, Koles ZP. Medical image segmentation: methods and software. 2007;140-143.

   Not in our library

3. 3.

   Prassni JS, Ropinski T, Hinrichs K. Uncertainty-aware guided volume segmentation. IEEE Trans Vis Comput Grap. 2010;16(6):1358-1365.

   Not in our library10.1109/tvcg.2010.208 (opens in a new tab)

4. 4.

   Wang G, Li W, Zuluaga MA, Pratt R, Patel PA, Aertsen M, Doel T, David AL, Deprest J, Ourselin S, Vercauteren T. Interactive medical image segmentation using deep learning with image-specific fine-tuning. IEEE Trans Med Imaging. 2018;37(7):1562-1573.

   Not in our library10.1109/tmi.2018.2791721 (opens in a new tab)

5. 5.

   Milletari F, Navab N, Ahmadi S-A. V-Net: fully convolutional neural networks for volumetric medical image segmentation. 2016;565-571.

   Not in our library

6. 6.

   Abdulkadir A, Lienkamp SS, Brox T, Ronneberger O. 3D U-Net: learning dense volumetric segmentation from sparse annotation. 2016;424-432.

   Not in our library

7. 7.

   Kamnitsas K, Ledig C, Newcombe VFJ, Simpson JP, Kane AD, Menon DK, Rueckert D, Glocker B. Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Med Image Anal. 2017;36:61-78.

   Not in our library10.1016/j.media.2016.10.004 (opens in a new tab)

8. 8.

   Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, Thrun S. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017;542(7639):115-118.

   Not in our library10.1038/nature21056 (opens in a new tab)

9. 9.

   Rajpurkar P, Irvin J, Bagul A, Ding D, Duan T, Mehta H, Yang B, Zhu K, Laird D, Ball RL, Langlotz C, et al. MURA dataset: towards radiologist-level abnormality eetection in musculoskeletal radiographs. arXiv: 1712.06957. 2017.

   Not in our library

10. 10.

    Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, Berg AC, et al. ImageNet large scale visual recognition challenge. Int J Comput Vis. 2015;115(3):211-252.

    Not in our library10.1007/s11263-015-0816-y (opens in a new tab)

11. 11.

    Kendall A, Gal Y. What uncertainties do we need in Bayesian deep learning for computer vision? 2017;5580-5590.

    Not in our library10.1109/tdei.2009.5211872 (opens in a new tab)

12. 12.

    Lakshminarayanan B, Pritzel A, Blundell C. Simple and scalable predictive uncertainty estimation using deep ensembles. 2017;6405-6416.

    Not in our library

13. 13.

    Zhu Y, Zabaras N. Bayesian deep convolutional encoder-decoder networks for surrogate modeling and uncertainty quantification. arXiv: 1801.06879. 2018.

    Not in our library

14. 14.

    Ayhan MS, Berens P. Test-time data augmentation for estimation of heteroscedastic aleatoric uncertainty in deep neural networks. 2018;1-9.

    Not in our library

15. 15.

    Nair T, Precup D, Arnold DL, Arbel T. Exploring uncertainty measures in deep networks for multiple sclerosis lesion detection and segmentation. 2018;655-663.

    Not in our library10.1016/j.media.2019.101557 (opens in a new tab)

16. 16.

    Roy AG, Conjeti S, Navab N, Wachinger C. Inherent brain segmentation quality control from fully convnet Monte Carlo sampling. 2018;664-672.

    Not in our library

17. 17.

    Matsunaga K, Hamada A, Minagawa A, Koga H. Image classification of melanoma, nevus and seborrheic keratosis by deep neural network ensemble. arXiv: 1703.03108. 2017.

    Not in our library

18. 18.

    Jin H, Li Z, Tong R, Lin L. A deep 3D residual CNN for false positive reduction in pulmonary nodule detection. Med Phys. 2018;45(5):2097-2107.

    Not in our library10.1002/mp.128 (opens in a new tab)

19. 19.

    Saad A, Hamarneh G, Möller T. Exploration and visualization of segmentation uncertainty using shape and appearance prior information. IEEE Trans Vis Comput Graph. 2010;16(6):1366-1375.

    Not in our library10.1109/tvcg.2010.152 (opens in a new tab)

20. 20.

    Shi W, Zhuang X, Wolz R, Simon D, Tung K, Wang H, Ourselin S, Edwards P, Razavi R, Rueckert D. A multi-image graph cut approach for cardiac image segmentation and uncertainty estimation. 2011;7085:178-187.

    Not in our library10.1007/978-3-642-28326-0_18 (opens in a new tab)

21. 21.

    Sankaran S, Grady L, Taylor CA. Fast computation of hemodynamic sensitivity to lumen segmentation uncertainty. IEEE Trans Med Imaging. 2015;34(12):2562-2571.

    Not in our library10.1109/tmi.2015.2445777 (opens in a new tab)

22. 22.

    Parisot S, Wells W, Chemouny S, Duffau H, Paragios N. Concurrent tumor segmentation and registration with uncertainty-based sparse non-uniform graphs. Med Image Anal. 2014;18(4):647-659.

    Not in our library10.1016/j.media.2014.02.006 (opens in a new tab)

23. 23.

    Top A, Hamarneh G, Abugharbieh R. Active learning for interactive 3D image segmentation. 2011;603-610.

    Not in our library10.1300/j172v06n01_05 (opens in a new tab)

24. 24.

    Gal Y, Ghahramani Z. Dropout as a Bayesian approximation: representing model uncertainty in deep learning. 2016;1050-1059.

    Not in our library

25. 25.

    Li W, Wang G, Fidon L, Ourselin S, Cardoso MJ, Vercauteren T. On the compactness, efficiency, and representation of 3D convolutional networks: brain parcellation as a pretext task. 2017;348-360.

    Not in our library

26. 26.

    Neal RM. Bayesian Learning for Neural Networks. 2012.

    Not in our library

27. 27.

    Teye M, Azizpour H, Smith K. Bayesian uncertainty estimation for batch normalized deep detworks. arXiv: 1802.06455. 2018.

    Not in our library

28. 28.

    Graves A. Practical variational inference for neural networks. Advances in Neural Information Processing Systems. 2011;1-9.

    Not in our library

29. 29.

    Louizos C, Welling M. Structured and efficient variational deep learning with matrix Gaussian posteriors. 2016;1708-1716.

    Not in our library

30. 30.

    Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems. 2012;1097-1105.

    Not in our library

31. 31.

    Ronneberger O, Fischer P, Brox T. U-Net: convolutional networks for biomedical image segmentation. 2015;234-241.

    Not in our library

32. 32.

    Radosavovic I, Dollár P, Girshick R, Gkioxari G, He K. Data distillation: towards omni-supervised learning. arXiv: 1712.04440. 2017.

    Not in our library

33. 33.

    Yue L, Shen H, Li J, Yuan Q, Zhang H, Zhang L. Image super-resolution: the techniques, applications, and future. Signal Process. 2016;128:389-408.

    Not in our library10.1016/j.sigpro.2016.05.002 (opens in a new tab)

34. 34.

    Tourbier S, Velasco-Annis C, Taimouri V, Hagmann P, Meuli R, Warfield SK, Bach Cuadra M, Gholipour A. Automated template-based brain localization and extraction for fetal brain MRI reconstruction. NeuroImage. 2017;155:460-472.

    Not in our library10.1016/j.neuroimage.2017.04.004 (opens in a new tab)

35. 35.

    Ebner M, Wang G, Li W, Aertsen M, Patel PA, Aughwane R, Melbourne A, Doel T, David AL, Deprest J, Ourselin S, et al. An automated localization, segmentation and reconstruction framework for fetal brain MRI. 2018;10433:313-320.

    Not in our library

36. 36.

    Rajchl M, Lee MCH, Schrans F, Davidson A, Passerat-Palmbach J, Tarroni G, Alansary A, Oktay O, Kainz B, Rueckert D. Learning under distributed weak supervision. arXiv: 1606.01100. 2016.

    Not in our library

37. 37.

    Salehi SSM, Erdogmus D, Gholipour A. Auto-context convolutional neural network (auto-net) for brain extraction in magnetic resonance imaging. IEEE Trans Med Imaging. 2017;36(11):2319-2330.

    Not in our library10.1109/tmi.2017.2721362 (opens in a new tab)

38. 38.

    Salehi SSM, Hashemi SR, Velasco-Annis C, Ouaalam A, Estroff JA, Erdogmus D, Warfield SK, Gholipour A. Real-time automatic fetal brain extraction in fetal MRI by deep learning. IEEE International Symposium on Biomedical Imaging. 2018;720-724.

    Not in our library

39. 39.

    Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation. 2015;3431-3440.

    Not in our library10.1109/cvpr.2015.7298965 (opens in a new tab)

40. 40.

    Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, Ghemawat S, Irving G, Isard M, Kudlur M, et al. TensorFlow: a system for large-scale machine learning. USENIX Symposium on Operating Systems Design and Implementation. 2016;265-284.

    Not in our library

41. 41.
    NiftyNet: a deep-learning platform for medical imaging · 2018

    No notice
42. 42.

    Menze BH, Jakab A, Bauer S, Kalpathy-Cramer J, Farahani K, Kirby J, Burren Y, Porz N, Slotboom J, Wiest R, Lanczi L, et al. The multimodal brain tumor image segmentation benchmark(BRATS). IEEE Trans Med Imaging. 2015;34(10):1993-2024.

    Not in our library10.1109/tmi.2014.2377694 (opens in a new tab)

43. 43.

    Wang G, Li W, Ourselin S, Vercauteren T. Automatic brain tumor segmentation using cascaded anisotropic convolutional neural networks. Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries. 2018;178-190.

    Not in our library

44. 44.
    Advancing The Cancer Genome Atlas glioma MRI collections with expert segmentation labels and radiomic features · 2017

    No notice

Cited by papers we hold 3

• Addressing architectural distortion in mammogram using AlexNet and support vector machine · 2021
  Not checked
• TorchIO: A Python library for efficient loading, preprocessing, augmentation and patch-based sampling of medical images in deep learning · 2021
  No notice
• Data Augmentation for Brain-Tumor Segmentation: A Review · 2019
  No notice

Questions about this paper

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.