DeepIGeoS: A Deep Interactive Geodesic Framework for Medical Image Segmentation

Abstract

Accurate medical image segmentation is essential for diagnosis, surgical planning and many other applications. Convolutional Neural Networks (CNNs) have become the state-of-the-art automatic segmentation methods. However, fully automatic results may still need to be refined to become accurate and robust enough for clinical use. We propose a deep learning-based interactive segmentation method to improve the results obtained by an automatic CNN and to reduce user interactions during refinement for higher accuracy. We use one CNN to obtain an initial automatic segmentation, on which user interactions are added to indicate mis-segmentations. Another CNN takes as input the user interactions with the initial segmentation and gives a refined result. We propose to combine user interactions with CNNs through geodesic distance transforms, and propose a resolution-preserving network that gives a better dense prediction. In addition, we integrate user interactions as hard constraints into a back-propagatable Conditional Random Field. We validated the proposed framework in the context of 2D placenta segmentation from fetal MRI and 3D brain tumor segmentation from FLAIR images. Experimental results show our method achieves a large improvement from automatic CNNs, and obtains comparable and even higher accuracy with fewer user interventions and less time compared with traditional interactive methods.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 65

1. 1.

   Withey D. and Koles Z., “Medical image segmentation: Methods and software,” in Proc. Joint Meeting 6th Int. Symp. Noninvasive Functional Source Imaging Brain Heart Int. Conf. Functional Biomedical Imaging, 2007, pp. 140–143.

   Not in our library

2. 2.
   Automated medical image segmentation techniques · 2010

   No notice
3. 3.

   Zhao F. and Xie X., “An overview of interactive medical image segmentation,” Ann. BMVA, vol. 2013, no. 7, pp. 1–22, 2013.

   Not in our library

4. 4.

   Hamamci A., Kucuk N., Karaman K., Engin K., and Unal G., “Tumor-Cut: Segmentation of brain tumors on contrast enhanced MR images for radiosurgery applications,” IEEE Trans. Medical Imaging, vol. 31, no. 3, pp. 790–804, Mar. 2012.

   Not in our library10.1109/tmi.2011.2181857

5. 5.

   Yushkevich P. A., Piven J., Hazlett H. C., Smith R. G., Ho S., Gee J. C., and Gerig G., “User-guided 3D active contour segmentation of anatomical structures: Significantly improved efficiency and reliability,” NeuroImage, vol. 31, no. 3, pp. 1116–1128, 2006.

   Not in our library10.1016/j.neuroimage.2006.01.015

6. 6.

   Wang G., Zuluaga M. A., Pratt R., Aertsen M., Doel T., Klusmann M., David A. L., Deprest J., Vercauteren T., and Ourselin S., “Slic-Seg: A minimally interactive segmentation of the placenta from sparse and motion-corrupted fetal MRI in multiple views,” Med. Image Anal., vol. 34, pp. 137–147, 2016.

   Not in our library10.1016/j.media.2016.04.009

7. 7.

   Grady L., “Random walks for image segmentation,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 11, pp. 1768–1783, Nov. 2006.

   Not in our library10.1109/tpami.2006.233

8. 8.

   Boykov Y. Y. and Jolly M. P., “Interactive graph cuts for optimal boundary & region segmentation of objects in N-D images,” in Proc. 8th IEEE Int. Conf. Comput. Vis., 2001, pp. 105–112.

   Not in our library

9. 9.

   Rother C., Kolmogorov V., and Blake A., “GrabCut: Interactive foreground extraction using iterated graph cuts,” ACM Trans. Graph., vol. 23, no. 3, pp. 309–314, 2004.

   Not in our library

10. 10.

    Litjens G., Kooi T., Bejnordi B. E., Setio A. A. A., Ciompi F., Ghafoorian M., van der Laak J. A., Ginneken B. V., and Sánchez C. I., “A survey on deep learning in medical image analysis,” Med. Image Anal., vol. 42, pp. 60–88, 2017.

    Not in our library10.1016/j.media.2017.07.005

11. 11.

    Long J., Shelhamer E., and Darrell T., “Fully convolutional networks for semantic segmentation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2015, pp. 3431–3440.

    Not in our library10.1109/tpami.2016.2572683

12. 12.

    Ronneberger O., Fischer P., and Brox T., “U-Net: Convolutional networks for biomedical image segmentation,” in Proc. Medical Image Comput. Comput.-Assisted Intervention, 2015, pp. 234–241.

    Not in our library

13. 13.

    Milletari F., Navab N., and Ahmadi S.-A., “V-Net: Fully convolutional neural networks for volumetric medical image segmentation,” in Proc. 4th Int. Conf. 3D Vis., 2016, pp. 565–571.

    Not in our library

14. 14.

    Li W., Wang G., Fidon L., Ourselin S., Cardoso M. J., and Vercauteren T., “On the compactness, efficiency, and representation of 3D convolutional networks: brain parcellation as a pretext task,” in Proc. Inf. Process. Medical Imaging, 2017, pp. 348–360.

    Not in our library

15. 15.

    Kamnitsas K., Ledig C., Newcombe V. F. J., Simpson J. P., Kane A. D., Menon D. K., Rueckert D., and Glocker B., “Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation,” Med. Image Anal., vol. 36, pp. 61–78, 2017.

    Not in our library10.1016/j.media.2016.10.004

16. 16.

    Krizhevsky A., Sutskever I., and Hinton G. E., “ImageNet classification with deep convolutional neural networks,” in Proc. 25th Int. Conf. Neural Inf. Process. Syst., 2012, pp. 1097–1105.

    Not in our library

17. 17.

    Simonyan K. and Zisserman A., “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556.

    Not in our library

18. 18.

    He K., Zhang X., Ren S., and Sun J., “Deep residual learning for image recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 770–778.

    Not in our library

19. 19.

    Havaei M., Davy A., Warde-Farley D., Biard A., Courville A., Bengio Y., Pal C., Jodoin P.-M., and Larochelle H., “Brain tumor segmentation with deep neural networks,” Med. Image Anal., vol. 35, pp. 18–31, 2016.

    Not in our library10.1016/j.media.2016.05.004

20. 20.

    Girshick R., Donahue J., Darrell T., and Malik J., “Rich feature hierarchies for accurate object detection and semantic segmentation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2014, pp. 580–587.

    Not in our library

21. 21.

    Abdulkadir A., Lienkamp S. S., Brox T., and Ronneberger O., “3D U-Net: Learning dense volumetric segmentation from sparse annotation,” in Proc. Medical Image Comput. Comput.-Assisted Intervention, 2016, pp. 424–432.

    Not in our library

22. 22.

    Chen L.-C., Papandreou G., Kokkinos I., Murphy K., and Yuille A. L., “Semantic image segmentation with deep convolutional nets and fully connected CRFs,” arXiv preprint arXiv:1412.7062.

    Not in our library10.1109/tpami.2017.2699184

23. 23.

    Yu F. and Koltun V., “Multi-scale context aggregation by dilated convolutions,” arXiv preprint arXiv:1511.07122.

    Not in our library

24. 24.

    Ondruska P., Dequaire J., Wang D. Z., and Posner I., “End-to-end tracking and semantic segmentation using recurrent neural networks,” arXiv preprint arXiv:1604.05091.

    Not in our library

25. 25.

    Dai J., He K., Li Y., Ren S., and Sun J., “Instance-sensitive fully convolutional networks,” in Proc. 14th Eur. Conf. Comput. Vis., 2016, pp. 534–549.

    Not in our library

26. 26.

    Lea C., Vidal R., Reiter A., and Hager G. D., “Temporal convolutional networks for action segmentation and detection,” in Proc. Eur. Conf. Comput. Vis., 2016, pp. 47–54.

    Not in our library

27. 27.

    Lin G., Shen C., Reid I., and van dan Hengel A., “Efficient piecewise training of deep structured models for semantic segmentation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 3194–3203.

    Not in our library

28. 28.

    Pinheiro P. and Collobert R., “Recurrent convolutional neural networks for scene labeling,” in Proc. 31st Int. Conf. Int. Conf. Mach. Learn., 2014, pp. 82–90.

    Not in our library

29. 29.

    Hariharan B., Arbeláez P., Girshick R., and Malik J., “Hypercolumns for object segmentation and fine-grained localization,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2015, pp. 447–456.

    Not in our library10.1109/tpami.2016.2578328

30. 30.

    Armstrong C. J., Price B. L., and Barrett W. A., “Interactive segmentation of image volumes with live surface,” Comput. Graph., vol. 31, no. 2, pp. 212–229, 2007.

    Not in our library

31. 31.

    Cates J. E., Lefohn A. E., and Whitaker R. T., “GIST: An interactive, GPU-based level set segmentation tool for 3D medical images,” Med. Image Anal., vol. 8, no. 3, pp. 217–231, 2004.

    Not in our library10.1016/j.media.2004.06.022

32. 32.

    Rajchl M., Lee M., Oktay O., Kamnitsas K., Passerat-Palmbach J., Bai W., Rutherford M., Hajnal J., Kainz B., and Rueckert D., “DeepCut: Object segmentation from bounding box annotations using convolutional neural networks,” IEEE Trans. Medical Imaging, vol. 36, no. 2, pp. 674–683, Feb. 2017.

    Not in our library10.1109/tmi.2016.2621185

33. 33.

    Haider S. A., Shafiee M. J., Chung A., Khalvati F., Oikonomou A., Wong A., and Haider M. A., “Single-click, semi-automatic lung nodule contouring using hierarchical conditional random fields,” in Proc. IEEE 12th Int. Symp. Biomed. Imaging, 2015, pp. 1139–1142.

    Not in our library

34. 34.

    Xu C. and Prince J. L., “Snakes, shapes, and gradient vector flow,” IEEE Trans. Image Process., vol. 7, no. 3, pp. 359–369, Mar. 1998.

    Not in our library10.1109/83.661186

35. 35.

    Bai X. and Sapiro G., “A geodesic framework for fast interactive image and video segmentation and matting,” in Proc. IEEE 11th Int. Conf. Comput. Vis., 2007, pp. 1–8.

    Not in our library

36. 36.

    Criminisi A., Sharp T., and Blake A., “GeoS: Geodesic image segmentation,” in Proc. Eur. Conf. Comput. Vis., 2008, pp. 99–112.

    Not in our library

37. 37.

    Barinova O., Shapovalov R., Sudakov S., and Velizhev A., “Online random forest for interactive image segmentation,” in Proc. Int. Workshop Exp. Economics Mach. Learn., 2012, pp. 1–8.

    Not in our library

38. 38.

    Luengo I., Darrow M. C., Spink M. C., Sun Y., Dai W., He C. Y., Chiu W., Pridmore T., Ashton A. W., Duke E. M., Basham M., and French A. P., “SuRVoS: Super-region volume segmentation workbench,” J. Structural Biol., vol. 198, no. 1, pp. 43–53, 2017.

    Not in our library10.1016/j.jsb.2017.02.007

39. 39.

    Lin D., Dai J., Jia J., He K., and Sun J., “ScribbleSup: Scribble-supervised convolutional networks for semantic segmentation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 3159–3167.

    Not in our library

40. 40.

    Xu N., Price B., Cohen S., Yang J., and Huang T., “Deep interactive object selection,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 373–381.

    Not in our library

41. 41.

    Kontschieder P., Kohli P., Shotton J., and Criminisi A., “GeoF geodesic forests for learning coupled predictors,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2013, pp. 65–72.

    Not in our library

42. 42.

    Boykov Y. and Kolmogorov V., “An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 26, no. 9, pp. 1124–1137, Sep. 2004.

    Not in our library10.1109/tpami.2004.60

43. 43.

    Yuan J., Bae E., and Tai X. C., “A study on continuous max-flow and min-cut approaches,” in Proc. IEEE Comput. Society Conf. Comput. Vis. Pattern Recognit., 2010, pp. 2217–2224.

    Not in our library

44. 44.

    Payet N. and Todorovic S., “(RF)^2 – random forest random field,” in Proc. Neural Inf. Process. Syst. Conf., 2010, pp. 1885–1893.

    Not in our library

45. 45.

    Krähenbühl P. and Koltun V., “Efficient inference in fully connected CRFs with gaussian edge potentials,” in Proc. Neural Inf. Process. Syst. Conf., 2011, pp. 109–117.

    Not in our library

46. 46.

    Szummer M., Kohli P., and Hoiem D., “Learning CRFs using graph cuts,” in Proc. Eur. Conf. Comput. Vis., 2008, pp. 582–595.

    Not in our library

47. 47.

    Orlando J. I. and Blaschko M., “Learning fully-connected CRFs for blood bessel segmentation in retinal images,” in Proc. Medical Image Comput. Comput.-Assisted Intervention, 2014, pp. 634–641.

    Not in our library10.1007/978-3-319-10404-1_79

48. 48.

    Domke J., “Learning graphical model parameters with approximate marginal inference,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 10, pp. 2454–2467, Oct. 2013.

    Not in our library10.1109/tpami.2013.31

49. 49.

    Krähenbühl P. and Koltun V., “Parameter learning and convergent inference for dense random fields,” in Proc. 30th Int. Conf. Int. Conf. Mach. Learn., 2013, pp. 513–521.

    Not in our library

50. 50.

    Zheng S., Jayasumana S., Romera-Paredes B., Vineet V., Su Z., Du D., Huang C., and Torr P. H. S., “Conditional random fields as recurrent neural networks,” in Proc. IEEE Int. Conf. Comput. Vis., 2015, pp. 1529–1537.

    Not in our library

51. 51.

    Adams A., Baek J., and Davis M. A., “Fast high-dimensional filtering using the permutohedral lattice,” Comput. Graph. Forum, vol. 29, no. 2, pp. 753–762, 2010.

    Not in our library

52. 52.

    Vemulapalli R., Tuzel O., Liu M.-y., and Chellappa R., “Gaussian conditional random field network for semantic segmentation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 3224–3233.

    Not in our library

53. 53.

    Liu F., Shen C., and Lin G., “Deep convolutional neural fields for depth estimation from a single image,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2014, pp. 5162–5170.

    Not in our library

54. 54.

    Kirillov A., Zheng S., Schlesinger D., Forkel W., Zelenin A., Torr P., and Rother C., “Efficient likelihood learning of a generic CNN-CRF model for semantic segmentation,” arXiv preprint arXiv:1511.05067.

    Not in our library

55. 55.

    Wang G., Zuluaga M. A., Pratt R., Aertsen M., Doel T., Klusmann M., David A. L., Deprest J., Vercauteren T., and Ourselin S., “Dynamically balanced online random forests for interactive scribble-based segmentation,” in Proc. Medical Image Comput. Comput.-Assisted Intervention, 2016, pp. 352–360.

    Not in our library

56. 56.

    Kolmogorov V. and Zabih R., “What energy functions can be minimized via graph cuts?” IEEE Trans. Pattern Anal. Mach. Intell., vol. 26, no. 2, pp. 147–159, Feb. 2004.

    Not in our library10.1109/tpami.2004.1262177

57. 57.

    Han D., Bayouth J., Song Q., Taurani A., Sonka M., Buatti J., and Wu X., “Globally optimal tumor segmentation in PET-CT images: A graph-based co-segmentation method,” in Proc. Inf. Process. Medical Imaging, 2011, pp. 245–256.

    Not in our library10.1007/978-3-642-22092-0_21

58. 58.

    Chen L.-C., Papandreou G., Kokkinos I., Murphy K., and Yuille A. L., “DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 40, no. 4, pp. 834–8481, Apr. 2018.

    Not in our library10.1109/tpami.2017.2699184

59. 59.

    Jia Y., Shelhamer E., Donahue J., Karayev S., Long J., Girshick R., Guadarrama S., and Darrell T., “Caffe: Convolutional architecture for fast feature embedding,” in Proc. 22nd ACM Int. Conf. Multimedia, 2014, pp. 675–678.

    Not in our library

60. 60.

    Abadi M., Barham P., Chen J., Chen Z., Davis A., Dean J., Devin M., Ghemawat S., Irving G., Isard M., Kudlur M., Levenberg J., Monga R., Moore S., Murray D. G., Steiner B., Tucker P., Vasudevan V., Warden P., Wicke M., Yu Y., Zheng X., and Brain G., “TensorFlow: A system for large-scale machine learning,” in Proc. 12th USENIX Conf. Operating Syst. Des. Implementation, 2016, pp. 265–284.

    Not in our library

61. 61.

    Deprest J. A., Flake A. W., Gratacos E., Ville Y., Hecher K., Nicolaides K., Johnson M. P., Luks F. I., Adzick N. S., and Harrison M. R., “The making of fetal surgery,” Prenatal Diagnosis, vol. 30, no. 7, pp. 653–667, 2010.

    Not in our library10.1002/pd.2571

62. 62.

    Alansary A., Kamnitsas K., Davidson A., Rajchl M., Malamateniou C., Rutherford M., Hajnal J. V., Glocker B., Rueckert D., and Kainz B., “Fast fully automatic segmentation of the human placenta from motion corrupted MRI,” in Proc. Medical Image Comput. Comput.-Assisted Intervention, 2016, pp. 589–597.

    Not in our library

63. 63.

    Keraudren K., Kuklisova-Murgasova M., Kyriakopoulou V., Malamateniou C., Rutherford M. A., Kainz B., Hajnal J. V., and Rueckert D., “Automated fetal brain segmentation from 2D MRI slices for motion correction,” NeuroImage, vol. 101, pp. 633–643, 2014.

    Not in our library10.1016/j.neuroimage.2014.07.023

64. 64.

    Menze B. H., Jakab A., Bauer S., Kalpathy-Cramer J., Farahani K., Kirby J., Burren Y., Porz N., Slotboom J., Wiest R., Lanczi L., Gerstner E., Weber M. A., Arbel T., Avants B. B., Ayache N., Buendia P., Collins D. L., Cordier N., Corso J. J., Criminisi A., Das T., Delingette H., Demiralp Ç., Durst C. R., Dojat M., Doyle S., Festa J., Forbes F., Geremia E., Glocker B., Golland P., Guo X., Hamamci A., Iftekharuddin K. M., Jena R., John N. M., Konukoglu E., Lashkari D., Mariz J. A., Meier R., Pereira S., Precup D., Price S. J., Raviv T. R., Reza S. M., Ryan M., Sarikaya D., Schwartz L., Shin H. C., Shotton J., Silva C. A., Sousa N., Subbanna N. K., Szekely G., Taylor T. J., Thomas O. M., Tustison N. J., Unal G., Vasseur F., Wintermark M., Ye D. H., Zhao L., Zhao B., Zikic D., Prastawa M., Reyes M., and Van Leemput K., “The multimodal brain tumor image segmentation benchmark (BRATS),” IEEE Trans. Medical Imaging, vol. 34, no. 10, pp. 1993–2024, Oct. 2015.

    Not in our library10.1109/tmi.2014.2377694

65. 65.

    Fidon L., Li W., Garcia-Peraza-Herrera L. C., Ekanayake J., Kitchen N., Ourselin S., and Vercauteren T., “Scalable multimodal convolutional networks for brain tumour segmentation,” in Proc. Medical Image Comput. Comput.-Assisted Intervention, 2017, pp. 285–293.

    Not in our library

Cited by papers we hold 4

• Segment anything in medical images · 2024
  No notice
• Introducing Biomedisa as an open-source online platform for biomedical image segmentation · 2020
  Corrected
• Automatic Segmentation of Human Placenta Images With U-Net · 2019
  Not checked
• NiftyNet: a deep-learning platform for medical imaging · 2018
  No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.