SonoNet: Real-Time Detection and Localisation of Fetal Standard Scan Planes in Freehand Ultrasound

Abstract

Identifying and interpreting fetal standard scan planes during 2-D ultrasound mid-pregnancy examinations are highly complex tasks, which require years of training. Apart from guiding the probe to the correct location, it can be equally difficult for a non-expert to identify relevant structures within the image. Automatic image processing can provide tools to help experienced as well as inexperienced operators with these tasks. In this paper, we propose a novel method based on convolutional neural networks, which can automatically detect 13 fetal standard views in freehand 2-D ultrasound data as well as provide a localization of the fetal structures via a bounding box. An important contribution is that the network learns to localize the target anatomy using weak supervision based on image-level labels only. The network architecture is designed to operate in real-time while providing optimal output for the localization task. We present results for real-time annotation, retrospective frame retrieval from saved videos, and localization on a very large and challenging dataset consisting of images and video recordings of full clinical anomaly screenings. We found that the proposed method achieved an average F1-score of 0.798 in a realistic classification experiment modeling real-time detection, and obtained a 90.09% accuracy for retrospective frame retrieval. Moreover, an accuracy of 77.8% was achieved on the localization task.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 36

1. 1.

   Abuhamad A., Falkensammer P., Reichartseder F., and Zhao Y., “Automated retrieval of standard diagnostic fetal cardiac ultrasound planes in the second trimester of pregnancy: A prospective evaluation of software,” Ultrasound Obstetrics, Gynecol., vol. 31, no. 1, pp. 30–36, 2008.

   Not in our library10.1002/uog.5228 (opens in a new tab)

2. 2.

   Baumgartner C. F., Kamnitsas K., Matthew J., Smith S., Kainz B., and Rueckert D., “Real-time standard scan plane detection and localisation in fetal ultrasound using fully convolutional neural networks,” in Medical Image Computing and Computer-Assisted Intervention—MICCAI. Cham, Switzerland: Springer, 2016, pp. 203–211.

   Not in our library

3. 3.

   Boykov Y., Veksler O., and Zabih R., “Fast approximate energy minimization via graph cuts,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 11, pp. 1222–1239, Nov. 2001.

   Not in our library

4. 4.

   Bridge C. P., Ioannou C., and Noble J. A., “Automated annotation and quantitative description of ultrasound videos of the fetal heart,” Med. Image Anal., vol. 36, pp. 147–161, Feb. 2017.

   Not in our library10.1016/j.media.2016.11.006 (opens in a new tab)

5. 5.

   Bridge C. P. and Noble J. A., “Object localisation in fetal ultrasound images using invariant features,” in Proc. ISBI, Apr. 2015, pp. 156–159.

   Not in our library

6. 6.

   Springett A., British Isles Network of Congenital Anomaly Registers, Congenital Anomaly Statistics 2010 England and Wales, London, U.K.: 2014.

   Not in our library

7. 7.

   Chan L. W., Fung T. Y., Leung T. Y., Sahota D. S., and Lau T. K., “Volumetric (3D) imaging reduces inter- and intraobserver variation of fetal biometry measurements,” Ultrasound Obstetrics, Gynecol., vol. 33, no. 4, pp. 447–452, 2009.

   Not in our library10.1002/uog.6321 (opens in a new tab)

8. 8.

   Chen H., “Automatic fetal ultrasound standard plane detection using knowledge transferred recurrent neural networks,” in Medical Image Computing and Computer-Assisted Intervention—MICCAI. Cham, Switzerland: Springer, 2015, pp. 507–514.

   Not in our library

9. 9.

   Chen H., Dou Q., Wang X., Qin J., and Heng P.-A., “Mitosis detection in breast cancer histology images via deep cascaded networks,” in Proc. AAAI, 2016, pp. 1160–1166.

   Not in our library

10. 10.

    Chen H., “Standard plane localization in fetal ultrasound via domain transferred deep neural networks,” IEEE J. Biomed. Health Inform., vol. 19, no. 5, pp. 1627–1636, Sep. 2015.

    Not in our library10.1109/jbhi.2015.2425041 (opens in a new tab)

11. 11.

    El-Zaart A., “Images thresholding using isodata technique with gamma distribution,” Pattern Recognit. Image Anal., vol. 20, no. 1, pp. 29–41, 2010.

    Not in our library

12. 12.

    Everingham M., Van Gool L., Williams C. K. I., Winn J., and Zisserman A., “The Pascal visual object classes (VOC) challenge,” Int. J. Comput. Vis., vol. 88, no. 2, pp. 303–338, Sep. 2009.

    Not in our library

13. 13.

    He H. and Garcia E. A., “Learning from imbalanced data,” IEEE Trans. Knowl. Data Eng., vol. 21, no. 9, pp. 1263–1284, Sep. 2009.

    Not in our library

14. 14.

    He K., Zhang X., Ren S., and Sun J. (Dec. 2015 ). “Deep residual learning for image recognition.” [Online]. Available: https://arxiv.org/abs/1512.03385

    Not in our library

15. 15.

    Hill G. D., Block J. R., Tanem J. B., and Frommelt M. A., “Disparities in the prenatal detection of critical congenital heart disease,” Prenatal Diagnosis, vol. 35, no. 9, pp. 859–863, 2015.

    Not in our library10.1002/pd.4622 (opens in a new tab)

16. 16.

    Ioffe S. and Szegedy C. (Feb. 2015 ). “Batch normalization: Accelerating deep network training by reducing internal covariate shift.” [Online]. Available: https://arxiv.org/abs/1502.03167

    Not in our library

17. 17.

    Kang K. and Wang X. (Nov. 2014 ). “Fully convolutional neural networks for crowd segmentation.” [Online]. Available: https://arxiv.org/abs/1411.4464

    Not in our library

18. 18.

    Krizhevsky A., Sutskever I., and Hinton G. E., “Imagenet classification with deep convolutional neural networks,” in Proc. Adv. Neural Inf. Syst., 2012, pp. 1097–1105.

    Not in our library

19. 19.

    Lin M., Chen Q., and Yan S. (Dec. 2013 ). “Network in network.” [Online]. Available: https://arxiv.org/abs/1312.4400

    Not in our library

20. 20.

    Maraci M. A., Napolitano R., Papageorghiou A., and Noble J. A., “Searching for structures of interest in an ultrasound video sequence,” in Machine Learning in Medical Imaging. Boston, MA, USA: Springer, 2014, pp. 133–140.

    Not in our library

21. 21.

    Maraci M. A., Napolitano R., Papageorghiou A., and Noble J. A., “Fisher vector encoding for detecting objects of interest in ultrasound videos,” in Proc. ISBI, Apr. 2015, pp. 651–654.

    Not in our library

22. 22.

    Fetal Anomalie Screen Programme Handbook, NHS Screening Programmes, London, U.K., 2015.

    Not in our library

23. 23.

    Ni D., “Selective search and sequential detection for standard plane localization in ultrasound,” in Proc. Int. MICCAI Workshop Comput. Clin. Challenges Abdominal Imag., 2013, pp. 203–211.

    Not in our library

24. 24.

    Ni D., “Standard plane localization in ultrasound by radial component model and selective search,” Ultrasound Med. Biol., vol. 40, no. 11, pp. 2728–2742, 2014.

    Not in our library10.1016/j.ultrasmedbio.2014.06.006 (opens in a new tab)

25. 25.

    Oquab M., Bottou L., Laptev I., and Sivic J., “Is object localization for free?—Weakly-supervised learning with convolutional neural networks,” in Proc. CVPR, 2015, pp. 685–694.

    Not in our library

26. 26.

    Pinheiro P. O. and Collobert R., “From image-level to pixel-level labeling with convolutional networks,” in Proc. CVPR, 2015, pp. 1713–1721.

    Not in our library

27. 27.

    Ren W., Huang K., Tao D., and Tan T., “Weakly supervised large scale object localization with multiple instance learning and bag splitting,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 38, no. 2, pp. 405–416, Feb. 2016.

    Not in our library10.1109/tpami.2015.2456908 (opens in a new tab)

28. 28.

    Salomon L. J., “Practice guidelines for performance of the routine mid-trimester fetal ultrasound scan,” Ultrasound Obstetrics, Gynecol., vol. 37, no. 1, pp. 116–126, 2011.

    Not in our library10.1002/uog.8831 (opens in a new tab)

29. 29.

    Simonyan K., Vedaldi A., and Zisserman A. (Dec. 2013 ). “Deep inside convolutional networks: Visualising image classification models and saliency maps.” [Online]. Available: https://arxiv.org/abs/1312.6034

    Not in our library

30. 30.

    Simonyan K. and Zisserman A. (Sep. 2014 ). “Very deep convolutional networks for large-scale image recognition.” [Online]. Available: https://arxiv.org/abs/1409.1556

    Not in our library

31. 31.

    Radiographers S. O., “Sonographer workforce survey analysis,” The Society Radiographers, Tech. Rep., 2014, pp. 28–35.

    Not in our library

32. 32.

    Springenberg J. T., Dosovitskiy A., Brox T., and Riedmiller M. (Dec. 2014 ). “Striving for simplicity: The all convolutional net.” [Online]. Available: https://arxiv.org/abs/1412.6806

    Not in our library

33. 33.

    Telea A., “An image inpainting technique based on the fast marching method,” J. Graph. Tools, vol. 9, no. 1, pp. 23–34, 2004.

    Not in our library

34. 34.

    Yaqub M., Kelly B., Papageorghiou A. T., and Noble J. A., “Guided random forests for identification of key fetal anatomy and image categorization in ultrasound scans,” in Medical Image Computing and Computer-Assisted Intervention—MICCAI. Cham, Switzerland: Springer, 2015, pp. 687–694.

    Not in our library

35. 35.

    Zhang L., Chen S., Chin C. T., Wang T., and Li S., “Intelligent scanning: Automated standard plane selection and biometric measurement of early gestational sac in routine ultrasound examination,” Med. Phys., vol. 39, no. 8, pp. 5015–5027, 2012.

    Not in our library10.1118/1.4736415 (opens in a new tab)

36. 36.

    Zhou Z.-H. and Liu X.-Y., “Training cost-sensitive neural networks with methods addressing the class imbalance problem,” IEEE Trans. Knowl. Data Eng., vol. 18, no. 1, pp. 63–77, Jan. 2006.

    Not in our library

Cited by papers we hold 1

• A Novel Ultrasound Robot With Force/Torque Measurement and Control for Safe and Efficient Scanning · 2023
  No notice

Questions about this paper

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.