Deep CT to MR Synthesis Using Paired and Unpaired Data

Abstract

Magnetic resonance (MR) imaging plays a highly important role in radiotherapy treatment planning for the segmentation of tumor volumes and organs. However, the use of MR is limited, owing to its high cost and the increased use of metal implants for patients. This study is aimed towards patients who are contraindicated owing to claustrophobia and cardiac pacemakers, and many scenarios in which only computed tomography (CT) images are available, such as emergencies, situations lacking an MR scanner, and situations in which the cost of obtaining an MR scan is prohibitive. From medical practice, our approach can be adopted as a screening method by radiologists to observe abnormal anatomical lesions in certain diseases that are difficult to diagnose by CT. The proposed approach can estimate an MR image based on a CT image using paired and unpaired training data. In contrast to existing synthetic methods for medical imaging, which depend on sparse pairwise-aligned data or plentiful unpaired data, the proposed approach alleviates the rigid registration of paired training, and overcomes the context-misalignment problem of unpaired training. A generative adversarial network was trained to transform two-dimensional (2D) brain CT image slices into 2D brain MR image slices, combining the adversarial, dual cycle-consistent, and voxel-wise losses. Qualitative and quantitative comparisons against independent paired and unpaired training methods demonstrated the superiority of our approach.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 35

1. 1.

   Guckenberger M., Meyer J., Wilbert J., Richter A., Baier K., Mueller G., Flentje M. Intra-fractional uncertainties in cone-beam CT based image-guided radiotherapy (IGRT) of pulmonary tumors. Radiother. Oncol. 2007;83:57-64.

   Not in our library10.1016/j.radonc.2007.01.012

2. 2.

   De Wagter C., Van Duyse B., Derycke S., Mersseman B., De Gersem W., Voet T., Achten E., De Neve W. Validation of MR-based polymer gel dosimetry as a preclinical three-dimensional verification tool in conformal radiotherapy. Magn. Reson. Med. Off. J. Int. Soc. Magn. Reson. Med. 2000;43:116-125.

   Not in our library10.1002/(sici)1522-2594(200001)43:1<116::aid-mrm14>3.0.co;2-5

3. 3.
   Systematisation of spatial uncertainties for comparison between a MR and a CT-based radiotherapy workflow for prostate treatments · 2009

   No notice
4. 4.

   Goodfellow I., Pouget-Abadie J., Mirza M., Xu B., Warde-Farley D., Ozair S., Courville A., Bengio Y. Generative adversarial nets. Advances in Neural Information Processing Systems. 2014;2672-2680.

   Not in our library

5. 5.

   Krizhevsky A., Sutskever I., Hinton G.E. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems. 2012;1097-1105.

   Not in our library

6. 6.

   Esteva A., Kuprel B., Novoa R.A., Ko J., Swetter S.M., Blau H.M., Thrun S. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017;542:115-118.

   Not in our library10.1038/nature21056

7. 7.

   Dai W., Doyle J., Liang X., Zhang H., Dong N., Li Y., Xing E.P. SCAN: Structure correcting adversarial network for chest x-rays organ segmentation. arXiv. 2017;1-10.

   Not in our library

8. 8.

   Alex V., Safwan K.P. M., Chennamsetty S.S., Krishnamurthi G. Generative adversarial networks for brain lesion detection. Medical Imaging 2017: Image Processing. 2017;Volume 10133.

   Not in our library

9. 9.

   Appan P., Sivaswamy J. Retinal image synthesis for cad development. Proceedings of the International Conference Image Analysis and Recognition. 613-621.

   Not in our library

10. 10.

    Zheng W., Kimg J.P., Kadbi M., Movsas B., Chetty I.J., Glide-Hurst C.K. Magnetic resonance—Based automatic air segmentation for generation of synthetic computed tomography scans in the head region. Int. J. Radiat. Oncol. Biol. Phys. 2015;93:497-506.

    Not in our library10.1016/j.ijrobp.2015.07.001

11. 11.

    Huynh T., Gao Y., Kang J., Wang L., Zhang P., Lian J., Shen D. Estimating CT image from MRI data using structured random forest and auto-context model. IEEE Trans. Med. Imaging. 2016;35:174-183.

    Not in our library10.1109/tmi.2015.2461533

12. 12.

    Catana C., Van der Kouwe A., Benner T., Michel C.J., Hamm M., Fenchel M., Fischl B., Rosen B., Schm M., Sorensen A.G. Toward implementing an MRI-based PET attenuation-correction method for neurologic studies on the MR-PET brain prototype. J. Nucl. Med. 2010;51:1431-1438.

    Not in our library10.2967/jnumed.109.069112

13. 13.

    Son J., Park S., Jung K.-H. Retinal vessel segmentation in fundoscopic images with generative adversarial networks. arXiv. 2017;1-9.

    Not in our library10.1007/s10278-018-0126-3

14. 14.

    Chen H., Qi X., Yu L., Dou Q., Qin J., Heng P.-A. DCAN: Deep contour-aware networks for object instance segmentation from histology images. Med. Image Anal. 2017;36:135-146.

    Not in our library10.1016/j.media.2016.11.004

15. 15.
    A review of substitute CT generation for MRI-only radiation therapy · 2017

    No notice
16. 16.

    Han X. MR-based synthetic CT generation using a deep convolutional neural network method. Med. Phys. 2017;44:1408-1419.

    Not in our library10.1002/mp.12155

17. 17.

    Zhao C., Carass A., Lee J., He Y., Prince J.L. Whole brain segmentation and labeling from CT using synthetic MR images. Proceedings of the International Workshop on Machine Learning in Medical Imaging. 291-298.

    Not in our library

18. 18.

    Ronneberger O., Fischer P., Brox T. U-Net: Concovlutional Networks for Biomedical Image Segmentation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention. 234-241.

    Not in our library

19. 19.

    Nie D., Trullo R., Lian J., Petitjean C., Ruan S., Wang Q., Shen D. Medical image synthesis with context-aware generative adversarial networks. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention. 417-425.

    Not in our library10.1007/978-3-319-66179-7_48

20. 20.

    Bi L., Kim J., Kumar A., Feng D., Fulham M. Synthesis of positron emission tomography (pet) images via multi-channel generative adversarial networks (gans). Molecular Imaging, Reconstruction and Analysis of Moving Body Organs, and Stroke Imaging and Treatment. 2017;43-51.

    Not in our library

21. 21.

    Isola P., Zhu J.-Y., Zhou T., Efros A.-A. Image-to-image translation with conditional adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1-10.

    Not in our library

22. 22.

    Ben-Cohen A., Klang E., Raskin S.-P., Amitai M.-M., Greenspan H. Virtual pet images from ct data using deep convolutional networks: Initial results. Proceedings of the International Workshop on Simulation and Synthesis in Medical Imaging. 49-57.

    Not in our library

23. 23.

    Long J., Shelhamer E., Darrell T. Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3431-3440.

    Not in our library

24. 24.

    Wolterink J.-M., Dinkla A.-M., Savenije M.-H., Seevinck P.-R., Berg C.-A., Išgum I. Deep MR to CT synthesis using unpaired data. Proceedings of the International Workshop on Simulation and Synthesis in Medical Imaging. 14-23.

    Not in our library

25. 25.

    Zhu J.-Y., Park T., Isola P., Efros A.-A. Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE International Conference on Computer Vision (ICCV). 1-10.

    Not in our library

26. 26.

    Saad Z.-S., Glen D.-R., Chen G., Beauchamp M.-S., Desai R., Cox R.-W. A new method for improving functional-to-structural MRI alignment using local Pearson correlation. Neuroimage. 2009;44:839-848.

    Not in our library10.1016/j.neuroimage.2008.09.037

27. 27.

    Mao X., Li Q., Xie H., Lau R.-Y., Wang Z., Smolley S.-P. Least squares generative adversarial networks. Proceedings of the IEEE International Conference on Computer Vision (ICCV). 2813-2821.

    Not in our library

28. 28.

    Pathak D., Krahenbuhl P., Donahue J., Darrell T., Efros A.-A. Context encoders: Feature learning by inpainting. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2536-2544.

    Not in our library

29. 29.

    Johnson J., Alahi A., Li F.-F. Perceptual losses for real-time style transfer and super-resolution. Proceedings of the European Conference on Computer Vision (ECCV). 694-711.

    Not in our library

30. 30.

    He K., Zhang X., Ren S., Sun J. Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). 770-778.

    Not in our library

31. 31.

    Ulyanov D., Vedaldi A., Lempitsky V.S. Instance normalization: The missing ingredient for fast stylization. arXiv. 2016;1-6.

    Not in our library

32. 32.

    Xu B., Wang N., Chen T., Li M. Empirical evaluation of rectified activations in convolutional network. arXiv. 2015;1-5.

    Not in our library

33. 33.

    Kingma D.-P., Ba J. Adam: A method for stochastic optimization. Proceedings of the International Conference of Machine Learning (ICML). 1-15.

    Not in our library

34. 34.

    Wang Z., Simoncelli E.P., Bovik A.C. Multiscale structural similarity for image quality assessment. Proceedings of the Thrity-Seventh Asilomar Conference on Signals, Systems & Computers. Volume 2;1398-1402.

    Not in our library

35. 35.

    Hinton G.-E., Salakhutdinov R.-R. Reducing the dimensionality of Data with neural networks. Science. 2006;313:504-507.

    Not in our library10.1126/science.1127647

Cited by papers we hold 2

• Fully automated dose prediction using generative adversarial networks in prostate cancer patients. · 2020
  Not checked
• A review on medical imaging synthesis using deep learning and its clinical applications · 2020
  No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.