SynthSeg: Segmentation of brain MRI scans of any contrast and resolution without retraining

Abstract

Despite advances in data augmentation and transfer learning, convolutional neural networks (CNNs) difficultly generalise to unseen domains. When segmenting brain scans, CNNs are highly sensitive to changes in resolution and contrast: even within the same MRI modality, performance can decrease across datasets. Here we introduce SynthSeg, the first segmentation CNN robust against changes in contrast and resolution. SynthSeg is trained with synthetic data sampled from a generative model conditioned on segmentations. Crucially, we adopt a domain randomisation strategy where we fully randomise the contrast and resolution of the synthetic training data. Consequently, SynthSeg can segment real scans from a wide range of target domains without retraining or fine-tuning, which enables straightforward analysis of huge amounts of heterogeneous clinical data. Because SynthSeg only requires segmentations to be trained (no images), it can learn from labels obtained by automated methods on diverse populations (e.g., ageing and diseased), thus achieving robustness to a wide range of morphological variability. We demonstrate SynthSeg on 5,000 scans of six modalities (including CT) and ten resolutions, where it exhibits unparallelled generalisation compared with supervised CNNs, state-of-the-art domain adaptation, and Bayesian segmentation. Finally, we demonstrate the generalisability of SynthSeg by applying it to cardiac MRI and CT scans.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 76

1. 1.

   Abadi M, Barham P, Chen J, Chen Z, Davis A, 2016. Tensorflow: A system for large-scale machine learning. In: Symposium on Operating Systems Design and Implementation. pp. 265–283.

   Not in our library

2. 2.

   Arsigny V, Commowick O, Pennec X, Ayache N, 2006. A log-Euclidean framework for statistics on diffeomorphisms. In: Medical Image Computing and Computer Assisted Intervention. pp. 924–931.

   Not in our library10.1007/11866565_113

3. 3.

   Ashburner J, Friston K, 2005. Unified segmentation. NeuroImage 26 (3), 839–851.

   Not in our library10.1016/j.neuroimage.2005.02.018

4. 4.

   Bengio Y, Bastien F, Bergeron A, Boulanger-Lewandowski N, Breuel T, Chherawala Y, Cisse M,, 2011. Deep learners benefit more from out-of-distribution examples. In: International Conference on Artificial Intelligence and Statistics. pp. 164–172.

   Not in our library

5. 5.

   Billot B, Cerri S, Van Leemput K, Dalca A, Iglesias JE, 2021. Joint segmentation of multiple sclerosis lesions and brain anatomy in MRI scans of any contrast and resolution with CNNs. In: IEEE International Symposium on Biomedical Imaging. pp. 1971–1974.

   Not in our library10.1109/isbi48211.2021.9434127

6. 6.

   Billot B, Greve D, Van Leemput K, Fischl B, Iglesias JE, Dalca A, 2020a. A learning strategy for contrast-agnostic MRI segmentation. In: Medical Imaging with Deep Learning. pp. 75–93.

   Not in our library

7. 7.

   Billot B, Robinson E, Dalca A, Iglesias JE, 2020b. Partial volume segmentation of brain MRI scans of any resolution and contrast. In: Medical Image Computing and Computer Assisted Intervention. pp. 177–187.

   Not in our library

8. 8.

   Chaitanya K, Erdil E, Karani N, Konukoglu E, 2020. Contrastive learning of global and local features for medical image segmentation with limited annotations. In: Advances in Neural Information Processing Systems, vol. 33, pp. 12546–12558.

   Not in our library

9. 9.

   Chaitanya K, Karani N, Baumgartner C, Becker A, Donati O, Konukoglu E, 2019. Semi-supervised and task-driven data augmentation. In: Information Processing in Medical Imaging. pp. 29–41.

   Not in our library10.1016/j.media.2020.101934

10. 10.

    Chartsias A, Joyce T, Giuffrida M, Tsaftaris S, 2018. Multimodal MR synthesis via modality-invariant latent representation. IEEE Trans. Med. Imaging 37, 803–814.

    Not in our library10.1109/tmi.2017.2764326

11. 11.

    Chen C, Dou Q, Chen H, Qin J, Heng PA, 2019. Synergistic image and feature adaptation: Towards cross-modality domain adaptation for medical image segmentation. Proc. AAAI Conf. Artif. Intell. 33, 65–72.

    Not in our library

12. 12.

    Chen C, Qin C, Ouyang C, Li Z, Wang S, Qiu H, Chen L, Tarroni G, Bai W, Rueckert D, 2022. Enhancing MR image segmentation with realistic adversarial data augmentation. Med. Image Anal. 82.

    Not in our library10.1016/j.media.2022.102597

13. 13.

    Choi H, Haynor D, Kim Y, 1991. Partial volume tissue classification of multichannel magnetic resonance images-a mixel model. IEEE Trans. Med. Imaging 10, 395–407.

    Not in our library10.1109/42.97590

14. 14.

    Chollet F, 2015. Keras. https://keras.io.

    Not in our library

15. 15.

    Chupin M, Gérardin E, Cuingnet R, Boutet C, Lemieux L, Lehéricy S,, 2009. Fully automatic hippocampus segmentation and classification in Alzheimer’s disease and mild cognitive impairment applied on data from ADNI. Hippocampus 19 (6), 579–587.

    Not in our library10.1002/hipo.20626

16. 16.

    Clevert DA, Unterthiner T, Hochreiter S, 2016. Fast and accurate deep network learning by exponential linear units (ELUs). arXiv:1511.07289 [cs].

    Not in our library

17. 17.

    Cohen J, 1988. Statistical Power Analysis for the Behavioural Sciences. Routledge Academic.

    Not in our library

18. 18.

    Dagley A, LaPoint M, Huijbers W, Hedden T, McLaren D,, 2017. Harvard aging brain study: Dataset and accessibility. NeuroImage 144, 255–258.

    Not in our library10.1016/j.neuroimage.2015.03.069

19. 19.

    Dempster A, Laird N, Rubin DB, 1977. Maximum likelihood from incomplete data via the EM algorithm. J. R. Stat. Soc. 39 (1), 1–22.

    Not in our library

20. 20.

    Di Martino A, Yan CG, Li Q, Denio E, Castellanos F,, 2014. The autism brain imaging data exchange: Towards large-scale evaluation of the intrinsic brain architecture in autism. Mol. Psychiatry 19 (6), 659–667.

    Not in our library10.1038/mp.2013.78

21. 21.

    Dou Q, Ouyang C, Chen C, Chen H, Glocker B, Zhuang X, Heng PA, 2019. PnP-AdaNet: Plug-and-play adversarial domain adaptation network at unpaired cross-modality cardiac segmentation. IEEE Access 7.

    Not in our library

22. 22.

    Fischl B, 2012. FreeSurfer. NeuroImage 62, 774–781.

    Not in our library10.1016/j.neuroimage.2012.01.021

23. 23.

    Fischl B, Salat D, Busa E, Albert M,, 2002. Whole brain segmentation: Automated labeling of neuroanatomical structures in the human brain. Neuron 33, 41–55.

    Not in our library10.1016/s0896-6273(02)00569-x

24. 24.

    Fischl B, Salat D, van der Kouwe A, Makris N, Ségonne F, Quinn B, Dale A, 2004. Sequence-independent segmentation of magnetic resonance images. NeuroImage 23, 69–84.

    Not in our library10.1016/j.neuroimage.2004.07.016

25. 25.

    Frid-Adar M, Klang E, Amitai M, Goldberger J, Greenspan H, 2018. Synthetic data augmentation using GAN for improved liver lesion classification. In: IEEE International Symposium on Biomedical Imaging. pp. 289–293.

    Not in our library

26. 26.

    Ganin Y, Ustinova E, Ajakan H, Germain P, Larochelle H, Laviolette F, Marchand M, Lempitsky V, 2017. Domain-adversarial training of neural networks. In: Domain Adaptation in Computer Vision Applications. In: Advances in Computer Vision and Pattern Recognition, pp. 189–209.

    Not in our library

27. 27.

    Ghafoorian M, Mehrtash A, Kapur T, Karssemeijer N, Marchiori E, Pesteie M, Fedorov A, Abolmaesumi P, Platel B, Wells W, 2017. Transfer learning for domain adaptation in MRI: Application in brain lesion segmentation. In: Medical Image Computing and Computer Assisted Intervention. pp. 516–524.

    Not in our library

28. 28.

    Gollub R, Shoemaker J, King M, White T, Ehrlich S,, 2013. The MCIC collection: A shared repository of multi-modal, multi-site brain image data from a clinical investigation of schizophrenia. Neuroinformatics 11 (3), 367–388.

    Not in our library10.1007/s12021-013-9184-3

29. 29.

    Havaei M, Guizard N, Chapados N, Bengio Y, 2016. HeMIS: Hetero-modal image segmentation. In: Medical Image Computing and Computer-Assisted Intervention. pp. 469–477.

    Not in our library

30. 30.

    He Y, Carass A, Zuo L, Dewey B, Prince J, 2021. Autoencoder based self-supervised test-time adaptation for medical image analysis. Med. Image Anal. 102136.

    Not in our library10.1016/j.media.2021.102136

31. 31.

    Hoffman J, Tzeng E, Park T, Zhu JY, Isola P, Saenko K, Efros A, Darrell T, 2018. CyCADA: Cycle-consistent adversarial domain adaptation. In: International Conference on Machine Learning. pp. 1989–1998.

    Not in our library

32. 32.
    Brain Genomics Superstruct Project initial data release with structural, functional, and behavioral measures · 2015

    No notice
33. 33.

    Huo Y, Xu Z, Moon H, Bao S, Assad A, Moyo T, Savona M, Abramson R, Landman B, 2019. SynSeg-Net: Synthetic segmentation without target modality ground truth. IEEE Trans. Med. Imaging 38 (4), 1016–1025.

    Not in our library10.1109/tmi.2018.2876633

34. 34.

    Hynd G, Semrud-Clikeman M, Lorys A, Novey E, Eliopulos D, Lyytinen H, 1991. Corpus callosum morphology in attention deficit-hyperactivity disorder: Morphometric analysis of MRI. J. Learn. Disabil. 24 (3), 141–146.

    Not in our library10.1177/002221949102400302

35. 35.

    Iglesias JE, Billot B, Balbastre Y, Tabari A, Conklin J, Gilberto González R, Alexander DC, Golland P, Edlow BL, Fischl B, 2021. Joint super-resolution and synthesis of 1 mm isotropic MP-RAGE volumes from clinical MRI exams with scans of different orientation, resolution and contrast. NeuroImage 237.

    Not in our library10.1016/j.neuroimage.2021.118206

36. 36.
    A probabilistic atlas of the human thalamic nuclei combining ex vivo MRI and histology · 2018

    No notice
37. 37.

    Ioffe S, Szegedy C, 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning. pp. 448–456.

    Not in our library

38. 38.

    Isensee F, Jaeger P, Kohl S, Petersen J, Maier-Hein K, 2021. nnU-Net: A self-configuring method for deep learning-based biomedical image segmentation. Nature Methods 18 (2), 203–211.

    Not in our library10.1038/s41592-020-01008-z

39. 39.

    Isola P, Zhu J, Zhou T, Efros A, 2017. Image-to-image translation with conditional adversarial networks. In: IEEE Conference on Computer Vision and Pattern Recognition. pp. 1125–1134.

    Not in our library

40. 40.

    Jakobi N, Husbands P, Harvey I, 1995. Noise and the reality gap: The use of simulation in evolutionary robotics. In: Advances in Artificial Life. pp. 704–720.

    Not in our library

41. 41.

    Jog A, Fischl B, 2018. Pulse sequence resilient fast brain segmentation. In: Medical Image Computing and Computer Assisted Intervention. pp. 654–662.

    Not in our library

42. 42.

    Kamnitsas K, Baumgartner C, Ledig C, Newcombe V, Simpson J, Nori A, Criminisi A, Rueckert D, Glocker B, 2017a. Unsupervised domain adaptation in brain lesion segmentation with adversarial networks. In: Information Processing in Medical Imaging. pp. 597–609.

    Not in our library

43. 43.

    Kamnitsas K, Ledig C, Newcombe V, Simpson J, Kane A, Menon D, Rueckert D, Glocker B, 2017b. Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Med. Image Anal. 36, 61–78.

    Not in our library10.1016/j.media.2016.10.004

44. 44.

    Karani N, Chaitanya K, Baumgartner C, Konukoglu E, 2018. A lifelong learning approach to brain MR segmentation across scanners and protocols. In: Medical Image Computing and Computer Assisted Intervention. pp. 476–484.

    Not in our library

45. 45.

    Karani N, Erdil E, Chaitanya K, Konukoglu E, 2021. Test-time adaptable neural networks for robust medical image segmentation. Med. Image Anal. 68, 101907.

    Not in our library10.1016/j.media.2020.101907

46. 46.

    Kingma D, Ba J, 2017. Adam: A method for stochastic optimization. arXiv:1412.6980 [cs].

    Not in our library

47. 47.

    Mahmood F, Borders D, Chen R, Mckay G, Salimian K, Baras A, Durr N, 2020. Deep adversarial training for multi-organ nuclei segmentation in histopathology images. IEEE Trans. Med. Imaging 39 (11), 3257–3267.

    Not in our library10.1109/tmi.2019.2927182

48. 48.

    Marcus D, Wang T, Parker J, Csernansky J, Morris J, Buckner R, 2007. Open access series of imaging studies: Cross-sectional MRI data in Young, middle aged, nondemented, and demented older adults. J. Cogn. Neurosci. 19, 498–507.

    Not in our library10.1162/jocn.2007.19.9.1498

49. 49.

    Marek K, Jennings D, Lasch S, Siderowf A, Tanner C, Simuni T,, 2011. The parkinson progression marker initiative (PPMI). Progress Neurobiol. 95 (4), 629–635.

    Not in our library10.1016/j.pneurobio.2011.09.005

50. 50.

    Milletari F, Navab N, Ahmadi S, 2016. V-Net: Fully convolutional neural networks for volumetric medical image segmentation. In: International Conference on 3D Vision. pp. 565–571.

    Not in our library

51. 51.

    Moshkov N, Mathe B, Kertesz-Farkas A, Hollandi R, Horvath P, 2020. Test-time augmentation for deep learning-based cell segmentation on microscopy images. Sci. Rep. 10, 182–192.

    Not in our library10.1038/s41598-020-61808-3

52. 52.

    Müller R, Shih P, Keehn B, Deyoe J, Leyden K, Shukla D, 2011. Underconnected, but How? A survey of functional connectivity MRI studies in autism spectrum disorders. Cerebral Cortex 21 (10), 2233–2243.

    Not in our library10.1093/cercor/bhq296

53. 53.

    Pan S, Yang Q, 2010. A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22, 45–59.

    Not in our library

54. 54.

    Puonti O, Iglesias JE, Van Leemput K, 2016. Fast and sequence-adaptive whole-brain segmentation using parametric Bayesian modeling. NeuroImage 143, 235–249.

    Not in our library10.1016/j.neuroimage.2016.09.011

55. 55.
    Accurate and robust whole-head segmentation from magnetic resonance images for individualized head modeling · 2020

    Not checked
56. 56.

    Richter SR, Vineet V, Roth S, Koltun V, 2016. Playing for data: Ground truth from computer games. In: Computer Vision – ECCV. pp. 102–118.

    Not in our library

57. 57.

    Ronneberger O, Fischer P, Brox T, 2015. U-Net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention. pp. 234–241.

    Not in our library

58. 58.
    Data augmentation using generative adversarial networks (CycleGAN) to improve generalizability in CT segmentation tasks · 2019

    No notice
59. 59.

    Tae W, Kim S, Lee K, Nam E, Kim K, 2008. Validation of hippocampal volumes measured using a manual method and two automated methods (FreeSurfer and IBASPM) in chronic major depressive disorder. Neuroradiology 50 (7), 569–581.

    Not in our library10.1007/s00234-008-0383-9

60. 60.
    The ADHD-200 Consortium: a model to advance the translational potential of neuroimaging in clinical neuroscience · 2012

    No notice
61. 61.

    Tobin J, Fong R, Ray A, Schneider J, Zaremba W, Abbeel P, 2017. Domain randomization for transferring deep neural networks from simulation to the real world. In: IEEE/RSJ International Conference on Intelligent Robots and Systems. pp. 23–30.

    Not in our library

62. 62.

    Tobon-Gomez C, Geers AJ, Peters J, 2015. Benchmark for algorithms segmenting the left atrium from 3D CT and MRI datasets. IEEE Trans. Med. Imaging 34 (7), 1460–1473.

    Not in our library10.1109/tmi.2015.2398818

63. 63.

    Tremblay J, Prakash A, Acuna D, Brophy M, Jampani V, Anil C, To T, Cameracci E, Boochoon S, Birchfield S, 2018. Training deep networks with synthetic data: Bridging the reality gap by domain randomization. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops. pp. 969–977.

    Not in our library

64. 64.

    Van Essen D, Ugurbil K, Auerbach E, Barch D, Behrens T,, 2012. The human connectome project: A data acquisition perspective. NeuroImage 62 (4), 2222–2231.

    Not in our library10.1016/j.neuroimage.2012.02.018

65. 65.

    Van Leemput K, Maes F, Vandermeulen D, Suetens P, 1999. Automated model-based tissue classification of MR images of the brain. IEEE Trans. Med. Imaging 87–98.

    Not in our library10.1109/42.811270

66. 66.

    Van Leemput K, Maes F, Vandermeulen D, Suetens P, 2003. A unifying framework for partial volume segmentation of brain MR images. IEEE Trans. Med. Imaging 22.

    Not in our library10.1109/tmi.2002.806587

67. 67.

    Warfield S, Zou K, Wells W, 2004. Simultaneous truth and performance level estimation: An algorithm for the validation of image segmentation. IEEE Trans. Med. Imaging 23 (7), 903–921.

    Not in our library10.1109/tmi.2004.828354

68. 68.

    Wells WM, Viola P, Atsumi H, Nakajima S, Kikinis R, 1996. Multi-modal volume registration by maximization of mutual information. Med. Image Anal. 1 (1), 35–51.

    Not in our library10.1016/s1361-8415(01)80004-9

69. 69.

    West J, Fitzpatrick J, Wang M, Dawant B, Maurer C, Kessler R, Maciunas R,, 1997. Comparison and evaluation of retrospective intermodality brain image registration techniques. J. Comput. Assist. Tomogr. 21 (4), 554–568.

    Not in our library10.1097/00004728-199707000-00007

70. 70.

    You C, Xiang J, Su K, Zhang X, Dong S, Onofrey J, Staib L, Duncan J, 2022a. Incremental learning meets transfer learning: Application to multi-site prostate MRI segmentation. In: Distributed, Collaborative, and Federated Learning, and Affordable AI and Healthcare for Resource Diverse Global Health. pp. 3–16.

    Not in our library10.1007/978-3-031-18523-6_1

71. 71.

    You C, Zhou Y, Zhao R, Staib L, Duncan J, 2022b. SimCVD: Simple contrastive voxel-wise representation distillation for semi-supervised medical image segmentation. IEEE Trans. Med. Imaging 41 (9), 2228–2237.

    Not in our library10.1109/tmi.2022.3161829

72. 72.

    Zhang L, Wang X, Yang D, Sanford T, Harmon S, Turkbey B, Wood BJ, Roth H, Myronenko A, Xu D, Xu Z, 2020. Generalizing deep learning for medical image segmentation to unseen domains via deep stacked transformation. IEEE Trans. Med. Imaging 39 (7), 2531–2540.

    Not in our library10.1109/tmi.2020.2973595

73. 73.

    Zhang Y, Yang L, Chen J, Fredericksen M, Hughes D, Chen D, 2017. Deep adversarial networks for biomedical image segmentation utilizing unannotated images. In: Medical Image Computing and Computer-Assisted Intervention. pp. 408–416.

    Not in our library

74. 74.

    Zhang Z, Yang L, Zheng Y, 2018. Translating and segmenting multimodal medical volumes with cycle- and shape-consistency generative adversarial network. In: IEEE Conference on Computer Vision and Pattern Recognition. pp. 242–251.

    Not in our library

75. 75.

    Zhao A, Balakrishnan G, Durand F, Guttag J, Dalca A, 2019. Data augmentation using learned transformations for one-shot medical image segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition. pp. 8543–8553.

    Not in our library

76. 76.
    Evaluation of algorithms for Multi-Modality Whole Heart Segmentation: An open-access grand challenge · 2019

    No notice

Cited by papers we hold 1

• A human brain network linked to restoration of consciousness after deep brain stimulation · 2025
  Not checked

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.