Repeatability of Multiparametric Prostate MRI Radiomics Features

Abstract

In this study we assessed the repeatability of radiomics features on small prostate tumors using test-retest Multiparametric Magnetic Resonance Imaging (mpMRI). The premise of radiomics is that quantitative image-based features can serve as biomarkers for detecting and characterizing disease. For such biomarkers to be useful, repeatability is a basic requirement, meaning its value must remain stable between two scans, if the conditions remain stable. We investigated repeatability of radiomics features under various preprocessing and extraction configurations including various image normalization schemes, different image pre-filtering, and different bin widths for image discretization. Although we found many radiomics features and preprocessing combinations with high repeatability (Intraclass Correlation Coefficient > 0.85), our results indicate that overall the repeatability is highly sensitive to the processing parameters. Neither image normalization, using a variety of approaches, nor the use of pre-filtering options resulted in consistent improvements in repeatability. We urge caution when interpreting radiomics features and advise paying close attention to the processing configuration details of reported results. Furthermore, we advocate reporting all processing details in radiomics studies and strongly recommend the use of open source implementations.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 66

1. 1.

   Kumar V, et al. Radiomics: the process and the challenges. Magn. Reson. Imaging. 2012;1234-1248.

   Not in our library10.1016/j.mri.2012.06.010 (opens in a new tab)

2. 2.

   Lambin P, et al. Radiomics: extracting more information from medical images using advanced feature analysis. Eur. J. Cancer. 2012;48:441-446.

   Not in our library10.1016/j.ejca.2011.11.036 (opens in a new tab)

3. 3.

   Aerts HJWL, et al. Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach. Nat. Commun. 2014;5:4006.

   Not in our library10.1038/ncomms5006 (opens in a new tab)

4. 4.

   Fehr D, et al. Automatic classification of prostate cancer Gleason scores from multiparametric magnetic resonance images. Proc. Natl. Acad. Sci. USA. 2015;112:E6265-73.

   Not in our library10.1073/pnas.1505935112 (opens in a new tab)

5. 5.

   Wibmer Andreas, Hricak Hedvig, Gondo Tatsuo, Matsumoto Kazuhiro, Veeraraghavan Harini, Fehr Duc, Zheng Junting, Goldman Debra, Moskowitz Chaya, Fine Samson W., Reuter Victor E., Eastham James, Sala Evis, Vargas Hebert Alberto. Haralick texture analysis of prostate MRI: utility for differentiating non-cancerous prostate from prostate cancer and differentiating prostate cancers with different Gleason scores. European Radiology. 2015;25(10):2840-2850.

   Not in our library10.1007/s00330-015-3701-8 (opens in a new tab)

6. 6.

   Peng Y, et al. Quantitative Analysis of Multiparametric Prostate MR Images: Differentiation between Prostate Cancer and Normal Tissue and Correlation with Gleason Score–A Computer-aided Diagnosis Development Study. Radiology. 2013.

   Not in our library10.1148/radiol.13121454 (opens in a new tab)

7. 7.

   Cameron A, Khalvati F, Haider MA, Wong A. MAPS: A Quantitative Radiomics Approach for Prostate Cancer Detection. IEEE Trans. Biomed. Eng. 2016;63:1145-1156.

   Not in our library10.1109/tbme.2015.2485779 (opens in a new tab)

8. 8.
   Machine Learning methods for Quantitative Radiomic Biomarkers · 2015

   No notice
9. 9.
   Radiomic Machine-Learning Classifiers for Prognostic Biomarkers of Head and Neck Cancer · 2015

   No notice
10. 10.

    Jain R, et al. Outcome prediction in patients with glioblastoma by using imaging, clinical, and genomic biomarkers: focus on the nonenhancing component of the tumor. Radiology. 2014;272:484-493.

    Not in our library10.1148/radiol.14131691 (opens in a new tab)

11. 11.

    Gevaert O, et al. Glioblastoma Multiforme: Exploratory Radiogenomic Analysis by Using Quantitative Image Features. Radiology. 2015;276:313.

    Not in our library10.1148/radiol.2015154019 (opens in a new tab)

12. 12.

    Carvalho S, et al. Prognostic value of metabolic metrics extracted from baseline positron emission tomography images in non-small cell lung cancer. Acta Oncol. 2013;52:1398-1404.

    Not in our library10.3109/0284186x.2013.812795 (opens in a new tab)

13. 13.
    Quantification of Heterogeneity as a Biomarker in Tumor Imaging: A Systematic Review · 2014

    No notice
14. 14.

    Barentsz JO, et al. ESUR prostate MR guidelines 2012. Eur. Radiol. 2012;22:746-757.

    Not in our library10.1007/s00330-011-2377-y (opens in a new tab)

15. 15.

    Weinreb JC, et al. PI-RADS Prostate Imaging – Reporting and Data System: 2015, Version 2. Eur. Urol. 2016;69:16-40.

    Not in our library10.1016/j.eururo.2015.08.052 (opens in a new tab)

16. 16.

    Turkbey B, et al. Correlation of magnetic resonance imaging tumor volume with histopathology. J. Urol. 2012;188:1157-1163.

    Not in our library10.1016/j.juro.2012.06.011 (opens in a new tab)

17. 17.

    Hambrock T, et al. Relationship between apparent diffusion coefficients at 3.0-T MR imaging and Gleason grade in peripheral zone prostate cancer. Radiology. 2011;259:453-461.

    Not in our library10.1148/radiol.11091409 (opens in a new tab)

18. 18.

    Langer DL, et al. Prostate Tissue Composition and MR Measurements: Investigating the relationship between ADC, T2, Ktrans, Ve and corresponding histological features. Radiology. 2010;255:485-494.

    Not in our library10.1148/radiol.10091343 (opens in a new tab)

19. 19.

    Smith Clayton P., Czarniecki Marcin, Mehralivand Sherif, Stoyanova Radka, Choyke Peter L., Harmon Stephanie, Turkbey Baris. Radiomics and radiogenomics of prostate cancer. Abdominal Radiology. 2018;44(6):2021-2029.

    Not in our library10.1007/s00261-018-1660-7 (opens in a new tab)

20. 20.

    Bonekamp David, Kohl Simon, Wiesenfarth Manuel, Schelb Patrick, Radtke Jan Philipp, Götz Michael, Kickingereder Philipp, Yaqubi Kaneschka, Hitthaler Bertram, Gählert Nils, Kuder Tristan Anselm, Deister Fenja, Freitag Martin, Hohenfellner Markus, Hadaschik Boris A., Schlemmer Heinz-Peter, Maier-Hein Klaus H. Radiomic Machine Learning for Characterization of Prostate Lesions with MRI: Comparison to ADC Values. Radiology. 2018;289(1):128-137.

    Not in our library10.1148/radiol.2018173064 (opens in a new tab)

21. 21.
    Reproducibility of radiomics for deciphering tumor phenotype with imaging · 2016

    No notice
22. 22.

    Balagurunathan Y, et al. Test-retest reproducibility analysis of lung CT image features. J. Digit. Imaging. 2014;27:805-823.

    Not in our library10.1007/s10278-014-9716-x (opens in a new tab)

23. 23.

    Leijenaar RTH, et al. Stability of FDG-PET Radiomics features: an integrated analysis of test-retest and inter-observer variability. Acta Oncol. 2013;52:1391-1397.

    Not in our library10.3109/0284186x.2013.812798 (opens in a new tab)

24. 24.

    Gudmundsson S, Runarsson TP, Sigurdsson S. Test–retest reliability and feature selection in physiological time series classification. Comput. Methods Programs Biomed. 2012;105:50-60.

    Not in our library10.1016/j.cmpb.2010.08.005 (opens in a new tab)

25. 25.
    Imaging biomarker roadmap for cancer studies · 2016

    No notice
26. 26.
    Robust Radiomics Feature Quantification Using Semiautomatic Volumetric Segmentation · 2014

    No notice
27. 27.

    Hunter Luke A., Krafft Shane, Stingo Francesco, Choi Haesun, Martel Mary K., Kry Stephen F., Court Laurence E. High quality machine-robust image features: Identification in nonsmall cell lung cancer computed tomography images. Medical Physics. 2013;40(12):121916.

    Not in our library10.1118/1.4829514 (opens in a new tab)

28. 28.

    Bologna M, et al. Assessment of Stability and Discrimination Capacity of Radiomic Features on Apparent Diffusion Coefficient Images. J. Digit. Imaging. 2018.

    Not in our library10.1007/s10278-018-0092-9 (opens in a new tab)

29. 29.

    Zwanenburg, A., Leger, S., Vallières, M., Löck, S. & for the Image Biomarker Standardisation Initiative. Image biomarker standardisation initiative. arXiv [cs.CV]. at http://arxiv.org/abs/1612.07003 (2016).

    Not in our library

30. 30.

    Zhao B, et al. Evaluating variability in tumor measurements from same-day repeat CT scans of patients with non-small cell lung cancer. Radiology. 2009;252:263-272.

    Not in our library10.1148/radiol.2522081593 (opens in a new tab)

31. 31.

    Sadinski M, et al. Short-term reproducibility of apparent diffusion coefficient estimated from diffusion-weighted MRI of the prostate. Abdom. Imaging. 2015;40:2523-2528.

    Not in our library10.1007/s00261-015-0396-x (opens in a new tab)

32. 32.

    Alonzi R, et al. Reproducibility and correlation between quantitative and semiquantitative dynamic and intrinsic susceptibility-weighted MRI parameters in the benign and malignant human prostate. J. Magn. Reson. Imaging. 2010;32:155-164.

    Not in our library10.1002/jmri.22215 (opens in a new tab)

33. 33.

    Traverso A, Wee L, Dekker A, Gillies R. Repeatability and Reproducibility of Radiomic Features: A Systematic Review. Int. J. Radiat. Oncol. Biol. Phys. 2018;102:1143-1158.

    Not in our library10.1016/j.ijrobp.2018.05.053 (opens in a new tab)

34. 34.

    Hu P, et al. Reproducibility with repeat CT in radiomics study for rectal cancer. Oncotarget. 2016;7:71440-71446.

    Not in our library10.18632/oncotarget.12199 (opens in a new tab)

35. 35.
    Repeatability of Radiomic Features in Non-Small-Cell Lung Cancer [18F]FDG-PET/CT Studies: Impact of Reconstruction and Delineation · 2016

    No notice
36. 36.

    Desseroit M-C, et al. Reliability of PET/CT Shape and Heterogeneity Features in Functional and Morphologic Components of Non–Small Cell Lung Cancer Tumors: A Repeatability Analysis in a Prospective Multicenter Cohort. J. Nucl. Med. 2017;58:406-411.

    Not in our library10.2967/jnumed.116.180919 (opens in a new tab)

37. 37.

    Emaminejad, N. et al The effects of variations in parameters and algorithm choices on calculated radiomics feature values: initial investigations and comparisons to feature variability across CT image acquisition conditions. In SPIE Medical Imaging 2018: Computer-Aided Diagnosis, 10.1117/12.2293864 (eds Mori, K. & Petrick, N.) 140 (SPIE, 2018).

    Not in our library

38. 38.
    False Discovery Rates in PET and CT Studies with Texture Features: A Systematic Review · 2015

    No notice
39. 39.

    Yip SSF, Aerts HJWL. Applications and limitations of radiomics. Phys. Med. Biol. 2016;61:R150-R166.

    Not in our library10.1088/0031-9155/61/13/r150 (opens in a new tab)

40. 40.

    Fedorov A, et al. An annotated test-retest collection of prostate multiparametric MRI. Scientific Data. 2018;5:180281.

    Not in our library10.1038/sdata.2018.281 (opens in a new tab)

41. 41.

    van Griethuysen JJM, et al. Computational Radiomics System to Decode the Radiographic Phenotype. Cancer Res. 2017;77:104-108.

    Not in our library10.1158/0008-5472.can-17-0339 (opens in a new tab)

42. 42.

    Fedorov A, Vangel MG, Tempany CM, Fennessy FM. Multiparametric Magnetic Resonance Imaging of the Prostate: Repeatability of Volume and Apparent Diffusion Coefficient Quantification. Invest. Radiol. 2017;52:538-546.

    Not in our library10.1097/rli.0000000000000382 (opens in a new tab)

43. 43.

    Somford DM, et al. Initial Experience With Identifying High-Grade Prostate Cancer Using Diffusion-Weighted MR Imaging (DWI) in Patients With a Gleason Score ≤ 3 + 3 = 6 Upon Schematic TRUS-Guided Biopsy: A Radical Prostatectomy Correlated Series. Invest. Radiol. 2012;47:153.

    Not in our library10.1097/rli.0b013e31823ea1f0 (opens in a new tab)

44. 44.

    Fedorov A, et al. 3D Slicer as an Image Computing Platform for the Quantitative Imaging Network. Magn. Reson. Imaging. 2012;30:1323-1341.

    Not in our library10.1016/j.mri.2012.05.001 (opens in a new tab)

45. 45.

    Rios Velazquez E, et al. Somatic Mutations Drive Distinct Imaging Phenotypes in Lung Cancer. Cancer Res. 2017;77:3922-3930.

    Not in our library10.1158/0008-5472.can-17-0122 (opens in a new tab)

46. 46.

    Tustison NJ, et al. N4ITK: Improved N3 bias correction. IEEE Trans. Med. Imaging. 2010;29:1310-1320.

    Not in our library10.1109/tmi.2010.2046908 (opens in a new tab)

47. 47.

    Huang W, et al. The Impact of Arterial Input Function Determination Variations on Prostate Dynamic Contrast-Enhanced Magnetic Resonance Imaging Pharmacokinetic Modeling: A Multicenter Data Analysis Challenge. Tomography. 2016;2:56-66.

    Not in our library10.18383/j.tom.2015.00184 (opens in a new tab)

48. 48.
    The effect of SUV discretization in quantitative FDG-PET Radiomics: the need for standardized methodology in tumor texture analysis · 2015

    No notice
49. 49.

    Tixier F, et al. Reproducibility of Tumor Uptake Heterogeneity Characterization Through Textural Feature Analysis in 18F-FDG PET. J. Nucl. Med. 2012;53:693-700.

    Not in our library10.2967/jnumed.111.099127 (opens in a new tab)

50. 50.

    Shrout PE, Fleiss JL. Intraclass correlations: Uses in assessing rater reliability. Psychol. Bull. 1979;86:420-428.

    Not in our library10.1037/0033-2909.86.2.420 (opens in a new tab)

51. 51.

    Raunig DL, et al. Quantitative imaging biomarkers: A review of statistical methods for technical performance assessment. Stat. Methods Med. Res. 2015;24:27-67.

    Not in our library10.1177/0962280214537344 (opens in a new tab)

52. 52.
    Repeatability of Multiparametric Prostate MRI Radiomics Features · 2019

    No notice
53. 53.

    Rosenblatt M. Remarks on Some Nonparametric Estimates of a Density Function. Ann. Math. Stat. 1956;27:832-837.

    Not in our library10.1214/aoms/1177728190 (opens in a new tab)

54. 54.

    Parzen E. On Estimation of a Probability Density Function and Mode. Ann. Math. Stat. 1962;33:1065-1076.

    Not in our library10.1214/aoms/1177704472 (opens in a new tab)

55. 55.

    Kuhl Christiane K., Bruhn Robin, Krämer Nils, Nebelung Sven, Heidenreich Axel, Schrading Simone. Abbreviated Biparametric Prostate MR Imaging in Men with Elevated Prostate-specific Antigen. Radiology. 2017;285(2):493-505.

    Not in our library10.1148/radiol.2017170129 (opens in a new tab)

56. 56.

    Malyarenko D, et al. Multi-system repeatability and reproducibility of apparent diffusion coefficient measurement using an ice-water phantom. J. Magn. Reson. Imaging. 2013;37:1238-1246.

    Not in our library10.1002/jmri.23825 (opens in a new tab)

57. 57.

    Kwon D, et al. Classification of suspicious lesions on prostate multiparametric MRI using machine learning. J. Med. Imag. 2018;5:034502.

    Not in our library10.1117/1.jmi.5.3.034502 (opens in a new tab)

58. 58.

    Wang HJ, et al. Value of normalized apparent diffusion coefficient for estimating histological grade of vesical urothelial carcinoma. Clin. Radiol. 2014;69:727-731.

    Not in our library10.1016/j.crad.2014.03.001 (opens in a new tab)

59. 59.

    Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986;1:307-310.

    Not in our library10.1016/s0140-6736(86)90837-8 (opens in a new tab)

60. 60.

    Barnhart Huiman X, Yow Eric, Crowley Anna Lisa, Daubert Melissa A, Rabineau Dawn, Bigelow Robert, Pencina Michael, Douglas Pamela S. Choice of agreement indices for assessing and improving measurement reproducibility in a core laboratory setting. Statistical Methods in Medical Research. 2016;25(6):2939-2958.

    Not in our library10.1177/0962280214534651 (opens in a new tab)

61. 61.
    Ultrastructural Characterization of the Lower Motor System in a Mouse Model of Krabbe Disease · 2016

    No notice
62. 62.

    Zhao B, et al. Evaluating Variability in Tumor Measurements from Same-day Repeat CT Scans of Patients with Non–Small Cell Lung Cancer. Radiology. 2009;252:263-272.

    Not in our library10.1148/radiol.2522081593 (opens in a new tab)

63. 63.

    Vaidya, P. et al. RaPtomics: integrating radiomic and pathomic features for predicting recurrence in early stage lung cancer. In Medical Imaging 2018: Digital Pathology10581, 105810M (International Society for Optics and Photonics, 2018).

    Not in our library

64. 64.

    Chirra, P. et al. Empirical evaluation of cross-site reproducibility in radiomic features for characterizing prostate MRI. In Medical Imaging 2018: Computer-Aided Diagnosis10575, 105750B (International Society for Optics and Photonics, 2018).

    Not in our library

65. 65.

    Nyúl LG, Udupa JK, Zhang X. New variants of a method of MRI scale standardization. IEEE Trans. Med. Imaging. 2000;19:143-150.

    Not in our library10.1109/42.836373 (opens in a new tab)

66. 66.

    Shinohara RT, et al. Australian Imaging Biomarkers Lifestyle Flagship Study of Ageing & Alzheimer’s Disease Neuroimaging Initiative. Statistical normalization techniques for magnetic resonance imaging. Neuroimage Clin. 2014;6:9-19.

    Not in our library10.1016/j.nicl.2014.08.008 (opens in a new tab)

Cited by papers we hold 5

• Artificial intelligence and machine learning in cancer imaging · 2022
  No notice
• A deep look into radiomics · 2021
  No notice
• Radiomics in medical imaging—“how-to” guide and critical reflection · 2020
  No notice
• Standardization of brain MR images across machines and protocols: bridging the gap for MRI-based radiomics · 2020
  No notice
• Gray-level discretization impacts reproducible MRI radiomics texture features · 2019
  No notice

Questions about this paper

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.