Stability of clinical prediction models developed using statistical or machine learning methods

Abstract

Clinical prediction models estimate an individual's risk of a particular health outcome. A developed model is a consequence of the development dataset and model‐building strategy, including the sample size, number of predictors, and analysis method (e.g., regression or machine learning). We raise the concern that many models are developed using small datasets that lead to instability in the model and its predictions (estimated risks). We define four levels of model stability in estimated risks moving from the overall mean to the individual level. Through simulation and case studies of statistical and machine learning approaches, we show instability in a model's estimated risks is often considerable, and ultimately manifests itself as miscalibration of predictions in new data. Therefore, we recommend researchers always examine instability at the model development stage and propose instability plots and measures to do so. This entails repeating the model‐building steps (those used to develop the original prediction model) in each of multiple (e.g., 1000) bootstrap samples, to produce multiple bootstrap models, and deriving (i) a prediction instability plot of bootstrap model versus original model predictions; (ii) the mean absolute prediction error (mean absolute difference between individuals’ original and bootstrap model predictions), and (iii) calibration, classification, and decision curve instability plots of bootstrap models applied in the original sample. A case study illustrates how these instability assessments help reassure (or not) whether model predictions are likely to be reliable (or not), while informing a model's critical appraisal (risk of bias rating), fairness, and further validation requirements.

References 61

1. 1.

   Altman, D. G., & Andersen, P. K. (1989 ). Bootstrap investigation of the stability of a Cox regression model. Statistics in Medicine, 8(7), 771–783.

   Not in our library10.1002/sim.4780080702

2. 2.

   Archer, L., Snell, K. I. E., Ensor, J., Hudda, M. T., Collins, G. S., & Riley, R. D. (2021 ). Minimum sample size for external validation of a clinical prediction model with a continuous outcome. Statistics in Medicine, 40(1), 133–146.

   Not in our library10.1002/sim.8766

3. 3.

   Breiman, L. (1996 ). Heuristics of instability and stabilization in model selection. Annals of Statistics, 24(6), 2350–2383.

   Not in our library10.1214/aos/1032181158

4. 4.

   Cleveland, W. S. (1979 ). Robust locally weighted regression and smoothing scatterplots. Journal of the American Statistical Association, 74(368), 829–836.

   Not in our library10.1080/01621459.1979.10481038

5. 5.
   Protocol for development of a reporting guideline (TRIPOD-AI) and risk of bias tool (PROBAST-AI) for diagnostic and prognostic prediction model studies based on artificial intelligence · 2021

   No notice
6. 6.

   Dankowski, T., & Ziegler, A. (2016 ). Calibrating random forests for probability estimation. Statistics in Medicine, 35(22), 3949–3960.

   Not in our library10.1002/sim.6959

7. 7.

   De Bin, R., Janitza, S., Sauerbrei, W., & Boulesteix, A.‐L. (2016 ). Subsampling versus bootstrapping in resampling‐based model selection for multivariable regression. Biometrics, 72(1), 272–280.

   Not in our library10.1111/biom.12381

8. 8.

   Dhiman, P., Ma, J., Navarro, C. L. A., Speich, B., Bullock, G., Damen, J. A. A., Hooft, L., Kirtley, S., Riley, R. D., Calster, B. V., Moons, K. G. M., & Collins, G. S. (2022 ). Methodological conduct of prognostic prediction models developed using machine learning in oncology: A systematic review. BMC Medical Research Methodology [Electronic Resource], 22(1), Article 101.

   Not in our library10.1186/s12874-022-01577-x

9. 9.

   Ensor, J., Riley, R. D., Moore, D., Snell, K. I., Bayliss, S., & Fitzmaurice, D. (2016 ). Systematic review of prognostic models for recurrent venous thromboembolism (VTE) post‐treatment of first unprovoked VTE. BMJ Open, 6(5), e011190.

   Not in our library10.1136/bmjopen-2016-011190

10. 10.

    Galea, M. H., Blamey, R. W., Elston, C. E., & Ellis, I. O. (1992 ). The Nottingham Prognostic Index in primary breast cancer. Breast Cancer Research and Treatment, 22(3), 207–219.

    Not in our library10.1007/bf01840834

11. 11.

    Ghassemi, M., Oakden‐Rayner, L., & Beam, A. L. (2021 ). The false hope of current approaches to explainable artificial intelligence in health care. Lancet Digital Health, 3(11), e745–e750.

    Not in our library10.1016/s2589-7500(21)00208-9

12. 12.

    Grote, T., & Keeling, G. (2022 ). On algorithmic fairness in medical practice. Cambridge Quarterly of Healthcare Ethics, 31(1), 83–94.

    Not in our library10.1017/s0963180121000839

13. 13.

    Harrell, F. E., Jr. (2015 ). Regression modeling strategies: With applications to linear models, logistic and ordinal regression, and survival analysis (2nd ed.). Springer.

    Not in our library

14. 14.
    A prognostic index in primary breast cancer · 1982

    No notice
15. 15.

    Heinze, G., Wallisch, C., & Dunkler, D. (2018 ). Variable selection—A review and recommendations for the practicing statistician. Biometrical Journal, 60(3), 431–449.

    Not in our library10.1002/bimj.201700067

16. 16.

    Hennig, C., & Sauerbrei, W. (2019 ). Exploration of the variability of variable selection based on distances between bootstrap sample results. Advances in Data Analysis and Classification, 13(4), 933–963.

    Not in our library10.1007/s11634-018-00351-6

17. 17.

    Houwelingen, H. C., & Sauerbrei, W. (2013 ). Cross‐validation, shrinkage and variable selection in linear regression revisited. Open Journal of Statistics, 3, 79–102.

    Not in our library10.4236/ojs.2013.32011

18. 18.

    Janitza, S., Binder, H., & Boulesteix, A. L. (2016 ). Pitfalls of hypothesis tests and model selection on bootstrap samples: Causes and consequences in biometrical applications. Biometrical Journal, 58(3), 447–473.

    Not in our library10.1002/bimj.201400246

19. 19.

    Leeuwenberg, A. M., van Smeden, M., Langendijk, J. A., van der Schaaf, A., Mauer, M. E., Moons, K. G. M., Reitsma, J. B., & Schuit, E. (2022 ). Performance of binary prediction models in high‐correlation low‐dimensional settings: A comparison of methods. Diagnostic and Prognostic Research, 6(1), Article 1.

    Not in our library10.1186/s41512-021-00115-5

20. 20.

    Localio, A., & Goodman, S. (2012 ). Beyond the usual prediction accuracy metrics: Reporting results for clinical decision making. Annals of Internal Medicine, 157(4), 294–295.

    Not in our library10.7326/0003-4819-157-4-201208210-00014

21. 21.

    Martin, G. P., Riley, R. D., Collins, G. S., & Sperrin, M. (2021 ). Developing clinical prediction models when adhering to minimum sample size recommendations: The importance of quantifying bootstrap variability in tuning parameters and predictive performance. Statistical Methods in Medical Research, 30(12), 2545–2561.

    Not in our library10.1177/09622802211046388

22. 22.

    Moons, K. G., Altman, D. G., Vergouwe, Y., & Royston, P. (2009 ). Prognosis and prognostic research: Application and impact of prognostic models in clinical practice. BMJ, 338, b606.

    Not in our library10.1136/bmj.b606

23. 23.

    Nashef, S. A. M., Roques, F., Michel, P., Gauducheau, E., Lemeshow, S., & Salamon, R. (1999 ). European system for cardiac operative risk evaluation (EuroSCORE). European Journal of Cardio‐Thoracic Surgery, 16(1), 9–13.

    Not in our library10.1016/s1010-7940(99)00134-7

24. 24.

    Nashef, S. A. M., Roques, F., Sharples, L. D., Nilsson, J., Smith, C., Goldstone, A. R., & Lockowandt, U. (2012 ). EuroSCORE II. European Journal of Cardio‐Thoracic Surgery, 41(4), 734–745.

    Not in our library10.1093/ejcts/ezs043

25. 25.
    Risk of bias in studies on prediction models developed using supervised machine learning techniques: systematic review · 2021

    No notice
26. 26.

    Pate, A., Emsley, R., Sperrin, M., Martin, G. P., & van Staa, T. (2020 ). Impact of sample size on the stability of risk scores from clinical prediction models: A case study in cardiovascular disease. Diagnostic and Prognostic Research, 4, Article 14.

    Not in our library10.1186/s41512-020-00082-3

27. 27.

    Platt, J. (2000 ). Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In A. Smola, P. Bartlett, B. Schölkopf, & D. Schuurmans (Eds.), Advances in large margin classifiers. Cambridge: MIT Press.

    Not in our library

28. 28.

    Reilly, B. M., & Evans, A. T. (2006 ). Translating clinical research into clinical practice: Impact of using prediction rules to make decisions. Annals of Internal Medicine, 144(3), 201–209.

    Not in our library10.7326/0003-4819-144-3-200602070-00009

29. 29.

    Riley, R. D., Collins, G. S., Ensor, J., Archer, L., Booth, S., Mozumder, S. I., Rutherford, M. J., van Smeden, M., Lambert, P. C., & Snell, K. I. E. (2022 ). Minimum sample size calculations for external validation of a clinical prediction model with a time‐to‐event outcome. Statistics in Medicine, 41(7), 1280–1295.

    Not in our library10.1002/sim.9275

30. 30.

    Riley, R. D., Debray, T. P. A., Collins, G. S., Archer, L., Ensor, J., van Smeden, M., & Snell, K. I. E. (2021 ). Minimum sample size for external validation of a clinical prediction model with a binary outcome. Statistics in Medicine, 40(19), 4230–4251.

    Not in our library10.1002/sim.9025

31. 31.

    Riley, R. D., Ensor, J., Snell, K. I. E., Harrell, F. E., Jr., Martin, G. P., Reitsma, J. B., Moons, K. G. M., Collins, G., & van Smeden, M. (2020 ). Calculating the sample size required for developing a clinical prediction model. BMJ, 368, m441.

    Not in our library10.1136/bmj.m441

32. 32.

    Riley, R. D., Snell, K. I. E., Ensor, J., Burke, D. L., Harrell, F. E., Jr., Moons, K. G. M., & Collins, G. S. (2019a ). Minimum sample size for developing a multivariable prediction model: Part II—Binary and time‐to‐event outcomes. Statistics in Medicine, 38(7), 1276–1296.

    Not in our library10.1002/sim.7992

33. 33.

    Riley, R. D., Snell, K. I. E., Ensor, J., Burke, D. L., Harrell, F. E., Jr., Moons, K. G. M., & Collins, G. S. (2019b ). Minimum sample size for developing a multivariable prediction model: Part I ‐ Continuous outcomes. Statistics in Medicine, 38(7), 1262–1275.

    Not in our library10.1002/sim.7993

34. 34.

    Riley, R. D., Snell, K. I. E., Martin, G. P., Whittle, R., Archer, L., Sperrin, M., & Collins, G. S. (2021 ). Penalization and shrinkage methods produced unreliable clinical prediction models especially when sample size was small. Journal of Clinical Epidemiology, 132, 88–96.

    Not in our library10.1016/j.jclinepi.2020.12.005

35. 35.

    Riley, R. D., Van Calster, B., & Collins, G. S. (2021 ). A note on estimating the Cox‐Snell R 2 from a reported C statistic (AUROC) to inform sample size calculations for developing a prediction model with a binary outcome. Statistics in Medicine, 40(4), 859–864.

    Not in our library10.1002/sim.8806

36. 36.

    Riley, R. D., van der Windt, D., Croft, P., & Moons, K. G. M. (Eds.). (2019 ). Prognosis research in healthcare: Concepts, methods and impact. Oxford University Press.

    Not in our library

37. 37.

    Riley, R. D., Tierney, J. F., & Stewart, L. A. (Eds). (2021 ). Individual participant data meta‐analysis: A handbook for healthcare research. Wiley.

    Not in our library

38. 38.

    Roberts, S., & Nowak, G. (2014 ). Stabilizing the lasso against cross‐validation variability. Computational Statistics & Data Analysis, 70, 198–211.

    Not in our library

39. 39.

    Royston, P., & Sauerbrei, W. (2003 ). Stability of multivariable fractional polynomial models with selection of variables and transformations: A bootstrap investigation. Statistics in Medicine, 22(4), 639–659.

    Not in our library10.1002/sim.1310

40. 40.

    Royston, P., & Sauerbrei, W. (2009 ). Bootstrap assessment of the stability of multivariable models. Stata Journal, 9(4), 547–570.

    Not in our library10.1177/1536867x0900900403

Cited by papers we hold 3

• Tackling algorithmic bias and promoting transparency in health datasets: the STANDING Together consensus recommendations · 2025
  No notice
• Evaluation of clinical prediction models (part 1): from development to external validation · 2024
  No notice
• Developing clinical prediction models: a step-by-step guide · 2024
  No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.