Stability of clinical prediction models developed using statistical or machine learning methods

Abstract

Clinical prediction models estimate an individual's risk of a particular health outcome. A developed model is a consequence of the development dataset and model‐building strategy, including the sample size, number of predictors, and analysis method (e.g., regression or machine learning). We raise the concern that many models are developed using small datasets that lead to instability in the model and its predictions (estimated risks). We define four levels of model stability in estimated risks moving from the overall mean to the individual level. Through simulation and case studies of statistical and machine learning approaches, we show instability in a model's estimated risks is often considerable, and ultimately manifests itself as miscalibration of predictions in new data. Therefore, we recommend researchers always examine instability at the model development stage and propose instability plots and measures to do so. This entails repeating the model‐building steps (those used to develop the original prediction model) in each of multiple (e.g., 1000) bootstrap samples, to produce multiple bootstrap models, and deriving (i) a prediction instability plot of bootstrap model versus original model predictions; (ii) the mean absolute prediction error (mean absolute difference between individuals’ original and bootstrap model predictions), and (iii) calibration, classification, and decision curve instability plots of bootstrap models applied in the original sample. A case study illustrates how these instability assessments help reassure (or not) whether model predictions are likely to be reliable (or not), while informing a model's critical appraisal (risk of bias rating), fairness, and further validation requirements.

References 61

1. 1.

   Altman, D. G., & Andersen, P. K. (1989 ). Bootstrap investigation of the stability of a Cox regression model. Statistics in Medicine, 8(7), 771–783.

   Not in our library10.1002/sim.4780080702

2. 2.

   Archer, L., Snell, K. I. E., Ensor, J., Hudda, M. T., Collins, G. S., & Riley, R. D. (2021 ). Minimum sample size for external validation of a clinical prediction model with a continuous outcome. Statistics in Medicine, 40(1), 133–146.

   Not in our library10.1002/sim.8766

3. 3.

   Breiman, L. (1996 ). Heuristics of instability and stabilization in model selection. Annals of Statistics, 24(6), 2350–2383.

   Not in our library10.1214/aos/1032181158

4. 4.

   Cleveland, W. S. (1979 ). Robust locally weighted regression and smoothing scatterplots. Journal of the American Statistical Association, 74(368), 829–836.

   Not in our library10.1080/01621459.1979.10481038

5. 5.
   Protocol for development of a reporting guideline (TRIPOD-AI) and risk of bias tool (PROBAST-AI) for diagnostic and prognostic prediction model studies based on artificial intelligence · 2021

   No notice
6. 6.

   Dankowski, T., & Ziegler, A. (2016 ). Calibrating random forests for probability estimation. Statistics in Medicine, 35(22), 3949–3960.

   Not in our library10.1002/sim.6959

7. 7.

   De Bin, R., Janitza, S., Sauerbrei, W., & Boulesteix, A.‐L. (2016 ). Subsampling versus bootstrapping in resampling‐based model selection for multivariable regression. Biometrics, 72(1), 272–280.

   Not in our library10.1111/biom.12381

8. 8.

   Dhiman, P., Ma, J., Navarro, C. L. A., Speich, B., Bullock, G., Damen, J. A. A., Hooft, L., Kirtley, S., Riley, R. D., Calster, B. V., Moons, K. G. M., & Collins, G. S. (2022 ). Methodological conduct of prognostic prediction models developed using machine learning in oncology: A systematic review. BMC Medical Research Methodology [Electronic Resource], 22(1), Article 101.

   Not in our library10.1186/s12874-022-01577-x

9. 9.

   Ensor, J., Riley, R. D., Moore, D., Snell, K. I., Bayliss, S., & Fitzmaurice, D. (2016 ). Systematic review of prognostic models for recurrent venous thromboembolism (VTE) post‐treatment of first unprovoked VTE. BMJ Open, 6(5), e011190.

   Not in our library10.1136/bmjopen-2016-011190

10. 10.

    Galea, M. H., Blamey, R. W., Elston, C. E., & Ellis, I. O. (1992 ). The Nottingham Prognostic Index in primary breast cancer. Breast Cancer Research and Treatment, 22(3), 207–219.

    Not in our library10.1007/bf01840834

11. 11.

    Ghassemi, M., Oakden‐Rayner, L., & Beam, A. L. (2021 ). The false hope of current approaches to explainable artificial intelligence in health care. Lancet Digital Health, 3(11), e745–e750.

    Not in our library10.1016/s2589-7500(21)00208-9

12. 12.

    Grote, T., & Keeling, G. (2022 ). On algorithmic fairness in medical practice. Cambridge Quarterly of Healthcare Ethics, 31(1), 83–94.

    Not in our library10.1017/s0963180121000839

13. 13.

    Harrell, F. E., Jr. (2015 ). Regression modeling strategies: With applications to linear models, logistic and ordinal regression, and survival analysis (2nd ed.). Springer.

    Not in our library

14. 14.
    A prognostic index in primary breast cancer · 1982

    No notice
15. 15.

    Heinze, G., Wallisch, C., & Dunkler, D. (2018 ). Variable selection—A review and recommendations for the practicing statistician. Biometrical Journal, 60(3), 431–449.

    Not in our library10.1002/bimj.201700067

16. 16.

    Hennig, C., & Sauerbrei, W. (2019 ). Exploration of the variability of variable selection based on distances between bootstrap sample results. Advances in Data Analysis and Classification, 13(4), 933–963.

    Not in our library10.1007/s11634-018-00351-6

17. 17.

    Houwelingen, H. C., & Sauerbrei, W. (2013 ). Cross‐validation, shrinkage and variable selection in linear regression revisited. Open Journal of Statistics, 3, 79–102.

    Not in our library10.4236/ojs.2013.32011

18. 18.

    Janitza, S., Binder, H., & Boulesteix, A. L. (2016 ). Pitfalls of hypothesis tests and model selection on bootstrap samples: Causes and consequences in biometrical applications. Biometrical Journal, 58(3), 447–473.

    Not in our library10.1002/bimj.201400246

19. 19.

    Leeuwenberg, A. M., van Smeden, M., Langendijk, J. A., van der Schaaf, A., Mauer, M. E., Moons, K. G. M., Reitsma, J. B., & Schuit, E. (2022 ). Performance of binary prediction models in high‐correlation low‐dimensional settings: A comparison of methods. Diagnostic and Prognostic Research, 6(1), Article 1.

    Not in our library10.1186/s41512-021-00115-5

20. 20.

    Localio, A., & Goodman, S. (2012 ). Beyond the usual prediction accuracy metrics: Reporting results for clinical decision making. Annals of Internal Medicine, 157(4), 294–295.

    Not in our library10.7326/0003-4819-157-4-201208210-00014

21. 21.

    Martin, G. P., Riley, R. D., Collins, G. S., & Sperrin, M. (2021 ). Developing clinical prediction models when adhering to minimum sample size recommendations: The importance of quantifying bootstrap variability in tuning parameters and predictive performance. Statistical Methods in Medical Research, 30(12), 2545–2561.

    Not in our library10.1177/09622802211046388

22. 22.

    Moons, K. G., Altman, D. G., Vergouwe, Y., & Royston, P. (2009 ). Prognosis and prognostic research: Application and impact of prognostic models in clinical practice. BMJ, 338, b606.

    Not in our library10.1136/bmj.b606

23. 23.

    Nashef, S. A. M., Roques, F., Michel, P., Gauducheau, E., Lemeshow, S., & Salamon, R. (1999 ). European system for cardiac operative risk evaluation (EuroSCORE). European Journal of Cardio‐Thoracic Surgery, 16(1), 9–13.

    Not in our library10.1016/s1010-7940(99)00134-7

24. 24.

    Nashef, S. A. M., Roques, F., Sharples, L. D., Nilsson, J., Smith, C., Goldstone, A. R., & Lockowandt, U. (2012 ). EuroSCORE II. European Journal of Cardio‐Thoracic Surgery, 41(4), 734–745.

    Not in our library10.1093/ejcts/ezs043

25. 25.
    Risk of bias in studies on prediction models developed using supervised machine learning techniques: systematic review · 2021

    No notice
26. 26.

    Pate, A., Emsley, R., Sperrin, M., Martin, G. P., & van Staa, T. (2020 ). Impact of sample size on the stability of risk scores from clinical prediction models: A case study in cardiovascular disease. Diagnostic and Prognostic Research, 4, Article 14.

    Not in our library10.1186/s41512-020-00082-3

27. 27.

    Platt, J. (2000 ). Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In A. Smola, P. Bartlett, B. Schölkopf, & D. Schuurmans (Eds.), Advances in large margin classifiers. Cambridge: MIT Press.

    Not in our library

28. 28.

    Reilly, B. M., & Evans, A. T. (2006 ). Translating clinical research into clinical practice: Impact of using prediction rules to make decisions. Annals of Internal Medicine, 144(3), 201–209.

    Not in our library10.7326/0003-4819-144-3-200602070-00009

29. 29.

    Riley, R. D., Collins, G. S., Ensor, J., Archer, L., Booth, S., Mozumder, S. I., Rutherford, M. J., van Smeden, M., Lambert, P. C., & Snell, K. I. E. (2022 ). Minimum sample size calculations for external validation of a clinical prediction model with a time‐to‐event outcome. Statistics in Medicine, 41(7), 1280–1295.

    Not in our library10.1002/sim.9275

30. 30.

    Riley, R. D., Debray, T. P. A., Collins, G. S., Archer, L., Ensor, J., van Smeden, M., & Snell, K. I. E. (2021 ). Minimum sample size for external validation of a clinical prediction model with a binary outcome. Statistics in Medicine, 40(19), 4230–4251.

    Not in our library10.1002/sim.9025

31. 31.

    Riley, R. D., Ensor, J., Snell, K. I. E., Harrell, F. E., Jr., Martin, G. P., Reitsma, J. B., Moons, K. G. M., Collins, G., & van Smeden, M. (2020 ). Calculating the sample size required for developing a clinical prediction model. BMJ, 368, m441.

    Not in our library10.1136/bmj.m441

32. 32.

    Riley, R. D., Snell, K. I. E., Ensor, J., Burke, D. L., Harrell, F. E., Jr., Moons, K. G. M., & Collins, G. S. (2019a ). Minimum sample size for developing a multivariable prediction model: Part II—Binary and time‐to‐event outcomes. Statistics in Medicine, 38(7), 1276–1296.

    Not in our library10.1002/sim.7992

33. 33.

    Riley, R. D., Snell, K. I. E., Ensor, J., Burke, D. L., Harrell, F. E., Jr., Moons, K. G. M., & Collins, G. S. (2019b ). Minimum sample size for developing a multivariable prediction model: Part I ‐ Continuous outcomes. Statistics in Medicine, 38(7), 1262–1275.

    Not in our library10.1002/sim.7993

34. 34.

    Riley, R. D., Snell, K. I. E., Martin, G. P., Whittle, R., Archer, L., Sperrin, M., & Collins, G. S. (2021 ). Penalization and shrinkage methods produced unreliable clinical prediction models especially when sample size was small. Journal of Clinical Epidemiology, 132, 88–96.

    Not in our library10.1016/j.jclinepi.2020.12.005

35. 35.

    Riley, R. D., Van Calster, B., & Collins, G. S. (2021 ). A note on estimating the Cox‐Snell R 2 from a reported C statistic (AUROC) to inform sample size calculations for developing a prediction model with a binary outcome. Statistics in Medicine, 40(4), 859–864.

    Not in our library10.1002/sim.8806

36. 36.

    Riley, R. D., van der Windt, D., Croft, P., & Moons, K. G. M. (Eds.). (2019 ). Prognosis research in healthcare: Concepts, methods and impact. Oxford University Press.

    Not in our library

37. 37.

    Riley, R. D., Tierney, J. F., & Stewart, L. A. (Eds). (2021 ). Individual participant data meta‐analysis: A handbook for healthcare research. Wiley.

    Not in our library

38. 38.

    Roberts, S., & Nowak, G. (2014 ). Stabilizing the lasso against cross‐validation variability. Computational Statistics & Data Analysis, 70, 198–211.

    Not in our library

39. 39.

    Royston, P., & Sauerbrei, W. (2003 ). Stability of multivariable fractional polynomial models with selection of variables and transformations: A bootstrap investigation. Statistics in Medicine, 22(4), 639–659.

    Not in our library10.1002/sim.1310

40. 40.

    Royston, P., & Sauerbrei, W. (2009 ). Bootstrap assessment of the stability of multivariable models. Stata Journal, 9(4), 547–570.

    Not in our library10.1177/1536867x0900900403

41. 41.

    Sauerbrei, W., Boulesteix, A. L., & Binder, H. (2011 ). Stability investigations of multivariable regression models derived from low‐ and high‐dimensional data. Journal of Biopharmaceutical Statistics, 21(6), 1206–1231.

    Not in our library10.1080/10543406.2011.629890

42. 42.

    Sauerbrei, W., Buchholz, A., Boulesteix, A. L., & Binder, H. (2015 ). On stability issues in deriving multivariable regression models. Biometrical Journal, 57(4), 531–555.

    Not in our library10.1002/bimj.201300222

43. 43.

    Schonlau, M., & Zou, R. Y. (2020 ). The random forest algorithm for statistical learning. Stata Journal, 20(1), 3–29.

    Not in our library10.1177/1536867x20909688

44. 44.

    Seibold, H., Bernau, C., Boulesteix, A.‐L., & De Bin, R. (2018 ). On the choice and influence of the number of boosting steps for high‐dimensional linear Cox‐models. Computational Statistics, 33(3), 1195–1215.

    Not in our library10.1007/s00180-017-0773-8

45. 45.

    Sinkovec, H., Heinze, G., Blagus, R., & Geroldinger, A. (2021 ). To tune or not to tune, a case study of ridge logistic regression in small or sparse datasets. BMC Medical Research Methodology [Electronic Resource], 21(1), Article 199.

    Not in our library10.1186/s12874-021-01374-y

46. 46.

    Steyerberg, E. W. (2019 ). Clinical prediction models: A practical approach to development, validation, and updating (2nd ed.). Springer.

    Not in our library

47. 47.

    Steyerberg, E. W., Eijkemans, W. J. C., Harrell, F. E., & Habbema, J. D. F. (2000 ). Prognostic modelling with logistic regression analysis: A comparison of selection and estimation methods in small datasets. Statistics in Medicine, 19, 1059–1079.

    Not in our library10.1002/(sici)1097-0258(20000430)19:8<1059::aid-sim412>3.0.co;2-0

48. 48.
    Prognosis Research Strategy (PROGRESS) 3: Prognostic Model Research · 2013

    No notice
49. 49.

    The GUSTO Investigators. (1993 ). An international randomized trial comparing four thrombolytic strategies for acute myocardial infarction. New England Journal of Medicine, 329(10), 673–682.

    Not in our library10.1056/nejm199309023291001

50. 50.
    Calibration: the Achilles heel of predictive analytics · 2019

    No notice
51. 51.

    Van Calster, B., Nieboer, D., Vergouwe, Y., De Cock, B., Pencina, M. J., & Steyerberg, E. W. (2016 ). A calibration hierarchy for risk models was defined: From utopia to empirical data. Journal of Clinical Epidemiology, 74, 167–176.

    Not in our library10.1016/j.jclinepi.2015.12.005

52. 52.

    Van Calster, B., van Smeden, M., De Cock, B., & Steyerberg, E. W. (2020 ). Regression shrinkage methods for clinical prediction models do not guarantee improved performance: Simulation study. Statistical Methods in Medical Research, 29(11), 3166–3178.

    Not in our library10.1177/0962280220921415

53. 53.

    Van Houwelingen, J. C. (2001 ). Shrinkage and penalized likelihood as methods to improve predictive accuracy. Statistica Neerlandica, 55, 17–34.

    Not in our library10.1111/1467-9574.00154

54. 54.

    Van Houwelingen, J. C., & Le Cessie, S. (1990 ). Predictive value of statistical models. Statistics in Medicine, 9(11), 1303–1325.

    Not in our library10.1002/sim.4780091109

55. 55.

    van Smeden, M., Moons, K. G., de Groot, J. A., Collins, G. S., Altman, D. G., Eijkemans, M. J., & Reitsma, J. B. (2019 ). Sample size for binary logistic prediction models: Beyond events per variable criteria. Statistical Methods in Medical Research, 28(8), 2455–2474.

    Not in our library10.1177/0962280218784726

56. 56.
    Extensions to decision curve analysis, a novel method for evaluating diagnostic tests, prediction models and molecular markers · 2008

    No notice
57. 57.

    Vickers, A. J., & Elkin, E. B. (2006 ). Decision curve analysis: A novel method for evaluating prediction models. Medical Decision Making, 26(6), 565–574.

    Not in our library10.1177/0272989x06295361

58. 58.

    Vickers, A. J., Van Calster, B., & Steyerberg, E. W. (2016 ). Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests. BMJ, 352, i6.

    Not in our library10.1136/bmj.i6

59. 59.

    Wallisch, C., Dunkler, D., Rauch, G., de Bin, R., & Heinze, G. (2021 ). Selection of variables for multivariable models: Opportunities and limitations in quantifying model stability by resampling. Statistics in Medicine, 40(2), 369–381.

    Not in our library10.1002/sim.8779

60. 60.

    Wolff, R. F., Moons, K. G. M., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., Mallett, S., & PROBAST Group. (2019 ). PROBAST: A tool to assess the risk of bias and applicability of prediction model studies. Annals of Internal Medicine, 170(1), 51–58.

    Not in our library10.7326/m18-1376

61. 61.
    Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal · 2020

    No notice

Cited by papers we hold 3

• Tackling algorithmic bias and promoting transparency in health datasets: the STANDING Together consensus recommendations · 2025
  No notice
• Evaluation of clinical prediction models (part 1): from development to external validation · 2024
  No notice
• Developing clinical prediction models: a step-by-step guide · 2024
  No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.