Cross-validation pitfalls when selecting and assessing regression and classification models

Abstract

BACKGROUND: We address the problem of selecting and assessing classification and regression models using cross-validation. Current state-of-the-art methods can yield models with high variance, rendering them unsuitable for a number of practical applications including QSAR. In this paper we describe and evaluate best practices which improve reliability and increase confidence in selected models. A key operational component of the proposed methods is cloud computing which enables routine use of previously infeasible approaches. METHODS: We describe in detail an algorithm for repeated grid-search V-fold cross-validation for parameter tuning in classification and regression, and we define a repeated nested cross-validation algorithm for model assessment. As regards variable selection and parameter tuning we define two algorithms (repeated grid-search cross-validation and double cross-validation), and provide arguments for using the repeated grid-search in the general case. RESULTS: We show results of our algorithms on seven QSAR datasets. The variation of the prediction performance, which is the result of choosing different splits of the dataset in V-fold cross-validation, needs to be taken into account when selecting and assessing classification and regression models. CONCLUSIONS: We demonstrate the importance of repeating cross-validation when selecting an optimal model, as well as the importance of repeating nested cross-validation when assessing a prediction error.

References 31

1. 1.

   This reference’s wording can’t be shown under the paper’s licence.

   Not in our library

2. 2.

   This reference’s wording can’t be shown under the paper’s licence.

   Not in our library

3. 3.

   This reference’s wording can’t be shown under the paper’s licence.

   Not in our library

4. 4.

   This reference’s wording can’t be shown under the paper’s licence.

   Not in our library

5. 5.

   This reference’s wording can’t be shown under the paper’s licence.

   Not in our library

6. 6.

   This reference’s wording can’t be shown under the paper’s licence.

   Not in our library

7. 7.

   This reference’s wording can’t be shown under the paper’s licence.

   Not in our library

8. 8.

   This reference’s wording can’t be shown under the paper’s licence.

   Not in our library

9. 9.

   This reference’s wording can’t be shown under the paper’s licence.

   Not in our library

10. 10.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

11. 11.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

12. 12.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

13. 13.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

14. 14.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

15. 15.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

16. 16.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

17. 17.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

18. 18.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

19. 19.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

20. 20.
    Bias in error estimation when using cross-validation for model selection · 2006

    No notice
21. 21.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

22. 22.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

23. 23.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

24. 24.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

25. 25.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

26. 26.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

27. 27.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

28. 28.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

29. 29.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

30. 30.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

31. 31.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

Cited by papers we hold 14

• Evaluating the performance of machine learning models for automatic diagnosis of patients with schizophrenia based on a single site dataset of 440 participants · 2022
  Not checked
• A multilayer multimodal detection and prediction model based on explainable artificial intelligence for Alzheimer’s disease · 2021
  No notice
• Evaluating the transcriptional fidelity of cancer models · 2021
  Not checked
• NBS-Predict: A prediction-based extension of the network-based statistic · 2021
  Not checked
• Machine Learning of Serum Metabolic Patterns Encodes Asymptomatic SARS-CoV-2 Infection · 2021
  Not checked
• Psycho-social factors associated with mental resilience in the Corona lockdown · 2021
  No notice
• AnOxPePred: using deep learning for the prediction of antioxidative properties of peptides · 2020
  No notice
• Machine learning algorithm validation with a limited sample size · 2019
  No notice
• Radiomics in Glioblastoma: Current Status and Challenges Facing Clinical Implementation · 2019
  No notice
• Identification of usual interstitial pneumonia pattern using RNA-Seq and machine learning: challenges and solutions · 2018
  Not checked
• Reliable Predictors of Arsenic Occurrence in the Southern Gulf Coast Aquifer of Texas · 2018
  Not checked
• Predicting Outcome of Endovascular Treatment for Acute Ischemic Stroke: Potential Value of Machine Learning Algorithms · 2018
  No notice
• Projection–Based Text Line Segmentation with a Variable Threshold · 2017
  Not checked
• Machine learning, statistical learning and the future of biological research in psychiatry · 2016
  No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.