A Comparison of MCC and CEN Error Measures in Multi-Class Prediction

Abstract

We show that the Confusion Entropy, a measure of performance in multiclass problems has a strong (monotone) relation with the multiclass generalization of a classical metric, the Matthews Correlation Coefficient. Analytical results are provided for the limit cases of general no-information (n-face dice rolling) of the binary classification. Computational evidence supports the claim in the general case.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 35

1. 1.

   Demšar J (2006 ) Statistical Comparisons of Classifiers over Multiple Data Sets. Journal of Machine Learning Research 7: 1–30.

   Not in our library

2. 2.

   García S, Herrera F (2008 ) An Extension on “Statistical Comparisons of Classifiers over Multiple Data Sets” for all Pairwise Comparisons. Journal of Machine Learning Research 9: 2677–2694.

   Not in our library

3. 3.

   Hand D (2009 ) Measuring classifier performance: a coherent alternative to the area under the ROC curve. Machine Learning 77: 103–123.

   Not in our library

4. 4.

   Sokolova M, Lapalme G (2009 ) A systematic analysis of performance measures for classification tasks. Information Processing and Management 45: 427–437.

   Not in our library

5. 5.

   Ferri C, Hernández-Orallo J, Modroiu R (2009 ) An experimental comparison of performance measures for classification. Pattern Recognition Letters 30: 27–38.

   Not in our library

6. 6.

   Felkin M (2007) Comparing Classification Results between N-ary and Binary Problems. In: Studies in Computational Intelligence, Springer-Verlag, volume 43. pp. 277–301.

   Not in our library

7. 7.

   Diri B, Albayrak S (2008 ) Visualization and analysis of classifiers performance in multi-class medical data. Expert Systems with Applications 34: 628–634.

   Not in our library

8. 8.

   Hanley J, McNeil B (1982 ) The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143: 29–36.

   Not in our library10.1148/radiology.143.1.7063747

9. 9.

   Bradley A (1997 ) The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition 30: 1145–1159.

   Not in our library

10. 10.

    Everson R, Fieldsend J (2006 ) Multi-class ROC analysis from a multi-objective optimisation perspective. Pattern Recognition Letters 27: 918–927.

    Not in our library

11. 11.

    Landgrebe T, Duin R (2005) On Neyman-Pearson optimisation for multiclass classifiers. In: Proceedings 16th Annual Symposium of the Pattern Recognition Association of South Africa. PRASA, pp. 165–170.

    Not in our library

12. 12.

    Landgrebe T, Duin R (2006) A simplified extension of the Area under the ROC to the multiclass domain. In: Proceedings 17th Annual Symposium of the Pattern Recognition Association of South Africa. PRASA, pp. 241–245.

    Not in our library

13. 13.

    Landgrebe T, Duin R (2008 ) Efficient multiclass ROC approximation by decomposition via confusion matrix perturbation analysis. IEEE Transactions Pattern Analysis Machine Intelligence 30: 810–822.

    Not in our library10.1109/tpami.2007.70740

14. 14.

    Ferri C, Hernández-Orallo J, Salido M (2003) Volume under the ROC surface for multi-class problems. In: Proceedings of 14th European Conference on Machine Learning. Springer-Verlag, pp. 108–120.

    Not in our library

15. 15.

    Van Calster B, Van Belle V, Condous G, Bourne T, Timmerman D, et al. (2008) Multi-class AUC metrics and weighted alternatives. In: Proceedings 2008 International Joint Conference on Neural Networks, IJCNN08. IEEE, pp. 1390–1396.

    Not in our library

16. 16.

    Li Y (2009) A generalization of AUC to an ordered multi-class diagnosis and application to lon-gitudinal data analysis on intellectual outcome in pediatric brain-tumor patients. Ph.D. thesis, College of Arts and Sciences, Georgia State University.

    Not in our library

17. 17.

    Hand D, Till R (2001 ) A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems. Machine Learning 45: 171–186.

    Not in our library

18. 18.

    Freitas C, De Carvalho J, Oliveira J Jr, Aires S, Sabourin R (2007) Confusion matrix disagreement for multiple classifiers. In: Rueda L, Mery D, Kittler J, editors, Proceedings of 12th Iberoamerican Congress on Pattern Recognition, CIARP 2007, LNCS 4756. Springer-Verlag, pp. 387–396.

    Not in our library

19. 19.

    Freitas C, De Carvalho J, Oliveira J Jr, Aires S, Sabourin R (2007) Distance-based Disagreement Classifiers Combination. In: Proceedings of the International Joint Conference on Neural Networks, IJCNN 2007. IEEE, pp. 2729–2733.

    Not in our library

20. 20.

    Matthews B (1975 ) Comparison of the predicted and observed secondary structure of T4 phage lysozyme. Biochimica et Biophysica Acta - Protein Structure 405: 442–451.

    Not in our library10.1016/0005-2795(75)90109-9

21. 21.

    Baldi P, Brunak S, Chauvin Y, Andersen C, Nielsen H (2000 ) Assessing the accuracy of prediction algorithms for classification: an overview. Bioinformatics 16: 412–424.

    Not in our library10.1093/bioinformatics/16.5.412

22. 22.

    The MicroArray Quality Control (MAQC) Consortium (2010 ) The MAQC-II Project: A comprehensive study of common practices for the development and validation of microarray-based predictive models. Nature Biotechnology 28: 827–838.

    Not in our library10.1038/nbt.1665

23. 23.

    Gorodkin J (2004 ) Comparing two K-category assignments by a K-category correlation coefficient. Computational Biology and Chemistry 28: 367–374.

    Not in our library10.1016/j.compbiolchem.2004.09.006

24. 24.

    Supper J, Spieth C, Zell A (2007) Reconstructing Linear Gene Regulatory Networks. In: Marchiori E, Moore J, Rajapakse J, editors, Proceedings of the 5th European Conference on Evolutionary Computation, Machine Learning and Data Mining in Bioinformatics, EvoBIO2007, LNCS 4447. Springer-Verlag, pp. 270–279.

    Not in our library

25. 25.

    Stokic D, Hanel R, Thurner S (2009 ) A fast and efficient gene-network reconstruction method from multiple over-expression experiments. BMC Bioinformatics 10: 253.

    Not in our library10.1186/1471-2105-10-253

26. 26.

    Shannon C (1948 ) A Mathematical Theory of Communication. The Bell System Technical Journal 27: 379–656, 379-423, 623-656.

    Not in our library

27. 27.

    van Son R (1994) A method to quantify the error distribution in confusion matrices. Technical Report IFA Proceedings 18, Institute of Phonetic Sciences, University of Amsterdam.

    Not in our library

28. 28.

    Abramson N (1963) Information theory and coding. McGraw-Hill, 201 pp.

    Not in our library

29. 29.

    Sindhwani V, Bhattacharge P, Rakshit S (2001) Information theoretic feature crediting in multiclass Support Vector Machines. In: Grossman R, Kumar V, editors, Proceedings First SIAM International Conference on Data Mining, ICDM01. SIAM, pp. 1–18.

    Not in our library

30. 30.

    Wei JM, Yuan XJ, Hu QH, Wang SQ (2010 ) A novel measure for evaluating classifiers. Expert Systems with Applications 37: 3799–3809.

    Not in our library

31. 31.

    Wei JM, Yuan XJ, Yang T, Wang SQ (2010 ) Evaluating Classifiers by Confusion Entropy. Information Processing & Management Submitted

    Not in our library

32. 32.

    Huang J, Ling C (2005 ) Using AUC and Accuracy in Evaluating Learning Algorithms. IEEE Transactions on Knowledge and Data Engineering 17: 299–310.

    Not in our library

33. 33.

    Sørlie T, Tibshirani R, Parker J, Hastie T, Marron JS, et al (2003 ) Repeated observation of breast tumor subtypes in independent gene expression data sets. Proceedings of the National Academy of Sciences 100: 8418–8423.

    Not in our library10.1073/pnas.0932692100

34. 34.

    Hastie T, Tibshirani R, Friedman JH (2003) The Elements of Statistical Learning. Springer.

    Not in our library

35. 35.

    Hand D (2010 ) Evaluating diagnostic tests: The area under the ROC curve and the balance of errors. Statistics in Medicine 29: 1502–1510.

    Not in our library10.1002/sim.3859

Cited by papers we hold 8

• The Matthews correlation coefficient (MCC) should replace the ROC AUC as the standard metric for assessing binary classification · 2023
  No notice
• Prediction of kinase inhibitors binding modes with machine learning and reduced descriptor sets · 2021
  Not checked
• The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation · 2021
  No notice
• Boosting methods for multi-class imbalanced data classification: an experimental review · 2020
  Not checked
• Machine learning can predict survival of patients with heart failure from serum creatinine and ejection fraction alone · 2020
  No notice
• The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation · 2020
  No notice
• Enhancing Confusion Entropy (CEN) for binary and multiclass classification. · 2019
  Not checked
• Probabilistic Confusion Entropy for Evaluating Classifiers · 2013
  Not checked

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.