Unmasking Clever Hans predictors and assessing what machines really learn

Abstract

Current learning machines have successfully solved hard application problems, reaching high accuracy and displaying seemingly intelligent behavior. Here we apply recent techniques for explaining decisions of state-of-the-art learning machines and analyze various tasks from computer vision and arcade games. This showcases a spectrum of problem-solving behaviors ranging from naive and short-sighted, to well-informed and strategic. We observe that standard performance evaluation metrics can be oblivious to distinguishing these diverse problem solving behaviors. Furthermore, we propose our semi-automated Spectral Relevance Analysis that provides a practically effective way of characterizing and validating the behavior of nonlinear learning machines. This helps to assess whether a learned model indeed delivers reliably for the problem that it was conceived for. Furthermore, our work intends to add a voice of caution to the ongoing excitement about machine intelligence and pledges to evaluate and judge some of these recent successes in a more nuanced manner.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 53

1. 1.
   Supervised group Lasso with applications to microarray data analysis · 2007

   Not checked
2. 2.
   Nonnegative Matrix Factorization: An Analytical and Interpretive Tool in Computational Biology · 2008

   No notice
3. 3.
   Comparing Statistical Methods for Constructing Large Scale Gene Networks · 2012

   No notice
4. 4.

   Haufe S, et al. On the interpretation of weight vectors of linear models in multivariate neuroimaging. Neuroimage. 2014;87:96-110.

   Not in our library10.1016/j.neuroimage.2013.10.067

5. 5.

   Mnih V, et al. Human-level control through deep reinforcement learning. Nature. 2015;518:529-533.

   Not in our library10.1038/nature14236

6. 6.

   Mnih, V. et al. Playing Atari with deep reinforcement learning. Preprint at https://arxiv.org/abs/1312.5602 (2013).

   Not in our library

7. 7.

   Silver D, et al. Mastering the game of Go with deep neural networks and tree search. Nature. 2016;529:484-489.

   Not in our library10.1038/nature16961

8. 8.

   Silver D, et al. Mastering the game of Go without human knowledge. Nature. 2017;550:354-359.

   Not in our library10.1038/nature24270

9. 9.

   Silver D, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science. 2018;362:1140-1144.

   Not in our library10.1126/science.aar6404

10. 10.

    Moravčk M, et al. DeepStack: expert-level artificial intelligence in heads-up no-limit poker. Science. 2017;356:508-513.

    Not in our library10.1126/science.aam6960

11. 11.

    Pfungst O. Clever Hans (the Horse of Mr. Von Osten): contribution to experimental animal and human psychology. J. Philos. Psychol. Sci. Method. 1911;8:663-666.

    Not in our library

12. 12.

    Guyon I, Elisseeff A. An introduction to variable and feature selection. J. Mach. Learn. Res. 2003;3:1157-1182.

    Not in our library

13. 13.

    Simonyan, K., Vedaldi, A. & Zisserman, A. Deep inside convolutional networks: Visualising image classification models and saliency maps. Preprint at https://arxiv.org/abs/1312.6034 (2013).

    Not in our library

14. 14.

    Yosinski, J., Clune, J., Nguyen, A., Fuchs, T. & Lipson, H. Understanding neural networks through deep visualization. Preprint at https://arxiv.org/abs/1506.06579 (2015).

    Not in our library

15. 15.

    Nguyen, A., Yosinski, J. & Clune, J. Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks. Preprint at https://arxiv.org/abs/1602.03616 (2016).

    Not in our library

16. 16.

    Nguyen, A., Dosovitskiy, A., Yosinski, J., Brox, T. & Clune, J. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. (D. D. Lee, M. Sugiyama, U. von Luxburg, I. Guyon, R. Garnett eds.) In Proc. Advances in Neural Information Processing Systems, 3387–3395 (Curran Associates, Inc., Red Hook, NY) (2016).

    Not in our library

17. 17.

    Zeiler, M. D. & Fergus, R. Visualizing and understanding convolutional networks. (D. Fleet, T. Pajdla, B. Schiele, T. Tuytelaars eds.) In Proc. European Conference on Computer Vision, 818–833 (Springer, Cham) (2014).

    Not in our library

18. 18.
    On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation · 2015

    No notice
19. 19.

    Baehrens D, et al. How to explain individual classification decisions. J. Mach. Learn. Res. 2010;11:1803-1831.

    Not in our library

20. 20.

    Ribeiro, M. T., Singh, S. & Guestrin, C. “Why should I trust you?”: explaining the predictions of any classifier. (B. Krishnapuram, M. Shah, A. J. Smola, C. C. Aggarwal, D. Shen, R. Rastogi eds.) In Proc. ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1135–1144 (ACM, New York, NY) (2016).

    Not in our library

21. 21.

    Zhou, B., Khosla, A., Lapedriza, À., Oliva, A. & Torralba, A. Learning deep features for discriminative localization. (T. Tuytelaars, F.-F. Li, R. Bajcsy eds.) In Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2921–2929 (IEEE, Piscataway, NJ) (2016).

    Not in our library

22. 22.

    Montavon G, Lapuschkin S, Binder A, Samek W, Müller K. Explaining nonlinear classification decisions with deep Taylor decomposition. Pattern Recognit. 2017;65:211-222.

    Not in our library10.1016/j.patcog.2016.11.008

23. 23.

    Sturm I, Lapuschkin S, Samek W, Müller KR. Interpretable deep neural networks for single-trial EEG classification. J. Neurosci. Methods. 2016;274:141-145.

    Not in our library10.1016/j.jneumeth.2016.10.008

24. 24.

    Greydanus, S., Koul, A., Dodge, J. & Fern, A. Visualizing and understanding Atari agents. (J. Dy, A. Krause eds.) In Proc. International Conference on Machine Learning, 1787–1796 (JMLR.org, Brookline, MA) (2018).

    Not in our library

25. 25.

    Zahavy, T., Ben-Zrihem, N. & Mannor, S. Graying the black box: Understanding DQNs. (M.-F. Balcan, K. Q. Weinberger eds.) In Proc. International Conference on Machine Learning, 1899–1908 (JMLR.org, Brookline, MA) (2016).

    Not in our library

26. 26.
    "What is relevant in a text document?": An interpretable machine learning approach. · 2017

    Not checked
27. 27.

    Turing AM. Mind. Mind. 1950;59:433-460.

    Not in our library10.1093/mind/lix.236.433

28. 28.

    Turing, A. M. Computing machinery and intelligence. (R. Epstein, G. Roberts, G. Beber eds.) In Parsing the Turing Test, 23–65 (Springer, 2009).

    Not in our library

29. 29.

    Legg S, Hutter M. Universal intelligence: a definition of machine intelligence. Mind Mach. 2007;17:391-444.

    Not in our library10.1007/s11023-007-9079-x

30. 30.

    Hernández-Orallo J. Evaluation in artificial intelligence: From task-oriented to ability-oriented measurement. Artif. Intell. Rev. 2017;48:397-447.

    Not in our library10.1007/s10462-016-9505-7

31. 31.

    Perronnin, F., Sánchez, J. & Mensink, T. Improving the Fisher kernel for large-scale image classification. (K. Daniilidis, P. Maragos, N. Paragios eds.) In Proc. European Conference on Computer Vision, 143–156 (Springer-Verlag, Berlin, Heidelberg) (2010).

    Not in our library

32. 32.

    Sánchez J, Perronnin F, Mensink T, Verbeek JJ. Image classification with the Fisher vector: theory and practice. Int. J. Comput. Vision. 2013;105:222-245.

    Not in our library10.1007/s11263-013-0636-x

33. 33.

    Everingham M, Van Gool L, Williams CK, Winn J, Zisserman A. The Pascal visual object classes (VOC) challenge. Int. J. Comput. Vision. 2010;88:303-338.

    Not in our library10.1007/s11263-009-0275-4

34. 34.

    Lapuschkin, S., Binder, A., Montavon, G., Müller, K.-R. & Samek, W. Analyzing classifiers: Fisher vectors and deep neural networks. (T. Tuytelaars, F.-F. Li, R. Bajcsy eds.) In Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2912–2920 (IEEE, Piscataway, NJ) (2016).

    Not in our library

35. 35.

    Chen, Y. F., Everett, M., Liu, M. & How, J. P. Socially aware motion planning with deep reinforcement learning. (H. Zhang, R. Vaughan eds.) In Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems, 1343–1350 (IEEE, Piscataway, NJ) (2017).

    Not in our library

36. 36.

    Rajalingham R, et al. Large-scale, high-resolution comparison of the core visual object recognition behavior of humans, monkeys, and state-of-the-art deep artificial neural networks. J. Neurosci. 2018;38:7255-7269.

    Not in our library10.1523/jneurosci.0388-18.2018

37. 37.

    Lake BM, Ullman TD, Tenenbaum JB, Gershman SJ. Building machines that learn and think like people. Behav. Brain Sci. 2017;40:e253.

    Not in our library10.1017/s0140525x16001837

38. 38.

    Tsividis, P. A., Pouncy, T., Xu, J. L., Tenenbaum, J. B. & Gershman, S. J. Human learning in Atari. (G. Sukthankar, C. Geib eds.) In Proc. AAAI Spring Symposium Series, 643–646 (AAAI Press, Palo Alto, CA) (2017).

    Not in our library

39. 39.

    Winston PH, Horn B. The Psychology of Computer Vision. 1975.

    Not in our library

40. 40.

    Smith LB, Jones SS, Landau B, Gershkoff-Stowe L, Samuelson L. Object name learning provides on-the-job training for attention. Psychol. Sci. 2002;13:13-19.

    Not in our library10.1111/1467-9280.00403

Cited by papers we hold 6

• Joint EANM/SNMMI guideline on radiomics in nuclear medicine · 2022
  No notice
• Deep learning in drug discovery: an integrative review and future challenges · 2022
  No notice
• Decoding subjective emotional arousal from EEG during an immersive virtual reality experience · 2021
  Not checked
• SpookyNet: Learning force fields with electronic degrees of freedom and nonlocal effects · 2021
  No notice
• Explainability for artificial intelligence in healthcare: a multidisciplinary perspective · 2020
  No notice
• Unifying machine learning and quantum chemistry with a deep neural network for molecular wavefunctions · 2019
  No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.