Unmasking Clever Hans predictors and assessing what machines really learn

Abstract

Current learning machines have successfully solved hard application problems, reaching high accuracy and displaying seemingly intelligent behavior. Here we apply recent techniques for explaining decisions of state-of-the-art learning machines and analyze various tasks from computer vision and arcade games. This showcases a spectrum of problem-solving behaviors ranging from naive and short-sighted, to well-informed and strategic. We observe that standard performance evaluation metrics can be oblivious to distinguishing these diverse problem solving behaviors. Furthermore, we propose our semi-automated Spectral Relevance Analysis that provides a practically effective way of characterizing and validating the behavior of nonlinear learning machines. This helps to assess whether a learned model indeed delivers reliably for the problem that it was conceived for. Furthermore, our work intends to add a voice of caution to the ongoing excitement about machine intelligence and pledges to evaluate and judge some of these recent successes in a more nuanced manner.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 53

1. 1.
   Supervised group Lasso with applications to microarray data analysis · 2007

   Not checked
2. 2.
   Nonnegative Matrix Factorization: An Analytical and Interpretive Tool in Computational Biology · 2008

   No notice
3. 3.
   Comparing Statistical Methods for Constructing Large Scale Gene Networks · 2012

   No notice
4. 4.

   Haufe S, et al. On the interpretation of weight vectors of linear models in multivariate neuroimaging. Neuroimage. 2014;87:96-110.

   Not in our library10.1016/j.neuroimage.2013.10.067

5. 5.

   Mnih V, et al. Human-level control through deep reinforcement learning. Nature. 2015;518:529-533.

   Not in our library10.1038/nature14236

6. 6.

   Mnih, V. et al. Playing Atari with deep reinforcement learning. Preprint at https://arxiv.org/abs/1312.5602 (2013).

   Not in our library

7. 7.

   Silver D, et al. Mastering the game of Go with deep neural networks and tree search. Nature. 2016;529:484-489.

   Not in our library10.1038/nature16961

8. 8.

   Silver D, et al. Mastering the game of Go without human knowledge. Nature. 2017;550:354-359.

   Not in our library10.1038/nature24270

9. 9.

   Silver D, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science. 2018;362:1140-1144.

   Not in our library10.1126/science.aar6404

10. 10.

    Moravčk M, et al. DeepStack: expert-level artificial intelligence in heads-up no-limit poker. Science. 2017;356:508-513.

    Not in our library10.1126/science.aam6960

11. 11.

    Pfungst O. Clever Hans (the Horse of Mr. Von Osten): contribution to experimental animal and human psychology. J. Philos. Psychol. Sci. Method. 1911;8:663-666.

    Not in our library

12. 12.

    Guyon I, Elisseeff A. An introduction to variable and feature selection. J. Mach. Learn. Res. 2003;3:1157-1182.

    Not in our library

13. 13.

    Simonyan, K., Vedaldi, A. & Zisserman, A. Deep inside convolutional networks: Visualising image classification models and saliency maps. Preprint at https://arxiv.org/abs/1312.6034 (2013).

    Not in our library

14. 14.

    Yosinski, J., Clune, J., Nguyen, A., Fuchs, T. & Lipson, H. Understanding neural networks through deep visualization. Preprint at https://arxiv.org/abs/1506.06579 (2015).

    Not in our library

15. 15.

    Nguyen, A., Yosinski, J. & Clune, J. Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks. Preprint at https://arxiv.org/abs/1602.03616 (2016).

    Not in our library

16. 16.

    Nguyen, A., Dosovitskiy, A., Yosinski, J., Brox, T. & Clune, J. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. (D. D. Lee, M. Sugiyama, U. von Luxburg, I. Guyon, R. Garnett eds.) In Proc. Advances in Neural Information Processing Systems, 3387–3395 (Curran Associates, Inc., Red Hook, NY) (2016).

    Not in our library

17. 17.

    Zeiler, M. D. & Fergus, R. Visualizing and understanding convolutional networks. (D. Fleet, T. Pajdla, B. Schiele, T. Tuytelaars eds.) In Proc. European Conference on Computer Vision, 818–833 (Springer, Cham) (2014).

    Not in our library

18. 18.
    On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation · 2015

    No notice
19. 19.

    Baehrens D, et al. How to explain individual classification decisions. J. Mach. Learn. Res. 2010;11:1803-1831.

    Not in our library

20. 20.

    Ribeiro, M. T., Singh, S. & Guestrin, C. “Why should I trust you?”: explaining the predictions of any classifier. (B. Krishnapuram, M. Shah, A. J. Smola, C. C. Aggarwal, D. Shen, R. Rastogi eds.) In Proc. ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1135–1144 (ACM, New York, NY) (2016).

    Not in our library

21. 21.

    Zhou, B., Khosla, A., Lapedriza, À., Oliva, A. & Torralba, A. Learning deep features for discriminative localization. (T. Tuytelaars, F.-F. Li, R. Bajcsy eds.) In Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2921–2929 (IEEE, Piscataway, NJ) (2016).

    Not in our library

22. 22.

    Montavon G, Lapuschkin S, Binder A, Samek W, Müller K. Explaining nonlinear classification decisions with deep Taylor decomposition. Pattern Recognit. 2017;65:211-222.

    Not in our library10.1016/j.patcog.2016.11.008

23. 23.

    Sturm I, Lapuschkin S, Samek W, Müller KR. Interpretable deep neural networks for single-trial EEG classification. J. Neurosci. Methods. 2016;274:141-145.

    Not in our library10.1016/j.jneumeth.2016.10.008

24. 24.

    Greydanus, S., Koul, A., Dodge, J. & Fern, A. Visualizing and understanding Atari agents. (J. Dy, A. Krause eds.) In Proc. International Conference on Machine Learning, 1787–1796 (JMLR.org, Brookline, MA) (2018).

    Not in our library

25. 25.

    Zahavy, T., Ben-Zrihem, N. & Mannor, S. Graying the black box: Understanding DQNs. (M.-F. Balcan, K. Q. Weinberger eds.) In Proc. International Conference on Machine Learning, 1899–1908 (JMLR.org, Brookline, MA) (2016).

    Not in our library

26. 26.
    "What is relevant in a text document?": An interpretable machine learning approach. · 2017

    Not checked
27. 27.

    Turing AM. Mind. Mind. 1950;59:433-460.

    Not in our library10.1093/mind/lix.236.433

28. 28.

    Turing, A. M. Computing machinery and intelligence. (R. Epstein, G. Roberts, G. Beber eds.) In Parsing the Turing Test, 23–65 (Springer, 2009).

    Not in our library

29. 29.

    Legg S, Hutter M. Universal intelligence: a definition of machine intelligence. Mind Mach. 2007;17:391-444.

    Not in our library10.1007/s11023-007-9079-x

30. 30.

    Hernández-Orallo J. Evaluation in artificial intelligence: From task-oriented to ability-oriented measurement. Artif. Intell. Rev. 2017;48:397-447.

    Not in our library10.1007/s10462-016-9505-7

31. 31.

    Perronnin, F., Sánchez, J. & Mensink, T. Improving the Fisher kernel for large-scale image classification. (K. Daniilidis, P. Maragos, N. Paragios eds.) In Proc. European Conference on Computer Vision, 143–156 (Springer-Verlag, Berlin, Heidelberg) (2010).

    Not in our library

32. 32.

    Sánchez J, Perronnin F, Mensink T, Verbeek JJ. Image classification with the Fisher vector: theory and practice. Int. J. Comput. Vision. 2013;105:222-245.

    Not in our library10.1007/s11263-013-0636-x

33. 33.

    Everingham M, Van Gool L, Williams CK, Winn J, Zisserman A. The Pascal visual object classes (VOC) challenge. Int. J. Comput. Vision. 2010;88:303-338.

    Not in our library10.1007/s11263-009-0275-4

34. 34.

    Lapuschkin, S., Binder, A., Montavon, G., Müller, K.-R. & Samek, W. Analyzing classifiers: Fisher vectors and deep neural networks. (T. Tuytelaars, F.-F. Li, R. Bajcsy eds.) In Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2912–2920 (IEEE, Piscataway, NJ) (2016).

    Not in our library

35. 35.

    Chen, Y. F., Everett, M., Liu, M. & How, J. P. Socially aware motion planning with deep reinforcement learning. (H. Zhang, R. Vaughan eds.) In Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems, 1343–1350 (IEEE, Piscataway, NJ) (2017).

    Not in our library

36. 36.

    Rajalingham R, et al. Large-scale, high-resolution comparison of the core visual object recognition behavior of humans, monkeys, and state-of-the-art deep artificial neural networks. J. Neurosci. 2018;38:7255-7269.

    Not in our library10.1523/jneurosci.0388-18.2018

37. 37.

    Lake BM, Ullman TD, Tenenbaum JB, Gershman SJ. Building machines that learn and think like people. Behav. Brain Sci. 2017;40:e253.

    Not in our library10.1017/s0140525x16001837

38. 38.

    Tsividis, P. A., Pouncy, T., Xu, J. L., Tenenbaum, J. B. & Gershman, S. J. Human learning in Atari. (G. Sukthankar, C. Geib eds.) In Proc. AAAI Spring Symposium Series, 643–646 (AAAI Press, Palo Alto, CA) (2017).

    Not in our library

39. 39.

    Winston PH, Horn B. The Psychology of Computer Vision. 1975.

    Not in our library

40. 40.

    Smith LB, Jones SS, Landau B, Gershkoff-Stowe L, Samuelson L. Object name learning provides on-the-job training for attention. Psychol. Sci. 2002;13:13-19.

    Not in our library10.1111/1467-9280.00403

41. 41.

    Lake BM, Salakhutdinov R, Tenenbaum JB. Human-level concept learning through probabilistic program induction. Science. 2015;350:1332-1338.

    Not in our library10.1126/science.aab3050

42. 42.

    Anselmi F, et al. Unsupervised learning of invariant representations. Theor. Comput. Sci. 2016;633:112-121.

    Not in our library10.1016/j.tcs.2015.06.048

43. 43.

    Chmiela S, et al. Machine learning of accurate energy-conserving molecular force fields. Sci. Adv. 2017;3:e1603015.

    Not in our library10.1126/sciadv.1603015

44. 44.
    Towards exact molecular dynamics simulations with machine-learned force fields · 2018

    No notice
45. 45.

    The European Parliament and the Council of the European Union.. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation). Off. J. Eur. Union. 2016;L119:1-88.

    Not in our library

46. 46.

    Goodman B, Flaxman SR. European union regulations on algorithmic decision-making and a “right to explanation”. AI Mag. 2017;38:50-57.

    Not in our library10.1609/aimag.v38i3.2741

47. 47.
    Quantum-chemical insights from deep tensor neural networks · 2017

    No notice
48. 48.
    Explaining the unique nature of individual gait patterns with deep learning · 2019

    No notice
49. 49.

    Yang, Y., Tresp, V., Wunderle, M. & Fasching, P. A. Explaining therapy predictions with layer-wise relevance propagation in neural networks. (Z. Lu, C. Yang eds.) In Proc. IEEE International Conference on Healthcare Informatics, 152–162 (IEEE, Piscataway, NJ) (2018).

    Not in our library

50. 50.

    Thomas, A. W., Heekeren, H. R., Müller, K.-R. & Samek, W. Interpretable LSTMs for whole-brain neuroimaging analyses. Preprint at https://arxiv.org/abs/1810.09945 (2018).

    Not in our library

51. 51.

    Gevrey M, Dimopoulos I, Lek S. Review and comparison of methods to study the contribution of variables in artificial neural network models. Ecol. Model. 2003;160:249-264.

    Not in our library10.1016/s0304-3800(02)00257-0

52. 52.

    Montavon G, Samek W, Müller KR. Methods for interpreting and understanding deep neural networks. Digit. Signal Process. 2018;73:1-15.

    Not in our library10.1016/j.dsp.2017.10.011

53. 53.

    Von Luxburg U. A tutorial on spectral clustering. Stat. Comput. 2007;17:395-416.

    Not in our library10.1007/s11222-007-9033-z

Cited by papers we hold 6

• Joint EANM/SNMMI guideline on radiomics in nuclear medicine · 2022
  No notice
• Deep learning in drug discovery: an integrative review and future challenges · 2022
  No notice
• Decoding subjective emotional arousal from EEG during an immersive virtual reality experience · 2021
  Not checked
• SpookyNet: Learning force fields with electronic degrees of freedom and nonlocal effects · 2021
  No notice
• Explainability for artificial intelligence in healthcare: a multidisciplinary perspective · 2020
  No notice
• Unifying machine learning and quantum chemistry with a deep neural network for molecular wavefunctions · 2019
  No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.