Keep it real: rethinking the primacy of experimental control in cognitive neuroscience

Abstract

Naturalistic experimental paradigms in neuroimaging arose from a pressure to test the validity of models we derive from highly-controlled experiments in real-world contexts. In many cases, however, such efforts led to the realization that models developed under particular experimental manipulations failed to capture much variance outside the context of that manipulation. The critique of non-naturalistic experiments is not a recent development; it echoes a persistent and subversive thread in the history of modern psychology. The brain has evolved to guide behavior in a multidimensional world with many interacting variables. The assumption that artificially decoupling and manipulating these variables will lead to a satisfactory understanding of the brain may be untenable. We develop an argument for the primacy of naturalistic paradigms, and point to recent developments in machine learning as an example of the transformative power of relinquishing control. Naturalistic paradigms should not be deployed as an afterthought if we hope to build models of brain and behavior that extend beyond the laboratory into the real world.

References 133

1. 1.

   Araújo D, Davids K, Passos P, 2007 Ecological validity, representative design, and correspondence between experimental task constraints and behavioral setting: comment on Rogers, Kadar, and Costall (2005). Ecol. Psychol. 19, 69–78. doi:.

   Not in our library10.1080/10407410709336951

2. 2.

   Attneave F, 1954 Some informational aspects of visual perception. Psychol. Rev. 61, 183–193. doi:.

   Not in our library10.1037/h0054663

3. 3.

   Baayen RH, Davidson DJ, Bates DM, 2008 Mixed-effects modeling with crossed random effects for subjects and items. J. Mem. Lang. 59, 390–412. doi:.

   Not in our library10.1016/j.jml.2007.12.005

4. 4.

   Baldassano C, Chen J, Zadbood A, Pillow JW, Hasson U, Norman KA, 2017 Discovering event structure in continuous narrative perception and memory. Neuron 95, 709–721. doi:.

   Not in our library10.1016/j.neuron.2017.06.041

5. 5.

   Bannister D, 1966 Psychology as an exercise in paradox. Bull. Br. Psychol. Soc. 19, 21–26.

   Not in our library

6. 6.

   Barker RG, 1965 Explorations in ecological psychology. Am. Psychol. 20, 1–14. doi:.

   Not in our library10.1037/h0021697

7. 7.

   Barlow HB, 1961 Possible principles underlying the transformation of sensory messages In: Rosenblith WA (Ed.), Sensory Communication. MIT Press, Cambridge, MA, pp. 217–234. doi:.

   Not in our library10.7551/mitpress/9780262518420.003.0013

8. 8.

   Bartels A, Zeki S, 2004 Functional brain mapping during free viewing of natural scenes. Hum. Brain Mapp. 21, 75–85. doi:.

   Not in our library10.1002/hbm.10153

9. 9.

   Bedny M, Aguirre GK, Thompson-Schill SL, 2007 Item analysis in functional magnetic resonance imaging. Neuroimage 35, 1093–1102. doi:.

   Not in our library10.1016/j.neuroimage.2007.01.039

10. 10.

    Boto E, Holmes N, Leggett J, Roberts G, Shah V, Meyer SS, Duque Muñoz L, Mullinger KJ, Tierney TM, Bestmann S, Barnes GR, Bowtell R, Brookes MJ, 2018 Moving magnetoencephalography towards real-world applications with a wearable system. Nature 555, 657–661. doi:.

    Not in our library10.1038/nature26147

11. 11.

    Braitenberg V, 1984 Vehicles: Experiments in Synthetic Psychology. MIT Press, Cambridge, MA http://www.worldcat.org/oclc/476621844.

    Not in our library

12. 12.

    Breiman L, 2001 Statistical modeling: the two cultures (with comments and a rejoinder by the author). Stat. Sci. 16, 199–231. doi:.

    Not in our library10.1214/ss/1009213726

13. 13.

    Bronfenbrenner U, 1977 Toward an experimental ecology of human development. Am. Psychol. 32, 513–531. doi:.

    Not in our library10.1037/0003-066x.32.7.513

14. 14.

    Brooks RA, 1991 Intelligence without representation. Artif. Intell. 47, 139–159. doi:.

    Not in our library10.1016/0004-3702(91)90053-m

15. 15.

    Brunswik E, 1943 Organismic achievement and environmental probability. Psychol. Rev. 50, 255–272. doi:.

    Not in our library10.1037/h0060889

16. 16.

    Brunswik E, 1947 Perception and the Representative Design of Psychological Experiments. University of California Press, Berkeley, CA http://www.worldcat.org/oclc/9551141.

    Not in our library

17. 17.

    Brunswik E, 1949 Systematic and representative design of psychological experiments with results in physical and social perception In: Neyman J (Ed.), Proceedings of the Berkeley Symposium on Mathematical Statistics and Probability. University of California Press, Berkeley, CA, pp. 143–202.

    Not in our library

18. 18.

    Brunswik E, 1955 Representative design and probabilistic theory in a functional psychology. Psychol. Rev. 62, 193–217. doi:.

    Not in our library10.1037/h0047470

19. 19.

    Calhoun AJ, Murthy M, 2017 Quantifying behavior to solve sensorimotor transformations: advances from worms and flies. Curr. Opin. Neurobiol. 46, 90–98. doi:.

    Not in our library10.1016/j.conb.2017.08.006

20. 20.

    Campbell DT, 1973 The social scientist as methodological servant of the experimenting society. Policy Stud. J. 72–75. doi:.

    Not in our library10.1111/j.1541-0072.1973.tb00128.x

21. 21.

    Carrasco M, 2011 Visual attention: the past 25 years. Vis. Res. 51, 1484–1525. doi:.

    Not in our library10.1016/j.visres.2011.04.012

22. 22.
    Endogenous variation in ventromedial prefrontal cortex state dynamics during naturalistic viewing reflects affective experience · 2021

    No notice
23. 23.

    Chen J, Leong YC, Honey CJ, Yong CH, Norman KA, Hasson U, 2017 Shared memories reveal shared structure in neural activity across individuals. Nat. Neurosci. 20, 115–125. doi:.

    Not in our library10.1038/nn.4450

24. 24.

    Chiel HJ, Beer RD, 1997 The brain has a body: adaptive behavior emerges from interactions of nervous system, body and environment. Trends Neurosci. 20, 553–557. doi:.

    Not in our library10.1016/s0166-2236(97)01149-1

25. 25.
    Resynthesizing behavior through phylogenetic refinement · 2019

    No notice
26. 26.

    Clark HH, 1973 The language-as-fixed-effect fallacy: a critique of language statistics in psychological research. J. Verbal Learn. Verbal Behav. 12, 335–359. doi:.

    Not in our library10.1016/s0022-5371(73)80014-3

27. 27.

    Coleman EB, 1964 Generalizing to a language population. Psychol. Rep. 14, 219–226. doi:.

    Not in our library10.2466/pr0.1964.14.1.219

28. 28.

    Cronbach LJ, 1975 Beyond the two disciplines of scientific psychology. Am. Psychol. 30, 116–127. doi:.

    Not in our library10.1037/h0076829

29. 29.

    Cronbach LJ, Rajaratnam N, Gleser GC, 1963 Theory of generalizability: a liberalization of reliability theory. Br. J. Math. Stat. Psychol. 16, 137–163. doi:.

    Not in our library10.1111/j.2044-8317.1963.tb00206.x

30. 30.

    Dang J, King KM, Inzlicht M, 2020 Why are self-report and behavioral measures weakly correlated? Trends Cogn. Sci. 24, 267–269. doi:.

    Not in our library10.1016/j.tics.2020.01.007

31. 31.

    David SV, Vinje WE, Gallant JL, 2004 Natural stimulus statistics alter the receptive field structure of V1 neurons. J. Neurosci. 24, 6991–7006. doi:.

    Not in our library10.1523/jneurosci.1422-04.2004

32. 32.

    de Heer WA, Huth AG, Griffiths TL, Gallant JL, Theunissen FE, 2017 The hierarchical cortical organization of human speech processing. J. Neurosci. 37, 6539–6557. doi:.

    Not in our library10.1523/jneurosci.3267-16.2017

33. 33.

    Deniz F, Nunez-Elizalde AO, Huth AG, Gallant JL, 2019 The representation of semantic information across human cerebral cortex during listening versus reading is invariant to stimulus modality. J. Neurosci. 39, 7722–7736. doi:.

    Not in our library10.1016/j.neuron.2011.08.026

34. 34.

    Dennett DC, 1995 Darwin’s Dangerous Idea: Evolution and the Meanings of Life. Simon and Schuster, New York, NY http://www.worldcat.org/oclc/892927037.

    Not in our library

35. 35.
    Nature abhors a paywall: How open science can realize the potential of naturalistic stimuli · 2020

    Not checked
36. 36.
    Uncovering the structure of self-regulation through data-driven ontology discovery · 2019

    No notice
37. 37.

    Elliott ML, Knodt AR, Ireland D, Morris ML, Poulton R, Ramrakha S, Sison ML, Moffitt TE, Caspi A, Hariri AR, 2020 What is the test-retest reliability of common task-functional MRI measures? New empirical evidence and a meta-analysis. Psychol. Sci. doi:.

    Not in our library10.1177/0956797620916786

38. 38.

    Elwert F, Winship C, 2014 Endogenous selection bias: the problem of conditioning on a collider variable. Annu. Rev. Sociol. 40, 31–53. doi:.

    Not in our library10.1146/annurev-soc-071913-043455

39. 39.

    Felsen G, Dan Y, 2005 A natural approach to studying vision. Nat. Neurosci. 8, 1643–1646. doi:.

    Not in our library10.1038/nn1608

40. 40.

    Fiedler K, 2011 Voodoo correlations are everywhere—Not only in neuroscience. Perspect. Psychol. Sci. 6, 163–171. doi:.

    Not in our library10.1177/1745691611400237

Cited by papers we hold 4

• In-degree centrality in a social network is linked to coordinated neural activity · 2022
  Not checked
• Challenging Paradigms Through Ecological Neuroscience: Lessons From Visual Models · 2021
  Not checked
• Naturalistic Stimuli in Affective Neuroimaging: A Review · 2021
  No notice
• The “Narratives” fMRI dataset for evaluating models of naturalistic language comprehension · 2021
  No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.