The Hippocampal Film Editor: Sensitivity and Specificity to Event Boundaries in Continuous Experience

Abstract

The function of the human hippocampus is normally investigated by experimental manipulation of discrete events. Less is known about what triggers hippocampal activity during more naturalistic, continuous experience. We hypothesized that the hippocampus would be sensitive to the occurrence of event boundaries, that is, moments in time identified by observers as a transition between events. To address this, we analyzed functional MRI data from two groups: one ( n = 253, 131 female) who viewed an 8.5 min film and another ( n = 15, 6 female) who viewed a 120 min film. We observed a strong hippocampal response at boundaries defined by independent observers, which was modulated by boundary salience (the number of observers that identified each boundary). In the longer film, there were sufficient boundaries to show that this modulation remained after covarying out a large number of perceptual factors. This hypothesis-driven approach was complemented by a data-driven approach, in which we identified hippocampal events as moments in time with the strongest hippocampal activity. The correspondence between these hippocampal events and event boundaries was highly significant, revealing that the hippocampal response is not only sensitive, but also specific to event boundaries. We conclude that event boundaries play a key role in shaping hippocampal activity during encoding of naturalistic events. SIGNIFICANCE STATEMENT Recent years have seen the field of human neuroscience research transitioning from experiments with simple stimuli to the study of more complex and naturalistic experience. Nonetheless, our understanding of the function of many brain regions, such as the hippocampus, is based primarily on the study of brief, discrete events. As a result, we know little of what triggers hippocampal activity in real-life settings when we are exposed to a continuous stream of information. When does the hippocampus “decide” to respond during the encoding of naturalistic experience? We reveal here that hippocampal activity measured by fMRI during film watching is both sensitive and specific to event boundaries, identifying a potential mechanism whereby event boundaries shape experience by modulation of hippocampal activity.

Shown for reading. Its publisher allows quoting it, not republishing it.

References 81

1. 1.

   Allen TA, Salz DM, McKenzie S, Fortin NJ (2016 ) Nonspatial sequence coding in CA1 neurons. J Neurosci 36:1547–1563.

   Not in our library10.1523/jneurosci.2874-15.2016

2. 2.

   Axmacher N, Cohen MX, Fell J, Haupt S, Dümpelmann M, Elger CE, Schlaepfer TE, Lenartz D, Sturm V, Ranganath C (2010 ) Intracranial EEG correlates of expectancy and memory formation in the human hippocampus and nucleus accumbens. Neuron 65:541–549.

   Not in our library10.1016/j.neuron.2010.02.006

3. 3.

   Baayen RH, Davidson DJ, Bates DM (2008 ) Mixed-effects modeling with crossed random effects for subjects and items. J Mem Lang 59:390–412.

   Not in our library10.1016/j.jml.2007.12.005

4. 4.

   Baker LJ, Levin DT (2015 ) The role of relational triggers in event perception. Cognition 136:14–29.

   Not in our library10.1016/j.cognition.2014.11.030

5. 5.

   Baldassano C, Chen J, Zadbood A, Pillow JW, Hasson U, Norman KA (2017 ) Discovering event structure in continuous narrative perception and memory. Neuron 95:709–721.e5.

   Not in our library10.1016/j.neuron.2017.06.041

6. 6.

   Ben-Yakov A, Dudai Y (2011 ) Constructing realistic engrams: poststimulus activity of hippocampus and dorsal striatum predicts subsequent episodic memory. J Neurosci 31:9032–9042.

   Not in our library10.1523/jneurosci.0702-11.2011

7. 7.

   Ben-Yakov A, Eshel N, Dudai Y (2013 ) Hippocampal immediate poststimulus activity in the encoding of consecutive naturalistic episodes. J Exp Psychol Gen 142:1255–1263.

   Not in our library10.1037/a0033558

8. 8.

   Ben-Yakov A, Rubinson M, Dudai Y (2014 ) Shifting gears in hippocampus: temporal dissociation between familiarity and novelty signatures in a single event. J Neurosci 34:12973–12981.

   Not in our library10.1523/jneurosci.1892-14.2014

9. 9.

   Bird CM, Capponi C, King JA, Doeller CF, Burgess N (2010 ) Establishing the boundaries: the hippocampal contribution to imagining scenes. J Neurosci 30:11688–11695.

   Not in our library10.1523/jneurosci.0723-10.2010

10. 10.

    Chen J, Olsen RK, Preston AR, Glover GH, Wagner AD (2011 ) Associative retrieval processes in the human medial temporal lobe: hippocampal retrieval success and CA1 mismatch detection. Learn Mem 18:523–528.

    Not in our library10.1101/lm.2135211

11. 11.

    Clark HH (1973 ) The language-as-fixed-effect fallacy: a critique of language statistics in psychological research. J Verbal Learning Verbal Behav 12:335–359.

    Not in our library10.1016/s0022-5371(73)80014-3

12. 12.

    Cusack R, Vicente-Grabovetsky A, Mitchell DJ, Wild CJ, Auer T, Linke AC, Peelle JE (2014 ) Automatic analysis (aa): efficient neuroimaging workflows and parallel processing using matlab and XML. Front Neuroinform 8:90.

    Not in our library10.3389/fninf.2014.00090

13. 13.

    Desikan RS, Ségonne F, Fischl B, Quinn BT, Dickerson BC, Blacker D, Buckner RL, Dale AM, Maguire RP, Hyman BT, Albert MS, Killiany RJ (2006 ) An automated labeling system for subdividing the human cerebral cortex on MRI scans into gyral based regions of interest. Neuroimage 31:968–980.

    Not in our library10.1016/j.neuroimage.2006.01.021

14. 14.

    Doeller CF, King JA, Burgess N (2008 ) Parallel striatal and hippocampal systems for landmarks and boundaries in spatial memory. Proc Natl Acad Sci U S A 105:5915–5920.

    Not in our library10.1073/pnas.0801489105

15. 15.

    DuBrow S, Davachi L (2013 ) The influence of context boundaries on memory for the sequential order of events. J Exp Psychol Gen 142:1277–1286.

    Not in our library10.1037/a0034024

16. 16.

    DuBrow S, Davachi L (2014 ) Temporal memory is shaped by encoding stability and intervening item reactivation. J Neurosci 34:13998–14005.

    Not in our library10.1523/jneurosci.2535-14.2014

17. 17.

    DuBrow S, Davachi L (2016 ) Temporal binding within and across events. Neurobiol Learn Mem 134:107–114.

    Not in our library10.1016/j.nlm.2016.07.011

18. 18.

    Dubrow S, Rouhani N, Niv Y, Norman KA (2017 ) Does mental context drift or shift? Curr Opin Behav Sci 17:141–146.

    Not in our library10.1016/j.cobeha.2017.08.003

19. 19.

    Eickhoff SB, Stephan KE, Mohlberg H, Grefkes C, Fink GR, Amunts K, Zilles K (2005 ) A new SPM toolbox for combining probabilistic cytoarchitectonic maps and functional imaging data. Neuroimage 25:1325–1335.

    Not in our library10.1016/j.neuroimage.2004.12.034

20. 20.

    Ezzyat Y, Davachi L (2011 ) What constitutes an episode in episodic memory? Psychol Sci 22:243–252.

    Not in our library10.1177/0956797610393742

21. 21.

    Güçlü U, van Gerven MA (2015 ) Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. J Neurosci 35:10005–10014.

    Not in our library10.1523/jneurosci.5023-14.2015

22. 22.

    Gupta AS, van der Meer MA, Touretzky DS, Redish AD (2012 ) Segmentation of spatial experience by hippocampal theta sequences. Nat Neurosci 15:1032–1039.

    Not in our library10.1038/nn.3138

23. 23.
    A high-resolution 7-Tesla fMRI dataset from complex natural stimulation with an audio movie · 2014

    No notice
24. 24.

    Hanke M, Adelhöfer N, Kottke D, Iacovella V, Sengupta A, Kaule FR, Nigbur R, Waite AQ, Baumgartner F, Stadler J (2016 ) A studyforrest extension, simultaneous fMRI and eye gaze recordings during prolonged natural stimulation. Sci Data 3:160092.

    Not in our library10.1038/sdata.2016.92

25. 25.

    Hasson U, Landesman O, Knappmeyer B, Vallines I, Rubin N, Heeger DJ (2008 ) Neurocinematics: The neuroscience of film. Projections 2:1–26.

    Not in our library

26. 26.

    Hasson U, Malach R, Heeger DJ (2010 ) Reliability of cortical activity during natural stimulation. Trends Cogn Sci 14:40–48.

    Not in our library10.1016/j.tics.2009.10.011

27. 27.
    An annotation of cuts, depicted locations, and temporal progression in the motion picture "Forrest Gump" [version 1; referees: 2 approved] · 2016

    Not checked
28. 28.

    Heusser AC, Ezzyat Y, Shiff I, Davachi L (2018 ) Perceptual boundaries cause mnemonic trade-offs between local boundary processing and across-trial associative binding. J Exp Psychol Learn Mem Cogn 44:1075–1090.

    Not in our library10.1037/xlm0000503

29. 29.

    Hsieh LT, Gruber MJ, Jenkins LJ, Ranganath C (2014 ) Hippocampal activity patterns carry information about objects in temporal context. Neuron 81:1165–1178.

    Not in our library10.1016/j.neuron.2014.01.015

30. 30.

    Huff M, Meitz TG, Papenmeier F (2014 ) Changes in situation models modulate processes of event perception in audiovisual narratives. J Exp Psychol Learn Mem Cogn 40:1377–1388.

    Not in our library10.1037/a0036780

31. 31.

    Kim H (2011 ) Neural activity that predicts subsequent memory and forgetting: a meta-analysis of 74 fMRI studies. Neuroimage 54:2446–2461.

    Not in our library10.1016/j.neuroimage.2010.09.045

32. 32.

    Köhler S, Danckert S, Gati JS, Menon RS (2005 ) Novelty responses to relational and non-relational information in the hippocampus and the parahippocampal region: a comparison based on event-related fMRI. Hippocampus 15:763–774.

    Not in our library10.1002/hipo.20098

33. 33.

    Krizhevsky A, Sutskever I, Hinton GE (2012 ) ImageNet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 25:1097–1105.

    Not in our library

34. 34.

    Kumaran D, Maguire EA (2006 ) An unexpected sequence of events: mismatch detection in the human hippocampus. PLoS Biol 4:e424.

    Not in our library10.1371/journal.pbio.0040424

35. 35.

    Kumaran D, Maguire EA (2009 ) Novelty signals: a window into hippocampal information processing. Trends Cogn Sci 13:47–54.

    Not in our library10.1016/j.tics.2008.11.004

36. 36.

    Kurby CA, Zacks JM (2008 ) Segmentation in the perception and memory of events. Trends Cogn Sci 12:72–79.

    Not in our library10.1016/j.tics.2007.11.004

37. 37.

    Kuznetsova A, Brockhoff PB, Christensen RHB (2017 ) lmerTest package: tests in linear mixed effects models. J Stat Softw 82:1–26.

    Not in our library

38. 38.

    Lee SA, Miller JF, Watrous AJ, Sperling MR, Sharan A, Worrell GA, Berry BM, Aronson JP, Davis KA, Gross RE, Lega B, Sheth S, Das SR, Stein JM, Gorniak R, Rizzuto DS, Jacobs J (2018 ) Electrophysiological signatures of spatial boundaries in the human subiculum. J Neurosci 38:3265–3272.

    Not in our library10.1523/jneurosci.3216-17.2018

39. 39.

    Luke SG (2017 ) Evaluating significance in linear mixed-effects models in R. Behav Res Methods 49:1494–1502.

    Not in our library10.3758/s13428-016-0809-y

40. 40.

    Magliano JP, Zacks JM (2011 ) The impact of continuity editing in narrative film on event segmentation. Cogn Sci 35:1489–1517.

    Not in our library10.1111/j.1551-6709.2011.01202.x

41. 41.

    Magliano JP, Miller J, Zwaan RA (2001 ) Indexing space and time in film understanding. Appl Cogn Psychol 15:533–545.

    Not in our library10.1002/acp.724

42. 42.

    McKenzie S, Buzsáki G (2016 ) Hippocampal mechanisms for the segmentation of space by goals and boundaries. New York: Springer.

    Not in our library

43. 43.

    Milivojevic B, Varadinov M, Vicente Grabovetsky A, Collin SH, Doeller CF (2016 ) Coding of event nodes and narrative context in the hippocampus. J Neurosci 36:12412–12424.

    Not in our library10.1523/jneurosci.2889-15.2016

44. 44.

    Nakagawa S, Schielzeth H (2013 ) A general and simple method for obtaining R2 from generalized linear mixed-effects models. Methods Ecol Evol 4:133–142.

    Not in our library10.1111/j.2041-210x.2012.00261.x

45. 45.

    Nakhmani A, Tannenbaum A (2013 ) A new distance measure based on generalized image normalized cross-correlation for robust video tracking and image recognition. Pattern Recognit Lett 34:315–321.

    Not in our library10.1016/j.patrec.2012.10.025

46. 46.

    Newtson D (1973 ) Attribution and the unit of perception of ongoing behavior. J Pers Soc Psychol 28:28–38.

    Not in our library10.1037/h0035584

47. 47.

    Patel AX, Kundu P, Rubinov M, Jones PS, Vértes PE, Ersche KD, Suckling J, Bullmore ET (2014 ) A wavelet method for modeling and despiking motion artifacts from resting-state fMRI time series. Neuroimage 95:287–304.

    Not in our library10.1016/j.neuroimage.2014.03.012

48. 48.
    PsychoPy—Psychophysics software in Python · 2007

    No notice
49. 49.

    Polyn SM, Norman KA, Kahana MJ (2009 ) A context maintenance and retrieval model of organizational processes in free recall. Psychol Rev 116:129–156.

    Not in our library10.1037/a0014420

50. 50.

    R Core Team (2017 ) R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing.

    Not in our library

51. 51.

    Radvansky GA (2012 ) Across the event horizon. Curr Dir Psychol Sci 21:269–272.

    Not in our library10.1177/0963721412451274

52. 52.

    Richmond LL, Zacks JM (2017 ) Constructing experience: event models from perception to action. Trends Cogn Sci 21:962–980.

    Not in our library10.1016/j.tics.2017.08.005

53. 53.

    Sargent JQ, Zacks JM, Hambrick DZ, Zacks RT, Kurby CA, Bailey HR, Eisenberg ML, Beck TM (2013 ) Event segmentation ability uniquely predicts event memory. Cognition 129:241–255.

    Not in our library10.1016/j.cognition.2013.07.002

54. 54.

    Satterthwaite FE (1941 ) Synthesis of variance. Psychometrika 6:309–316.

    Not in our library10.1007/bf02288586

55. 55.

    Schapiro AC, Turk-Browne NB, Norman KA, Botvinick MM (2016 ) Statistical learning of temporal community structure in the hippocampus. Hippocampus 26:3–8.

    Not in our library10.1002/hipo.22523

56. 56.

    Schielzeth H, Nakagawa S (2013 ) Nested by design: model fitting and interpretation in a mixed model era. Methods Ecol Evol 4:14–24.

    Not in our library10.1111/j.2041-210x.2012.00251.x

57. 57.

    Sengupta A, Kaule FR, Guntupalli JS, Hoffmann MB, Häusler C, Stadler J, Hanke M (2016 ) A studyforrest extension, retinotopic mapping and localization of higher visual areas. Sci Data 3:160093.

    Not in our library10.1038/sdata.2016.93

58. 58.
    The Cambridge Centre for Ageing and Neuroscience (Cam-CAN) study protocol: a cross-sectional, lifespan, multidisciplinary examination of healthy cognitive ageing · 2014

    No notice
59. 59.

    Smith TJ, Henderson JM (2008 ) Edit blindness: the relationship between attention and global change blindness in dynamic scenes. J Eye Mov Res 2:1–17.

    Not in our library

60. 60.

    Smith TJ, Levin D, Cutting JE (2012 ) A window on reality: perceiving edited moving images. Curr Dir Psychol Sci 21:107–113.

    Not in our library10.1177/0963721412437407

61. 61.

    Sols I, DuBrow S, Davachi L, Fuentemilla L (2017 ) Event boundaries trigger rapid memory reinstatement of the prior events to promote their representation in long-term memory. Curr Biol 27:3499–3504.e4.

    Not in our library10.1016/j.cub.2017.09.057

62. 62.

    Speer NK, Zacks JM, Reynolds JR (2007 ) Human brain activity time-locked to narrative event boundaries. Psychol Sci 18:449–455.

    Not in our library10.1111/j.1467-9280.2007.01920.x

63. 63.

    Squire LR (1992 ) Memory and the hippocampus: a synthesis from findings with rats, monkeys, and humans. Psychol Rev 99:195–231.

    Not in our library10.1037/0033-295x.99.2.195

64. 64.

    Staresina BP, Davachi L (2009 ) Mind the gap: binding experiences across space and time in the human hippocampus. Neuron 63:267–276.

    Not in our library10.1016/j.neuron.2009.06.024

65. 65.

    Strange BA, Dolan RJ (2001 ) Adaptive anterior hippocampal responses to oddball stimuli. Hippocampus 11:690–698.

    Not in our library10.1002/hipo.1084

66. 66.

    Swallow KM, Barch DM, Head D, Maley CJ, Holder D, Zacks JM (2011 ) Changes in events alter how people remember recent information. J Cogn Neurosci 23:1052–1064.

    Not in our library10.1162/jocn.2010.21524

67. 67.

    Swallow KM, Kemp JT, Candan Simsek A (2018 ) The role of perspective in event segmentation. Cognition 177:249–262.

    Not in our library10.1016/j.cognition.2018.04.019

68. 68.
    The Cambridge Centre for Ageing and Neuroscience (Cam-CAN) data repository: Structural and functional MRI, MEG, and cognitive data from a cross-sectional adult lifespan sample · 2017

    No notice
69. 69.

    Terada S, Sakurai Y, Nakahara H, Fujisawa S (2017 ) Temporal and rate coding for discrete event sequences in the hippocampus. Neuron 94:1248–1262.e4.

    Not in our library10.1016/j.neuron.2017.05.024

70. 70.

    Tulving E, Markowitsch HJ, Craik FE, Habib R, Houle S (1996 ) Novelty and familiarity activations in PET studies of memory encoding and retrieval. Cereb Cortex 6:71–79.

    Not in our library10.1093/cercor/6.1.71

71. 71.

    Wagner AD, Schacter DL, Rotte M, Koutstaal W, Maril A, Dale AM, Rosen BR, Buckner RL (1998 ) Building memories: remembering and forgetting of verbal experiences as predicted by brain activity. Science 281:1188–1191.

    Not in our library10.1126/science.281.5380.1188

72. 72.

    Wang L, Zhang Y, Feng J (2005 ) On the euclidean distance of images. IEEE Trans Pattern Anal Mach Intell 27:1334–1339.

    Not in our library10.1109/tpami.2005.165

73. 73.

    Westfall J, Nichols TE, Yarkoni T (2016 ) Fixing the stimulus-as-fixed-effect fallacy in task fMRI. Wellcome Open Res 1:23.

    Not in our library10.12688/wellcomeopenres.10298.1

74. 74.

    Yushkevich PA, Piven J, Hazlett HC, Smith RG, Ho S, Gee JC, Gerig G (2006 ) User-guided 3D active contour segmentation of anatomical structures: significantly improved efficiency and reliability. Neuroimage 31:1116–1128.

    Not in our library10.1016/j.neuroimage.2006.01.015

75. 75.

    Zacks JM, Braver TS, Sheridan MA, Donaldson DI, Snyder AZ, Ollinger JM, Buckner RL, Raichle ME (2001 ) Human brain activity time-locked to perceptual event boundaries. Nat Neurosci 4:651–655.

    Not in our library10.1038/88486

76. 76.

    Zacks JM, Swallow KM, Vettel JM, McAvoy MP (2006 ) Visual motion and the neural correlates of event perception. Brain Res 1076:150–162.

    Not in our library10.1016/j.brainres.2005.12.122

77. 77.

    Zacks JM, Speer NK, Swallow KM, Braver TS, Reynolds JR (2007 ) Event perception: a Mind/Brain perspective. Psychol Bull 133:273–293.

    Not in our library10.1037/0033-2909.133.2.273

78. 78.

    Zacks JM, Speer NK, Reynolds JR (2009 ) Segmentation in reading and film comprehension. J Exp Psychol Gen 138:307–327.

    Not in our library10.1037/a0015305

79. 79.
    The brain’s cutting-room floor: segmentation of narrative cinema · 2010

    Not checked
80. 80.

    Zwaan RA, Radvansky GA (1998 ) Situation models in language comprehension and memory. Psychol Bull 123:162–185.

    Not in our library10.1037/0033-2909.123.2.162

81. 81.

    Zwaan RA, Langston MC, Graesser AC (1995 ) The construction of situation models in narrative comprehension: an event-indexing model. Psychol Sci 6:292–297.

    Not in our library10.1111/j.1467-9280.1995.tb00513.x

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.