Follow-up event of the biannual Baltic NeuroCine kick-off conference 2022 (next scheduled to 2024).

Organizers Elen Lotman & Pia Tikka, Baltic Film, Media and Arts School, Tallinn Unversity

Neurocine Seminar is part of FilmEU_RIT Pilot activities.

Photo by Heikki Leis.

Open call for PARTICIPATION

Tallinn invites students and film professionals to join the Baltic NeuroCine seminar & workshop days in May 5-7, 2023.  Registration opens in mid April. 

The event aims to enhance interdisciplinary research endeavors at European film schools on cinematic storytelling, film viewing, and related first-person experiences.

LINK TO Neurocine seminar PROGRAM May 5-7 HERE

Motivation

Since the beginning of this millennium, narratives mediated by films have allowed researchers to simulate complex socio-emotional events in behavioral and neuroimaging  laboratories, accumulating new insights to the human behavior, emotion, and memory, to name a few of many topics. The proponents of the so-called naturalistic neuroscience and, in particular its subfield neurocinematics (Hasson et al. 2008) have shown how experiencing naturally unfolding events evokes synchronized activations in the large-scale brain networks across different test participants (see, Jaaskelainen et al. 2021, for review). The  tightly framed contextual settings in cinematic narratives have opened a fresh window for researchers interested in understanding linkages between individual subjective experiences and intersubjectively shared experiences.

So far, neurocinematic studies have nearly exclusively focused on mapping  the correlations between narrative events and observed physiological behaviors of uninitiated test participants.. The knowledge accumulated so far does not tell much of the affective or cognitive functions of the experts of audiovisual storytelling, with few exceptions (e.g. de Borst et al 2016; Andreu-Sánchez et al. 2021).  Along with this Special Issue we want to extend the scope of studies to the embodied cognitive processes  of storytellers themselves. As an example, consider the term “experiential heuristics” proposed by cinematographer Elen Lotman (2021) to describe the practice-based knowledge accumulation by cinematographers, or the embodied dynamics of the filmmaker in the process of simulating the experiences of the fictional protagonists and/or that of  imagined viewers described as “enactive authorship”(Tikka 2022). 

References:

• Andreu-Sánchez, C., Martín-Pascual, M.A., Gruart, A. and Delgado-García, J.M. (2021). The effect of media professionalization on cognitive neurodynamics during audiovisual cuts. Frontiers in Systems Neuroscience, 15: 598383. doi: https://doi.org/10.3389/fnsys.2021.598383
• de Borst, A. W., Valente, G., Jääskeläinen, I. P., & Tikka, P. (2016). Brain-based decoding of mentally imagined film clips and sounds reveals experience-based information patterns in film professionals. NeuroImage, 129, 428–438. https://doi.org/10.1016/j.neuroimage.2016.01.043
• Hasson, U., Landesman, O., Knappmeyer, B. Vallines, I., Rubin, N., & Heeger, D. (2008). Neurocinematics: The Neuroscience of Film. Projections, 2, 1–26. https://doi.org/10.3167/proj.2008.020102.
• Jääskeläinen, I. P., Sams, M., Glerean, E., & Ahveninen, J. (2021). Movies and narratives as naturalistic stimuli in neuroimaging. NeuroImage, 224, 117445. https://doi.org/10.1016/j.neuroimage.2020.117445
• Lotman, E. (2021) Experiential heuristics of fiction film cinematography. PhD Diss. Tallinn University.
• Tikka, Pia (2022). Enactive Authorship Second-Order Simulation of the Viewer Experience— A Neurocinematic Approach. Projections: the Journal for Movies & Mind, 16 (1), 47−66. DOI: 10.3167/proj.2022.160104.

MISSION FOR NEUROCINE SEmINAR

We take as the starting point the assumption that filmmakers and viewers alike build their experience and sense-making on the same embodied foundation, as their occasional references to “instincts”, or “gut feelings” may suggest. These experiences and actions could be likewise addressed using neurocinematic methods. The question our NeuroCine seminar puts forward is, in what ways could neuroimaging observations of viewer experience be extended to accumulating novel insights on the professional practices?  

We ENCOURAGE PhD studentS TO JOIN*

During the NeuroCine seminar & workshop days,  we will discuss a range of research questions that could help to understand more of the creative processes of professional filmmaking practitioners, e.g., screenwriters, editors, sound-designers, cinematographers, etc.

The talks will focus on the film experience of the film professionals and/or film viewers. The theoretical, conceptual and practical approaches to the experiential nature of filmmaking and viewing may draw, for instance, from social and cognitive sciences, psychophysiology, neurosciences, ecological psychology, affective computing, cognitive semantics, aesthetics, or empirical phenomenology. 

The focus of the talk may also be on a specific expertise, for example that of the writer, editor, cinematographer, scenographer or sound designer. The papers may describe, for example 1) subjective experiences, 2) intersubjectively shared experiences; 3) context or content annotation, 4) semantic description; 5) first-person phenomenal description, and/or 6) physiological observation (e.g. neuroimaging, eye-tracking, psychophysiological measures).

Additional possibility is that you may offer your topic to be published as an article in the Baltic Screen Media Review, Special Issue, Guest editors Pia Tikka and Elen Lotman with Maarten Coëgnarts, entitled: “Cinematic minds in making – Investigation of subjective and intersubjective experiences of storytelling” – All journal articles go through a separate submission and a rigorous peer-review process. Contact: NeuroCineBFM@tlu.ee

*Possibility to gain PhD study credits:  By request, upon participation PhD students receive 1 ECT for participation and reading assignment and 3 EAP for participation, presentation, reading and writing assignment. International PhD students are advised to negotiate eligibility of these study points with their own local study coordinators.

Organizers:

Elen Lotman  https://www.filmeu.eu/alliance/people/elen-lotman

Pia Tikka https://www.etis.ee/CV/Pia_Tikka/eng

SEE Neurocine seminar PROGRAM May 5-7 HERE

BALTIC NEUROCINE PROGRAM MAY 5-7, 2023

PRESENTERS MAY 5

WORKSHOP MAY 5

Maude Sills-Néron

Workshop: Creating cognitive overload: image density in the script

Bio: Maude Sills-Néron is a Ph. D candidate in film studies, research-creation, Université de Montréal. After completing a bachelor’s degree in philosophy and political science at the Université de Montréal, Maude Sills-Néron worked at Radio-Canada as a content producer and an executive producer while she completed a master’s degree in philosophy, and training in fiction feature film writing at INIS (National Institute of Image and Sound). Today, she is a doctoral candidate in film studies, a research assistant, a content producer and a screenwriter.

(Alphabetical; to be completed)

CREATIVITY TALK May 5

Josie Barnard

The Multimodal Writer: on storytelling and writing visuals

Dr. Josie Barnard SFHEA; Associate Professor in Creative and Digital Practice, De Montfort University, UK

This lecture views film storytelling through the prism of the theory of creativity that Barnard provides in her book, The Multimodal Writing: Creative Writing Across Genres and Media (Bloomsbury, 2019).  Writing for film is complex and dynamic, with multiple external factors involved.  To develop a creative flexibility that enables negotiation of such external multimodality, ‘a writer must acknowledge and learn how to deploy an internal multimodality – that is, all the mental work (conscious and unconscious) a writer does before operationalising any medium. The channels of communication that work together are, in this context, primarily internal channels – intellectual enquiry, moods, sensations, for example – which are brought to work together by a co-ordinating unconscious capacity which [Barnard conceptualises] as the “inner auteur”’ (ibid, p. 6). The lecture draws on a programme of academic research starting 2012 and Barnard’s extensive practitioner experience as award-winning author of novels, creative non-fiction books and scripts for publishers including Virago/LittleBrown, BBC Radio and Film 4.

Barnard, Josie (2019) The Multimodal Writer: Creative Writing Across Genres and Media, London: Bloomsbury/Red Globe.

Talk is facilitated by the official BFM PhD program.

Robert McNamara

MetaHuman with Wav2lip: New Virtual Representations and their Ethical Implications

The proposed paper will address ethical concerns surrounding representation of vulnerable groups as well as the methodological challenges inherent in using virtual characters in human studies which involve representing such groups. Such concerns focus on consequences arising from the technological affordances of new systems for creating graphical and audio representations which are capable of portraying human-like beings with near identical resemblance. Enacting such virtual representations of humans inevitably gives rise to important ethical questions: (1) Who has the right to tell certain stories? (2) Is it ethical to change the medium of a narrative and the identity of a narrator? (3) Do such changes, or technological mediations, affect whether a vulnerable group will be fairly and accurately portrayed? (4) And what are the implications, either way? While the backdrop of the paper involves discussing the potential of virtual representation as a tool for moral and social change, the ethical implications inherent in the use of new technologies such as Unreal Engine’s MetaHuman to create human-like virtual characters in conjunction with software such as Wav2lip, which allows for accurate lip syncing of audio by virtual characters, call for needed theoretical scrutiny from a methodological perspective.

Bio: Robert McNamara has a background in American criminal law as well as degrees related to audiovisual ethnography and eastern classics. He is from New York State, but has lived in Tallinn for the last five years. Currently, he specifically focuses on researching related to ethical, socio-political, anthropological, and legal aspects in the context of employing human-like VR avatars and related XR technology. He is co-authoring journal papers in collaboration with other experts on the team related to the social and legal issues surrounding the use of artificial intelligence, machine learning, and virtual reality avatars for governmental immigration regimes. During 2020-2023 Robert G. McNamara has worked with the MOBTT90 team. Currently he focuses on his PhD project as a doctoral student.

PRESENTERS MAY 6

ELEN LOTMAN

Perceptual professionalization – eye tracking pilot experiment

As a passionate proponent of including tacit knowledge from filmmakers to neurocinematic studies, Elen has experienced first hand the struggles of bringing filmmakers into the context of academic studies. The two worlds often seem to be too far apart. Challenges include both the terminology and mutual understanding, but also technical and human resources for conducting large studies. During the last couple of years, Elen has started to test a model of small pilot studies, that are less scientifically rigorous, but offer a possibility to test original hunches that come from both filmmaking and film teaching experience. In her talk, she will discuss her experiences of conducting small pilot experiments, elaborate on the challenges and opportunities that come with this model and present one of the studies in more detail. The pilot experiment that will be addressed in more detail is the one where she studied perceptual professionalization in an eye tracking experiment.

Bio:Elen Lotman is a cinematographer who has shot numerous feature films, shorts and documentaries. For various film projects she has travelled to Japan, Tibet, China, Thailand, India, Nicaragua, Costa Rica, Lapland and the Russian Arctic. Her films have won awards and have been shown in competition programs of A-class film festivals like Camerimage, IDFA, Tampere Film Festival, Black Nights Film Festival, Moscow International Film Festival etc. Most recently, her full-length documentary ‘Dear Mother’ was nominated for Best Cinematography at the ReelHeart Filmfestival in Toronto and won the Best Feature Documentary in 2020. Full-length fiction ‘Good Bye Soviet Union’ was nominated for Camerimage Golder Frog in the Directors’ Debuts Competition and won the Black Nights Film Festival audience award in 2021. Her feature film screenplay ‘Container’ made it to the quarterfinalist at the American Film Academy Nicholl screenwriting competition 2021 and her virtual exhibitions filmed for Tallinn Art Hall were chosen among world’s 10 best virtual exhibitions of 2020 by both New York Times and Wallpaper Magazine.
Elen Lotman is the Associate Professor of Film Arts at Baltic Film, Media and Arts School (BFM) of Tallinn University and curates the artistic research branch in BFM PhD studies. Elen was elected co-President of the European Federation of Cinematographers (IMAGO) after serving as IMAGO Board Member and also Diversity and Inclusion Committee co-chair. She defended her PhD thesis titled “Experiential Heuristics in Fiction Film Cinematography” with laudatur in 2021.

Mircea Valeriu Deaca

POV shots in light of Cognitive Grammar

The paper applies Ronald Langacker’s Cognitive Grammar notions to cinematic editing constructions. Notions like head, complement, autonomous and dependent structures and the concept of elaboration site (e-site) are applied to editing constructions. The e-site represents a schematic region in the situation evoked by a simulation, and plays the role of a simulator and a goal concept.  Simulations can contain schematic components that only partially represent entities and event frames, not just detailed representations of them. Starting from grammatical constructions, we will detail the couplings between control models in sequences of POV shots. The visual expression is a composite structure that “inherits’ its profile from a component structure. Different construals and constructional schemas are instantiated by the juxtapositions of shots. The e-site plays the role a goal-concept. Goal-derived categories constitute the mappings between roles in event sequences, and instantiations in the environment. As structures of these mappings emerge, the conceptual system becomes organized around the action–environment interface (Barsalou 2003: 522). Since conceptual representations are dynamically contextualized to support diverse courses of goal pursuit, situated conceptualizations support conceptual inferences via pattern completion. When one part of a situated conceptualization is perceived, the remainder of the conceptualization becomes active, constituting inferences about the current situation. In short, the e-site can be defined as a summary conceptualization that inhabits a schematic region of a simulation, and prompts the film viewer to construct concepts that will categorize other components down the chain of a sequence of shots. In this paper, I describe different semantic construals evoked by a POV sequence of shots. The claim is that sequences of shots are conceptual systems bound by circular loops at several levels. Several configurations of the system coexist and the viewer has the executive control necessary for implementing alternative meaning construals.

Bio: Mircea Valeriu Deaca teaches film analysis and theory, film history and contemporary Romanian film at the University of Bucharest, Faculty of History, Department of Ancient History, Archaeology and History of Art. He has a Ph.D thesis on the carnival tradition and Federico Fellini’s films at the Paris 3 – Sorbonne nouvelle. A key focus of his research is the cognitive approach to film studies. He had in 2018 a Fulbright Grant for research at the Centre for Literature and Mind (University of California at Santa Barbara). Among his books are: The Carnival and Federico Fellini’s Films (2009), The Anatomy of Film (2013), Investigations in Cognitive Film Analysis (2015), and The Kitchen Scenes in the New Romanian Cinema (2017). He recently published a monographical study on the The Control Cycle in Film. Circular Coupling from Brain to Culture (2020). He is hosting the international conference Cinema: Cognition and Arts since 2016, and was editor for the 2018 issue of Ekphrasis dedicated on Cinema and cognition.

IIro P. Jääskeläinen

Online assessment of emotional and cognitive states of movie viewers with neuroimaging: a tool for testing whether the minds of the viewership synchronize with the mind of the director?

In my talk I will go over some recent findings where we have been able to estimate the (mix of) emotional states of movie viewers, including tracking the emotional states over time, based on their brain activity recorded with fMRI. In practice, to achieve this we have used a separate emotion localizer session for the machine learning algorithms that are based on the assumption that information, including emotional states, is represented in the brain as distributed fingerprint patterns. In principle, with similar methods it is possible to also estimate cognitive states and though processes of viewers as they live through the events on the silver screen. Tentatively, these methods, especially when combined with self-report methods that together provide complementary information, open up new possibilities also for estimating to what extent the emotions and cognition of viewers correspond with what the director intends to relate.

 

Bio: Iiro P. Jääskeläinen is a Full Professor of Systems Neuroscience at Aalto University School of Science, Espoo, Finland. He obtained his PhD in Psychology at the University of Helsinki in 1995, after which he worked as a post-doc and as a junior faculty member at Harvard Medical School, Boston, MA. He joined Helsinki University of Technology, the predecessor of Aalto University, in 2003. Prof. Jääskeläinen’s interests currently encompass the use of movies and narratives as media-based stimuli for elicitation of social cognition and emotions in neuroimaging studies. The publications of Prof. Jääskeläinen can be found in this link: https://scholar.google.fi/citations?hl=en&user=F-i6heMAAAAJ ”

ASSIM KALOUAZ

Contribution of priming on the emotional appraisal of VR experiences

 

Awe as an emotional experience can be transformative and perspective-changing. Storytellers use this powerful emotional connection with audiences to capture their attention and move them. However, we don’t fully understand how such intense emotions work when designed into more immersive media like virtual reality (VR). Designing awe in VR can provide incredible experiences with potential positive benefits, but it also creates the capacity for supercharged and overwhelming experiences that would not achieve the desired effect. For this reason, this study aims at exploring how people construct awe from the scope of the constructed emotions theory (Barrett, 2014) and how the framing of the experience affects its emotional appraisal. Participants (N=38) were presented with 4 experiences (2 calm-inducing and 2 awe-inducing), half of the sample was told that all the experiences were awe-inducing, and the other half was told that they were all calm-inducing. Awe was measured through self-reports, duration perception, and physiological measures (HRV and EDA). Results from self-reports show a significant effect of the experience ( F(1, 36) = 13.787, p < .001), duration perception showed no effect, and HRV measures, while not reaching significance, consistently showed trends of a main effect of priming in the frequency measures (HRV LF: F(1, 32) = 3.705), p=.063, HRV HF: F(1, 32) = 3.310), p=.078, HRV LF/HF: F(1, 32) = 3.444), p=.073). Results suggest that how an experience is presented may affect physiological responses even if people are not aware of it, but they also highlight the need for a more holistic approach in measuring awe and a higher sample size to evaluate effect size and its potential implications.

 

Bio: Assim Kalouaz is a PhD researcher in psychology at UCD. His research interests revolve around storytelling, social cognition, and experiential design. He believes that VR has incredible potential as a storytelling medium for positive impact, but this impact can only be attained if the design of immersive experiences is driven by scientific evidence. To contribute to this discourse, his research focuses on understanding how to prompt, capture, and design awe (in its different flavours) in VR, how people appreciate and regulate it, and how it can be used for positive benefits.

JUHA SALMITAIVAL

Naturalistic neuroscience in clinical research: insights from virtual reality research

Naturalistic neuroscience has provided new means to study cognitive brain functions in a lifelike setting. In this talk, I will show several examples how we have applied virtual reality technologies to examine manifestation of the symptoms of neurodevelopmental disorders in situations resembling those where the symptoms manifest themselves. I will also show some data where we have compared active virtual reality situations with matching passive video viewing methods that are widely used in neurocinematics. Together, the new methods allow more holistic analysis of complex cognitive processes from different perspectives, including inter-individual differences and similarities, as well as intra-individual dynamics. Pre-recorded narrative stimuli and open-ended virtual reality conditions may have complementary benefits, while both naturalistic research approaches have wealth of underexploited opportunities to unravel the mysteries of the human mind.

Bio: Juha Salmi (aka Salmitaival) is an Academy of Finland funded Research Fellow in the Department of Neuroscience and Biomedical Engineering at Aalto University, Finland. He obtained his PhD in cognitive neuroscience in 2009 and he is a Docent of clinical neuropsychology at the University of Helsinki since 2015. In 2019 he started to run his own lab, Translational Cognitive Neuroscience Laboratory, that aims at developing novel digital methods for treatment of neurodevelopmental disorders. He is particularly interested in naturalistic methods for behavioural assessment and neuroimaging.

Mehmet Burak Yilmaz

Embodiment of Cinematographer 

The talk focuses on the cognitive-perceptual implications of camera movement techniques and how cinematographers effectively utilize their embodied knowledge while creating camera movements. The cognitive and emotional significance of cinematography is well-acknowledged in film studies. However, different techniques of camera movements (e.g. dolly, Steadicam, handheld) in narrative storytelling is a surprisingly sparsely studied topic, even though communication via moving images is increasing in an exponential manner. The audience experiences may largely vary, depending on how the camera is moved, and what type of viewpoint it generates for the viewers. Drawing from the tacit knowledge accumulated by cinematographers, I argue that when producing specific movements, cinematographers extend their perceptual bodily senses via the camera onto the screen, and in this sense, these movements may be further seen to become embodied in the viewer’s experience. In my talk, I will also share the findings from a practice-based experiment conducted to test my hypothesis, where the effects of different camera movement methods on viewers’ degree of immersion and emotional responses were studied. 

 

Bio: Mehmet Burak Yilmaz is a trained cinematographer and a Ph.D. student at Tallinn University. He obtained his bachelor’s in filmmaking (2018) and his master’s degree in cinematography (2020). He started his Ph.D. studies at Tallinn University, Baltic Film and Media School in 2020. His practice-based artistic research focuses on the affective functions of the camera movements in cinematic storytelling. As a cinematographer, he shot various short films, documentaries and commercials in different countries. He also works in the camera and lighting departments on professional productions.