team news – Enactive Virtuality Lab

Affective Computing ACII 2019 conference Cambridge

The 8th International Conference on. Affective Computing & Intelligent Interaction ( ACII 2019). 3rd-6th September, 2019. Cambridge, United Kingdom.

Enactive Virtuality team has two abstracts accepted for multidisciplinary Special Tracks :

(1) Neural and Psychological Models of Affect and (2) Technological and Biological Bodies in Dialogue: Multidisciplinary Perspectives on Multisensory Embodied Emotion and Cognition.

Image: Jelena Rosic in action during an TLU ‘s internal pre-presentation of the ACII conference talk “Phenomenology of the Artificial” by Kosunen, Rosic, Kaipainen and Tikka,

Neuroadaptive dance project “Trisolde”

TRISOLDE – Neuroadaptive Gesamtkunstwerk: The Biocybernetic Symbiosis of Tristan and Isolde”

Exploring the final frontier of human-computer interaction with a neuroadaptive opera…performed by the audience, dancers and computational creativity .

Team of “TRISOLDE” (Tiina Ollesk, Simo Kruusement, Renee Nõmmik, Ilkka Kosunen, Hans-Günther Lock, Giovanni Albini) performed in Festival “IndepenDance” in Göteborg, nov 29 and Dec 2, 2019.

A symbiotic dance version of Wagner´s “Tristan and Isolde” where dancers are controlling the music via body movements and implicit psychophysiological signals. This work explores the next step in this coming-together of man and machine: the symbiotic interaction paradigm where the computer can automatically sense the cognitive and affective state of the user and adapt appropriately in real-time. It brings together many exciting fields of research from computational creativity to physiological computing. To measure audience and to use the audience’s reactions to module the orchestra is new way of doing “participatory theatre” where audience becomes part of the performance.

“Tristan and Isolde” is widely considered both as one of the greatest operas of all time as well as beginning of modernism in music, introducing techniques such as chromaticism dissonance and even atonality. It has sometimes been described as a “symphony with words”; the opera lacks major stage action, large choruses or wide range of characters. Most of the “action” in the opera happens inside the heads of Tristan and Isolde. This provides amazing possibilities for a biocybernetic system: I this case, Tristan and Isolde will communicate both explicitly (through movement of the dancers) but also implicitly via the measured psychophysiological signals.

Dance artists: Tiina Ollesk, Simo Kruusement

Choreographer-director: Renee Nõmmik

Dramaturgy and science of biocybernetic symbiosis: Ilkka Kosunen

Composers for interactive audio media: Giovanni Albini, Hans-Gunter Lock

Video interaction: Valentin Siltsenko

Duration: 40’

This performance is supported by: The Cultural Endowment of Estonia, and Enactive Virtuality Lab and Digital Technology Insitute (biosensors), Tallinn University.

Presentation of project: November 29th-30th and December 1st, 2018 at 3:e Våningen Göteborg (Sweden) at festival Independance. The event is dedicated to the centenary of the Republic of Estonia and supported by program “Estonia100-EV100”.

PREMIER IN TALLINN FEBRUARY 2019 (see more Fine 5 Theater)

Compassion Cultivation Training in Bio-Adaptive Virtual Environments Paris, CRI June 8

As part of the Frontiers du Vivant, Lynda Joy Gerry presented and defended her project “Compassion Cultivation Training in Bio-Adaptive Virtual Environments” at the Centre for Research and Interdisciplinarity (CRI) in Paris. This project involves using perspective-taking in virtual environments using biofeedback relating to emotion regulation (heart rate variability) to manage the recovery from empathic distress. Empathic distress is conceived as a step in the empathic process towards the understanding of another person’s affective, bodily, and psychological state, but one that can lead to withdrawal and personal distress for the empathizer. Thus, this project implements instruction techniques adopted from Compassion Cultivation Training guided meditation practices cued by biofeedback to entrain better self-other boundaries and distinctions, as well as emotion regulation.

Lynda also participated in a workshop on VR and Empathy led by Philippe Bertrand from BeAnother Lab (BAL, The Machine to Be Another). See Philippe Bertrand TEDx talk “Standing in the shoes of the others with VR”

Embodying Creative Expertise in Virtual Reality Zurich ZhDK May 29-31

In collaboration with BeAnotherLab (The Machine to Be Another), Lynda Joy Gerry taught a workshop, “Embodying Creative Expertise in Virtual Reality” to Masters in Interaction Design students at Zürcher Hochschule der Künste (ZhDK), as part of a course, “Ecological perception, embodiment, and behavioral change in immersive design” led by BAL members.

Image: Poster for the students’ final project presentation and exhibition.

Lynda specifically taught students design approaches using a semi-transparent video overlay of another person’s first-person, embodied experience, as in First-Person Squared. The focus of the workshop was on Leap Motion data tracking and measurements, specifically how to calculate compatibility and interpersonal motor coordination through a match score between the two participants, and how to send this data over a network. The system provides motor feedback regarding imitative gestures that are similar in form and position, and also for gestures that occur synchronously (at the same time), ideally trying to support both types of interpersonal motor coordination. Lynda taught students the equations used and data input necessary to calculate this algorithm for the different match scores, and also how to add interaction effects to this data. Lynda showed students how to implement Leap motion hand tracking on top of stereoscopic point-of-view video and how to record user hand movements. On the 31^st, students premiered their final projects at an event entitled “Scattered Senses.”

Mimic Yourself: Mo-cap Workshop Zurich ZhDK May 30

On March 30^th, Lynda Joy Gerry visited the Innovation Lab at Zürcher Hochschule der Künste (ZhDK) for a workshop entitled “Mimic Yourself.”

This workshop involved collaborations between psychologists, motion-tracking and capture experts, and theater performers. The performers wore the Perception Neuron motion capture suit within an opti-track system. The data from the performer’s motion was tracked onto virtual avatars in real-time. Specifically, the team had used the Structure Sensor depth-field camera to create photogrammetry scans of members of the lab. These scans were then used as the avatar “characters” put into the virtual environment to have the mocap actors’ movements tracked onto. A screen was also programmed into the Unity environment, such that the screen could move around the real world in different angles and three-dimensional planes and show different views and perspectives of the virtual avatar being tracked relative to the human actor’s movements. Two actors playfully danced and moved about while impacting virtual effects with their tracked motion – specifically, animating virtual avatars but also cueing different sound effects and experiences.

Image above: Motion capture body suit worn by human actor and tracked onto a virtual avatar. Multiple avatar “snap shots” can be taken to create visual effects and pictures. Images below: Creating a many-arm shakti pose with avatar screen captures created through mocap.

Image above shows examples of photogrammetry scans taken with the Structure Sensor.

Watch Your Bubble Conference in Berlin May 17-19

On May 17^th, Lynda Joy Gerry attended a conference organized by the Berlin School of Mind and Brain entitled “Watch Your Bubble!” The conference title refers to the information bubble conceptualization of social dynamics, wherein which groups of individuals are becoming increasingly nested within a bubble that reinforces only their own worldview. The conference brought in speakers on neuroaesthetics and social neuroscience, specifically Vittorio Gallese, Joerg Fingerhut, Andreas Roepstorpff, Olafur Eliasson, and Vincent Hendricks.

Neuroscientist Vittorio Gallese’s lecture specifically explored the ways in which individuals are reciprocally connected and the inter-dependence of self and other. Embodied simulations when imagining actions activate similar neural pathways as actually performing the same action. This makes film an especially evocative medium, specifically for haptic-vision and what Gallese calls “embodying technesis” wherein common action representations exist between imagination and action and also between self and other. The conference also addressed the formation and plasticity of personal identity, where identity comes from and how it is formed. Joerg Fingerhut specifically addressed the types of changes that individuals believe would make them a new person. For instance, a change in one’s musical tastes and preferences is perceived as a change in one’s identity.

Collaboration meeting @ Virtual Cinema Lab, Aalto University

Our Enactive Avatar team Victor Pardinho, Lynda Joy Gerry, Eeva R Tikka, Tanja Bastamow, and Maija Paavola planning the volumetric video capture of a screen character with a collaborator in Berlin. The team’s work is supported by the Finnish Cultural Foundation, Huhtamäki Fund, and Virtual Cinema Lab (VCL), School of Film, Television and Scenography, Aalto University, and by Pia Tikka’s EU Mobilitas Top Researcher Grant.

“Sümbiootiline helilooming” – New publication by Ilkka Kosunen

The manifesto for the new field of science, Symbiotic Composing, by Dr. Ilkka Kosunen, was recently published in the Teater. Muusika. Kino.

Symbiotic composing connects the topics of deep learning, physiological computing and computational creativity to facilitate new type of creative process where technology and human aesthetic judgment merge into one.

The article is in Estonian language.

TMK_aprill_2018

Invited young lecture Ilkka Kosunen at Tohoku Forum for Creativity, Tokyo

Image: Ilkka Kosunen presenting as the Invited Young Lecturer at the Tohoku Universal Acoustical Communication Month 2018 October 20.

Seminar on music, sound, speech and artificial intelligence

TOKYO ELECTRON House of Creativity 3F, Lecture Theater, Katahira Campus, Tohoku University [Access]

Doctoral Defense: Ilkka Kosunen “Exploring the dynamics of the biocybernetic loop in physiological computing”

ILKKA KOSUNEN: EXPLORING THE DYNAMICS OF THE BIOCYBERNETIC LOOP IN PHYSIOLOGICAL COMPUTING

Enactive Virtuality team member Ilkka Kosunen defended the doctoral dissertation entitled “Exploring the dynamics of the biocybernetic loop in physiological computing” in the Faculty of Science, University of Helsinki, on 22 March 2018 at 12:00. The public examination took place at the following address: Exactum, D 122, Kumpula. Professor Fairclough Stephen, Liverpool John Moores University, will serve as the opponent, and Professor Jacucci Giulio as the custos.

The dissertation is published in the series Series of publications A / Department of Computer Science, University of Helsinki. The dissertation is also available in electronic form through the E-thesis service.