Effects of Manipulating the Coordination of Gesture and Speech In Computer Animations of Storytelling

Background:  Gestures typically accompany speech and are known to help in the understanding of the meaning of speech.  It is thought that speech and gesture are tightly coupled and that the precise coordination of gesture and speech is different between typically developed individuals and those on the autism spectrum (de Marchena & Eigsti, 2010).  However, the perceptual factors that influence speech-gesture coordination are not well understood.

Objectives:  We wished to explore the perceptual factors that lead to gesture not being perceived as coordinated with speech and the effects this has on understanding and appreciating someone gesturing while telling a story.  To obtain precise control of the visual stimuli we used techniques of computer animation to capture the motion of storytellers and play back the story on a computer character.

Methods:  The voice and body motion of 4 typically developed storytellers was captured using a audio recorder and a 12-camera Vicon motion capture system using a standard marker set.  The stories told were taken from the cartoon task of Modules 3/4 of the ADOS.  In this task participants see a series of cards representing a sequence of events and then are asked to tell the story.  The motion capture data was used to drive the behaviour of a computer-generated mannequin using the program MotionBuilder.  Motionbuilder allows editing the motion of the character and we created an animation without any manipulation and one in which the elbows were restricted from moving, resulting in straight stiff arms throughout the story.  These two animations were combined either with the standard audio or audio misaligned temporally by 330 ms.  This resulted in a total of 4 displays that were shown to 15 participants.  Participants viewed and listened to the displays and made judgments about the quailty of the story and the storyteller.  Each participant viewed each of the 4 conditions once given by a different individual.

Results:  The different ratings about the perceived quality of the storyteller and story were combined to provide an average for each condition of each participant.  An effect of order of presentation was obtained that complicated the interpretation.  However, the unaltered original animation was rated significantly different from the animation with the asynchronous audio and the one with the locked elbow.  Surprisingly, the combined asynchronous and elbow locked display was not rated different from original animation.  

Conclusions:  These results provide an important first step in establishing a rigorous procedure for analysing the perceptual factors which are key to perceiving coordinated gesture and speech.