Engineering systems provide controlled platforms for interaction and automatic tools to quantify aspects of the resulting behavior. The measures derived from these tools can be used to aid clinicians in diagnostics and in the development of individualized therapeutic interventions. In this work we present Rachel, an Embodied Conversational Agent (ECA), for the collection of interaction data from children with autism and their parents. This work was demoed at IMFAR 2010. ECAs are an important tool for natural, repeatable, and structured interaction behavior collection. The goal of this work is to explore how an ECA platform can elicit conversational behavior for automatic and manual analysis.
Objectives:
The ECA used in the study is designed to elicit natural affective communication through the interaction protocol and accompanying scenarios. The recorded communication is automatically assessed using: speaker clustering, interaction modeling, and a statistical analysis of prosody. The manual analyses include transcription and speech act coding.
Methods:
Rachel is an ECA designed to elicit affective and social child-parent-computer interaction behavior for analysis in a four-session emotion problem-solving study. The ECA serves as the moderator and coach, introducing the scenario and querying the child for detail. The interactions use the Wizard-of-Oz paradigm allowing for controlled and repeatable interactions while avoiding the technological challenges associated with speech recognition and understanding. The child and parent interact with the ECA either audibly or using a touch monitor. The interaction is recorded using audio-visual sensors and the ECA behavior is logged.
The audio data were manually segmented by speaker and transcribed as a ground truth. Speaker clustering was conducted using the audio data and Rachel log files, allowing us to identify the speaker at each instant in time (the transcription data is used to validate the accuracy). The results of the speaker clustering data were leveraged to model the child-parent-ECA interaction patterns, providing a quantitative description of the observed speech patterns. The results of the speaker clustering can also be used to isolate the child's speech for prosodic assessments. The transcription data were manually coded with speech act tags to provide statistical relationships between the ECA's speech acts and the child's and parent's responses.
Results:
The ECA technologies were tested on two children with autism. The inclusion criteria included: 1) a diagnosis of autism (confirmed through an administration of the Autism Diagnostic Observation Scale (ADOS) by a psychologist with a research certification in the ADOS); 2) age 5-13 years; 3) a score on the Vineland Behavior Scales at or above an age equivalent of 2 years, 6 months, and 4) both the parent and the child were English-speaking. The children included two boys: one 6 and one 12. The results of the automatic assessment will be discussed at the workshop.
Conclusions:
ECA technologies can elicit social interactions for post-hoc analysis. The Rachel interaction data were recorded using audio-visual and log files to facilitate post-hoc analysis to identify aspects of social deficits in children with autism. This work is supported by the National Science Foundation and Autism Speaks.