27764
Autism Data Goes Big: A Publicly-Accessible Multi-Modal Database of Child Interactions for Behavioural and Machine Learning Research

Poster Presentation
Friday, May 11, 2018: 10:00 AM-1:30 PM
Hall Grote Zaal (de Doelen ICC Rotterdam)
J. Shen1, E. Ainger2, A. M. Alcorn2, S. Babović Dimitrijevic3, A. Baird4, P. Chevalier5, N. Cummins6, J. J. Li5, E. Marchi7, E. Marinoiu8, V. Olaru8, M. Pantic1, E. Pellicano9, S. Petrović3, V. Petrović3, B. R. Schadenberg5, B. Schuller6, S. Skendžić3, C. Sminchisescu10, T. Tavassoli11, L. Tran1, B. Vlasenko7,12, M. Zanfir8, V. Evers5 and C. De-Enigma13, (1)Department of Computing, Imperial College London, London, United Kingdom, (2)Centre for Research in Autism and Education, University College London, London, United Kingdom, (3)Serbian Society of Autism, Belgrade, Serbia, (4)Chair for Embedded Intelligence for Health Care and Wellbeing, University of Augsburg, Augsburg, Germany, (5)Human-Media Interaction, University of Twente, Enschede, Netherlands, (6)Informatics, University of Augsburg, Augsburg, Germany, (7)Complex Intelligent Systems, University of Passau, Passau, Germany, (8)"Simion Stoilow" Institute of Mathematics of the Romanian Academy (IMAR), Bucharest, Romania, (9)Centre for Research in Autism and Education (CRAE), UCL Institute of Education, University College London, London, United Kingdom, (10)Institute of mathematics Simion Stoilow of the Romanian Academy (IMAR), Bucharest, Romania, (11)Centre for Autism, School of Psychology & Clinical Language Sciences, University of Reading, Reading, United Kingdom, (12)Idiap Research Institute, Martigny, Switzerland, (13)DE-ENIGMA project consortium, Enschede, Netherlands
Background: Advances in artificial intelligence (AI), machine learning and “big data” have been almost exclusively developed based on neurotypical adult data, meaning that these tools may substantially misinterpret or fail to recognise the facial, vocal, and physical behaviours of autistic children. Existing (non-medical, non-genetic) autism datasets have been “small data”: insufficient to train autism-specific algorithms, and infrequently shared between researchers. Thus, the autism and technology subfield has been largely excluded from harnessing the cutting-edge techniques present elsewhere in academic research and industry, limiting our ability to deliver personalised, intelligent technologies for education, intervention, and daily life. More—and shareable—data is needed to advance research in this area. Previous projects with adults’ data have demonstrated that publicly accessible databases are logistically feasible, can operate on ethical terms acceptable to participants, and can accelerate research in related areas (e.g. https://semaine-db.eu/).

Objectives: To develop the first free, large-scale, publicly available multi-modal database of autistic children’s interactions that is suitable for both behavioural and AI research. It will use data collected in DE-ENIGMA project studies on autistic children’s emotion learning with a humanoid robot (http://de-enigma.eu/).

Methods: 62 British and 66 Serbian children aged between 5 and 12 years (19 female), participated in DE-ENIGMA studies on emotion recognition teaching. Each child was randomly assigned to robot-assisted or adult-assisted activities. These were based on steps 1-4 of the emotion training programme, “Teaching Children with Autism to Mind Read” (Howlin, Baron-Cohen, & Hadwin, 1999). Each child participated in 4-5 sessions, all recorded by multiple audio, video, and depth recording devices (see Table 1 and Figure 1). The parents of 121 of these children have granted consent for database inclusion.

Results: The DE-ENIGMA project has created a multi-modal database accessible via a web portal (http://db.de-enigma.eu), to which academic researchers worldwide may apply for access under a licensing agreement that prohibits commercial or governmental use. It includes ~13 TB of multi-modal data, representing 152 hours of interaction. The database is filterable based on data type (audio, video, depth, annotations) and session characteristics (child age group, study condition, country). Furthermore, 49 children’s data have been annotated by experts for emotional valence, arousal, audio features in English or Serbian, and body gestures.

Conclusions: This database will be the largest existing dataset of its kind (i.e. autistic interaction, rather than genetic or medical data). The audio and video in particular represent a rich resource for behavioural research questions about autism, such as child-robot or child-adult interactions, emotion recognition, social and communicative behaviours, and cross-cultural comparison. It also provides the required scale of data needed for furthering machine learning, computer vision, audio processing and other technical techniques that include autistic behaviours. The annotated data are in effect ready-labelled training data for future autism-focused machine learning research. Finally, the DE-ENIGMA database should accelerate both new behavioural and technological work on autism by providing free starting data to researchers, a potentially enormous saving of time and resources that may also reduce the many obstacles to participation in this area.