Autism Data Goes Big: A Publicly-Accessible Multi-Modal Database of Child Interactions for Behavioural and Machine Learning Research

Background: Advances in artificial intelligence (AI), machine learning and “big data” have been almost exclusively developed based on neurotypical adult data, meaning that these tools may substantially misinterpret or fail to recognise the facial, vocal, and physical behaviours of autistic children. Existing (non-medical, non-genetic) autism datasets have been “small data”: insufficient to train autism-specific algorithms, and infrequently shared between researchers. Thus, the autism and technology subfield has been largely excluded from harnessing the cutting-edge techniques present elsewhere in academic research and industry, limiting our ability to deliver personalised, intelligent technologies for education, intervention, and daily life. More—and shareable—data is needed to advance research in this area. Previous projects with adults’ data have demonstrated that publicly accessible databases are logistically feasible, can operate on ethical terms acceptable to participants, and can accelerate research in related areas (e.g. https://semaine-db.eu/).

Objectives: To develop the first free, large-scale, publicly available multi-modal database of autistic children’s interactions that is suitable for both behavioural and AI research. It will use data collected in DE-ENIGMA project studies on autistic children’s emotion learning with a humanoid robot (http://de-enigma.eu/).

Methods: 62 British and 66 Serbian children aged between 5 and 12 years (19 female), participated in DE-ENIGMA studies on emotion recognition teaching. Each child was randomly assigned to robot-assisted or adult-assisted activities. These were based on steps 1-4 of the emotion training programme, “Teaching Children with Autism to Mind Read” (Howlin, Baron-Cohen, & Hadwin, 1999). Each child participated in 4-5 sessions, all recorded by multiple audio, video, and depth recording devices (see Table 1 and Figure 1). The parents of 121 of these children have granted consent for database inclusion.

Results: The DE-ENIGMA project has created a multi-modal database accessible via a web portal (http://db.de-enigma.eu), to which academic researchers worldwide may apply for access under a licensing agreement that prohibits commercial or governmental use. It includes ~13 TB of multi-modal data, representing 152 hours of interaction. The database is filterable based on data type (audio, video, depth, annotations) and session characteristics (child age group, study condition, country). Furthermore, 49 children’s data have been annotated by experts for emotional valence, arousal, audio features in English or Serbian, and body gestures.

Conclusions: This database will be the largest existing dataset of its kind (i.e. autistic interaction, rather than genetic or medical data). The audio and video in particular represent a rich resource for behavioural research questions about autism, such as child-robot or child-adult interactions, emotion recognition, social and communicative behaviours, and cross-cultural comparison. It also provides the required scale of data needed for furthering machine learning, computer vision, audio processing and other technical techniques that include autistic behaviours. The annotated data are in effect ready-labelled training data for future autism-focused machine learning research. Finally, the DE-ENIGMA database should accelerate both new behavioural and technological work on autism by providing free starting data to researchers, a potentially enormous saving of time and resources that may also reduce the many obstacles to participation in this area.