On the Potential of Automated Emotional Facial Expression Recognition for Screening/Diagnosis of ASD: Evidence from Comparing Human Vs. Machine Detection of Emotional Expressions of High Functioning Young Children with ASD

Background: Improving early recognition of ASD has been a key focus of research and development in the last decades. Recently, technological developments seem to have given new impetus to it. Both clinical and experimental studies indicate differences between emotional facial expressions of neurotypical (NT) children and children with ASD. Although differences are often subtle and varying, screening/diagnostic technologies may potentially be able to exploit them.

Objectives: The present work was aimed at exploring whether commercially available emotional facial expression recognition technology has a potential for being a part of technologically-aided early screening/recognition of ASD, vis-á-vis recognizing emotional facial expressions by human coders. This work is a part of a larger research-and-development project with the objective of developing and validating a multi-modal, social serious game-based digital system for screening for ASD in 3-6 year old high functioning (HF) children.

Methods: Video recordings of 13 HF ASD children (mean age 58.4 months, range 43-70) and 13 NT children (mean age 57.15 months, range 43-68) were analyzed; all ASD children had a clinical diagnosis confirmed by ADOS and ADI-R; all NT children had negative ADOS and ADI-R results. There were no significant differences between the two groups along age and IQ. Each video-recording showed the face of a child while playing with the fully playable prototype of the screening serious game under development, but did not show any content of the game. Three sessions were selected from the recording for analysis: from the beginning the play session (140 secs), from the middle (195 secs), and from the closure of the game (15 secs). Each video-recording was analyzed by a machine system (Noldus FaceReader v5.1, Noldus Information Technology) and by two independent human raters. For human coding, we adapted the judgment-based emotion coding system of Kring és Sloan (2007; Facial Expression Coding System, FACES). Although sign-vehicle-based human emotion coding systems (such as the Facial Action Coding System by Ekman & Friesen, 1978) seem more accurate, their time and resource consumptiveness makes them imperfect tools in screening contexts. First, inter-rater reliability was calculated for human, test-retest reliability was calculated for machine codings. Secondly, we intended to compare the two groups along the intensities of the displayed emotions, based on human and machine codings, separately.

Results: Human coding of emotional facial expressions showed very poor inter-rater reliability, on both ASD and NT data, rendering any further analysis of this dataset methodologically unsupported. Machine coding, as expected, showed practically total test-retest reliability, and group comparisons showed significant differences in mean intensities of two emotions (scared, surprised), ASD group showing these more intensively.

Conclusions: While human emotional facial expressions coding has proven to be unreliable in this study, machine coding was practically totally reliable and was able to reveal ASD/NT differences. These are consistent with findings from earlier studies on emotion expressions in children with ASD. Although further confirmation and resolving methodological issues are needed for firm conclusions, our results suggest that automated facial expression recognition may potentially play a role in technologically-aided early screening/diagnosis of ASD.