Automated Detection of Mutual Eye Contact and Joint Attention Using a Single Wearable Camera System

Background:  

Typically developing toddlers use means such as eye gaze, sound and gesture to express social interactions and joint attention. Autism is often characterized by pervasive differences in communication and social interaction, particularly in terms of visual attention patterns. Thus, when a child is evaluated for developmental progress, a clinician will examine the child's patterns of mutual eye contact and joint attention. Our goal is to explore how automated techniques of mutual gaze and joint attention can be developed for use in real physical settings. Current commercial mobile eye tracking solutions offer an interesting opportunity to explore this challenge. There have been attempts on automating the detection of joint attention using audio and video processing technologies mainly by instrumenting the environment with microphones and cameras. However, it is very hard to capture the patterns of kid's and examiner’s attention simultaneously using cameras mounted on tables or the ceiling. A wearable camera not only always captures where the examiner is attending and records her interaction with the toddler, but also provides an accurate estimate of what are the important events and objects in the scene to which examiner is looking.

Objectives:  

We combine a current commercial mobile eye tracking solution with gaze estimation technology to automatically detect patterns of mutual eye contact and joint attention.  This solution only requires one person in a dyadic interaction to wear the system.

Methods:  

Current commercial mobile eye-tracking solutions provide continuous estimations of the visual gaze patterns of the wearer. In our experiments, an adult wears the eye-tracking technology while interacting with a child in the semi-structured RapidABC protocol. We further process the video recorded from the camera mounted on the eye-tracker, continuously estimating the child's face orientation. We then analyze adult's and child's gaze patterns to identify mutual eye contact. These computed mutual gaze instances are then compared against ground truth data from the adult as well as third party observers. Similarly, we can monitor patterns of gaze to estimate when the child and adult shift between periods of looking towards each other and then towards a common area.

Results:  

We have recorded a number of mock RapidABC sessions with the adult wearing the Tobii mobile eye-tracking glasses. Our preliminary results show that a combination of the Tobii eye-tracking data and standard computer vision algorithms result in reasonable estimates of mutual eye gaze. We are in the process now of collecting RapidABC sessions with a trained evaluator and children ranging in age from 9 to 30 months. We will use ground truth indications of mutual gaze and joint attention indicated by the adult during the session.

Conclusions:  

In this work we use a single wearable eye-tracking camera for detecting patterns of mutual attention and eye contact between an adult and a child. Our preliminary results show the promise of applying this technology towards the study of dyadic interactions in physical space. This should lead to valuable extensions of the study of contingent mutual gaze in behavioral science.