Use of Crowd-Sourcing to Assess Social Communicative Behavior in Toddlers with and without ASD

Background:

Social communication deficits are key behaviors associated with Autism Spectrum Disorders (ASDs). The assessment of these behaviors is complex and requires highly trained experts especially in very young children. Current practice standards assume that only experienced professionals can assess differences in social communication among young children. Consequently, waitlists are often long in most communities. The technique of crowd sourcing has been used to efficiently answer problems requiring human intelligence. It relies on the concepts of collective intelligence, i.e., that large groups of individuals can create solutions equal in quality to a few experts.

Objectives:

To compare ratings of toddlers' social-communicative behavior as judged by crowd-workers with those made by clinicians with expertise in ASD. We hypothesized that the crowd-sourced assessments would differentiate between varying abilities of social communication in a manner similar to expert-based assessments.

Methods:

A single video clip for each of eight, 18-month-old children participating in a longitudinal research study were selected for this study. All videos showed a child engaging with an adult during administration of the Screening Tool for Autism in Toddlers (STAT). The videos were chosen to show a range of typical and atypical behaviors during simple play-based activities. A 12-item Likert-based questionnaire was created to identify a range of behaviors that might be observed in 18 month old children, with the sum of the 12 items used as an aggregate score and higher scores indicative of more typical behavior. An additional question included a global judgement of whether or not the toddler’s behavior was typical for his/her age. Each video was viewed by 3 experts and at least 68 crowd-workers. We assessed inter-rater reliability between expert ratings with an intra-class correlation coefficient (ICC). Rank order mean aggregate scores for each video clip were compared between crowd-workers and experts using a Spearman correlation coefficient. We used a Mann-Whitney U test to compare the aggregate score between clips that were rated as typical versus atypical.

Results:

Spearman correlation for rankings of total social communication behavior between crowd-workers and experts was very high (0.93; p<0.001) (See Fig.1). Inter-rater reliability between expert raters was excellent (ICC = 0.80. Crowd-workers completed 587 reviews in 6 hours and 50 minutes. Expert reviewers completed 24 reviews in 57 days. Aggregate summary scores were higher for videos rated as typical behavior than those rated as atypical for both crowd-workers (median score: 43 vs. 33; p<0.001) and experts (median score: 49 vs. 32; p<0.001).

Conclusions:

In this pilot study, we demonstrated the feasibility and validity of using a crowd-sourcing approach to measure social communication behaviors in very young children. Such a method may be a useful additional screening tool to discriminate typical from atypical behavior rapidly, with the goal of improving specificity of screening for developmental delays in social communication. Further studies with larger populations will be needed to confirm the utility of this approach.