Promises and Limits: Exploring Relationships Between the First Year Inventory (FYI) and Autism Diagnostic Observation Schedule (ADOS) from 12 to 18 Months of Age with Machine Learning

Background: Machine-learning, when used in combination with clinical expertise, has great potential to enhance diagnostic procedures and to improve our understanding of heterogeneity in autism spectrum disorder (ASD). A recently published study has shown that machine-learning techniques can achieve comparable classification performance on the Autism Diagnostic Observation Schedule (ADOS) compared to pre-existing algorithms (Wall et al., 2012). However, the usefulness of machine-learning for more heterogeneous groups of children, such as infants at high-risk (HR) for ASD, for other instruments including parent reports, and for the more difficult problem of prediction of future clinical presentations compared to concurrent presentations, is not known yet.  

Objectives: To evaluate the performance of machine-learning techniques for (1) using the First Year Inventory (FYI), a parent questionnaire given to parents of children 12 months of age, to predict levels of autism symptoms as given by the ADOS-Toddler Module (ADOS-T) at 12 and 18 months; and (2) using ADOS scores at 12 months to predict scores at 18 months.

Methods: Based on 76 HR children, we first examined Pearson’s correlation between the FYI totals at 12 months (FYI.12) and ADOS algorithm totals at 12 (ADOS.12) and 18 months (ADOS.18). Then, we used two widely-used machine-learning techniques, support vector machines (SVM) and random forests using variable selection, for the same predictions with all items from both measures (FYI.12–ADOS.12 and FYI.12-ADOS.18). The same methods were repeated to examine the correlation between ADOS.12 and ADOS.18. For machine-learning techniques, we performed 1000 repeated random sub-sampling validations, utilizing 80% of data for training and 20% for testing. We also report results while using all data as training sample [reported in [brackets]).

Results: We found a moderate correlation between FYI.12 and ADOS.12 (r=0.38). Both SVM and random forest resulted in comparable correlations, r=0.47[0.91 for training sample] and r=0.51[0.96] respectively. We found a milder correlation between the FYI.12 and ADOS.18 (r=0.13), but the SVM and random forest improved the correlations significantly, r=0.52[0.94] and r=0.58[0.98] (differences significant at p’s<0.01). Not surprisingly, we found a moderate correlation between ADOS.12 and ADOS.18 (r=0.53). The SVM and random forest resulted in comparable correlations, r=0.61[0.88] and r=0.63[0.96].

Conclusions: As the FYI is designed to capture current developmental profiles at 12 months, it was already moderately associated with concurrent clinical presentations measured by the “gold standard” clinician observations (ADOS) without implementing machine-learning techniques. Similarly, as the ADOS is designed to assess core autism symptoms that are fairly stable over time, the use of the ADOS algorithm was sufficient to predict clinical outcomes at 18 months. However, the prediction of clinical outcomes measured by clinician observations while using parent reports obtained 6 months earlier was more challenging. For this prediction, machine learning resulted in improved correlations, suggesting that it can be especially helpful when adaptations to measures are needed to make predictions that go beyond what they are designed for. Therefore, understanding the role of machine learning to add to ongoing explorations regarding measurement in ASD requires care, collaboration, open discussion, and humility regarding strengths and limits of statistical tools.