Testing the Ecological Validity of Executive Function Assessment in Autism with a Novel ‘Tea-Making’ Task

Background: People on the autism spectrum often struggle to achieve personal and occupational outcomes commensurate with their intellectual potential. One possible explanation for this mismatch could be difficulties with executive function (EF): a collective term for the higher-order cognitive skills that facilitate goal-directed behavior. Over 250 peer-reviewed papers have reported on EF assessments of autistic people but the findings have been markedly inconsistent. One criticism levelled at current EF assessment is that it lacks ecological validity. Indeed, research testing the ecological validity of these assessments in autism is severely limited and so it is unknown if the relationship between performance on neuropsychological EF tasks and everyday executively demanding tasks differs in autism compared with non-autistic people. Further, it is relatively unknown the extent to which current EF tasks are empirically related to the personal and occupational outcomes that have been so widely documented in autism.

Objectives: The current study sought to test the ecological validity of current neuropsychological EF assessment in autism in two ways. First, we tested the verisimilitude of EF by comparing the performance of autistic and non-autistic adolescents on two measures of EF: a battery of common EF tasks (National Institutes of Health (NIH) Examiner battery) and a novel, ecologically-valid ‘tea-making’ task. Second, we assessed the veridicality of EF by testing whether performance on the NIH battery was related to autistic participants’ quality of life and level of disability.

Methods: Data collection is ongoing. Participants assessed so far include autistic (n=20) and non-autistic (n=19) adolescents, between 12 and 19 years, group-matched by age (p=.15) and IQ (p=.30). A composite EF variable was generated by the NIH battery. The tea-making task involved preparing food and materials for a group study session while a researcher rated the number of executive errors committed (e.g. boiling the kettle extra times, failing to adapt to a change in task instructions). Performance on both tasks was scaled and standardized such that higher scores were indicative of better EF ability. Parents also completed The Pediatric Quality of Life Inventory (PedsQL) and The World Health Organization Disability Assessment Schedule (WHODAS-2).

Results: A 2x2 repeated-measures ANOVA comparing group (autistic and non-autistic) and EF assessment type (NIH and ‘tea-making’ task) yielded a significant main effect of group, F(1,36)=8.00, p=.008, η_p²=.18. Autistic participants showed, on average, more EF difficulties compared to non-autistic participants. There was, however, no main effect of EF assessment type, F(1,37)=.001, p=.99, η_p²<.001 and no significant group x EF assessment type interaction, F(1,37)=.29, p=.59, η_p²=.008. There were also no significant correlations between performance on the NIH battery and PedsQL, r(16)=-.28, p=.26 or WHODAS-2, r(16)=.23, p=.36, scores when age and IQ were controlled for among autistic participants.

Conclusions: Both the neuropsychological and ecological assessments of EF employed here were sensitive to group differences. This study demonstrated that, for autistic people, the verisimilitude of EF assessment is comparable with typical development. Yet, the absence of any relationship with parent reported quality of life or level of disability draws into question the veridicality of EF assessment in autism.