An Automated Measure of Conversational Semantic Coherence

Background: Complex conversational impairments in people with autism (focus on special interests or topically inappropriate responses) are well identified but their measurement requires time-consuming manual annotation of language samples. Natural Language Processing (NLP), and especially vector-based semantic methods, have shown promise to identify semantic difficulties in ASD on tasks such as semantic fluency and narrative retelling when compared to a clinician-annotated TD reference transcript. Whether or not such tools can establish language-based differences in conversational transcripts without recourse to a reference document remains to be established.

Objectives: Our goal was to develop a novel NLP measure of semantic coherence that could be employed with transcripts of conversational language in children. We hypothesized that semantic coherence, as measured by this method, would discriminate between children with or without ASD, and that more variability would be found in the group with ASD.

Methods: Participants: We used data from 70 subjects (38 ASD, 32 TD) age 5 to 8, all males, enrolled in a language study (ERPA). All participants were administered a battery of standardized diagnostic and neuropsychological tests, including the ADOS Module 3, the WPPSI-III, and WISC-IV. Measures: ADOS were recorded and subsequently transcribed verbatim using Praat software. The language output of each child during the emotion/conversation section was used in these analyses. Analyses: Transcripts were converted to vectors via application of a word2vec model trained on the Google News Corpus. Pairwise similarity across all subjects and a sample grand mean were first calculated. Using a leave-one-out algorithm a pseudo-value representing each subject’s contribution to the grand mean was generated, and means of pseudo-values were then compared between the 2 clinical groups. Analyses were co-varied for non-verbal IQ (NVIQ), Mean length of utterance in morphemes (MLUM) and number of distinct word roots (NDR).

Results:

Statistically significant differences in mean of the pseudo-values between TD and ASD groups (Wilcoxon rank sum test, p = 0.007).

• TD subjects typically have a higher pseudo-value score suggesting that similarity scores of TD subjects are more similar to the overall group mean.
• Greater variance in the pseudo-values of the ASD group.
• Analysis of covariance suggests that none of NVIQ, MLUM, or NDR account for the difference between group means.

Conclusions:

The findings suggest that NLP methods can be effectively used to identify specific semantic difficulties that characterize children with ASD. The method is automated and does not require costly and time-consuming annotation by expert clinicians. Our results are preliminary and need to be replicated in larger samples, and older age groups. Likewise, we would need to replicate our findings in language samples collected in more natural ecological settings. Future developments of NLP methods might help to provide fine-grained differentiation of types of semantic divergence in a conversation or locate in a conversation the time and context when divergence from topics occur. Future improvements of NLP methodology may also yield new, cost-effective and sensitive outcome measures applicable to treatment research. Furthermore, more precise documentation of semantic impairments in ASD may inform new intervention strategies in language therapy.