An Automated Measure of Conversational Semantic Coherence
Objectives: Our goal was to develop a novel NLP measure of semantic coherence that could be employed with transcripts of conversational language in children. We hypothesized that semantic coherence, as measured by this method, would discriminate between children with or without ASD, and that more variability would be found in the group with ASD.
Methods: Participants: We used data from 70 subjects (38 ASD, 32 TD) age 5 to 8, all males, enrolled in a language study (ERPA). All participants were administered a battery of standardized diagnostic and neuropsychological tests, including the ADOS Module 3, the WPPSI-III, and WISC-IV. Measures: ADOS were recorded and subsequently transcribed verbatim using Praat software. The language output of each child during the emotion/conversation section was used in these analyses. Analyses: Transcripts were converted to vectors via application of a word2vec model trained on the Google News Corpus. Pairwise similarity across all subjects and a sample grand mean were first calculated. Using a leave-one-out algorithm a pseudo-value representing each subject’s contribution to the grand mean was generated, and means of pseudo-values were then compared between the 2 clinical groups. Analyses were co-varied for non-verbal IQ (NVIQ), Mean length of utterance in morphemes (MLUM) and number of distinct word roots (NDR).
Statistically significant differences in mean of the pseudo-values between TD and ASD groups (Wilcoxon rank sum test, p = 0.007).
- TD subjects typically have a higher pseudo-value score suggesting that similarity scores of TD subjects are more similar to the overall group mean.
- Greater variance in the pseudo-values of the ASD group.
- Analysis of covariance suggests that none of NVIQ, MLUM, or NDR account for the difference between group means.
The findings suggest that NLP methods can be effectively used to identify specific semantic difficulties that characterize children with ASD. The method is automated and does not require costly and time-consuming annotation by expert clinicians. Our results are preliminary and need to be replicated in larger samples, and older age groups. Likewise, we would need to replicate our findings in language samples collected in more natural ecological settings. Future developments of NLP methods might help to provide fine-grained differentiation of types of semantic divergence in a conversation or locate in a conversation the time and context when divergence from topics occur. Future improvements of NLP methodology may also yield new, cost-effective and sensitive outcome measures applicable to treatment research. Furthermore, more precise documentation of semantic impairments in ASD may inform new intervention strategies in language therapy.