An Autism Spectrum Disorder (ASD) Clinical Trial: Rater Training Program Overview and Initial Findings

Background: Ensuring reliable and valid screening and outcome assessments in Autism Spectrum Disorders (ASD) trials is challenging; ASD is a behaviorally-defined disorder, measures are complex and diverse, including cognitive/neuropsychological tests, clinical global impression ratings, caregiver interviews, and caregiver- and patient-reported outcomes (PROs). A further challenge is many of these assessments require unique rater skills. For this phase II clinical trial involving 26 US sites, the customized rater training/assessment program included rater prequalification, five rater roles, didactic training for all scales, including expert consultants, required practice, and rater certification procedures assessing rater competency. Raters also completed a post-Investigators’ Meeting (IM) training experience survey.

Objectives:  Summarize key considerations for rater training programs in ASD, present information about site/rater experience in an ASD clinical trial, summarize relationships between pre-qualified rater experience and rater performance on certification assessments.

Methods: All potential raters completed a Rater Experience Survey (RES) of education, clinical experience with ASD, and scale-specific experience.  Rater certification data included scores on mock scoring exercises, quizzes, and practice administrations. Relationships among these variables were examined using analysis of variance and correlation techniques. A rater survey post-IM gathered structured and open-ended responses, which were summarized descriptively and qualitatively.

Results: Thus far, 125 potential raters completed a RES, of which 118 met pre-qualification criteria and entered the rater training/certification program. Pre-qualified raters’ educational background varied: 13 raters with bachelor degrees, 34 with master degrees, 29 with doctoral degrees (PhD, PsyD), 42 with medical degrees (MD). ASD clinical experience varied, ranging from 0 to 40 years (mean =9.56, SD=7.90). Scale-specific experience was also evaluated and varied from no experience to extensive experience, depending on the scale. Most raters performed well on training vignettes for a clinician-rated scale (Ohio Autism Clinical Impression Scale; OACIS) and a quiz on administration practices for the Vineland Adaptive Behavior Scale-II; VABS-II). Rater performance on certification materials for neuropsychological tests (Wechsler Abbreviated Scale of Intelligence; WASI-II, Repeatable Battery for the Assessment of Neuropsychological Status; RBANS) varied from no errors to many. Results of the rater survey after the IM indicated differences in the value raters see in hands-on training. Less experienced raters stated need for more individual time with the trainers; more experienced raters requested abbreviated training requirements.  However, in the hands-on training, several experienced raters made unexpected errors, validating the need for their participation. Preliminary analyses examining the relationship between ASD clinical experience, education, and performance on the training/certification materials for pre-qualified raters suggest no significant relationship among these variables. For one test, the WASI-II, previous experience with the scale was associated with fewer scoring errors on the rater assessment materials. In addition, we also observed that a small number of raters were responsible for a disproportionate number of total rater errors across all raters.

Conclusions: Even clinical trial raters who meet pre-qualification criteria are highly diverse in their clinical and scale-specific experience, and not always aware of the errors they make despite their experience. Comprehensive training and individual assessment are required to prepare raters for testing clinical trial study subjects.