16437
An Autism Spectrum Disorder (ASD) Clinical Trial: Rater Training Program Overview and Initial Findings

Saturday, May 17, 2014
Atrium Ballroom (Marriott Marquis Atlanta)
L. Kingery1, P. Ventola2, X. Liogier D'ardhuy3, M. Deraët3, A. E. Veroff1, S. Zaragoza4, V. Krishna1, A. P. Nomikos1, A. Miner1, M. del Valle Rubido3 and O. Khwaja5, (1)Cogstate, New Haven, CT, (2)Child Study Center, Yale University, New Haven, CT, (3)Roche, Basel, Switzerland, (4)Cogstate, Barcelona, Spain, (5)Neurosciences, F.Hoffmann-La Roche AG, Basel, Switzerland
Background: Ensuring reliable and valid screening and outcome assessments in Autism Spectrum Disorders (ASD) trials is challenging; ASD is a behaviorally-defined disorder, measures are complex and diverse, including cognitive/neuropsychological tests, clinical global impression ratings, caregiver interviews, and caregiver- and patient-reported outcomes (PROs). A further challenge is many of these assessments require unique rater skills. For this phase II clinical trial involving 26 US sites, the customized rater training/assessment program included rater prequalification, five rater roles, didactic training for all scales, including expert consultants, required practice, and rater certification procedures assessing rater competency. Raters also completed a post-Investigators’ Meeting (IM) training experience survey.

Objectives:  Summarize key considerations for rater training programs in ASD, present information about site/rater experience in an ASD clinical trial, summarize relationships between pre-qualified rater experience and rater performance on certification assessments.

Methods: All potential raters completed a Rater Experience Survey (RES) of education, clinical experience with ASD, and scale-specific experience.  Rater certification data included scores on mock scoring exercises, quizzes, and practice administrations. Relationships among these variables were examined using analysis of variance and correlation techniques. A rater survey post-IM gathered structured and open-ended responses, which were summarized descriptively and qualitatively.

Results: Thus far, 125 potential raters completed a RES, of which 118 met pre-qualification criteria and entered the rater training/certification program. Pre-qualified raters’ educational background varied: 13 raters with bachelor degrees, 34 with master degrees, 29 with doctoral degrees (PhD, PsyD), 42 with medical degrees (MD). ASD clinical experience varied, ranging from 0 to 40 years (mean =9.56, SD=7.90). Scale-specific experience was also evaluated and varied from no experience to extensive experience, depending on the scale. Most raters performed well on training vignettes for a clinician-rated scale (Ohio Autism Clinical Impression Scale; OACIS) and a quiz on administration practices for the Vineland Adaptive Behavior Scale-II; VABS-II). Rater performance on certification materials for neuropsychological tests (Wechsler Abbreviated Scale of Intelligence; WASI-II, Repeatable Battery for the Assessment of Neuropsychological Status; RBANS) varied from no errors to many. Results of the rater survey after the IM indicated differences in the value raters see in hands-on training. Less experienced raters stated need for more individual time with the trainers; more experienced raters requested abbreviated training requirements.  However, in the hands-on training, several experienced raters made unexpected errors, validating the need for their participation. Preliminary analyses examining the relationship between ASD clinical experience, education, and performance on the training/certification materials for pre-qualified raters suggest no significant relationship among these variables. For one test, the WASI-II, previous experience with the scale was associated with fewer scoring errors on the rater assessment materials. In addition, we also observed that a small number of raters were responsible for a disproportionate number of total rater errors across all raters.

Conclusions: Even clinical trial raters who meet pre-qualification criteria are highly diverse in their clinical and scale-specific experience, and not always aware of the errors they make despite their experience. Comprehensive training and individual assessment are required to prepare raters for testing clinical trial study subjects.