30252
Concordance between Community Supervisor and Provider Ratings of Fidelity: Examination of Predictors and Outcomes
Fidelity is a key mechanism impacting treatment outcomes (McLeod et al., 2013), yet community examination is limited, partly due to limited feasible tools for community use (Schoenwald et al., 2011). Provider-report represents a feasible method for improving routine fidelity monitoring, but this method may be biased, with limited concordance with other fidelity measures (Hurlburt et al., 2010). The literature points to several provider and client factors known to impact both evidenced-based intervention implementation (Aarons et al., 2011) as well as interrater concordance (Dickson et al., 2018) but their impact on fidelity measurement is poorly understood. Further, data exploring the impact of concordance on provider’s actual implementation are limited.
Objectives:
The objectives of this study are to examine the impact of these factors on concordance, data were drawn from a train-the-trainer study evaluating the feasibility of an adapted fidelity tool for pivotal response training (PRT) developed for community use.
Methods:
Participants were five supervisors enrolled with corresponding therapist-child dyad participants. PRT intervention sessions (N=110; M=6.4 per therapist) were independently coded by supervisors and therapists using our PRT fidelity tool, which involved rating therapist fidelity using a 3-point Likert scale (1- limited/no use; 3-appropriate use). Trained observational coders coded recorded sessions, which served as the “research-standard” for fidelity comparison. Concordance was examined using percent of agreement in fidelity ratings. Predictors of concordance included: child cognitive and language level, autism symptom severity, therapist education and experience.
Results:
Results suggest variable but good concordance (Mean supervisor-coder = 61.11%; Mean therapist-coder =53.74%), with a trend towards supervisor-coder concordance being significantly higher than therapist-coder agreement (t(109)=1.97, p=.05). Child language level had a marginal impact on therapist-coder agreement. Therapist years’ experience also impacted on supervisor-coder and therapist-coder agreement. Therapist education also impacted therapist-coder agreement. Finally, supervisor-coder agreement predicted provider PRT implementation.
Conclusions:
Our results suggest that supervisors and therapists were able to learn to use our PRT fidelity tool, providing initial support for its feasibility. Therapist and child factors impacted the use of this tool and supervisor accuracy predicted therapist implementation, highlighting the importance of considering cross-level factors impacting the adoption and implementation of fidelity tools.