27499
What Matters?: An Examination of Implementation Factors in the Effectiveness of Two Tier 2 School-Based Interventions
The intervention research field for children with neurodevelopmental disorders is increasingly moving beyond efficacy towards effectiveness, dissemination, and implementation trials. As this shift occurs, we must continually examine which implementation factors do – and do not - influence outcomes in order to consider the likelihood of broad dissemination and implementation.
Objectives:
This study presents the exploratory analyses of the effects of possible implementation factors on classroom-based outcomes from a comparative effectiveness research (CER) trial, factors that may be used for hypothesis generation for future studies.
Methods:
The CER trial compared two Tier 2 interventions for students with ASD or ADHD, Unstuck and On Target (UOT) and Contingency Behavior Management (CBM), in 21 Title I (low-income) elementary schools in three districts. Both interventions targeted executive functioning skills in the classroom. The primary outcome for these analyses are change in pre-post academic classroom observations on the following behaviors: demonstrates social reciprocity, follows rules, transitions appropriately, gets stuck, displays negativity, and participates in class (total raw scores ranging from 0 (worst) to 6 (best)). Family income ranged from $8,796-400,000. There were fewer than 30% White Non-Hispanic/Latino students in the study and only about 51% of the families spoke only English at home.
Results:
Diagnosis was an important moderator for this outcome, with ADHD students making significant treatment gains in both interventions, while ASD students only showed a positive effect in UOT. The following implementation factors did not have a significant impact on effectiveness (whether separated by intervention, diagnosis or when combined): Treatment fidelity, number of intervention sessions completed, profession of the school-based interventionist, and amount of knowledge that the parent gained.
Some important acceptability factors were significant, though. Students reported that they enjoyed UOT more than CBM (t=2.018,df=128,p=.046), though there was no significant relationship between how much they enjoyed the group and how much they improved within intervention group. Parents also reported that they felt that UOT helped their child more (t=2.767,df=117,p=.007), they were more satisfied with UOT (t=3.015,df=116,p=.003) and are more likely to use UOT techniques in the future (t=2.055,df=90,p=.043). Improvements measured by masked classroom observations were significantly correlated with parents’ report of how much they felt the interventions helped their child (r=.267,p=.005,N=109) and how satisfied they were with the intervention (r=.336,p<.001,N=108). Qualitative analyses of student and parent responses will also be presented.
Conclusions:
The lack of relationship between most standard implementation factors, such as fidelity, training of interventionist, and completion of the intervention, and outcome is unexpected and difficult to explain. First, it is possible that the participating schools delivered both interventions with exceptional skill, though the fidelity and completion rates ranged considerably. It may also be that these implementation factors do not matter as much in Tier 2 (middle intensity, group-based) as in Tier 3 (High-intensity, individualized, and more frequently studied) interventions designed to be delivered in schools to students experiencing disparities. Finally, it could be that providing any explicit training of EF skills can have positive effects for these at-risk students.