Evaluating Student Social Behavior: An Initial Comparison of Systematic Direct Observation and Direct Teacher Behavior Ratings

Background:  Currently in the field of special education, federal mandates (IDEA, NCLB) not only promote, but also require, the implementation of evidence based practices (EBPs) for students with disabilities (Cook & Cook, 2011). In response to closing research to practice gaps, organizations such as What Works Clearinghouse seek to identify potential EBPs for specific populations. One specific area of education for which the need for clarity regarding appropriate EBPs has been increasingly felt, is for students with autism. As professionals seek to identify and implement strong manualized interventions targeting this population, it becomes increasingly necessary to effectively monitor student’s progress, providing implications for tailored dosage to meet individualized needs. Systematic direct observation (SDO) is a common method for ongoing assessment used to monitor behavior and associated intervention effects. While SDOs are generally considered the most appropriate and accepted means of measuring student behavior, SDO produces a number of limitations such as Hawthorn effects and are time intensive. A related measure, Direct Behavior Rating (DBR) is a scale that provides a direct rating of behavior(s) immediately following the observation of a student in regard to domains identified as significant moderators of student success. This measure is meant to be brief and immediate in nature, allowing for repeated use over time and implications to inform practice, similar to SDO. As EBP implementation increases, means of effectively and efficiently monitoring progress and informing practice in a feasible manner is vital.

Objectives:   The Social Competence Intervention for Adolescents (SCI-A) is a targeted EBP designed to meet the specific social needs (Stichter, et al., 2010) of youth with ASD or similar challenges. The current study seeks to examine the reliability of SDO and DBR as a potential efficient assessment of progress in relevant domains of student functioning in response to SCI-A intervention effects.

Methods:   This project will examine the agreement across five target behaviors through 15-minute systematic direct observations at 3 time points. Concurrently, classroom teachers rated the same target students (DBR). Student behaviors included active engagement, disruptiveness, respect, and appropriate social interactions with peers and teachers. This examination occurred for thirty- three students middle students as part of a larger RCT investigation of the efficacy of the (SCI) program. 

Results:   A review of overall correlations indicated a strong relationship between SDO and DBR for active engagement (AE), but smaller correlations for remaining pairings. DBR and OCF data were then disaggregated and compared within time point. Results indicated that the relationships between DBR and OCF scales were moderate to strong at baseline. Yet, correlations attenuated over time within each DBR-OCF pairing. This was with the exception of AE, which remained strong across all time points. 

Conclusions:  

A brief, immediate measure that informs practice is as essential as EBPs for students with ASD. Without an effective manner to progress monitor, the research to practice gap can’t close. The current N draws caution, nevertheless, suggests the DBR compares well to the SDO. Furthermore, teachers may be less responsive to ongoing social change.