Skip to main content
SpringerLink
Log in

Initial Development of Multi-item Direct Behavior Rating Measures of Academic Enablers

  • Original Paper
  • Published:
School Mental Health Aims and scope Submit manuscript

Abstract

Although there is a growing body of evidence to support the use of direct behavior rating (DBR) as a formative behavioral assessment method in school-based problem-solving models, this work has centered largely on the assessment of attending and problem behaviors and on the use of single-item DBR scales. The primary purpose of the current study was to report on the development and initial validation of teacher-completed multi-item DBR scales (DBR-MIS) designed to expand upon prior DBR research to assess three constructs representing behaviors that are widely considered to support student academic achievement (academic engagement, interpersonal skills, and study skills), only one of which (academic engagement) has been the target of DBR assessment methods. Development of the scales involved a content validity study utilizing feedback from a panel of consumers (teachers, parents, and school administrators) and a panel of researchers with expertise in school-based behavioral assessment. Results of an exploratory factor analysis of ratings of students completed by teachers in Grades K through 3 (N = 307) supported a one-factor solution for each of the aforementioned constructs. Favorable internal consistency was found for each scale.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Anthony, C. J., & DiPerna, J. C. (2017). Identifying sets of maximally efficient items from the Academic Competence Evaluation Scales—Teacher Form. School Psychology Quarterly,32, 552–559. https://doi.org/10.1037/spq0000205.

    Article  PubMed  Google Scholar 

  • Anthony, C. J., & DiPerna, J. C. (2018). Piloting a short form of the academic competence evaluation scales. School Mental Health,10, 1–8. https://doi.org/10.1007/s12310-018-9254-7.

    Article  Google Scholar 

  • Atkins, M., Hoagwood, K., Kutash, K., & Seidman, E. (2010). Toward the integration of education and mental health in schools. Administration and Policy in Mental Health,37, 40–47. https://doi.org/10.1007/s10488-010-0299-7.

    Article  PubMed  PubMed Central  Google Scholar 

  • Barkley, R. A. (Ed.). (2015). Attention-deficit hyperactivity disorder: A handbook for diagnosis and treatment (4th ed.). New York, NY: The Guilford Press.

    Google Scholar 

  • Briesch, A. M., Chafouleas, S. M., & Riley-Tillman, T. C. (2010). Generalizability and dependability of behavior assessment methods to estimate academic engagement: A comparison of systematic direct observation and direct behavior rating. School Psychology Review,39, 408–421.

    Google Scholar 

  • Briesch, A. M., Chafouleas, S. M., & Riley-Tillman, T. C. (2016). Direct Behavior Rating (DBR): Linking assessment, communication, and intervention. New York, NY: Guilford Press.

    Google Scholar 

  • Briesch, A. M., Ferguson, T. D., Volpe, R. J., & Briesch, J. M. (2013). Examining teachers’ perceptions of social-emotional and behavioral referral concerns. Remedial and Special Education,34, 249–256.

    Article  Google Scholar 

  • Briesch, A. M., & Volpe, R. J. (2007). Important considerations in the selection of progress monitoring measures for classroom behaviors. School Psychology Forum,1, 59–74.

    Google Scholar 

  • Briesch, A. M., Volpe, R. J., & Ferguson, T. D. (2014). The influence of student characteristics on the dependability of observation data. School Psychology Quarterly, 29, 171–181.

    Article  PubMed  Google Scholar 

  • Chafouleas, S. M. (2011). Direct Behavior Rating: A review of the issues and research in its development. Education and Treatment of Children,34(4), 575–591.

    Article  Google Scholar 

  • Chafouleas, S. M., Briesch, A. M., Riley-Tillman, T. C., Christ, T. C., Black, A. C., & Kilgus, S. P. (2010a). An investigation of the generalizability and dependability of Direct Behavior Rating Single Item Scales (DBR-SIS) to measure academic engagement and disruptive behavior of middle school students. Journal of School Psychology,48, 219–246. https://doi.org/10.1016/j.jsp.2010.02.001.

    Article  PubMed  Google Scholar 

  • Chafouleas, S. M., Christ, T. J., Riley-Tillman, T. C., Briesch, A. M., & Chanese, J. A. (2007a). Generalizability and dependability of direct behavior ratings to assess social behavior of preschoolers. School Psychology Review,36, 63–79.

    Google Scholar 

  • Chafouleas, S. M., Riley-Tillman, T. C., & Christ, T. J. (2009). Direct behavior rating (DBR) an emerging method for assessing social behavior within a tiered intervention system. Assessment for Effective Intervention,34, 195–200.

    Article  Google Scholar 

  • Chafouleas, S., Riley-Tillman, T. C., & Sugai, G. M. (2007b). School-based behavioral assessment: Informing intervention and instruction. New York, NY: Guilford Press.

    Google Scholar 

  • Chafouleas, S. M., Sanetti, L. M. H., Kilgus, S. P., & Maggin, D. M. (2012). Evaluating sensitivity to behavioral change across consultation cases using Direct Behavior Rating Single-Item Scales (DBR-SIS). Exceptional Children, 78, 491–505.

    Article  Google Scholar 

  • Chafouleas, S. M., Volpe, R. J., Gresham, F. M., & Cook, C. R. (2010b). School-based behavioral assessment within problem-solving models: Current status and future directions. School Psychology Review,39, 343–349.

    Google Scholar 

  • Christ, T. J., Riley-Tillman, T. C., & Chafouleas, S. M. (2009). Foundation for the development and use of Direct Behavior Rating (DBR) to assess and evaluate student behavior. Assessment for Effective Intervention,34, 201–213.

    Article  Google Scholar 

  • Comrey, A. L., & Lee, H. B. (1992). A first course in factor analysis (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • DiPerna, J. C., & Elliott, S. N. (1999). The development and validation of the Academic Competence Evaluation Scales. Journal of Psychoeducational Assessment,17, 207–225. https://doi.org/10.1177/073428299901700302.

    Article  Google Scholar 

  • DiPerna, J. C., & Elliott, S. N. (2000). Academic competence evaluation scales. San Antonio, TX: The Psychological Corporation.

    Google Scholar 

  • DiPerna, J. C., & Elliott, S. N. (2002). Promoting academic enablers to improve student achievement: An introduction to the miniseries. School Psychology Review,31, 293–297.

    Google Scholar 

  • DiPerna, J. C., Volpe, R. J., & Elliott, S. N. (2002). A model of academic enablers and elementary reading/language arts achievement. School Psychology Review,31, 298–312.

    Google Scholar 

  • DiPerna, J. C., Volpe, R. J., & Elliott, S. N. (2005). A model of academic enablers and mathematics achievement in the elementary grades. Journal of School Psychology,43, 379–392. https://doi.org/10.1016/j.jsp.2005.09.002.

    Article  Google Scholar 

  • Every Student Succeeds Act (ESSA) of 2015, Pub. L. 114-95, § 114.

  • Fabiano, G. A., Pyle, K., Kelty, M. B., & Parham, B. R. (2017). Progress monitoring using direct behavior rating single item scales in a multiple baseline design study of the daily report card intervention. Assessment for Effective Intervention, 43, 21–33.

    Article  Google Scholar 

  • Fabiano, G. A., Vujnovic, R., Pelham, W. E., Waschbusch, D. A., Massetti, G. M., Yu, J., et al. (2010). Enhancing the effectiveness of special education programming for children with ADHD using a daily report card. School Psychology Review,39, 219–239.

    Google Scholar 

  • Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods,4, 272–299. https://doi.org/10.1037/1082-989X.4.3.272.

    Article  Google Scholar 

  • Ferguson, T. D., Briesch, A. M., Volpe, R. J., & Daniels, B. (2012). The influence of observation length on the dependability of data. School Psychology Quarterly,27, 187–197. https://doi.org/10.1037/spq0000005.

    Article  Google Scholar 

  • Gettinger, M., & Seibert, J. K. (2002). Contributions of study skills to academic competence. School Psychology Review,31, 350–365.

    Google Scholar 

  • Greenwood, C. R. (1996). The case for performance-based instructional models. School Psychology Quarterly,11, 283–296. https://doi.org/10.1037/h0088935.

    Article  Google Scholar 

  • Gresham, F. M., Cook, C. R., Collins, T., Dart, E., Rasetshwane, K., Truelson, E., et al. (2010). Developing a change-sensitive brief behavior rating scale as a progress monitoring tool for social behavior: An example using the Social Skills Rating System—Teacher Form. School Psychology Review,39, 364–379.

    Google Scholar 

  • Harvey, V. S., & Chickie-Wolfe, L. A. (2007). Fostering independent learning: Practical strategies to promote student success. New York, NY: Guilford Press.

    Google Scholar 

  • Henricsson, L., & Rydell, A. (2006). Children with behaviour problems: The influence of social competence and social relations on problem stability, school achievement and peer acceptance across the first six years of school. Infant and Child Development,15, 347–366. https://doi.org/10.1002/icd.448.

    Article  Google Scholar 

  • Hintze, J. M., & Matthews, W. J. (2004). The generalizability of systematic direct observations across time and setting: A preliminary investigation of the psychometrics of behavioral observation. School Psychology Review,33, 258–270.

    Google Scholar 

  • Horn, J. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika,30, 179–185.

    Article  PubMed  Google Scholar 

  • Individuals with Disabilities Education Act (IDEA) of 2004, Pub. L. 108-446. 118 U.S.C. § 2647.

  • Jenkins, L. N., & Demaray, M. K. (2015). An investigation of relations among academic enablers and reading outcomes. Psychology in the Schools,52, 379–389. https://doi.org/10.1002/pits.21830.

    Article  Google Scholar 

  • Malecki, C. K., & Elliott, S. N. (2002). Children’s social behaviors as predictors of academic achievement: A longitudinal analysis. School Psychology Quarterly,17, 1–23. https://doi.org/10.1521/scpq.17.1.1.19902.

    Article  Google Scholar 

  • Massetti, G. M., Lahey, B. B., Pelham, W. E., Loney, J., Ehrhardt, A., Lee, S. S., et al. (2008). Academic achievement over 8 years among children who met modified criteria for attention-deficit/hyperactivity disorder at 4–6 years of age. Journal of Abnormal Child Psychology,36, 399–410. https://doi.org/10.1007/s10802-007-9186-4.

    Article  PubMed  Google Scholar 

  • McDonald, R. P. (2013). Test theory: A unified treatment. Mahwah, NJ: Erlbaum.

    Book  Google Scholar 

  • Meade, A. W., & Bauer, D. J. (2007). Power and precision in confirmatory factor analytic tests of measurement invariance. Structural Equation Modeling,14, 611–635. https://doi.org/10.1080/10705510701575461.

    Article  Google Scholar 

  • National Center on Intensive Intervention. (2018). Behavioral progress monitoring tools. Retrieved June 26, 2018, from https://charts.intensiveintervention.org/chart/behavioral-progress-monitoring-tools.

  • National Center on Response to Intervention. (2010). Essential components of RTI-A closer look at response to intervention. Washington, DC: U.S. Department of Education, Office of Special Education Programs.

    Google Scholar 

  • Osborne, J. W., & Costello, A. B. (2009). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Pan-Pacific Management Review,12, 131–146.

    Google Scholar 

  • Owens, J. S., Holdaway, A. S., Zoromski, A. K., Evans, S. W., Himawan, L. K., Girio-Herrera, E., et al. (2012). Incremental benefits of a daily report card intervention over time for youth with disruptive behavior. Behavior Therapy,43, 848–861. https://doi.org/10.1016/j.beth.2012.02.002.

    Article  PubMed  Google Scholar 

  • Pelham, W. E., Jr., Fabiano, G. A., & Massetti, G. M. (2005). Evidence-based assessment of attention deficit hyperactivity disorder in children and adolescents. Journal of Clinical Child and Adolescent Psychology,34, 449–476. https://doi.org/10.1207/s15374424jccp3403_5.

    Article  PubMed  Google Scholar 

  • Rabiner, D., & Coie, J. D. (2000). Early attention problems and children’s reading achievement: A longitudinal investigation. Journal of the American Academy of Child and Adolescent Psychiatry,39, 859–867. https://doi.org/10.1097/00004583-200007000-00014.

    Article  PubMed  PubMed Central  Google Scholar 

  • Reyes, M. R., Brackett, M. A., Rivers, S. E., White, M., & Salovey, P. (2012). Classroom emotional climate, student engagement, and academic achievement. Journal of Educational Psychology,104, 700–712. https://doi.org/10.1037/a0027268.

    Article  Google Scholar 

  • Reynolds, C. R., & Kamphaus, R. W. (2015). BASC-3 flex monitor. Minneapolis, MN: NCS Pearson.

    Google Scholar 

  • Reynolds, A. J., & Walberg, H. J. (1992). A process model of mathematics achievement and attitude. Journal for Research in Mathematics Education,23, 306–328. https://doi.org/10.2307/749308.

    Article  Google Scholar 

  • Riley-Tillman, T. C., Chafouleas, S. M., Sassu, K. A., Chanese, J. A. M., & Glazer, A. D. (2008). Examining the agreement of Direct Behavior Ratings and Systematic Direct Observation for on-task and disruptive behavior. Journal of Positive Behavior Interventions,10, 136–143. https://doi.org/10.1177/1098300707312542.

    Article  Google Scholar 

  • Shapiro, E. S., & Heick, P. F. (2004). School psychologist assessment practices in the evaluation of students referred for social/behavioral/emotional problems. Psychology in the Schools,41, 551–561. https://doi.org/10.1002/pits.10176.

    Article  Google Scholar 

  • Spectrum K12 School Solutions (2011). Response to intervention adoption survey 2011. Retrieved June 26, 2018, from http://www.globalscholar.com/2011RTI.

  • Stevens, J. (1992). Applied multivariate statistics for the social sciences. Hillsdale, NJ: Erlbaum.

    Google Scholar 

  • Sugai, G., & Horner, R. H. (2009). Responsiveness-to-intervention and school-wide positive behavior supports: Integration of multi-tiered system approaches. Exceptionality,17, 223–237. https://doi.org/10.1080/09362830903235375.

    Article  Google Scholar 

  • Tabachnick, B. G., & Fidell, L. S. (2007). Using multivariate statistics (5th ed.). New York: Pearson Education Inc.

    Google Scholar 

  • Tilly, D. (2008). The evolution of school psychology to science based practice. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology, V (pp. 17–36). Bethesda, MD: National Association of School Psychologists.

    Google Scholar 

  • U.S. Department of Education. (2018). National Center for Education Statistics, Common Core of Data (CCD), “Public Elementary/Secondary School Universe Survey,” 2017-2018. Washington, DC: National Center for Education Statistics.

  • Velicer, W. F. (1976). Determining the number of components from the matrix of partial correlations. Psychometrika,41, 321–327.

    Article  Google Scholar 

  • Velicer, W. F., Eaton, C. A., & Fava, J. L. (2000). Construct explication through factor or component analysis: A review and evaluation of alternative procedures for determining the number of factors or components. In R. D. Goffin & E. Helmes (Eds.), Problems and solutions in human assessment (pp. 41–71). Boston, MA: Springer.

    Chapter  Google Scholar 

  • Volpe, R. J., & Briesch, A. M. (2012). Generalizability and dependability of single-item and multiple-item direct behavior rating scales for engagement and disruptive behavior. School Psychology Review,41, 246–261.

    Google Scholar 

  • Volpe, R. J., & Briesch, A. M. (2015). Multi-item Direct Behavior Ratings: Dependability of two levels of assessment specificity. School Psychology Quarterly,30, 431–442. https://doi.org/10.1037/spq0000115.

    Article  PubMed  Google Scholar 

  • Volpe, R. J., DiPerna, J. C., Hintze, J. M., & Shapiro, E. S. (2005). Observing students in classroom settings: A review of seven coding schemes. School Psychology Review,34, 454–474.

    Google Scholar 

  • Volpe, R. J., DuPaul, G. J., DiPerna, J. C., & Jitendra, A. K. (2006). Attention deficit hyperactivity disorder and scholastic achievement: A model of mediation via academic enablers. School Psychology Review,35, 47–61.

    Google Scholar 

  • Volpe, R. J., & Fabiano, G. A. (2013). Daily behavior report cards: An evidence-based system of assessment and intervention. New York, NY: The Guilford Press.

    Google Scholar 

  • Volpe, R. J., & Gadow, K. D. (2010). Creating abbreviated rating scales to monitor classroom inattention-overactivity, aggression, and peer conflict: Reliability, validity, and treatment sensitivity. School Psychology Review,39, 350–363.

    Google Scholar 

  • Volpe, R. J., Gadow, K. D., Blom-Hoffman, J., & Feinberg, A. B. (2009a). Factor-analytic and individualized approaches to constructing brief measures of ADHD behaviors. Journal of Emotional and Behavioral Disorders,17, 118–128. https://doi.org/10.1177/1063426608323370.

    Article  Google Scholar 

  • Volpe, R. J., McConaughy, S. H., & Hintze, J. M. (2009b). Generalizability of classroom behavior problem and on-task scores from the Direct Observation Form. School Psychology Review,38, 382–401.

    Google Scholar 

  • Wentzel, K. R. (1993). Does being good make the grade? Social behavior and academic competence in middle school. Journal of Educational Psychology, 85, 357–364.

    Article  Google Scholar 

  • Wolf, M. M. (1978). Social validity: The case for subjective measurement of how behavior analysis is finding its heart. Journal of Applied Behavior Analysis,11, 203–214. https://doi.org/10.1901/jaba.1978.11-203.

    Article  PubMed  PubMed Central  Google Scholar 

  • Wood, B. K., Hojnoski, R. L., Laracy, S. D., & Olson, C. L. (2015). Comparison of observational methods and their relation to ratings of engagement in young children. Topics in Early Childhood Special Education,35, 211–222. https://doi.org/10.1177/0271121414565911.

    Article  Google Scholar 

  • Zirkel, P. A. (2011). State laws and guidelines for RTI: Additional implementation features. Communiqué,39(7), 30–32.

    Google Scholar 

Download references

Funding

Preparation of this article was supported by funding provided by the Institute of Education Sciences, US Department of Education (R324A150071). Opinions expressed herein do not necessarily reflect the position of the US Department of Education, and such endorsements should not be inferred.

Author information

Authors and Affiliations

  1. Department of Applied Psychology, Northeastern University, 404 International Village, Boston, MA, 02115, USA

    Robert J. Volpe, Ruth K. Chaffee, Tat Shing Yeung & Amy M. Briesch

Authors
  1. Robert J. Volpe
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ruth K. Chaffee
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tat Shing Yeung
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Amy M. Briesch
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Robert J. Volpe.

Ethics declarations

Conflict of interest

The authors declare that they have no conflicts of interest.

Ethical Approval

All procedures performed were in accordance with the ethical standards of the institutional research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. The study was deemed exempt from obtaining signed written consent from participants in accordance with HHS regulations at 45 CFR 46.117(c).

Human and Animal Rights

This article does not contain any studies with animals performed by any of the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Volpe, R.J., Chaffee, R.K., Yeung, T.S. et al. Initial Development of Multi-item Direct Behavior Rating Measures of Academic Enablers. School Mental Health 12, 77–87 (2020). https://doi.org/10.1007/s12310-019-09338-w

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12310-019-09338-w

Keywords

Search

Navigation