Multilevel validity: assessing the validity of school-level inferences from student achievement test data

Public Deposited
Analytics

Downloadable Content

Download PDF
Request Version for Screen Reader
Last Modified
  • March 21, 2019
Creator
  • Rosenberg, Sharyn L.
    • Affiliation: School of Education
Abstract
  • Psychometric theory is clear about the central role of validity and the importance of gathering evidence for a particular purpose. State achievement tests are generally developed with ample validity evidence for their intended inferences about student achievement. Such evidence may not be sufficient for drawing group-level inferences, a crucial point that is often ignored when student achievement scores are used in multilevel analyses to study effects at the school level. This study explores the process of gathering multilevel validity evidence necessary to make school-level inferences from student achievement tests. Using data from approximately 28,000 students in grades 3, 5, and 8 in a northeastern U.S. state, this study examined the multilevel factor structure of mathematics achievement tests. Multilevel exploratory factor analyses were used to determine the optimal number of factors at both the student and the school levels of analysis. Multilevel confirmatory factor analyses were used to assess the extent to which the one-factor solutions on each level were feasible. Both standard (single level) confirmatory factor analyses and multilevel confirmatory factor analyses were used to compare the size and relative importance of factor loadings at the different levels of analysis. The statistical significance of the school-level factor loadings provided evidence about the extent to which the mathematics achievement test items were effective for discriminating between schools. For each of the three grades studied, there was only one meaningful factor identified (presumably mathematics achievement) at both the student and school levels of analysis. At each grade level, items differed in terms of both their absolute and relative size of their factor loadings at the student and school levels of analysis, suggesting that when school-level inferences are of interest, standard factor analyses provide insufficient information about test development and validation. The majority of items in this study were more discriminating at the school level than at the student level. Interpretations of these findings are discussed in the context of relevant research on validity and student achievement. Implications for educational measurement and ideas for future research are also addressed.
Date of publication
DOI
Resource type
Rights statement
  • In Copyright
Advisor
  • Cizek, Gregory J.
Degree granting institution
  • University of North Carolina at Chapel Hill
Language
Access right
  • Open access
Date uploaded
  • October 11, 2010

Relations

Parents:

This work has no parents.

Items

Thumbnail Title Date Uploaded Visibility Actions
file details Multilevel validity : assessing the validity of school-level inferences from student achievement test data 2019-04-08 Public Download