Arguing for Validity When Using a Small Scale Study to Determine the Content of a Montessori Teacher Credentialing Test Public Deposited

Downloadable Content

Download PDF
Last Modified
  • March 19, 2019
  • Talley, Diane
    • Affiliation: School of Education
  • Measuring competency of a Montessori teacher for the purpose of credentialing requires a deep understanding of the role of these teachers in the classroom and school environment. In the credentialing industry, large-scale studies are conducted in order to determine the essential tasks performed on the job and the critical knowledge, skills, and abilities required for competent performance. Literature suggests a strong reliance on a single method for determining content of a credentialing test (Raymond, 2001, 2016), but not all professions have the same characteristics or can support use of large-scale studies. This research investigated whether a content framework developed using a small-scale study that relies on the work of a carefully selected panel of experts provides adequate content-related validity evidence. Such evidence is essential to a high quality test that aligns with industry standards (AERA, APA, & NCME, 2014). The method selected was a three-phase process, referred to as a practice analysis study: 1) planning, 2) in-person meeting to develop the content framework, and 3) survey. Thirteen early childhood Montessori teachers, teacher trainers, and administrators in South Carolina were carefully selected to represent the profession in that state. These experts identified essential tasks of an entry-level teacher. A survey was then administered to both the panel and a broader sample of professionals in South Carolina to determine the importance of each task in the content framework. Results from the panel of experts were compared to those from the broader professional population using parametric and non-parametric statistical analysis. In addition, results were evaluated within a validity framework to determine the impact when relying on a small-scale study. Results of non-parametric analyses indicated agreement between the two groups for all but 2 of 19 tasks. Results of correlation analysis indicated overall moderate to strong agreement between groups. Selecting a small-scale study method using a panel-only practice analysis is supported based on the results, assuming the panel can be carefully selected to represent the diversity of the profession and stakes are low to moderate. There may be some concerns for using panel results alone when stakes are high, such as those for licensure testing.
Date of publication
Resource type
Rights statement
  • In Copyright
  • Johnson, Stephen
  • Ware, William
  • Buckendahl, Chad
  • Hughes, Sherick
  • O'Sullivan, Rita G.
  • Doctor of Philosophy
Degree granting institution
  • University of North Carolina at Chapel Hill Graduate School
Graduation year
  • 2016

This work has no parents.