Calculating Item Discrimination Values Using Samples of Examinee Scores Around Real and Anticipated Cut Scores: Effects on Item Discrimination, Item Selection, Examination Reliability, and Classification Decision Consistency Public Deposited

Downloadable Content

Download PDF
Last Modified
  • March 22, 2019
Creator
  • Earnest, Darin
    • Affiliation: School of Education
Abstract
  • This study examined the degree to which limiting the calculation of item discrimination values to groups of examinee scores near real and anticipated cut scores affected item discrimination, item selection, examination reliability, and classification decision consistency. Three examinations used to credential individuals in health-related professions were used to answer the research questions. To replicate as closely as possible the context in which many credentialing examinations are developed, each of the examinations consisted of small samples of examinees and were analyzed using classical test theory procedures. Item discrimination values, as expressed by the point-biserial statistic, were calculated for each examination item. Restricted item discrimination values were then calculated for each item using subsets of examinee scores. The restricted values were based on scores within 0.50 SD, 0.75 SD, and 1.00 SD of five unique cut score locations. Differences between unrestricted and restricted item discrimination values were measured. Two 50-item test variants for each examination were created to evaluate the effect restricted item discrimination values had on item selection, examination reliability, and classification decision consistency. Form A variants included the 50 most discriminating items using unrestricted discrimination values. Form B variants included the 50 most discriminating items using restricted discrimination values. The results of the study indicated that (a) item discrimination values were lower when their calculation was limited to groups of scores near cut scores; (b) using restricted item discrimination values as the criterion by which items were selected for test variants resulted in the selection of items that were different than those selected when unrestricted values were used as the selection criterion; (c) differences in examination reliability between test variants were found to be statistically significant, with scores of variants based on restricted item discrimination values producing lower estimates; and (d) test variants based on restricted item discrimination values produced slightly lower observed classification decision consistency estimates than variants based on unrestricted item discrimination values. The results of the study were tied to several aspects of the test development process for credentialing examinations, including issues related to sample size, cut score location, and examination validity.
Date of publication
Keyword
DOI
Resource type
Rights statement
  • In Copyright
Advisor
  • Cizek, Gregory J.
Degree
  • Doctor of Philosophy
Degree granting institution
  • University of North Carolina at Chapel Hill
Graduation year
  • 2014
Language
Publisher
Parents:

This work has no parents.

Items