Observer agreement paradoxes in 2x2 tables: comparison of agreement measures

Public Deposited
Analytics

Downloadable Content

Download PDF
Request Version for Screen Reader
Creator
  • Shankar, Viswanathan
    • Other Affiliation: Division of Biostatistics, Department of Epidemiology and Population Health, Albert Einstein College of Medicine, Bronx, NY 10461, USA
  • Bangdiwala, Shrikant
    • Affiliation: Gillings School of Global Public Health, Department of Biostatistics
Abstract
  • Abstract Background Various measures of observer agreement have been proposed for 2x2 tables. We examine the behavior of alternative measures of observer agreement for 2x2 tables. Methods The alternative measures of observer agreement and the corresponding agreement chart were calculated under various scenarios of marginal distributions (symmetrical or not, balanced or not) and of degree of diagonal agreement, and their behaviors are compared. Specifically, two specific paradoxes previously identified for kappa were examined: (1) low kappa values despite high observed agreement under highly symmetrically imbalanced marginals, and (2) higher kappa values for asymmetrical imbalanced marginal distributions. Results Kappa and alpha behave similarly and are affected by the marginal distributions more so than the B-statistic, AC1-index and delta measures. Delta and kappa provide values that are similar when the marginal totals are asymmetrically imbalanced or symmetrical but not excessively imbalanced. The AC1-index and B-statistics provide closer results when the marginal distributions are symmetrically imbalanced and the observed agreement is greater than 50%. Also, the B-statistic and the AC1-index provide values closer to the observed agreement when the subjects are classified mostly in one of the diagonal cells. Finally, the B-statistic is seen to be consistent and more stable than kappa under both types of paradoxes studied. Conclusions The B-statistic behaved better under all scenarios studied as well as with varying prevalences, sensitivities and specificities than the other measures, we recommend using B-statistic along with its corresponding agreement chart as an alternative to kappa when assessing agreement in 2x2 tables.
Date of publication
DOI
Identifier
Resource type
  • Article
Rights statement
  • In Copyright
Rights holder
  • Viswanathan Shankar et al.; licensee BioMed Central Ltd.
License
Journal title
  • BMC Medical Research Methodology
Journal volume
  • 14
Journal issue
  • 1
Page start
  • 100
Language
  • English
Is the article or chapter peer-reviewed?
  • Yes
ISSN
  • 1471-2288
Bibliographic citation
  • BMC Medical Research Methodology. 2014 Aug 28;14(1):100
Publisher
  • BioMed Central Ltd
Access right
  • Open Access
Date uploaded
  • July 15, 2016

Relations

Parents:

This work has no parents.

Items

Thumbnail Title Date Uploaded Visibility Actions
file details 12874_2014_article_1117.pdf 2019-05-06 Public Download
file details 12874_2014_1117_moesm6_esm.tif 2019-05-06 Public Download
file details 12874_2014_1117_moesm4_esm.tif 2019-05-06 Public Download
file details 12874_2014_1117_moesm5_esm.tif 2019-05-06 Public Download
file details 12874_2014_1117_moesm1_esm.pdf 2019-05-06 Public Download
file details 12874_2014_1117_moesm2_esm.tif 2019-05-06 Public Download
file details 12874_2014_1117_moesm3_esm.tif 2019-05-06 Public Download