Original article
Bias, prevalence and kappa

https://doi.org/10.1016/0895-4356(93)90018-VGet rights and content

Abstract

Since the introduction of Cohen's kappa as a chance-adjusted measure of agreement between two observers, several “paradoxes” in its interpretation have been pointed out. The difficulties occur because kappa not only measures agreement but is also affected in complex ways by the presence of bias between observers and by the distributions of data across the categories that are used (“prevalence”). In this paper, new indices that provide independent measures of bias and prevalence, as well as of observed agreement, are defined and a simple formula is derived that expresses kappa in terms of these three indices. When comparisons are made between agreement studies it can be misleading to report kappa values alone, and it is recommended that researchers also include quantitative indicators of bias and prevalence.

References (14)

  • D.V. Cicchetti et al.

    High agreement but low kappa: II. Resolving the paradoxes

    J Clin Epidemiol

    (1990)
  • W.D. Thompson et al.

    A reappraisal of the kappa coefficient

    J Clin Epidemiol

    (1988)
  • J.A. Cohen

    A coefficient of agreement for nominal scales

    Educ Psychol Meas

    (1960)
  • R. Zwick

    Another look at interrater agreement

    Psychol Bull

    (1988)
  • E.M. Bennett et al.

    Communications through limited response questioning

    Public Opinion Q

    (1954)
  • R.L. Brennan et al.

    Coefficient kappa: Some uses, misuses, and alternatives

    Educ Psychol Meas

    (1981)
  • A.E. Maxwell

    Coefficients of agreement between observers and their interpretation

    Br J Psychiatry

    (1977)
There are more references available in the full text version of this article.

Cited by (0)

View full text