Apologies for non-SPLUS related post.
>From: SoSoSanta@aol.com [SMTP:SoSoSanta@aol.com]
>I'm using Kappa values to look at interrater reliability for a
psychological
>tool and I am having a few problems.
>The tool uses yes/no questions and thus I have problems with Kappa scores
not
>reflecting very good levels of agreement when many of the rating scores
are
>Yes and there are few no's. Any suggestions on playing around with the
>values? I thought of keeping the yes/no ratio the same and just getting
the
>total yes and no scores to be about 50% each of all the scores.This seems
to
>get a more realistic Kappa value, but I'm not sure if it is allowed?
The dependence of Cohen's kappa on prevalence has been noted. There is no
easy solution. There is too little data in a 2x2 table to estimate
chance-correct agreement (kappa). So you have to make assumptions, which
may be plausible depending on circumstances. If the assumptions are
plausible, just use Cohen's kappa.
Two articles discuss kappa-type measures in general (1,2). IMHO, they're
required reading for anybody using kappa. Article (3) summarizes some of
the discussion about kappa and prevalence. There is a large body of work
(4-6) about prevalence and kappa which are, IMHO, made significantly less
interesting by articles 1 and 2:
Regarding "equalizing" prevalences:
We tried this (7) for purposes of comparing subgroups (smokers vs exsmokers
vs neversmokers) but I'm not sure how valid it would be to equalize
prevalences between different questions. Also, you run into sample size
problems, especially for rare prevalences when the problem is most acute.
If it were possible, you could construct a scale out of your yes-no
questions - then you'd have more degrees of freedom and could estimate more
parameters.
Yours,
Jan Brogger
PhD student, Respiratory Epidemiology Group, University of Bergen, Norway
(1) Guggenmoos-Holzmann, I. (1996). "The meaning of kappa: probabilistic
concepts of reliability and validity revisited." J Clin Epidemiol 49(7):
775-782.
(2) Guggenmoos-Holzmann, I. and R. Vonk (1998). "Kappa-like indices of
observer
agreement viewed from a latent class perspective." Stat Med 17(8 Apr 30):
797-812.
(3) Cook RJ. Kappa. In: Armitage P, Colton P, editors. Encyclopedia of
biostatistics. John Wiley & Sons Ltd, Chichester; 1998. p. 2160-2169.
(4) Byrt T, Bishop J, Carlin JB. Bias, prevalence and kappa. J Clin
Epidemiol 1993; 46(5):423-429.
Feinstein AR, Cicchetti DV. High agreement but low kappa: I. The problems
of two paradoxes. J Clin Epidemiol 1990; 43(6):543-549.
(5) Cicchetti DV, Feinstein AR. High agreement but low kappa: II.
Resolving the paradoxes. J Clin Epidemiol 1990; 43(6):551-558.
(6) Lantz CA, Nebenzahl E. Behavior and interpretation of the kappa
statistic: resolution of the two paradoxes. J Clin Epidemiol 1996;
49(4):431-434.
(7) Brogger J, Bakke PS, Gulsvik A. Comparison of two respiratory symptoms
questionnaries. Int J Tuberc Lung Dis 2000; 4(1):83-90.
|