Fleiss' kappa

Fleiss' kappa is a variant of Cohen's kappa, a statistical measure of inter-rater reliability. Where Cohen's kappa works for only two raters, Fleiss' kappa works for any constant number of raters giving categorical ratings (see nominal data), to a fixed number of items. It is a measure of the degree of agreement that can be expected above chance. Agreement can be thought of as follows, if a fixed number of people assign numerical ratings to a number of items then the kappa will give a measure for how consistent the ratings are. The kappa, Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \kappa\,} , can be defined as

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \kappa = \frac{\bar{P} - \bar{P_e}}{1 - \bar{P_e}}}

The factor Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle 1 - \bar{P_e}} gives the degree of agreement that is attainable above chance, and Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \bar{P} - \bar{P_e}} gives the degree of agreement actually achieved above chance. The statistic takes values between 0 and 1, where a value of 1 means complete agreement.

An example of the use of Fleiss' kappa may be the following: Consider fourteen psychiatrists are asked to look at ten patients. Each gives one of possibly five diagnoses to each patient. The Fleiss' kappa can be computed from the data matrix (see example below) to show the degree of agreement between the psychiatrists above the level of agreement expected by chance. Fleiss' kappa has benefits over the standard Cohen's kappa as it works for multiple raters, and it is an improvement over a simple percentage agreement calculation as it takes into account the amount of agreement that can be expected by chance.

Definition

Let N be the number of subjects, n the number of ratings per subject, and k the number of categories into which assignments are made. The subjects are indexed by i = 1, ... N and the categories are indexed by j = 1, ... k. Let n_ij, represent the number of raters who assigned the i-th subject to the j-th category.

First calculate p_j, the proportion of all assignments which were to the j-th category:

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle p_{j} = \frac{1}{N n} \sum_{i=1}^N n_{i j} }

Now calculate Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle P_{i}\,} , the extent to which raters agree for the i-th subject:

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle P_{i} = \frac{1}{n(n - 1)} \sum_{j=1}^k n_{i j} (n_{i j} - 1)}

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle = \frac{1}{n(n - 1)} \left(\sum_{j=1}^k n_{i j}^2 - n_{i j}\right)}

Now compute Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \bar{P}} , the mean of the Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle P_i\,} 's, and Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \bar{P_e}} which go into the formula for Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \kappa\,} :

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \bar{P} = \frac{1}{N} \sum_{i=1}^N P_{i}}

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle = \frac{1}{N n (n - 1)} \left(\sum_{i=1}^N \sum_{j=1}^k n_{i j}^2 - N n\right)}

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \bar{P_e} = \sum_{j=1}^k p_{j} ^2}

Worked example

Table of values for computing the worked example

	1	2	3	4	5	Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle P_i\,}
1	0	0	0	0	14	1.000
2	0	2	6	4	2	0.253
3	0	0	3	5	6	0.308
4	0	3	9	2	0	0.440
5	2	2	8	1	1	0.330
6	7	7	0	0	0	0.462
7	3	2	6	3	0	0.242
8	2	5	3	2	2	0.176
9	6	5	2	1	0	0.286
10	0	2	2	3	7	0.286
Total	20	28	39	21	32
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle p_j\,}	0.143	0.200	0.279	0.150	0.229

In the following example, fourteen raters (Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle n} ) assign ten subjects (Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle N} ) to a total of five categories (Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle k} ). The categories are presented in the columns, while the subjects are presented in the rows.

Data

See table to the right.

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle N} = 10, Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle n} = 14, Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle k} = 5

Sum of all cells = 140
Sum of Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle P_{i}\,} = 3.780

Equations

For example, taking the first column,

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle p_1 = \frac{(0+0+0+0+2+7+3+2+6+0)}{140} = 0.143}

And taking the second row,

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle P_2 = \frac{1}{14(14 - 1)} \left((0^2 - 0) + (2^2 - 2) + (6^2 - 6) + (4^2 - 4) + (2^2 - 2)\right) = 0.253}

In order to calculate Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \bar{P}} , we need to know the sum of Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle P_i} ,

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle ~ = 1.000 + 0.253 + ... + 0.286 + 0.286 = 3.780}

Over the whole sheet,

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \bar{P} = \frac{1}{((10) ((14) (14 - 1)))} \left((3.780) (14) (14-1)\right) = 0.378}

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \bar{P}_{e} = 0.143^2 + 0.200^2 + 0.279^2 + 0.150^2 + 0.229^2 = 0.210}

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \kappa = \frac{0.378 - 0.211}{1 - 0.211} = 0.211}

Significance

Landis and Koch Template:Ref give the following table for interpreting the significance of the Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \kappa} value. This table is however no means universally accepted as a guide for interpreting the value. It has been noted that this benchmark may be more harmful than helpfulTemplate:Ref, as the number of categories and subjects will affect the magnitude of the value. The kappa will be higher when there are fewer categories. Template:Ref

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \kappa}	Interpretation
< 0	No agreement
0.0 — 0.19	Poor agreement
0.20 — 0.39	Fair agreement
0.40 — 0.59	Moderate agreement
0.60 — 0.79	Substantial agreement
0.80 — 1.00	Almost perfect agreement

See also

• Cohen's kappa
• Pearson product-moment correlation coefficient

Notes

1. Template:Note Landis JR, Koch GG (1977) "The measurement of observer agreement for categorical data" in Biometrics 33:159-74
2. Template:Note Gwet K (2001) Statistical Tables for Inter-Rater Agreement. (Gaithersburg : StatAxis Publishing)
3. Template:Note Sim J, Wright CC (2005) "The Kappa Statistic in Reliability Studies: Use, Interpretation, and Sample Size Requirements" in Physical Therapy 85:257-68

References

• Fleiss JL (1971) Measuring nominal scale agreement among many raters. Psychological Bulletin 76:378--382

Further reading

• Fleiss JL, Cohen J (1973) The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability" Educational and Psychological Measurement 33:613-9
• Fleiss JL (1981) Statistical methods for rates and proportions. 2nd ed. (New York: John Wiley) pp. 38-46

External links

• Kappa: Pros and Cons contains a good bibliography of articles about the coefficient.