Fleiss' Kappa Calculator
Measure agreement between three or more raters, corrected for chance. Paste one row per subject with the count of raters who chose each category.
Learn how to calculate it yourselfEnter your values
Results update as you type.
Fleiss' kappa
Check your inputs
Enter valid values to see your result. Use the field hints or open the data guide for help.
Calculate it yourself
Three raters classify each of three subjects into two categories. The category-count rows are 3,0; 2,1; and 0,3.
Follow this worked example, then repeat the steps with your own numbers.
Calculate agreement for each subject
Pᵢ = [sum of squared category counts − 3] ÷ [3 × (3 − 1)]. The three results are 1, 1/3 and 1.Average the subject agreements
P̄ = (1 + 1/3 + 1) ÷ 3 = 7/9.Find expected agreement from all category assignments
Category proportions = 5/9 and 4/9. Pₑ = (5/9)² + (4/9)² = 41/81.Adjust for chance agreement
κ = (7/9 − 41/81) ÷ (1 − 41/81) = 22/40 = 0.55.
Each row contains category counts, not individual rater scores. Every row must add to the same number of raters.
Check your answer in the calculatorUse your PaperSurvey.io results
Export the collected ratings, count each subject’s category assignments and calculate Fleiss’ kappa here.
When to use Fleiss' kappa
Cohen's kappa handles exactly two raters. Fleiss' kappa extends the idea to any fixed number of raters, and the raters do not have to be the same people for every subject. You supply how many raters assigned each category to each subject rather than individual rating pairs.
Interpreting the result
Kappa ranges up to 1, where 1 is perfect agreement and 0 is agreement no better than chance. Following Landis and Koch, values below 0.20 are slight, 0.21 to 0.40 fair, 0.41 to 0.60 moderate, 0.61 to 0.80 substantial and above 0.80 almost perfect. Negative values mean less agreement than chance alone would produce.
Frequently asked questions
How should I lay out my data?
One row per subject, one column per rating category. Each cell is the number of raters who picked that category for that subject, and every row must add up to the same total number of raters.
Get Started with PaperSurvey.io Software
Start your 14-day free trial now, no credit card required.