Cohen's Kappa Calculator
Measure agreement between two raters, corrected for chance. Paste the agreement matrix, one row per rater-A category and one column per rater-B category.
How to enter your data: Enter your numbers as a table, one row per line. Each row stands for one of the first rater's categories, and the counts on that row, separated by commas, show how many items the second rater placed in each category. List the categories in the same order on every row so the table stays square, with the same number of rows as columns.
Cohen's Kappa is a simple score that shows how well two people agree when they each sort the same items into categories, such as pass or fail. It goes further than just counting matches by subtracting the agreement you would expect from lucky guessing, so it gives a fairer picture. The result is usually a number between 0 and 1, where higher means the two people agreed more strongly.
Built into PaperSurvey.io
Skip the copy-paste. Scan your paper or web surveys and PaperSurvey computes these metrics automatically on your real data, ready to export to Excel, SPSS and R.
Where it is used
- Teachers: Two teachers grade the same batch of essays as pass or fail and check how closely their marking agrees before finalising results.
- HR and hiring staff: Two interviewers rate the same shortlist of candidates as suitable or not, then check they judged people consistently.
- Researchers: Two people sort the same open-ended survey answers into themes and confirm they applied the labels the same way.
Interpreting kappa
Cohen’s kappa measures how much two raters agree beyond what would be expected by chance. Landis and Koch suggest: 0.01–0.20 slight, 0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 substantial and 0.81–1.00 almost perfect agreement.
When should you use it?
Use this calculator when two people each look at the same set of items and sort every item into categories, and you want to know how much they truly agree. The categories should be labels, not numbers, for example pass or fail, yes or no, or a set of theme names. Both people must judge the exact same items. It works for two raters only. If you are scoring things on a number scale or sliding rating, a different measure fits better than Kappa.
What does the result mean?
The result is a single number called Kappa, usually between 0 and 1. A value of 1 means the two people agreed on everything. A value of 0 means they agreed only as often as random guessing would predict, and a value below 0 means they disagreed more than chance. A widely used guideline from Landis and Koch reads it like this: up to 0.20 is slight, 0.21 to 0.40 is fair, 0.41 to 0.60 is moderate, 0.61 to 0.80 is substantial, and above 0.81 is almost perfect. Many fields look for 0.61 or higher.
Mistakes to avoid
The most common mistake is not judging the same items, so line up one shared list for both people before you start. Keep your table square, meaning the same number of rows and columns, and list the categories in the same order across the top and down the side. Do not use this for three or more raters, since Cohen's Kappa is built for exactly two. Avoid using it for number scores or ratings on a sliding scale. Very small samples can also give shaky results, so gather enough items first.
How to use this calculator
- Count your results into a grid: for every pair of category choices the two raters made, write how many items fell into it.
- Enter the grid one row per line, with the counts on each row separated by commas, keeping the categories in the same order.
- Check the table is square, the same number of rows as columns, so every category appears on both sides.
- Read the Kappa number and compare it to the guideline, where 0.61 and above is usually considered strong agreement.
Worked example
Two teachers each mark the same 50 homework sheets as pass or fail. They both said pass on 20 sheets and both said fail on 15. On 8 sheets the first teacher said pass while the second said fail, and on 7 the first said fail while the second said pass. You would enter "20, 8" on the first line and "7, 15" on the second. Even though they matched on 35 of the 50 sheets, the calculator returns a Kappa of about 0.39, which counts as only fair agreement.
Frequently asked questions
What do I type in each cell of the table?
Each cell holds a count: how many items the first rater put in that row's category and the second rater put in that column's category. The cells where both categories match are where the two people agreed.
Where do I get these numbers?
Have your two raters each label the same items, then tally the results into a grid. Count how many items landed in each combination of the two people's choices.
What counts as a good Kappa score?
Higher is better. Using the common Landis and Koch guideline, 0.61 to 0.80 is substantial agreement and above 0.81 is almost perfect. Many fields treat 0.61 and up as solid.
Can I use this for three or more raters?
No. Cohen's Kappa compares exactly two raters. If you have three or more, a related measure called Fleiss' Kappa is the right tool.
We matched on most items, so why is my Kappa low?
Kappa removes the agreement you would expect by chance. When almost everything falls into one category, a lot of matching is just luck, so the score can look low even when the raw match rate is high.
Related calculators
Get Started with PaperSurvey.io Software
Start your 14-day free trial now, no credit card required.