If your program scores essays, performance tasks, interviews or practical demonstrations, at least part of every score is a person’s judgment. Inter-rater reliability is how you show that the judgment belongs to your rubric, not to whichever scorer a candidate happened to get.
This article explains the common measures in plain terms, works through a kappa calculation with numbers, and sets out how a program lead can monitor scorers during a live window.
What it measures
Inter-rater reliability is the degree to which different trained scorers give the same score to the same response. High reliability means a candidate’s score would barely change if a different scorer read their work. Low reliability means the scorer is part of the score.
It is one source of evidence about the consistency of scores, and consistency is a precondition for any claim the score makes. It does not show that the rubric measures the right thing. Two scorers can agree perfectly on a rubric that rewards length.
A related idea is intra-rater reliability: whether one scorer gives the same score to the same response on two occasions. Fatigue and drift during a long scoring session usually show up here first.
Percent agreement
The simplest measure is the share of responses where two scorers gave the same score.
- Exact agreement. Same score on the same response.
- Adjacent agreement. Scores within one point.
Percent agreement is easy to explain and easy to calculate, which makes it the right first number for most committees. Its weakness is that it ignores chance. If almost every response is a pass, two scorers who barely read the work would still agree most of the time.
Adjacent agreement has a second weakness. On a 4-point scale, being within one point covers half the scale. Report it alongside exact agreement, never alone.
Cohen’s kappa
Cohen’s kappa corrects percent agreement for the agreement you would expect by chance, given how often each scorer uses each score. The formula is:
kappa = (observed agreement - chance agreement) / (1 - chance agreement)
A kappa of 1 means perfect agreement. A kappa of 0 means agreement no better than chance. Negative values mean scorers agree less often than chance would predict, which usually points to a misunderstanding of the rubric.
For rubrics with ordered levels (1, 2, 3, 4), a weighted kappa treats a 1-point disagreement as less serious than a 3-point one. Quadratic weighting, which penalizes large disagreements heavily, is a common choice for essay scores.
For scores from more than two scorers, or for continuous scores, programs often use the intraclass correlation coefficient instead. It answers the same question with a different calculation.
Example: a kappa calculation
Say two trained scorers each rate the same 100 practical demonstrations as pass or fail. The numbers are illustrative.
| Scorer B: pass | Scorer B: fail | Total | |
|---|---|---|---|
| Scorer A: pass | 60 | 12 | 72 |
| Scorer A: fail | 8 | 20 | 28 |
| Total | 68 | 32 | 100 |
Observed agreement. They agree on 60 passes and 20 fails: 80 of 100, or .80.
Chance agreement. Scorer A passes 72 percent of responses and scorer B passes 68 percent. If they scored at random with those rates, they would both pass a response .72 × .68 = .4896 of the time, and both fail it .28 × .32 = .0896 of the time. Chance agreement is .4896 + .0896 = .5792.
Kappa. (.80 - .5792) / (1 - .5792) = .2208 / .4208, or about .52.
Eighty percent agreement sounds strong. A kappa of about .52 says that a good part of that agreement is what you would expect from two scorers who pass most people anyway. That is exactly why kappa is worth calculating.
The table also shows the direction of disagreement: scorer A passed 12 responses that scorer B failed, and the reverse happened 8 times. Scorer A is slightly more lenient. That is a specific, fixable finding.
Setting a working target
There is no universal threshold for acceptable inter-rater reliability. What is acceptable depends on the stakes, the scale and how scores are combined. Published rules of thumb exist, but they were written for other purposes and are often applied too literally.
As common practice, many programs set their own working target before scoring begins and write it down. Example: a program might require exact agreement of at least 70 percent on a 4-point scale, nearly all scores within one point, and a weighted kappa it has agreed with its psychometrician. It would apply stricter targets for scores near the pass line.
What matters is that the target exists before the data does, so it cannot be bent to fit the result.
What lowers agreement
When agreement is low, the cause is usually one of these.
- Vague descriptors. Adjectives such as “clear” or “adequate” invite personal standards.
- Missing anchors. Scorers have no shared example of what a 3 looks like.
- Untrained scorers. Scorers start without a qualifying set.
- Fatigue. Agreement drops late in long sessions.
- Drift. Scorers slowly shift their standard over days, often toward leniency.
- Order effects. A weak response read just after a strong one is scored harsher than it would be otherwise.
Each cause has a different fix. Vague descriptors need rewriting. Missing anchors need choosing. Fatigue needs shorter sessions. Drift needs ongoing checks, which is the next section.
When one scorer is a machine
The same measures apply when one of the two scorers is an automated system. Exact agreement, weighted kappa and the direction of disagreement tell you how closely the machine tracks trained people on your rubric.
One rule matters most: compare machine to human agreement with human to human agreement on the same responses. If two trained people reach a weighted kappa your psychometrician considers acceptable, the machine should be judged against that figure, on held-out responses, and separately for each candidate group you report on. A machine is never a reason to stop double scoring a human sample; that sample is how you notice when the machine drifts. Our article on automated essay scoring with human review covers the review designs in more detail.
Monitoring a live window
Reliability is not measured once. It is watched through the whole scoring window.
- Qualify first. Each scorer scores a set of previously agreed responses and must meet the target before live scoring.
- Double score a sample. A set share of live responses, chosen at random, is scored by two people. Many programs choose 10 to 20 percent as common practice.
- Seed check papers. Previously scored responses are mixed into each queue. Scorers do not know which ones they are.
- Report daily. Agreement per scorer, per criterion, on double-scored and seeded responses.
- Recalibrate early. A scorer below target on seeded responses stops, reviews the anchors with a lead, and requalifies.
- Resolve disagreements. When double scores differ by more than the program allows, a third trained scorer adjudicates.
A daily report table with one row per scorer and one column per criterion is enough. The program lead looks for the one cell that is out of line, not for a single overall number.
What to do this week
- Find out whether any of your scored tasks have double-scored data, and calculate exact agreement.
- If they do, calculate kappa for one task using the method above.
- Write down a working target for exact agreement before your next window.
- Check whether scorers qualify before live scoring, and on what set.
- Plan a daily agreement report for your next window, even a simple spreadsheet.
Our assessment glossary defines kappa, inter-rater reliability and related terms in a few lines each, which helps when sharing this with a committee.
Where we can help
When agreement is low, our Rubric Calibration Sprint diagnoses which criteria and scorers drive it, rewrites descriptors, chooses anchors and retrains scorers, then measures agreement again. It is indicative from $3,200.
Our automated grading service builds double scoring, seeded check responses and per-scorer agreement reports into the scoring queue, so the daily report already exists when the program lead opens it.
