Most rubrics look reasonable on paper and fall apart the first time two people score the same response. One scorer reads “clear and well organized” as a 4, another as a 3, and both are right by the rubric’s own words.
Consistency comes from the level descriptors: the sentences that say what work at each level looks like. This article shows how to write them, how to pick anchor papers, and how to pilot a rubric before it counts.
Start from the task’s claim
Before choosing criteria, write down what a score on this task is meant to say about the candidate. One sentence is enough.
Example: “A candidate who scores 3 or higher can write an incident report that a supervisor could act on without follow-up questions.”
That sentence tells you the criteria. A supervisor needs to know what happened, when, who was involved and what was done. It also tells you what not to score: spelling matters only if it gets in the way of the supervisor understanding the report.
Criteria that do not connect to the claim add noise. A rubric with nine criteria is not more rigorous than one with four. It is usually harder to apply consistently.
Analytic or global scoring
Two common rubric types suit different jobs.
- Global. One overall score for the whole response. Faster to apply and often reliable for experienced scorers. Gives candidates less detail.
- Analytic. A separate score for each criterion. Slower, but shows where a response is strong or weak, and makes disagreements easier to locate.
For high-stakes decisions and for candidate feedback, analytic rubrics are usually the better fit. For a large volume of short responses where one judgment is enough, global rubrics can work well.
Describe evidence, not adjectives
This is where most rubrics go wrong. Descriptors built on adjectives (“excellent,” “good,” “adequate,” “weak”) ask scorers to bring their own standard. Descriptors built on observable evidence ask them to look for something specific.
Compare two versions of one criterion, “Sequence of events,” on a 4-point scale.
Adjective version:
- 4: Excellent sequence of events
- 3: Good sequence of events
- 2: Adequate sequence of events
- 1: Poor sequence of events
Evidence version:
- 4: All events in order, each with a time; no gaps a reader must fill
- 3: All events in order; one or two missing times, but the order is clear
- 2: Most events in order; at least one gap or reversal a reader must work out
- 1: Events out of order or missing, so a reader cannot reconstruct what happened
The evidence version tells two scorers exactly what to look for and what separates adjacent levels. That is the difference between a rubric that reads well and one that scores well.
Rules for level descriptors
Guidelines most scoring programs follow:
- One criterion, one quality. Do not mix organization and accuracy in one row.
- Change one thing between levels. Each step should differ in a way a scorer can point to.
- Count where you can. “Two or more errors” is easier to apply than “several errors.”
- Describe the top level as attainable. It should describe strong work candidates actually produce, not perfection.
- Describe the bottom level by what is present. “Lists events without times” is more useful than “fails to meet expectations.”
- Keep the language parallel. Use the same terms across levels so scorers can compare them line by line.
On the number of levels, four is a common choice. An even number removes the safe middle option and makes scorers decide which side of the line a response falls on. More than six levels usually adds disagreement without adding information.
Choose anchor papers
Descriptors define levels in words. Anchor papers define them in examples: real responses, chosen and agreed by expert scorers, that show what each level looks like.
For each criterion and level, choose at least one clear anchor and, ideally, one borderline anchor that sits just above the line. Borderline anchors are the most useful, because scorers disagree most at the boundaries.
Keep a short written rationale with each anchor: which words in the descriptor it meets and why it does not reach the level above. That rationale is what trains new scorers.
Remove anything that identifies a candidate from every anchor, and get permission to use responses in training where your policies require it.
Weight the criteria on purpose
If one criterion matters more to the claim, weight it, and write down why. In the incident report example, a program might double the weight of accuracy and sequence, because a supervisor cannot act on a report that gets the facts or order wrong.
Unweighted analytic rubrics are a weighting decision too. Four equal criteria mean presentation counts as much as accuracy. Make that choice deliberately.
Example: piloting a new rubric
Say a program has written a four-criterion analytic rubric for a written incident report, scored on a 4-point scale. Before using it live, it runs a pilot. The numbers are illustrative.
- Select 30 responses from a practice window, covering a range of quality.
- Two trained scorers score all 30 independently, without discussing them.
- Calculate exact agreement for each criterion. Example results: sequence 87 percent, accuracy 83 percent, actions taken 60 percent, clarity 53 percent.
- Find the disagreements. For actions taken and clarity, list every response where the scorers differed and what each scorer saw.
- Rewrite the descriptors. The clarity row used “easy to follow” at level 3; it becomes “a supervisor can identify the main issue in the first two sentences.”
- Rescore a fresh set of 20 responses and recheck agreement.
The pilot tells you which criteria are working and which need rewriting, before any candidate’s score depends on them. Our article on inter-rater reliability explains how to go beyond percent agreement once the rubric is stable.
Signs a rubric needs work
Rubrics already in use show their problems in the scoring data and in scorer behavior. Watch for these.
- Scores bunch at one level. If most responses land on 3, the descriptors around 3 are probably too broad.
- One criterion drives disagreement. Agreement is high on three rows and low on one; rewrite that row first.
- Scorers ask the same question. If several scorers ask what “sufficient detail” means, the rubric has not said.
- Criteria move together. If every response gets the same level on two criteria, they may be measuring one thing.
- Appeals cluster. Candidates challenge scores on the same criterion window after window.
- Notes in the margin. Scorers keep private notes on how to apply a row, which means the real rubric is not the written one.
Each sign points to a specific row to rewrite, which is much faster than rewriting the whole rubric. Fix one row, pilot it on a small set, then move to the next.
Train scorers on the rubric
A good rubric still needs trained scorers. A typical training session covers the claim, each criterion, the anchors with their rationales, then practice scoring with discussion. Scorers then score a qualifying set and must reach your agreement threshold before scoring live work.
During live scoring, mix previously scored responses into each scorer’s queue. If a scorer drifts from the agreed scores on those, they recalibrate before continuing.
What to do this week
- Pick your most-used rubric and write the one-sentence claim for its task.
- Circle every adjective in its descriptors and rewrite one row as observable evidence.
- Choose one clear and one borderline anchor for that row.
- Ask two scorers to score ten responses independently and compare.
- Check whether any criterion mixes two qualities.
If your team uses terms like analytic rubric, anchor paper or inter-rater reliability differently, our assessment glossary gives each a plain definition.
Where we can help
Our Rubric Calibration Sprint rewrites descriptors, selects and annotates anchor papers, trains your scorers and measures their agreement before and after, indicative from $3,200.
Once the rubric scores consistently, our automated grading service can apply it at volume, with human review queues, seeded check responses and an audit trail on every score.
