Twenty scorers, one reading of the rubric
Anchor papers, practice rounds and a rewritten rubric in three weeks.
- Length
- 3 weeks
- Price, indicative
- From $3,200
When two scorers read the same essay and land two points apart, the rubric is usually the cause, not the people. The sprint finds the rows they read differently, rewrites them, and measures agreement again.
How the three weeks run
- Week 1
Choose anchor papers
Your lead and our psychometrician pick anchors at every score point from 60 to 100 real responses.
- Week 2, day 1
Score a baseline
Every scorer scores the same 20 papers alone, before any discussion.
- Week 2, days 2 to 5
Discuss and rewrite
A two-hour session on the papers with most disagreement, then the weak rubric rows rewritten.
- Week 3
Practice and measure
A second round of 20 papers on the new rubric, with agreement compared to the baseline.
- Week 3, last day
Hand over
Anchor set, rubric and practice set delivered, with a 60-minute session for scorer leads.
What you receive
- Who it is for
- Teams of up to 20 scorers sharing one rubric
- Format
- Online sessions, or on site in New Mexico
- An anchor set with annotated papers at every score point
- A rewritten rubric with each change explained
- Exact and adjacent agreement before and after, by scorer and rubric row
- Cohen’s kappa (agreement corrected for chance) for both rounds
- A practice set and key for training new scorers
Is this the right program?
Right for you if
- Up to 20 scorers share one rubric for essays, portfolios or performances
- Rescoring requests or appeals point to scorers reading the rubric differently
- You can gather at least 60 real responses
Look elsewhere if
- You need several rubrics calibrated; each extra rubric is quoted separately
- Your exam is multiple-choice only; a Test Health Check fits better
Example: a district’s grade 8 writing task
A school district scores a grade 8 argument essay each spring, with 18 teacher scorers over two Saturdays. Baseline agreement on its 4-point rubric is lowest on the evidence row, where scorers read “relevant” differently. The sprint rewrites that row with two anchor papers per score point, and the second round is measured against the first.
Rubric Calibration Sprint questions, answered
What do exact and adjacent agreement mean?
Exact agreement is the share of papers two scorers give the identical score; adjacent agreement also counts scores one point apart. Both are reported by scorer and by rubric row. Kappa adjusts exact agreement for matches expected by chance, which matters on rubrics with only three or four score points.
How does this connect to automated grading?
It is often the first step. Automated grading suggests scores from your anchors, so a calibrated rubric and a clean anchor set make the suggestions more useful. The sprint stands on its own, and its anchors carry straight into a Grading Pilot or the automated grading service if you go on.
What does it cost for more than 20 scorers?
The sprint is from $3,200 for up to 20 scorers and one rubric. Larger groups add a second practice round and a second discussion session, quoted as a fixed fee. Every program price is on the pricing page, and on-site days in New Mexico carry no travel fee.
Can scorers take part remotely?
Yes. Scorers work in the platform from wherever they are, and the discussion session runs on video for two hours. Practice rounds are blind, so no scorer sees another’s score until the round closes. Districts and boards in New Mexico can hold the discussion session in person instead.
Plan this program with us
A 30-minute call to see whether Rubric Calibration Sprint fits your teams, your calendar and your budget.