Ask a vendor how accurate its essay scoring is and you will often hear a single number. That number usually describes someone else’s essays, someone else’s rubric and someone else’s candidates. It says little about how the system will score yours.
This article explains what accuracy should mean for automated essay scoring, how to measure it on your own responses, and three ways to keep a trained person accountable for every score that matters.
Accuracy means agreement
An automated scorer does not know what a good essay is. It learns to predict the score a trained human would give, based on scored examples. So its accuracy is a measure of agreement: how closely its scores match those of trained scorers using your rubric.
That makes three things matter more than any headline figure.
- Your rubric. A scorer trained on a different rubric is predicting a different thing.
- Your responses. Essays from a nursing licensure exam and a ninth-grade writing task differ in length, vocabulary and structure.
- Your human scorers. If two trained people disagree often, no machine can agree with “the human score,” because there is no single one.
Agreement is necessary but not enough. The Standards for Educational and Psychological Testing (AERA, APA and NCME, 2014) define validity as “the degree to which evidence and theory support the interpretations of test scores for proposed uses of tests,” as quoted in Sireci and Benítez’s 2023 review in Psicothema. A machine that matches humans by rewarding length would agree well and still undercut what the score claims.
Four numbers to ask for
When you evaluate a scoring engine on your own data, ask for these, computed on responses the engine has not seen during training.
- Exact agreement. The share of responses where machine and human gave the same score.
- Adjacent agreement. The share within one point. On a wide scale, this is often high even for weak systems, so read it alongside exact agreement.
- Quadratic weighted kappa. A measure of agreement that corrects for chance and penalizes large disagreements more than small ones. It runs from below 0 to 1, where 1 is perfect agreement.
- Standardized mean difference. Whether the machine scores higher or lower on average than humans, expressed in standard deviation units.
Then ask for the same figures from two trained humans scoring the same responses. Human to human agreement is your benchmark. A machine that agrees with humans about as well as humans agree with each other is behaving like another trained scorer, no better. If kappa or standardized mean difference are new terms on your team, our assessment glossary defines both in plain words.
Check every group separately
Overall agreement can hide trouble in subgroups. A system may match humans well on average and still score one group of candidates systematically higher or lower.
Run the four numbers separately for groups your program reports on or is responsible to, such as candidates testing in a second language, candidates using accommodations, and candidates from different regions or training routes. Look especially at the standardized mean difference by group. A consistent gap is a reason to investigate before going live.
Also check unusual responses. Very short, very long, off-topic or copied essays are where engines most often go wrong, and they are exactly the responses a person should see.
Three review designs
Where a person sits in the process is the most important design choice. Here are three common patterns, from most to least human effort.
Machine as second reader. A trained person scores every response. The machine scores it too. When they disagree by more than a set amount, a second person adjudicates. This suits high-stakes exams and gives the clearest audit trail.
Machine first, people review. The machine scores every response. People review all low-confidence scores, all unusual responses, all scores near the cut, and a random sample of the rest. This suits large volumes where every score still counts.
Machine with spot checks. The machine scores practice or formative work. People review a sample and any response a candidate or instructor questions. This suits low-stakes feedback, not decisions about credentials or grades.
Whichever design you choose, a person can override any score, and the override is logged with a reason.
Example: routing rules for 1,200 essays
Say a program scores 1,200 essays a window on a 6-point rubric and chooses the second design. Its routing rules might read like this. The percentages are illustrative choices, not targets.
- Low confidence. Any response where the engine’s confidence falls below the program’s set threshold goes to a person. Example: 15 percent of responses.
- Near the cut. Any machine score of 3 or 4, where the pass line sits, goes to a person.
- Unusual responses. Under 50 words, off-topic flags or high similarity to another response go to a person.
- Random sample. A further 10 percent of the remaining responses go to a person, chosen at random, to keep checking agreement.
- Discrepancy rule. If person and machine differ by 2 points or more, a second person adjudicates.
The program then reports, each window, how many responses each rule routed, the agreement on the random sample, and every override. If agreement on the sample drops, the program widens human review until the cause is found.
Watch for drift
An engine trained on one window’s essays may score the next window differently if the prompt, the population or the instruction changes. Common causes:
- A new prompt that the engine was not trained on
- A change in the rubric wording or anchor papers
- A new training provider whose candidates write in a different style
- Candidates learning what the engine rewards
The random sample in the second design is your drift detector. Keep it running every window, and retrain or recalibrate when agreement moves.
Questions to ask any vendor
Before you sign with any scoring provider, including us, ask for written answers to these.
- Whose data? Were the reported agreement figures computed on our responses or someone else’s?
- Held out? Were the test responses kept out of training?
- Human benchmark? What was human to human agreement on the same responses?
- By group? Can we see agreement and mean differences for each group we report on?
- Overrides? Can a person change any score, and is the change logged with a name and reason?
- Retraining? What triggers recalibration, and who approves the new model before it scores live work?
- Exit? Can we export every score, review and override if we leave?
A provider that answers each in writing gives you material for your own validity file. A provider that offers only a single headline number gives you a marketing claim.
Explain it to candidates
Candidates deserve to know how their work is scored. A short statement in the candidate handbook covers most questions: that an automated system assists scoring, that trained people review scores in defined situations, and how a candidate can request a review.
When a candidate appeals, a person rescores the response without seeing the machine score. The audit trail shows the original score, every review and the final decision.
What to do this week
- Write down every place your program uses, or plans to use, automated scoring.
- For each, note which review design applies and whether scores count toward a decision.
- Collect 200 to 300 double-scored responses so you can measure human to human agreement.
- Draft your routing rules, including the discrepancy rule and random sample size.
- Draft the two-paragraph candidate statement.
Where we can help
Our automated grading service scores essays, short answers and rubric tasks with human review queues and a full audit trail, indicative from $3 per candidate per exam. Our responsible AI policy sets out what the system does, what a person does and what is logged.
If you want evidence on your own essays before committing, a Grading Pilot runs automated scoring alongside your scorers for one course or exam for one term, indicative from $2,400, and reports agreement by group at the end. When human scorers disagree too often to serve as a benchmark, start with a Rubric Calibration Sprint.
