Every exam window produces data that most programs never read. Item analysis is the habit of reading it: checking each item to see how hard it was, whether it separated stronger candidates from weaker ones, and whether its wrong options did their job.
This article explains the two core numbers, difficulty and discrimination, walks through five example items, and gives a routine for deciding what to do with the ones that look wrong.
What item analysis tells you
Item analysis answers three questions about every item on a form.
- Difficulty. What share of candidates answered it correctly?
- Discrimination. Did candidates who did well on the whole exam do better on this item than candidates who did poorly?
- Distractor function. Did each wrong option attract some candidates, and mainly weaker ones?
An item can fail on any of the three. The numbers do not tell you why it failed. They tell you where to look, and a subject matter expert then reads the item to find the reason.
Difficulty: the p-value
In classical item analysis, an item’s difficulty is its p-value: the proportion of candidates who answered it correctly. It runs from 0 to 1.
- A p-value of .90 means 90 percent answered correctly. The item is easy for this group.
- A p-value of .30 means 30 percent answered correctly. The item is hard.
The name is backward, since a higher “difficulty” means an easier item, and it has nothing to do with the p-value in a significance test. It is simply a proportion.
Difficulty is always relative to the group that took the item. The same item might have a p-value of .85 for experienced practitioners and .50 for new graduates.
Very easy and very hard items are not automatically bad. An easy item on a critical safety rule may belong on the exam because every certified person must know it. But items near the extremes tell you little about who is ready and who is not, because almost everyone gets them right, or almost everyone gets them wrong.
Discrimination: the point-biserial
Discrimination asks whether an item separates candidates the way the exam as a whole does. The most common index is the point-biserial correlation: the correlation between getting the item right (1 or 0) and the candidate’s total score.
- A positive value means stronger candidates tended to get it right more often. That is what you want.
- A value near zero means the item does not separate candidates at all.
- A negative value means weaker candidates did better on it than stronger ones. Something is wrong.
Many programs use a corrected point-biserial, which removes the item itself from the total score before correlating. On short forms, this gives a more honest figure.
A simpler alternative is the upper-lower index: the p-value among the top-scoring group minus the p-value among the bottom group. It is easy to explain to a committee and tells a similar story.
Distractor counts
For multiple-choice items, look at how many candidates chose each option, split by high and low scorers.
A healthy pattern: the key is chosen mostly by high scorers, and each distractor is chosen by some candidates, mostly low scorers. Two warning patterns:
- A dead distractor. An option chosen by almost no one is not doing any work. The item is effectively one option shorter.
- A magnet distractor. An option that high scorers choose more often than the key may be defensible, or the key may be wrong.
Example: five items from one window
Here is an illustrative report for a form taken by 400 candidates. The numbers are invented for teaching, not drawn from any real exam.
| Item | p-value | Point-biserial | Note |
|---|---|---|---|
| 12 | .94 | .08 | Almost everyone correct |
| 27 | .61 | .38 | No issues |
| 33 | .22 | -.12 | Option C chosen by 48% of top scorers |
| 41 | .48 | .11 | Two distractors chosen by under 2% |
| 58 | .35 | .31 | Hard, but separates well |
Reading them one at a time:
- Item 12 is very easy and barely discriminates. If it tests a must-know safety rule, the program may keep it on purpose. If not, it is a candidate for revision or retirement.
- Item 27 behaves well: moderate difficulty and clear separation. No action.
- Item 33 is the urgent one. A negative point-biserial and a distractor that pulls the strongest candidates point to a miskey or a second defensible answer. An expert should check it against the reference before scores are released.
- Item 41 has middling discrimination and two dead distractors, so it works like a two-option item. Rewriting the distractors from real candidate errors is the usual fix.
- Item 58 is hard and discriminates well. Hard items that separate are valuable, especially near the cut score.
Flags as a starting point
Programs usually set screening flags so reviewers do not read every item’s numbers by hand. As common practice, not a research finding, many programs flag items with:
- A p-value above about .90 or below about .25
- A point-biserial below about .15
- Any negative point-biserial
- A distractor chosen by fewer than about 5 percent of candidates
- A distractor chosen more often than the key by the top-scoring group
These cut-offs vary by program and purpose. A mastery exam where most candidates are expected to pass will have many high p-values by design. Write your flags down before the window and apply them the same way each time.
Mind the sample size
Item statistics from small groups move around. With 40 candidates, one or two people choosing differently can shift a p-value by five points and swing a point-biserial noticeably.
As common practice, many programs wait for at least 100 to 200 responses before acting on an item’s statistics, and treat smaller samples as early signals only. If your windows are small, combine data across windows for the same item before making a decision, provided the item has not changed.
Handling a flagged item
Flags start a review, not a deletion. A routine that holds up:
- Check the key. Confirm the scored answer against the cited reference. Fix data errors first.
- Read the item. An expert who did not write it reads the stem and options with the statistics in hand.
- Decide and record. Keep, revise, retire, or (for a live miskey) rescore. Record who decided, why and when.
- Rescore consistently. If an item is dropped or double-keyed after a window, apply the change to every candidate on that form and document it.
- Retest revisions. A revised item is a new item. Run it unscored first if you can.
Do not drop items only because they are hard or easy. Dropping items can shift the blueprint balance, and a form can drift from its weights one deletion at a time.
From items to the whole form
Item statistics feed form-level ones. Reliability, often reported as Cronbach’s alpha, estimates how consistently the items on a form measure the same thing. Weak or negative items pull it down, and a few well-discriminating items near the cut score help it most.
The standard error of measurement turns reliability into score points. It describes how far a candidate’s observed score might sit from the score they would get on a perfectly consistent test. For candidates scoring just below or above the cut, that band matters more than any single item.
Review item flags and form reliability together, in the same meeting, so a fix to one is checked against the other.
What to do this week
- Pull item statistics for your most recent form and sort by point-biserial, lowest first.
- Check every negative value against the key before anything else.
- List items where a distractor drew fewer than about 5 percent of candidates.
- Write down your screening flags and your minimum sample size.
- Note any key changes made in the last year and confirm they were applied to every candidate.
If terms like point-biserial, Cronbach’s alpha or standard error of measurement are new to your committee, our assessment glossary defines each in a few lines.
Where we can help
The online testing platform produces p-values, point-biserials, distractor counts and reliability for every form within one business day of a window closing, with flags you set.
If you run an exam elsewhere and want an outside read, a Test Health Check reviews one exam’s item statistics, reliability and accessibility and returns a written report in ten business days, indicative from $1,900 per exam. For deeper work, our psychometrics service covers item review, form assembly and technical reports.
