Language models can draft a multiple-choice item in seconds. Many of those drafts read well: clean stems, parallel options, a confident key. That polish is the risk. An AI draft looks finished long before it has been checked, and a reviewer who is pressed for time can mistake fluent for correct.

This article sets out where AI drafting helps, where it tends to fail, and a review workflow that treats every AI draft the way a careful program treats any new item: as raw material.

A draft is raw material

The evidence a program needs for an AI-drafted item is the same as for a human-written one. The Standards for Educational and Psychological Testing (AERA, APA and NCME, 2014) define validity as “the degree to which evidence and theory support the interpretations of test scores for proposed uses of tests,” as quoted in Sireci and Benítez’s 2023 review in Psicothema. Nothing in that definition changes because a machine wrote the first version.

So the question is not whether AI items are acceptable. It is whether each item, however it started, has a verified key, a blueprint tag, completed reviews and a record of who approved it.

Where AI drafting helps

Used with care, drafting tools can save experts time on the parts of item writing that are slow but not hard.

  • Scenario stems. A realistic case with the right details, which the expert then corrects.
  • Variants. Several versions of a strong item with different numbers or contexts, to reduce exposure.
  • Distractor ideas. A list of plausible errors for the expert to accept, reject or sharpen.
  • Reading level. A rewrite of a dense stem at a plainer reading level, with the meaning checked by an expert.
  • Coverage gaps. First drafts for thin blueprint areas, so the workshop starts from something rather than a blank page.

In each case the tool proposes and an expert decides.

Where AI drafts go wrong

The failures are predictable, which makes them checkable.

  • Wrong or outdated keys. The key sounds right but reflects an older code, a different jurisdiction or a common myth.
  • Two defensible answers. A distractor is true under some reasonable reading.
  • Cues. The key is the longest or most qualified option, or repeats a word from the stem.
  • Generic content. Items test textbook facts instead of the specific tasks in your blueprint.
  • Near copies. A draft closely resembles a published item or a widely shared practice question.
  • Unfamiliar contexts. Names, settings or idioms that some candidates will find foreign, or that lean on stereotypes.
  • Reading load. Long stems that test reading speed as much as the skill.

An item writing checklist catches most of these, provided reviewers actually use it on AI drafts rather than skimming.

A review workflow with named roles

Here is a workflow that keeps a person accountable at every step. Each role should be filled by someone other than the person who requested the draft.

  1. Request from the blueprint. Each request names one blueprint line, one cognitive level and the reference source. No open-ended “write me 50 questions.”
  2. Draft. The tool produces the draft. The system logs the request, the tool and version, the date and the raw output.
  3. Key verification. A subject matter expert confirms the key against the named reference and writes the citation into the item record.
  4. Content edit. An item writer rewrites the stem and distractors to your standard, checking that every distractor is clearly wrong and plausible.
  5. Bias and accessibility review. A trained reviewer checks context, language and screen reader behavior.
  6. Editorial review. An editor checks style, reading level and cues.
  7. Pretest. The item runs unscored on a live form. It counts only after its statistics are reviewed.

The same workflow applies to items written by people. The difference is emphasis: AI drafts need closer attention at steps 3 and 4, because their errors are harder to spot.

Keep a provenance log

For each item, the record should show how it began and how it changed. A practical provenance log includes:

  • Whether the first draft came from a person or a tool, and which tool and version
  • The request, including the blueprint tag and reference
  • The raw draft, stored and never shown to candidates
  • Each reviewer, their role, the date and what they changed
  • The final approver

This log answers the questions an accreditor, a candidate’s lawyer or your own board might ask later. It also lets you compare AI-drafted and human-drafted items after a few windows: if one group shows more flagged statistics, you will know where to focus review.

Example: 200 new domain items

Say a program adds a new domain on electric vehicle service to a technician certification and needs 200 new items. The numbers below are illustrative planning figures, not a prediction of any program’s results.

  • Drafts requested: 300, because some will be discarded. Each request names a blueprint line and a reference.
  • Key verification: three experts split the drafts and verify keys. Drafts with wrong keys are discarded, not patched.
  • Content edit: item writers rework the surviving drafts in a two-day workshop, writing new items where drafts are weak.
  • Reviews: bias, accessibility and editorial reviews run in the platform, each logged against the item.
  • Pretest: approved items run unscored across two windows before any of them count.

The time saved, if any, comes from starting the workshop with drafts on the table. It does not come from skipping review. If a program finds that experts spend longer fixing drafts than writing from scratch, that is useful evidence too, and the drafting step can be narrowed to the tasks where it helps.

Questions every reviewer asks

A reviewer facing a polished draft needs a short script to slow down. Five questions work well, asked in order:

  1. Can I prove the key? Point to the page, section or code clause that makes it correct.
  2. Could a strong candidate argue for another option? If yes, rewrite or discard.
  3. Does it test the blueprint task? Or a nearby fact that is easier to write about?
  4. Have I seen this before? Search your bank and common public sources for near matches.
  5. Would every candidate understand the context? Replace local references, brand names and idioms.

An item that fails any one goes back, with the reason logged. Rejection rates by reason are worth tracking over time, because they show which drafting requests need tighter instructions.

Three practical cautions.

  • Keep secure content out of public tools. Do not paste live items, candidate data or confidential blueprints into tools whose terms allow the provider to keep or train on inputs. Use tools under a contract that addresses data handling.
  • Ask counsel about ownership. Who owns AI-drafted content, and what your contracts with item writers say about it, are legal questions. Ask your counsel before you rely on an answer.
  • Say what you do. Candidates and stakeholders increasingly ask how items are made. A short public statement of where AI is used, and where a person always decides, answers that before it becomes a complaint.

What to do this week

  • Write down where, if anywhere, your program already uses AI drafting, including informal use by item writers.
  • Add a “draft source” field to your item records.
  • Add AI-specific checks to your review checklist: key against reference, near copies, and cues.
  • Confirm that no live items have been pasted into public tools.
  • Decide who signs off on an AI-drafted item, by role.

Our item review checklist is a good base to add those checks to.

Where we can help

Our responsible AI policy sets out exactly where we use AI in item drafting and scoring, what a person does at each step and what is logged. In the online testing platform, AI-assisted drafting comes with mandatory human review: drafts are marked as drafts, carry their provenance, and wait for the reviews your program requires before they can be placed on a form.

If your experts are new to reviewing drafts critically, our Item Writing Workshop trains them in item construction, bias and accessibility review, and in reworking AI drafts to your standard, with a facilitator in the room.