Vendors often call a test “validated,” as if validity were a stamp applied once at the factory. It is not. Validity is an argument you make about a specific use of a score, and it is only as strong as the evidence you can put on the table.
This article explains what validity means in practice, how to write the claim your exam makes, and what a small program can do this week to start a validity file.
Validity belongs to a use
The Standards for Educational and Psychological Testing (AERA, APA and NCME, 2014) define validity as “the degree to which evidence and theory support the interpretations of test scores for proposed uses of tests.” You can read that definition in context in Sireci and Benítez’s 2023 review in Psicothema.
Three words in that sentence do most of the work. Evidence means you have to show something, not assert it. Interpretations means the question is what a score is taken to mean. Uses means the same test can be well supported for one purpose and poorly supported for another.
Example: a 100-item pharmacology exam written for a nursing course. As a course final, it may support the claim that a student learned the course content. Used by an employer to screen applicants for a hospital role, it makes a different claim, and the course evidence does not automatically carry over.
So the first question is never “Is this test valid?” It is “Valid for what decision, about whom?”
Write the claim down first
Most programs have never written their claim in one sentence. Doing so takes ten minutes and changes every conversation that follows.
A useful template:
A candidate who passes can [do this work] at [this level] in [this setting], and the score can be used to [make this decision].
Example: say a program certifies residential HVAC technicians. Its claim might read, “A candidate who passes can install, diagnose and maintain residential heating and cooling systems safely at entry level, and the score can be used to award a certification valid for three years.”
That one sentence now tells you what the blueprint must cover (install, diagnose, maintain, safety), who the exam is for (entry level), and what decision rides on it (a three-year credential). It also tells you what the exam does not claim: it says nothing about commercial systems or supervisory skill.
This is the Define phase of the Exampit Proof Cycle, and it comes before a single item is written.
Five kinds of evidence
The Standards group validity evidence into five sources. You do not need all five in equal measure, but you should know which ones your claim depends on.
- Test content. Does the exam cover what the claim names, in the right proportions? A job task analysis and a weighted blueprint are the usual evidence.
- Response processes. Are candidates using the skill you intend, or something else, such as reading speed or test-taking tricks? Think-aloud sessions and timing data help here.
- Internal structure. Do the items hang together the way your blueprint says they should? Item statistics and reliability estimates sit here.
- Relations to other variables. Do scores relate to other measures the way you would expect, such as supervisor ratings or performance on a practical?
- Consequences. What happens when the score is used? Are pass rates very different across groups, and if so, is there a content reason?
For most certification and licensing exams, content evidence carries the most weight. For hiring tests, relations to job performance matter more. For classroom tests, content and consequences usually lead.
Reliability is necessary, not enough
Reliability is the consistency of scores: whether a candidate would get a similar score on a different form, on a different day, or from a different scorer. A common index is Cronbach’s alpha, which estimates how consistently the items on a form measure the same thing. It ranges from 0 to 1, with higher values meaning more internal consistency.
A test cannot support a claim if its scores bounce around. But a test can be very consistent and still measure the wrong thing.
Example: a 60-item safety exam written at a dense reading level might produce a high alpha because strong readers do well on every item. The scores are consistent, but part of what they consistently measure is reading, not safety knowledge. That is a validity problem that a reliability number will never reveal.
Treat reliability as one piece of internal structure evidence, not as the whole argument.
Two threats to watch for
Most validity problems in working programs fall into two families. Psychometricians have names for both.
Construct underrepresentation. The exam leaves out part of what the claim names. If the HVAC claim says “diagnose” but only 6 of 120 items involve diagnosis, the score says less about diagnosis than the credential implies. A blueprint built from job data is the main defense.
Construct-irrelevant variance. Something other than the target skill moves scores. Common sources:
- Reading load higher than the job requires
- An unfamiliar interface or item type, used for the first time on test day
- Cultural references or idioms that some candidates will not know
- Accommodations that are missing, late or set up by hand on the day
- Time limits tighter than the skill itself demands
Each of these can make a qualified candidate fail for reasons unrelated to the claim. Item review for bias and accessibility, practice items in the real interface, and accommodations configured before the window all reduce this kind of noise.
Example: a one-page validity file
You do not need a 90-page technical report to begin. A one-page table, kept current, is a good start. Here is what one might look like for a program that tests about 1,500 candidates a year on a 120-item exam. The entries are illustrative.
| Claim element | Evidence | Where it lives | Status |
|---|---|---|---|
| Content matches the job | Job task analysis survey and blueprint | Blueprint file, version 3 | Done, five years old |
| Items reviewed for bias | Review panel sign-off per item | Item bank metadata | Done for 80 percent of items |
| Scores are consistent | Alpha and standard error per form | Window reports | Done each window |
| Cut score has a method | Standard setting report | Not on file | Missing |
| Accommodations applied | Approval log per candidate | Platform records | Done |
| Pass rates reviewed by group | Annual summary | Not yet produced | Missing |
Two rows say “Missing.” That is useful, not embarrassing. It turns a vague worry about validity into two specific tasks with owners and dates. If the blueprint is five years old, a third task appears: check whether the job has changed.
Notice what the table does not say. It does not say the exam is valid. It says what evidence exists for each part of the claim and what does not exist yet. A regulator, accreditor or court decides what is sufficient; your job is to have the evidence ready and honest.
Common misreadings to avoid
A few phrases come up often and are worth retiring.
- “The test was validated.” Validated for which use, with what evidence, and when?
- “Validity and reliability are the same.” Consistent scores are required, but consistency is not meaning.
- “Face validity is enough.” An exam that looks right to experts is a start, not evidence of how scores behave.
- “We bought a validated test.” A vendor’s evidence supports the vendor’s intended use. Your use may differ.
- “It is valid because candidates pass.” Pass rates are an outcome to explain, not proof of anything by themselves.
What to do this week
You can make real progress in a few hours.
- Write the claim. One sentence, using the template above. Share it with two subject matter experts and see if they agree.
- List the decisions. Write down every use of the score, including informal ones such as hiring screens or course placement.
- Start the table. Copy the one-page validity file and fill in what you already have. Be honest about gaps.
- Check the blueprint date. If it is older than the last real change in the job or curriculum, flag it.
- Pull one window of statistics. Look at item difficulty, discrimination and reliability for your most recent form.
- Define your terms. If your team uses words like construct, cut score or standard error loosely, share our assessment glossary so everyone means the same thing.
Where we can help
Our psychometrics and exam development service builds the evidence behind a score: job task analysis, blueprints, item review, standard setting and technical reports, all led by a psychometrician with a graduate degree in measurement.
If you already run an exam and want an outside read before investing more, a Test Health Check reviews one exam’s item statistics, reliability and accessibility and returns a written report in ten business days. It is indicative from $1,900 per exam.
Neither one makes an exam valid by declaration. What they do is help you build, organize and maintain the evidence so that when someone asks what your score means, you can show them.
