Every assessment platform demos well. Questions appear, timers count down, a results chart fills in. The differences show up later: when a candidate appeals, when an accessibility complaint arrives, when the contract ends and you want your items back.
These twelve questions are built to surface those differences before you sign. Ask them of every vendor, including us. Write the answers down, because the written answer is the one that ends up in the contract.
Start with the decision
Before any vendor call, write one sentence about what your scores decide. “Who is licensed to practice.” “Who moves to the next course.” “Who we hire for the night shift.”
That sentence shapes everything else. The Standards for Educational and Psychological Testing (AERA, APA and NCME, 2014) define validity as “the degree to which evidence and theory support the interpretations of test scores for proposed uses of tests.” Software cannot make a test valid. It can make the evidence for your use easier or harder to build and keep.
Questions about fit
1. Can it deliver the items we actually use? List your item types: multiple choice, multiple response, hotspot, ordering, essay, file upload, case sets. Ask to see each one working with a keyboard and a screen reader, not just with a mouse. Ask which standards it imports and exports, such as QTI 2.1 and 3.0.
2. How are accommodations configured and recorded? Extra time, breaks, screen readers, larger text and separate rooms should be set per candidate before the window, linked to the approval, and visible to proctors. Ask to see the record for one accommodated candidate from approval to score.
3. What accessibility standard does the candidate interface meet, and how do you know? Ask for the WCAG version and level, the date of the last test, who did it, and the list of known issues. Public bodies facing ADA Title II dates have particular reason to ask; our article on Title II deadlines and online exams covers why.
Questions about evidence
4. What item statistics do we get, and when? Difficulty, discrimination, distractor counts and reliability should be available for every form, soon after a window closes. Ask how long that takes and whether you can export the raw response data to run your own analysis.
5. Who helps when we need psychometric work? Job task analysis, standard setting, equating and technical reports need qualified people. Ask whether the vendor has them, what their credentials are, and whether the work is priced separately. A platform with no route to this help leaves the evidence gap for you to fill.
6. Where is AI used, and where does a person decide? If the platform drafts items, scores essays or flags proctoring events with AI, ask for the review step and the audit trail. A good answer names the human role at each point and shows the log. A vague answer is a warning.
Questions about data
7. Who owns our items, candidate records and results? The answer should be you. Ask for the clause in the contract, not just a verbal assurance.
8. What happens when we leave? Ask what you receive at exit, in what format, how long it takes, and when the vendor deletes its copies. Item banks exported as QTI and results as CSV are a common baseline. If moving out looks hard, moving in will be the last easy step.
9. Where is our data hosted, and how is it protected? Ask where data sits, how it is encrypted in transit and at rest, who on the vendor’s staff can see it, and how you will be told about an incident. Ask for written security documentation. If a vendor claims a certification or audit report, ask for the document itself and its date.
Questions about delivery and connection
10. How does proctoring work, and who reviews a flag? Ask which modes are offered, how candidates are told what is recorded, how long recordings are kept, and who looks at a flag before anything happens to a candidate. Proctoring deters, detects and documents. It does not stop all misconduct, and a vendor that says otherwise is overselling. Our article on remote proctoring that respects candidates lists what a fair process includes.
11. What does it connect to? List the systems you run: an LMS, single sign-on, an HR system, a credential registry. Ask how each connection works (for example LTI 1.3, SAML 2.0 or OpenID Connect, a REST API, webhooks) and which ones are live with other customers today rather than on a roadmap.
The question about cost
12. What will this cost over three years, all in? Ask for a fixed, itemized price covering the platform, sessions, overage, proctoring, support, implementation and any psychometric work. Then model your real volume, including retakes and growth.
Per-session and per-seat pricing can look similar in year one and diverge sharply by year three. Ask how prices change at renewal and what support hours are included. Our session cost estimator shows how volume, attempts and proctoring mode drive an indicative annual cost.
How to score the answers
Turn the twelve questions into a simple scoring sheet before the first vendor call, so you are not scoring the demo you liked best.
- Weight each question by how much it matters for your decision. Questions 1, 3, 7 and 8 often matter most.
- Score each answer on a short scale: not met, partly met, met with evidence.
- Count only what is shown or written. A promise of a future feature scores as not met.
- Have two people score each vendor alone, then compare.
The same logic that makes a structured interview fair makes a vendor selection fair: same questions, set scoring rules, independent ratings.
Run a real pilot
A scored demo narrows the list. A pilot decides it. Ask the top one or two vendors for a short pilot with your own items, your accommodations and a few volunteer candidates.
Watch for the small things. How long did import take? Did the screen reader read the math? Did a volunteer with extra time actually get it? How quickly did support answer a question at 4 p.m. on a weekday? The answers will tell you more than any feature list.
Our answers, for the record
Since we suggest asking us too, here are ours in brief. Twelve item types; QTI 2.1 and 3.0 import and export. Accommodations set per candidate and logged against the approval. A candidate interface built to WCAG 2.1 AA and tested with screen readers each release. Item statistics within one business day of a window closing. Psychometricians on staff for job analysis, standard setting and technical reports. Data hosted in the United States, encrypted with TLS 1.2 or later in transit and AES-256 at rest. A full export on exit and deletion of our copies within 30 days.
Our online testing platform is priced from $6,000 a year for up to 2,000 sessions, then from $2 a session; these are indicative starting prices, with 15 percent off platform fees for nonprofit, public and education programs. The pricing page lists every service and program. If you are moving from another system, our Platform Migration program (indicative, from $6,500) covers the item bank and history.
What to do this week
- Write the one-sentence decision your scores support.
- List your item types, accommodations and the systems the platform must connect to.
- Weight the twelve questions for your program.
- Send them in writing to each vendor on your list.
- Price your real volume over three years, including retakes.
Our assessment software buying guide turns these questions into a full selection process with a vendor question table. If you are weighing one platform against a set of separate tools, our comparison page lays out when each makes sense.
