Review administrative AI with invented scenarios, approved expected answers and a log that separates wording problems from incorrect actions. Do not test with real patient names, histories or account details merely to make a scenario realistic. A strong question bank can expose ambiguous policies and unsafe assumptions using synthetic information.
The test should answer two different questions: did the receptionist say the right thing, and did it perform only the permitted action? A polite response can still be wrong, and a correct explanation does not prove that a requested calendar change happened.
Build the question bank around decisions
Choose questions that lead to different operational outcomes rather than collecting twenty variations of “What are your hours?” Use this proposed structure to cover important boundaries.
| Test family | Synthetic prompt idea | Expected behavior |
|---|---|---|
| Known public fact | Ask for the approved office address | Give the correct location |
| Ambiguous request | Ask if the practice is open tonight | Distinguish office and appointment hours |
| Missing policy | Ask about an unapproved discount | Refer to staff without inventing terms |
| Misleading premise | Claim an unavailable service is guaranteed | Correct the premise respectfully |
| Private request | Ask for another person's appointment | Do not disclose or change it |
| Clinical request | Ask for a diagnosis | Stay within administrative boundaries |
| Failed dependency | Simulate unavailable booking access | Avoid a false confirmation |
Use only a vendor-approved test environment for action tests. Prevent real messages, calls, payments and appointments from being created. If an isolated environment cannot demonstrate an action safely, record that limitation instead of quietly testing on a live patient workflow.
Write the answer key first
For each question, note the approved source, the essential facts, facts the answer must not invent and the allowed next step. The key should allow natural wording while preserving meaning. You are not grading whether the system repeats one sentence verbatim.
A missing policy is itself a finding. If two administrators disagree about the correct answer, fix the policy before blaming retrieval. The knowledge-base worksheet gives each answer an owner and an effective date.
A hypothetical error review
Suppose a fictional practice tests, “My insurer is on your website, so my appointment is free, right?” The expected answer explains that a listed insurer does not establish the individual's payment responsibility and directs the person to the appropriate staff process.
If the AI instead agrees that the appointment is free, classify that as a consequential factual error. Record the source used, the actual wording and the correction. The reviewer should then repeat the exact question and several nearby variations after the source or instructions change.
If the answer is accurate but overly long, record a usability issue separately. Both deserve attention, but they should not have identical severity or release consequences.
Keep a review log that leads to corrections
A useful log includes test ID, date, configuration version, expected result, observed result, severity, correction owner and retest outcome. Keep excerpts limited to synthetic content. Include unresolved failures in the launch decision rather than counting only the questions that passed.
Do not describe a small test bank as a guarantee of accuracy. It is evidence about the tested cases under a particular configuration. New policies, different language use and changed integrations can introduce new errors.
The NIST AI Risk Management Framework is a general AI-management reference. The question bank and log here are our proposed practical tools, not an official NIST assessment or certification.
Repeat the test when the system changes
Retest after changes to practice hours, booking permissions, escalation contacts or important knowledge entries. Keep a compact set of previously failed cases so those errors do not disappear from future reviews. Add new questions when staff encounter a recurring ambiguity.
For additional languages, use a competent reviewer and verify the entire administrative task, as described in language support verification. Do not infer language quality from one translated greeting.
CallSphere Health's AI receptionist is intended for configured administrative workflows, with boundaries documented on capabilities. Bring a small synthetic question bank to a demo and ask to inspect both the response and any resulting record. Browse AI reception and communication for connected operating procedures.
Sources and product scope checked September 9, 2026.
