Test whether your website AI assistant gives useful answers
"Looks good" is not a repeatable acceptance test. A useful answer must address the question, stay within the approved source, preserve important conditions, and help the visitor verify or continue.
Build the test sheet before launch and keep it for regression checks.
Create five question groups
- Direct facts: one current page contains the answer.
- Cross-page tasks: two compatible sources are needed.
- Vocabulary differences: visitors use a term absent from the page title.
- Ambiguous questions: clarification is necessary.
- Unsupported requests: no approved source contains the answer.
Use real wording from support, sales, site search, or onboarding where available.
Record an expected result
| Field | Example |
|---|---|
| Question | "Where do I allow my production domain?" |
| Expected source | Assistant security documentation |
| Required facts | Open Security; add exact domain under allowed origins |
| Important condition | Wildcard should be narrowed before launch |
| Prohibited claim | Do not describe an origin as user authentication |
| Expected next step | Save and test the embed |
| Result | Pass, partial, or fail with a note |
This makes the review about evidence rather than writing preference.
Score the answer
Use a simple rubric from 0 to 2:
| Criterion | 0 | 1 | 2 |
|---|---|---|---|
| Addresses the question | Misses it | Partly answers | Directly answers |
| Source support | Unsupported | Source is related | Every material claim is supported |
| Completeness | Omits a critical condition | Minor omission | Preserves required conditions |
| Fallback | Invents an answer | Vague limitation | Clear limitation and next step |
| Citation | Missing or wrong | Useful but incomplete | Opens the supporting source |
| Clarity | Hard to follow | Understandable | Concise and actionable |
Set the passing threshold based on risk. A technical limit or policy answer may require full source support and completeness even if the total score looks acceptable.
Diagnose failures by stage
- Search the dataset for the expected fact.
- Check whether retrieval selected the correct source.
- Compare each claim with that source.
- Check response instructions and fallback behavior.
- Look for conflicting pages.
- Confirm the source is current.
Change one stage at a time and rerun the failed question. Otherwise you may improve one example while hiding the original cause.
Keep results comparable
Record the date, dataset version or capture, assistant configuration, and reviewer note. Re-run the same critical questions after content and configuration changes.
A test set does not guarantee every future question will pass. It gives the team a stable way to detect regressions and discuss answer quality with evidence.