A practical checklist for testing whether a business AI assistant uses current policies and handles older sources correctly.
Before testing
An AI assistant can cite a real document and still give the wrong answer if the document has been replaced. A service desk might update its collection hours while an old PDF still appears in search results. Both documents look plausible and both may use the same terms. Test how the assistant handles that conflict before it answers customers. Start with one rule that has actually changed and work through what should happen before, during and after the change.
Choose a rule with a clear date and owner
Pick a rule that staff can explain without relying on the assistant. It might be a service cut-off, a returns instruction or the list of documents someone needs to submit. The test should have a clearly correct answer for a specific date. Ask the person responsible for the rule to identify the official wording, when it takes effect and whether any requests already in progress follow the older process.
Give the change a small record that includes the rule, the effective date, its owner and the documents it supersedes. Include a note about where the old wording is still likely to appear. A help page, staff handbook and saved reply may all contain a copy. The team should be able to identify the current source from the record without deciding which PDF has the most recent file name.
Prepare a realistic set of conflicting sources
In a test environment, give the assistant a current policy, an older copy and an example of a draft that staff have yet to approve. Use fictional values and customer details. Make the documents similar enough that a search can find more than one version. Keep a record of the passages you expect it to use and of the documents it may retrieve but should ignore for a current answer.
Try a case where the newer document is missing an important detail that the old version contains. For example, a notice may announce that a collection counter has moved while the previous leaflet lists the old counter and its opening hours. Decide whether the assistant can carry over the hours from another approved source or whether it needs to ask staff. Combining fragments from two documents can create a confident answer nobody actually authorised.
Test ordinary questions, follow-ups and dates
Write a question a customer would actually ask, such as whether an item can be collected on Friday afternoon. Add a shorter version that leaves out the day and another that asks how the process used to work. If the service follows different rules for orders placed before a change, include a case with an order date and a case where that date is unknown. The useful response may be a single clarifying question.
Use follow-ups to see whether the assistant sticks with the context. Ask about Friday, then change the date to next month. If the question crosses the policy change date, check that the answer changes with it. Run the same scenario in a new conversation and in an existing one. The first test can expose problems in the source search, while the second can reveal that the assistant is carrying an earlier answer forward.
Check the source the answer actually relies on
Read the cited passage after each test. Does it support the service detail in the answer, or does it only establish that the service exists? A link to the right document is useful only if the passage behind it supports the claim. Ask the reviewer to note the source, relevant wording, effective date and the exact sentence the assistant produced. That record makes it easier to reproduce a failure.
When an answer uses the wrong source, check what the system retrieved before rewriting the instructions. An old passage may have been retrieved because a file was reuploaded with a new modification date. Or the current rule may have been split from its effective date during indexing. Give document status and dates explicit fields where the system supports them. Keep the old source accessible to authorised staff if it is useful for historic queries.
Test the change as an operating procedure
Rehearse a policy update from the owner’s approval through to a customer answer. Find out how the replacement is uploaded, when the search index refreshes and how staff can tell the refresh has finished. Then test from the same channel the customer would use. A passing test in an editor may still miss a production cache or a separate help centre connection.
Write down what the team does during the refresh window. It may direct questions about the changing rule to staff until the new source is confirmed, while the rest of the assistant continues to answer routine questions. Make it clear who turns the affected answers back on. Verify that staff can see the customer’s question and the relevant policy context during the handover.
Set a pass condition staff can repeat
A test passes when the answer reflects the applicable rule, the source supports it and the assistant asks for clarification when the date or details could change the answer. Include a case in which the documents genuinely conflict and a case in which the current answer has no approved source. Agree on the wording for escalation so customers know what staff can confirm and how to follow up.
Repeat the tests after the team changes a source, an instruction or a search configuration. Add new cases when a real enquiry reveals a gap. Save the test date, source versions and decision so a different person can run the same check later. A small set of reliable checks is easier to maintain than a long script that nobody reviews when the service changes.
Bring one example to the project brief
Before commissioning an assistant, bring one policy change, examples of the older and current wording, and the names of the people responsible for approving the answer. Include the channels customers actually use and the point at which a person should take over. If you need help defining that work, Blackstone Intelligence Sarawak’s AI agency services describe support for business AI projects. The Blackstone Intelligence Sarawak website provides company information and contact details.