Internal Knowledge Assistant: Sources, Access and Review

Choose the content boundary.

Identify the documents the assistant should use and the questions it is intended to help with. Check which versions are current and who owns them. Exclude material that the pilot audience should not access. A large collection of unreviewed documents can create more uncertainty than a small, well-defined source set.

Make unsupported answers visible.

Agree how the assistant should respond when the source material does not support an answer. Review whether references lead to the relevant passage and whether the response reflects it. Include questions that cannot be answered from the selected documents so the evaluation checks the boundary as well as useful answers.

Plan content maintenance.

Decide how revised or withdrawn documents reach the source set. Assign a route for reporting misleading answers and outdated references. Treat the assistant as part of a content-maintenance process, with review responsibilities that continue after the first demonstration.

Establish the source and permission model first

List the documents the assistant may use and identify their owners. Check which versions are current, which should be excluded and how permissions differ between users. Choose a narrow starting collection whose boundaries the team can explain.

Prepare questions answerable from the material, questions requiring multiple documents and questions the collection cannot answer. Review whether the response preserves important qualifications and points readers to the relevant source. Test access restrictions independently of answer quality. Define how revised or withdrawn documents are handled and how unanswered questions reach a content owner. Expanding the source collection should be a reviewed change, since new material can introduce conflicting information or broader access requirements.

A knowledge-assistant source register

Keep this register with the content maintenance process. It should show both the information boundary and the people responsible for it.

• Source identity: document title, owner, approved version and intended audience.
• Access scope: permitted users, restricted sections and material excluded from the pilot.
• Freshness: review date, update process and treatment of withdrawn or superseded information.
• Answer evidence: expected source references and examples of supported questions.
• Unanswered route: what the assistant says and who receives the unresolved question.
• Change control: approval for new collections, revised permissions and repeated evaluation.

A larger document collection is not automatically more useful

Adding outdated or conflicting material can make it harder to establish which answer is authoritative. Resolve source ownership and version decisions rather than expecting the assistant to invent a policy hierarchy.

Do not give the system broad document access simply because the pilot team can access those files. Review the permissions of each intended user group. Also keep source references visible so staff can inspect the underlying material when a decision depends on precise wording or a current policy.

What should happen when no source answers the question?

The assistant should clearly state that the approved material does not establish an answer and provide the agreed route for help. Capture the question for the source owner without silently treating it as permission to search unrelated or restricted information.

ILLUSTRATIVE PLANNING EXAMPLE

Example: searching an employee handbook

Start with a maintained handbook and a named content owner. Check which sections employees are allowed to access and how revised policies replace old versions. Test questions whose answers are absent as well as questions that depend on a date. Ask the assistant to show the source used so a reviewer can inspect it.

Agree the route for unanswered or disputed questions. A knowledge assistant should not invent a company policy when the approved material does not contain an answer.

FURTHER READING

Official resources to explore

Explore the related official guidance below. The templates and examples are practical starting points to adapt to your own project.

• NIST AI Risk Management Framework Playbook