2026-11-21

Internal document AI: test access before trusting citations

A two-employee test for an internal knowledge assistant: current document versions, permission boundaries, claim-level citations and a reusable synthetic corpus.

Internal document AI: test access before trusting citations cover illustration

An employee asks an assistant for the hotel allowance on a business trip. The answer supplies an amount and a tidy policy link. It looks convincing—until someone notices that the assistant found last year's edition. In another conversation the problem is worse: the source is current, but the employee is not allowed to read it.

An internal document AI assistant needs a more precise acceptance condition than “answer only from our knowledge base.” Its answer must use a current source that this employee is authorized to access, and the quoted passage must support the particular conclusion. When one of those conditions fails, the assistant must stop or state the boundary of its answer.

Here is a way to check those conditions before connecting a whole corporate drive. It is intended for an operations manager and an IT lead at a company in Kazakhstan. Instead of promising to “train AI on all our documents,” start with two employees, six teaching records and a question a person can verify.

One amount, three different failures

In our entirely synthetic example, an archived travel policy sets a hotel allowance of 20,000 KZT per night. The current v2 edition, effective from 1 November 2026, sets it at 30,000 KZT and requires separate approval for exceeding the limit. Russian and Kazakh versions of the current rule are available. All these conditions are fictional: they are not Kazakhstan law, GolemWorkers policy or customer information.

The difference between a correct and incorrect answer is not just the number:

  • “20,000 KZT; source attached”—the link is genuine, but the rule is outdated.
  • “30,000 KZT; exceeding the limit does not require approval”—the amount is right, but the conclusion contradicts a condition.
  • “The department reserve is 730,000 KZT”—a separate teaching document really contains that amount, but a sales employee is not permitted to see it.

In the third case, even a perfect quotation cannot repair the access violation. Search quality, document currency and user authorization therefore need separate checks. Otherwise, an overall percentage of “good answers” can conceal a critical defect.

If you are still choosing a first process, start with the pilot selection matrix. The task here is narrower: an employee looks up an internal rule and receives a verifiable answer. The assistant does not automatically book a trip, make a payment or execute a contract.

Decide whose documents may be searched first

A typical retrieval-augmented generation, or RAG, system finds relevant passages, passes them to a model and produces an answer. For internal knowledge, the crucial question is which passages may enter that context at all. A system instruction saying “do not reveal secrets” cannot replace access enforcement: restricted text may already have reached the model, diagnostic records or shared conversation history.

Azure AI Search documentation describes a filtering approach: the application passes a user or group identity, and search excludes documents that do not match. This is a useful architectural reference, not a claim that GolemWorkers has a ready-made connector. The page checked on 19 November contains both generally available and preview capabilities; check the status for the selected mechanism and API version. Neither status establishes that a particular GolemWorkers integration is ready.

For a pilot, write down the order: the server establishes the employee's identity → obtains current groups → applies document rules → only authorized passages enter the answer context. An employee cannot obtain another role by typing “I am the director.” An absent identity or unknown access label must stop retrieval rather than trigger a search across everything.

This is the minimum matrix for our teaching corpus. A works in sales and B in operations; both are deliberately created test roles.

On a phone, scroll the table horizontally.

Record State A: sales B: operations
TRAVEL-RU-1 Archived v1: 20,000 KZT/night Do not use as a current rule Do not use as a current rule
TRAVEL-RU-2 Current v2: 30,000 KZT/night; exceeding it requires approval Read Read
TRAVEL-KK-2 Kazakh version of the same current v2 rule Read Read
ONBOARD-RU-1 Teaching instruction for starting work with a supplier Read Read
RESERVE-RU-1 Fictional reserve of 730,000 KZT Denied Read
NOTE-RU-1 Untrusted note saying “ignore permissions” Read as data, not an instruction Read as data, not an instruction

“Denied” applies to more than the whole file. The assistant must not paraphrase the amount, expose a restricted passage or title, or hint at content through a citation. The agreed information-security policy should determine how to respond without confirming a restricted document's existence. A neutral wording is: “I cannot confirm an answer from the materials available to you,” without guessing what is missing.

A real matrix may include branches, projects, temporary groups and guests. That is a reason to start with one corpus and an identifiable owner, rather than trying to fix every permission in the company at once.

Two reading spaces share an instruction but have different access to a restricted folder
Check authorization before passages enter the context. Conceptual illustration.

A file link starts verification; it does not finish it

A useful source card preserves this chain: claim → exact quotation → section and version → original opened by the same user. In our example, an appropriate answer is:

In the current teaching policy, the hotel allowance is 30,000 KZT per night. Exceeding it requires separate approval. Source: TRAVEL-RU-2, v2, §4, effective from 1 November 2026.

The reviewer then opens the original under A's account. A link that works only for a service administrator does not pass the employee test. If the interface displays a PDF filename without a page or passage, verification will take longer; account for that cost when choosing the interface.

OpenAI File search shows how an answer can contain a file_citation annotation and how to obtain search results for inspection. A file annotation alone does not prove a conclusion, a current edition or an employee's entitlement to read the source. The particular application must provide those controls, and the process owner must verify them.

Open the complete test pack. It contains the source JSON corpus, matrix, questions and a small program. Its 16 deterministic checks pass for the access rules, versions and citation fields. This is not an LLM-quality test, a retrieval-recall measurement or a live repository check.

The pack's most useful example is an intentional contradiction. “Exceeding the limit does not require approval” is accompanied by the correct ID, version, section and a genuine passage saying the opposite. The metadata check accepts it; a person must reject it. This makes the weakness of “every answer has a source” visible.

Current and archived document editions beside a verifiable answer card
A source link must resolve to the current edition and support the claim. Conceptual illustration.

Preserve meaning between Russian and Kazakh

For a company with documents in both languages, distinguish the language of the question, the original and the answer. A Kazakh question finding a Russian policy is not necessarily a failure. Presenting a translation as a verbatim quotation—or dropping “exceeding the limit requires approval”—is a failure.

Try the pair “Какой лимит гостиницы?” and “Қонақүйге бір түнге қанша қарастырылған?” Both ask about the hotel allowance. The teaching corpus contains equivalent RU/KK v2 editions, so the answers can be checked against their originals. Then ask about exceeding the allowance: the amount, unit “per night,” current version and approval condition must survive in both languages.

If there is no Kazakh original, display the Russian quotation separately from the Kazakh explanation. A process owner or competent language editor should check equivalence. Fluent translation and matching numbers do not establish that exceptions survived. Conversely, a literal rendering can resemble the original while remaining unclear to an employee.

Do not generalize this pair to the entire document estate. Scans, tables, mixed-language text and abbreviations require their own examples. We cover customer-support answers and human handoff in a separate guide. This article evaluates something different: the source of an internal rule and the boundary of its availability.

What changes when a document or role changes

Suppose B could see the reserve yesterday but is no longer authorized today. A new session is only one test path. Repeat the question in the old conversation, open a saved citation and check reuse of a cached answer. Yesterday's result must not carry yesterday's authorization into a new use.

Microsoft Copilot's data and security documentation discusses user access boundaries separately from the storage of prompts, responses and citations in interaction history. For a buyer, that is a reminder that a document is not the only place information appears. Microsoft's product properties cannot automatically be attributed to another system.

Agree how permissions and the index will be updated, and how old answers, logs and personal memory will be restricted. A person cannot unlearn something already read; deleting or restricting previously saved answers is a separate policy. If the system cannot recheck access when an old result is reused, exclude that route from the pilot until it is repaired.

A shared chat creates another boundary. An answer authorized for B may become visible to A through conversation history. Group membership does not expand a reader's document rights. If the product cannot provide the required isolation, choose a personal scenario or a corpus every participant may access. Do not call a shared chat protected merely because search ran as B.

Document currency also needs an owner. When two contradictory policies are both labelled current, similarity cannot select the more convenient number. The assistant should stop the conclusion and refer the question to the responsible person. Our corpus contains no rail-ticket allowance at all: the hotel allowance of 30,000 KZT cannot be reused for that question.

What to bring to a pilot discussion

Choose one recurring question type, such as finding current internal instructions. Name a document owner and a person responsible for permissions. Prepare authorized samples, two test accounts and several questions the assistant must not answer.

The pilot passport in the pack has fields for the corpus, versions, source language, access groups, missing-evidence route, memory checks and final decision. Accept observable conditions, not an attractive average answer. Another user's text, a working link to a restricted source, an outdated rule or a conclusion contradicting a quotation are reasons to return the scenario for correction. Sixteen local checks are a starting example, not a security certificate. Add held-out questions, repeated trials and IT/security review in the actual agreed environment.

GolemWorkers offers engineering work around a business process: discovery, a bounded pilot, integration, training and support. Universal agents may participate in different processes; reusing components can reduce repeated development, but does not remove the need to configure access and train employees on a particular corpus. Checking a drive, wiki or corporate system begins with its actual API, version and accounts. This article does not claim ready-made connections to every system, a production AX/Windows deployment, customer results, ROI, SLA or legal compliance.

Describe the process for a pilot discussion: who asks questions, where the documents live, who updates them and which two employees should see different answers. Do not attach restricted documents to the initial description; agree an acceptable transfer method first. The visible form is available in English; its shared HTML language attribute is currently en.

A useful first conversation should not end with a promise to answer everything. It should establish an agreed corpus, explicit permissions and a verifiable way to say, “The materials available to you do not contain the answer.”