Lab
Why an assistant can cite evidence and miss its consequence
Short answer
Providing evidence does not ensure that an assistant will use it correctly. A model can cite a source and miss a contradiction, or read a qualification without changing its answer. We investigated this through experiments on guidance placement and alerts tied to the question.

The intuitive idea: repeat the rule at the right time
If important guidance sits far from the final answer, moving it closer should help. An experiment compared guidance at the start, immediately before answering, in both places and absent altogether. Business information remained identical. The test used synthetic corpora and three models.
The main result was negative: moving the guidance did not improve contradiction reporting. Secondary outcomes looked more promising, especially citations and justified abstention, the ability to state that the available evidence does not support an answer.
Those improvements were tempting. But they were not the experiment’s primary criteria. Examining many secondary effects increases the chance of finding a favourable result by accident. They needed a fresh confirmation rather than an immediate product conclusion.
Confirmation separated the effects: scope and limits
A new campaign compared two placements on two fresh corpora using three models. Citation rates rose from 0.95 to 0.99. Accuracy did not show an established improvement, while correct abstention fell from 0.74 to 0.67. The success rule written before testing was not met.
This does not establish that nearby guidance is always useless. It shows that improving a local action, citing, does not necessarily improve judgement about whether evidence answers a question or conflicts. Small samples and the benchmark’s budget stop also limit the conclusion.
An alert cannot replace missing evidence
The Lab also investigated question-dependent alerts. In another campaign, an alert assessed by the host model reduced unnecessary warnings, but the gain on difficult traps fell short of the predefined threshold. The benefit did not replicate with the second model.
Error analysis revealed two different causes. Sometimes the decisive note was buried in a meeting record and never retrieved. Sometimes it was provided but misinterpreted. An alert added after retrieval cannot solve the first problem. A persuasive alert can also influence another model in an unhelpful direction.
Measurement needs its own evidence
Automatic metrics also required checking. In the first guidance experiment, a blind review found that many numbers flagged as invented were correct calculations. Treating the raw metric as a confirmed error would have led to another false conclusion.
We therefore separate the presence of a source, the fidelity of a citation and the validity of a conclusion. A citation provides a route to evidence. It does not establish that the evidence supports every claim in an answer. This distinction matters for both evaluating an assistant and designing its context.
The consequence for Nawame
Useful memory should make sources, uncertainty and contradictions accessible. The receiving model still interprets them. The Lab can investigate better presentation without treating persuasive wording as demonstrated improvement.
This is why we also study multi-step retrieval and the status of rewritten decisions. Each experiment needs to target the right part of the chain: finding information, returning it, reading it or drawing a conclusion. An improvement at one stage is valuable, but must be tested again at the level of the answer.
Related reading: Source, validation and authority are three different things and Can an AI judge whether another AI remembered correctly?.


