Lab

Why we deliberately test topics that do not exist

Short answer

We deliberately ask Nawame about topics that were never stored, to check that it does not make up an answer. A memory that always answers passes every test where the answer exists, yet a false memory is worse than no memory. These tests are work in progress, not a guarantee.

A copper line crosses an empty space and ends at an empty glass ring, with fine graphite tick marks below.

Testing a memory usually starts with questions whose answer exists: “What did we decide about this supplier?” If it finds the right note, the test passes. It is easy to set up and it is flattering.

But the most instructive question is often the opposite one: the question whose answer does not exist.

Why tests where the answer exists are not enough

A positive test checks that the system finds what was stored. A negative test, which we call here a nonexistent-topic test, asks about a subject that was never stored and checks that the system does not make up an answer.

A smoke detector that rings all the time catches every fire. Nobody calls it reliable for that. In the same way, a memory that always offers something passes every positive test. Only the negative test shows what happens when there is nothing to find, which is a common situation: people often ask a memory about things they never put into it.

Why a false memory is worse than no memory

With no memory, you know it and you start from zero. With a false memory, you build on sand.

Imagine a small consulting firm. Someone asks the AI: “Have we already set a price for the training aimed at nonprofits?” Nothing was ever discussed on that subject, but there is a note about conference pricing. An overzealous memory links the two and answers with confidence. The quote goes out based on a price nobody approved.

It is the same danger described in our article on the false precedent that semantic search can produce. Testing nonexistent topics is the most direct way of hunting for it at the source.

What we are looking to observe

We do not detail our protocols: they are part of how we build the product. The principles, however, are no secret.

The easiest case is a subject completely foreign to everything stored. The hardest is the neighboring subject: close in vocabulary to real memories, yet distinct. That is where wrong answers slip in. A third case deserves attention: the subject of which only an old version remains, perhaps replaced since.

In each case, we look at what the system announces. The right outcome is not always the same: sometimes “no convincing precedent”, sometimes “possible connection” along with what differs, depending on what is actually stored. We are not chasing a spotless score. We want to understand where the system goes wrong and in which direction, because a mistake in one direction and a mistake in the other call for different fixes.

Here is a fictional example of how to prepare such a test, which you can repeat on your own notes. A small agency has kept decisions about its conference fees and its invoicing tool. To build a neighboring subject, take one of those topics and change a single element: “the fee for a conference abroad”, when nothing was ever written down about abroad. Write the right answer before asking the question: here “no precedent” or, at best, “the note about conferences in France, which says nothing about abroad”. Then ask the question in two different wordings and compare. If the answer changes from one wording to the other, that is already a signal. Finally, record the gap between the expected answer and the one you got, without rushing to explain it: the point of the test is to spot cases, not to excuse them.

Calibration, in both directions

Calibration is the match between the confidence a system displays and what it actually knows. It degrades in two directions. An overly talkative system makes things up. An overly silent one answers “nothing” even when there was something. The first does badly on negative tests, the second does badly on positive tests.

That is why a negative test alone would not be enough either: a memory that never says anything would pass every nonexistent-topic test. Both sides have to be held at once, and that is what makes the exercise hard. The different types of errors and how to count them are described in how we measure memory errors.

What we can say about our measurements

On a neighboring problem, we do have one public data point: our first synthetic non-writing protocol correctly classified 26 events out of 26; the real world will be more complicated.

A few precautions when reading that figure. It concerns the decision not to keep certain things (we discuss it in the article on why a good memory knows when not to write), not the response to a nonexistent topic. We do not claim it shows anything about the latter. It is also synthetic: the events were prepared for the test, so they are cleaner than real life, where things are ambiguous and often partial.

We do not publish a figure on nonexistent topics. We would rather have no number than one that suggests the question is settled. The method for testing non-writing itself is described step by step in how to test non-writing.

Current limits

Several limits, stated plainly.

  • A synthetic test is not reality. It measures what we thought to test. The situations that will surprise us are, by definition, the ones we did not think of.
  • Nawame does not eliminate errors. It may answer “no convincing precedent” when one existed, or flag a connection that was not worth flagging. Found does not mean true, and finding nothing does not mean nothing exists.
  • You cannot verify our protocols. Since we do not publish them, you take our word for it. That is a real limit of this article, and we prefer it to a false transparency that would spell everything out.

These topics are among the problems that the Lab studies and measures: false precedents, noise, context selection, timing. We also explain why we publish Nawame’s limitations rather than keep quiet about them.

What this changes for you

You can run this test yourself, on any memory, including the one you already use. Ask it a question you know has no answer in what you have stored. Note what it says: does it admit it has nothing, or does it offer something confident?

Then try it with a subject close to a real one. That is the most revealing trial, and it takes only a few minutes. To avoid fooling yourself while testing, see how to test an AI memory without cheating on the benchmark.

If you want to try it with Nawame, the free plan gives you up to 10 memories. The FAQ spells out what the tool does and does not do.

Your next prompt should not have to contain your whole story.

Start building a memory your AI tools can find again when it matters.

Start free