Lab

How we measure memory errors

Short answer

An AI memory can go wrong in at least seven ways: false recall, omission, out-of-context recall, an old fact taken for current, duplication, an authority error, and writing what should never have been kept. Each error has its own detection question and gets counted separately, because one overall score hides where the memory fails and in which direction.

A graduated glass ruler lying along a row of grey marbles: a few marbles carry a small amber spark, and clean empty slots break up the row.

Saying an AI memory “works well” means nothing until you say which error you are talking about. It can forget what it should have found, invent what you never told it, or bring back a fact that was true once and no longer is. Those are three different failures, and they are neither spotted nor fixed the same way.

The short answer of this article: sort the errors before you count them. Below is the grid we use in the Lab, seven kinds of error with one way to catch each. It is a method, with no numeric result about Nawame in it, and you can apply it to any memory, including a plain notebook you share with an AI.

Why a single score misleads

Picture two memories with the same overall grade. The first fails by making things up. The second fails by staying silent when it should have answered. On paper they are equal. In practice, one walks you into a wall with full confidence, and the other makes you waste time hunting for something it already had.

There is a trade-off underneath. A memory tuned to be more cautious invents less but goes quiet more often. One tuned to be chattier finds more but also gets more wrong. A single score blends both effects, so you can “improve” the grade just by pushing errors from one side to the other. Counting each type of error separately makes that shift visible.

Seven errors, and how to spot each one

For each error we ask the same three things: what is it, how do you detect it, and what does it look like in a made-up case. The examples are invented for illustration.

False recall

A memory is invented, or pinned on the wrong person or the wrong date. To detect it, take each retrieved memory and ask where it sits in what was actually written. No source means an invention. A source that says something else means a misattribution.

Example: the AI states that “Karim approved the quote”, when the note only says Karim asked for it. A very useful variant is asking questions that have no answer at all, which we cover in our piece on topics that do not exist.

Omission

The memory finds nothing although the information was there. To catch it, write the expected answer in advance for a question whose source note you know, then ask it in words different from the note’s. An omission that only shows up with different vocabulary is the most common and the least visible.

Example: the note says “Ines runs the server”, and the question “who looks after hosting?” gets no answer.

Out-of-context recall

What comes back is true, but has nothing to do with the situation. Detection is a manual read with a single question: would this item have changed my decision? If not, it is clutter.

Example: while preparing a quote for one client, the AI surfaces another client’s invoicing rule. It is accurate and belongs somewhere else. It is a cousin of the false precedent of semantic search, where two situations look alike without being comparable.

An old fact taken for current

Something was replaced, and the memory brings back the old version as if it still held. To test for it, build pairs: a fact, then its replacement, then a question. The right answer gives the recent version or clearly flags the replacement.

Example: in March the printer changed. In September the AI still suggests the old one.

Duplication

The same information exists in several slightly different versions, and retrieval returns three at once. You spot it by counting duplicates at retrieval time, and above all by checking whether the variants contradict each other.

Example: three notes mention the budget, as “4,000”, “4,000 EUR” and “forecast 4,500”. Which one is the reference?

Authority error

A hypothesis is presented as a decision, or a draft as a reference. To detect it, the test set carries an expected status on each item (hypothesis, decision), and you check that retrieval preserves it.

Example: “we could move to monthly payments” turns, two weeks later, into “the decision to move to monthly”. Nobody decided anything, but the sentence gained confidence on the way.

Writing what should not have been written

Something was kept that should not have been: a pleasantry, a draft attempt, a confidence, a fact good for one day. This is a writing error, not a retrieval error, and it needs its own test. We gave it a full article: how to test non-writing.

Writing and retrieval: two separate measurements

These seven errors do not happen in the same place. Some are born when the memory decides what to keep (writing what should not be written, freezing a hypothesis into a decision). Others are born when it answers (omission, out-of-context recall, an old fact taken for current). Duplication can come from either side.

If you only look at the final answer, you do not know what to fix. A memory can write perfectly and retrieve badly, or the other way round. So we give each stage its own expected truth and compare each stage against it, not against the final answer. It is more work, and it is the only way to know what to repair.

Measuring when the truth is ambiguous

Real life offers few clean cases. Is “we’ll see in September” a decision to postpone, or a hypothesis? Two honest people sometimes answer differently.

Our answer is not to force the case. Two reviewers label each item on their own, without seeing the other’s call. When they disagree, the case goes into an “ambiguous” bucket that counts neither as a pass nor as a failure. What we look at instead is something else: did the memory flag the uncertainty, or did it settle the question with confidence? This is the same logic as the blind review that also protects labels from our own hopes.

What we do not publish

We publish the method, the types of error, and the limits that come with them. We do not publish the detail of our test sets, the internal workings or our tuning. That is part of how we build the product, and we say so plainly in why we publish Nawame’s limitations.

One honest consequence: you cannot replay our tests exactly. You can replay the method on your own notes, which is often more instructive, because your real subjects are messier than ours.

Limits of this method

  • The grid is ours. There is no standard taxonomy of memory errors. The categories overlap: a hypothesis turned decision is both an authority error and, often, a false recall.
  • Labels stay subjective. Having two reviewers reduces the problem without removing it.
  • Counting is not weighing. An error about a tone preference and an error about a budget decision count the same in a table, and not in real life.
  • Synthetic data is cleaner than reality. A test set built for the occasion lacks the half-sentences, the U-turns and the innuendo of real conversations.
  • Rare errors slip through small samples. A short test set can miss an error type that does exist.
  • A memory without errors is not necessarily useful. Measuring errors does not tell you whether it saves work. That other question is the subject of our piece on measuring the rework a memory avoids.

A grid to try this week

You do not need a lab. An hour is enough.

  1. Pick ten items from your notes or from the memory you use: a decision, a hypothesis, a replaced piece of information, an unimportant detail.
  2. For each one, write the expected answer to a question in advance, with a status (decision, hypothesis, replaced).
  3. Ask the questions in words different from those in the notes.
  4. File each gap under one of the seven families, and count per family without adding them up.
  5. See which family dominates. That one tells you what to change: the notes, the writing habit, or the tool.

If you want to follow what we publish on these questions, the Lab gathers our work.

Your next prompt should not have to contain your whole story.

Start building a memory your AI tools can find again when it matters.

Start free