Lab

How to test an AI memory without cheating on the benchmark

Short answer

People rarely cheat on purpose with a benchmark: they test on what they tuned for, keep the cases that work, change the metric afterwards. To avoid it, write the test sets before tuning the system, add negative and ambiguous cases, have answers reviewed blind, run several passes and keep the failures.

A straight stone path leads to a clear target; beside it, a second path visibly detours around an obstacle. An amber thread marks the honest path, and dull marbles, the failures, sit on a visible shelf.

A benchmark gives you a number, and a number is reassuring. That is exactly what makes it risky: you want it to be good, so you believe it faster than you check it. In nearly every case nobody lied. People simply drifted toward the choices that produce a nice result.

This article lists the ways people fool themselves when testing an AI memory, then offers a method to guard against them. It applies to us in the Lab as much as to you, if you are evaluating a tool before adopting it. Further down, one section explains why we also publish our imperfect results.

Six ways to cheat without meaning to

  • Testing on what you tuned for. You adjust the system after each failure, then rerun the same test. In the end the score measures how well you fitted the test, not how good the system is.
  • Using a known test set. If the set circulates, or if its authors are also the people tuning the system, the cases end up shaping design choices, even with no bad intent.
  • Asking questions that are too clean. A real conversation is full of half-sentences and U-turns. A test written in crisp sentences rewards systems that only work on crisp text.
  • Picking the cases that work. You drop the odd case “because it isn’t representative”. Sometimes that is true. Often it was the one that would have taught you something.
  • Changing the metric afterwards. The result disappoints on the planned measure, so you discover another one under which it looks better. The metric has to be fixed before you see the answers.
  • Only evaluating the successes. You read the wins closely to understand why they worked, and skim the failures. But the failures hold nearly all the information.

A benchmark that corrects itself

Picture a small team building a memory for an internal tool. It writes thirty or so questions, runs them, fixes what fails, runs them again. After three weeks everything passes. The team announces: “all our tests are green.”

An outside colleague then writes their own questions, without seeing the first ones, in their own words. Several fail. Nothing scandalous: the first set had served as a guide, not as a judge. What the team had measured was its ability to pass its own questions.

This story is made up, and it is probably the most common one there is.

A method that keeps you honest

No method is foolproof, but a few habits cut the risk a lot.

Separate test sets, written first. One set is for tuning, another for judging. The second is written before you start tuning, and you do not peek at it along the way. If you reuse it several times, renew it.

Negative cases. Questions with no answer, events that should not be kept. Without them, a system that always answers or writes everything wins easily. We explained the logic for topics that do not exist, and the writing-side version is described in how to test non-writing.

Ambiguous cases, kept as ambiguous. You do not force a fuzzy case into a success or a failure. You note what the system does when faced with ambiguity: does it flag the uncertainty, or does it pick a side?

Blind review. The person judging the answers does not know which version of the system produced them, or which answer the test’s author “expected”. It is the best antidote to our own hopes.

Several passes. A system that answers differently from one run to the next cannot be judged on a single run. You rerun, look at the variation, keep both results.

Failures kept on file. Every failure stays in the record, with its date and what you learned from it. A growing list of failures is more likely a good sign than a bad one: it suggests someone is really looking.

Finally, you separate the kinds of errors instead of folding them into one score, as laid out in how we measure memory errors.

What can be published, and what stays in-house

A public benchmark has three useful parts: the protocol (what was tested and how), the limits (what it does not say), and the failures. Without those, a number is a claim, not a result.

On the other hand, a team can legitimately keep some things to itself: detailed test sets (which would stop being useful once known), the system’s internals and its tuning. We do that too, and we say so. It has a cost: readers cannot redo everything, and have to take part of it on trust. That is a real limit, and we would rather write it down than hide it.

Why we also publish our imperfect benchmarks

The temptation is strong to show a result only when it is flattering. We think you should do the opposite, for three reasons.

Trust. A reader knows every test has flaws. A result shown with its flaws is more credible than a perfect one, because it shows someone looked for what is wrong.

The illusion of control. A number published alone becomes a target, then an argument. Before long people believe the problem is settled because a grade exists. Publishing the imperfections keeps the question open, including for us.

Criticism. An imperfect result shown in detail can be challenged, corrected, redone by others. A result shown without its flaws can only be believed or ignored.

Here is what that looks like for us: our first synthetic non-writing protocol correctly classified 26 events out of 26; the real world will be more complicated. That sentence carries the figure, and also what makes it less flattering than it sounds: it is a synthetic protocol, prepared by us, and real life is messier. We draw no wider conclusion from it.

It is the same reasoning as why we publish Nawame’s limitations: a memory whose weaknesses are hidden invites too much trust.

Limits of this method

  • It reduces bias, it does not remove it. Even with a set written in advance and a blind review, the test’s authors share habits and blind spots.
  • It takes time. Writing two sets, reviewing blind, rerunning: it is slower than a single test. For a small use case, it is out of proportion.
  • An honest benchmark is still a benchmark. It measures behavior on prepared cases. It does not say whether the memory helps you day to day, a question we take up in how to measure whether a memory avoids rework.
  • Publishing is not proving. A public protocol can be criticized, but that is no guarantee its results will hold for you.
  • We can be wrong too. This article describes what we aim for, not an infallibility we do not have.

Before you believe a score

When you see a benchmark result, yours or someone else’s, ask these questions:

  1. Who wrote the test cases, and was it before or after tuning the system?
  2. Are there negative and ambiguous cases, or only questions that have a right answer?
  3. Was the metric fixed before the results came in?
  4. Are the failures shown, or only the successes?
  5. What does the result say about the real world, and what does it not say?

If three of the answers are missing, treat the score as a hint, not as proof. The Lab collects our method work, with its limits.

Your next prompt should not have to contain your whole story.

Start building a memory your AI tools can find again when it matters.

Start free