Lab

Can an AI judge whether another AI remembered correctly?

Short answer

AI judges are useful for large-scale evaluation, but they should not be the only arbiters. Ambiguous cases, superseded decisions and final-answer usefulness still require human judgment.

Two identical structures made of clear and dark panels face each other across a smooth floor, joined only by a few fine threads, one of them amber.

Improving memory requires testing it.

As the number of tests grows, one temptation appears quickly: ask another AI to grade the answers.

That is useful.

It is not enough.

AI judges can inspect many cases quickly

A model can compare an expected answer with the answer that was produced.

It can detect:

  • missing information;
  • contradictions;
  • irrelevant answers;
  • an old decision used instead of the current one;
  • missing reasoning.

For hundreds of cases, that first layer of evaluation is valuable.

Memory contains ambiguous situations

Consider a decision that was later replaced.

If the question is “What are we doing now?”, the old answer is wrong.

If the question becomes “Why did we change?”, that same old decision becomes relevant again.

An automatic judge may struggle to decide whether memory behaved correctly without understanding the exact intention of the question.

Exact does not always mean useful

Memory can retrieve a perfectly exact sentence that contributes nothing to the answer.

Conversely, it can combine several memories into a useful answer without matching the expected wording.

That is why memory evaluation has to look beyond textual similarity.

What we actually want to measure

Useful evaluation asks several questions:

  1. was the right context retrieved?
  2. did bad context influence the answer?
  3. was the current state separated from history?
  4. did the final answer become more useful?
  5. should the AI have abstained?

No single score perfectly captures all of that.

A concrete example

Test question: “Where do we stand on choosing the host?” The memory holds a March decision (host A) and a June decision (host B). The AI answers B. An automatic judge compares it with the expected answer, sees B, and approves.

But if the answer does not say that A was dropped for a specific reason, and that this reason explains the next step, it is exact and of little use. A reviewer who knows the subject notices it more easily.

The role of human judgment

AI judges make it possible to explore faster.

Humans still matter for cases where meaning depends on context, intent or the authority of a piece of information.

This is not unique to Nawame.

It is a general limitation of automated evaluation: the system doing the judging can also be wrong.

The useful discipline is therefore to automate enough to measure more, without confusing evaluation speed with scientific truth.

Our methods and their flaws are described in how to test an AI memory without cheating on the benchmark and how we measure memory errors. We explain why we also publish imperfections in why we also publish Nawame’s limitations, and the studies are gathered in the Lab.

Limits of this approach

We do not claim that any one evaluation method is the right one. Combining an automatic judge with human review costs time, and human review has its own biases. No figure from these tests should be read without knowing the method that produced it.

Your next prompt should not have to contain your whole story.

Start building a memory your AI tools can find again when it matters.

Start free