Lab

How to measure whether a memory really avoids rework

Short answer

To find out whether a memory avoids work, count the rework: what you re-explain, re-decide, look up again or correct for the second time. Keep a repetition log for a few weeks before and after, with durations marked as estimates. Compare how often repeats happen rather than how long they take, and watch out for the novelty effect.

A glass hourglass whose sand partly climbs back up an amber thread; beside it, a pile of small grey loops, the redone tasks, is lower than the taller pile next to it.

An AI memory easily feels useful. You find an old note, you pick up a reflex, you feel more organized. The real question is more down to earth: are you redoing fewer things than before? A feeling of efficiency does not answer that. Counting does.

This article offers a simple way to do it yourself or with a team, using a notebook and a few weeks. It needs no tooling and works for any memory. We give no numeric result about Nawame here: whether a memory is useful gets judged on your work, not on someone else’s figures.

What we call “rework”

Rework is what you do a second time when the first should have been enough. It comes in four forms:

  • Re-explaining: giving the AI the context, the goal and the constraints it already received once.
  • Re-deciding: settling a question that was already settled, because nobody can find the reason behind the choice.
  • Looking up again: rerunning a search, reopening a document, rereading an exchange to recover what you knew.
  • Correcting a mistake you already saw: fixing the same AI error for the second time, because the first correction did not survive.

These four families share a useful trait: you notice them as they happen (“huh, I’m doing this again”). That is what makes measurement possible without fancy instruments.

A repetition log over a few weeks

The protocol fits on a page.

Baseline period, two weeks. Work as usual. Whenever you catch yourself redoing something, write one line: the date, the family (one of the four above), the subject, and an estimated duration in minutes. Put “estimated” at the top of the column: these are orders of magnitude, not measurements.

Period with a memory, two more weeks. Set up the memory you want to evaluate, give it a week to settle in, then pick up the same log again, with the same categories. Two weeks per period, four if you can manage it, is enough to smooth out odd weeks.

A sample log, entirely made up:

Date Family Subject Estimated time
Nov 3 Re-explain Newsletter audience 10 min
Nov 4 Re-decide Site typeface 15 min
Nov 6 Look up again Printer’s quote 10 min
Nov 7 Correct Client name misspelled, again 2 min

A line takes thirty seconds. If the log turns into a chore you will drop it after three days, and the measurement will be wrong before it starts. On a team, agree on the four families beforehand and keep one shared sheet, so that “re-decide” means the same thing for everyone. Otherwise two people will file the same repeat under different headings and the comparison will blur.

Reading the log without telling yourself stories

Look at frequency first, not time. Time is estimated, so it is fragile. The number of lines per week can actually be counted. Compare the two periods family by family: did the “re-explain” repeats go down? And the “re-decide” ones?

A few qualitative signals round out the picture:

  • is the start of a session shorter?
  • does “what did we decide?” get an answer on the first try?
  • do the same mistakes keep coming back, or did the correction hold?
  • are there topics nobody reopens anymore, without anyone having forbidden it?

One caveat: a drop in “re-explain” alongside a rise in “correct” can mean the memory is supplying context, just the wrong one. Rework does not say everything about quality, it is one indicator among several. The errors themselves are tracked with another grid, described in how we measure memory errors.

Traps

The novelty effect. During the first days with a new tool you pay attention, try harder and write everything down. The second period looks better for reasons that have nothing to do with memory. Let a week go by before you start counting.

Measuring time lost rather than time saved. Saved time does not exist in the log: it only holds what you did. Do not conclude “we saved three hours” from “we logged three hours fewer of repeats”. Say only that you logged fewer repeats.

Crediting the memory for what a better prompt does. If, between the two periods, you also got better at writing your requests, the drop is not the memory’s doing. A careful prompt fixes part of the cases, and our piece on what a great prompt cannot contain shows where it stops. To isolate the effect, change one thing at a time.

Comparing weeks that do not look alike. A launch week and a holiday week do not produce the same repeats. Compare comparable work.

Skipping the embarrassing repeats. You remember the moments when repeating yourself made you feel silly. You forget the small ones. Counter the bias by picking a fixed time of day to fill in the notebook.

When it is not worth measuring

Measuring has a cost. If your project is short, if the use is occasional, or if the annoyance is glaring (you re-explain the same thing every morning), a log is extra work for a conclusion you already have.

In many cases a simpler fix also does the job: a “project state” file reread at the start of each session. We describe that tool-free method in how to keep AI project context for months. If that file is enough to make your repeats drop, you need nothing more.

If you later try a persistent memory, the same log will tell you whether it brings anything beyond the file. Our guide to using Nawame on a long-running project describes one example of a setup, which you can measure this way.

Limits of this method

  • Durations are estimates. They move with your mood that day. Count occurrences before minutes.
  • It is self-reported. You are both the subject and the observer. On a team, have someone else read the log.
  • There is no control group. You compare “before” and “after”, so anything else that changed in between muddies the reading.
  • Four weeks is a small sample. The result tells you something about your weeks, not about memory in general.
  • It does not measure answer quality. A memory can reduce your repeats while serving you wrong memories.
  • It is not a benchmark. To evaluate a memory more rigorously, without telling yourself stories, see our article on how to test without cheating.

A two-week protocol to copy

  1. Weeks 1 and 2: log every repeat (date, family, subject, estimated time), changing nothing in your habits.
  2. End of week 2: count the lines per family. Do not compute a gain yet.
  3. Week 3: set up the memory. Do not count, let the novelty effect pass.
  4. Weeks 4 and 5: resume the log with the same categories.
  5. Compare frequencies per family, then reread the lines that remain: they show what the memory did not fix.
  6. Decide: keep it, adjust it, or drop it.

The Lab approaches these measurement questions with the same caution: methods you can rerun at home rather than figures to take on trust.

Your next prompt should not have to contain your whole story.

Start building a memory your AI tools can find again when it matters.

Start free