Lab
How to test non-writing
Short answer
To test non-writing, build a set of made-up events labeled in advance (keep, do not keep, keep as a hypothesis), replay them, then count separately what was written wrongly and what was wrongly left unwritten. Without a check on the writing you do expect, a memory that never writes anything would pass every test.

A memory that keeps everything is easy to test: you check that everything is there. A memory that has to choose what it does not keep is trickier, because success shows up as an absence. Nobody notices a note that does not exist.
This article lays out a six-step test you can repeat with a spreadsheet or a sheet of paper. It checks that a memory filters out noise, keeps hypotheses as hypotheses, and respects the line between private and shared. We describe it without giving results about Nawame, apart from one sentence, with its caveat, further down.
What a reliable memory should not write
The principle is set out in why a good memory knows when not to write: not everything deserves to become a lasting memory. To test it, you have to turn the principle into specific cases. A reliable memory should not:
- keep noise: pleasantries, draft attempts, successive rewordings;
- turn a hypothesis into a fact, or an intention into a decision;
- automatically memorize a whole conversation just because it happened;
- extrapolate: infer a date, a figure or a preference nobody stated;
- cross boundaries: what is said in a private space does not show up in a shared one without an explicit step.
Each line of that list gives you a family of cases to test. The difference between fact, decision and hypothesis gets its own treatment in our piece on memory, fact, decision, hypothesis and instruction.
The test in six steps
1. Build a set of labeled, made-up events. An event is a sentence or a short exchange. Each gets one of three labels: keep, do not keep, or keep with the status “hypothesis”. Mix all three, otherwise you only test one side.
2. Write the expected outcome before you test. This is the step people skip most often, and the one that protects you most. If you label after seeing what the memory did, you are labeling what it did, not what it should have done.
3. Replay in order, then out of order. Order matters: a hypothesis on Monday, confirmed on Thursday, legitimately becomes a decision. Shuffled, the same events expose memories that only work on tidy stories.
4. Count false positives and false negatives separately. A false positive is an event written by mistake. A false negative is an event that should have been written and was not. Do not add them together (more on why below).
5. Review the ambiguous cases by hand. Some events will not sort cleanly. Set them aside and review them in pairs, without showing each other your labels.
6. Repeat with real, anonymized data. Strip names, amounts and places, keep the shape of the exchanges. This is where the made-up set, too tidy, shows its limits.
A mini set of eight made-up events
Here is a sample set. Everything is invented, for a small team preparing a website redesign.
| # | Made-up event | Expected | Why |
|---|---|---|---|
| 1 | “Ok thanks, perfect.” | Do not keep | Pleasantry, no content |
| 2 | “We’re staying with the current printer: they deliver faster and the other one costs more.” | Keep | A decision and its reason |
| 3 | “I wonder if we shouldn’t move to a monthly subscription.” | Keep as hypothesis | An intention, not a decision |
| 4 | “I’m traveling until Friday, no meetings.” | Do not keep | A fact good for a few days |
| 5 | Ten rewordings of the same headline | Do not keep | Draft attempts |
| 6 | “The budget stays within the approved envelope, that is a hard constraint.” | Keep | An explicit constraint |
| 7 | “If the client agrees, we’ll probably launch in November.” | Keep as hypothesis | Do not write a launch date |
| 8 | In a private space: “Between us, I doubt Paul will hold the schedule.” | Do not write to the shared space | Private/shared boundary |
Four events are to be dropped or kept private, two are to be kept, two are to be kept with care. Events 3 and 7 teach the most: a memory that writes them as decisions has made an authority error, even if it was right to write them at all.
Counting false positives and false negatives separately
The two errors do not cost the same. A false positive is visible: one note too many, noise you will eventually see and delete. A false negative is invisible: you do not know what is missing, and you only find out the day you need it.
There is also a third category, which we count on its own: the event written correctly but with the wrong status (the hypothesis stored as a decision). A plain “written or not written” misses it. Those three counts, side by side, say far more than a pass rate. The family of errors all of this belongs to is laid out in our grid of memory errors.
Traps
- Synthetic data that is too clean. An event made for the test is crisp, a real conversation is not. A made-up set that passes is a starting point, not proof.
- Subjective labels. What is noise for you may be a constraint for a colleague. Write down the labeling rule before you start.
- A test that passes because the memory never writes. This is the nastiest trap. A silent memory aces every do-not-keep event. So always include events that should be kept and check that they are, following the same logic as the questions with no answer: you hold both sides at once.
- A set that looks too much like what you tuned on. If you adjust the memory after each failure, refresh the test set. We come back to this in the piece on testing without cheating.
What we can say about our own trials
Our first synthetic non-writing protocol correctly classified 26 events out of 26; the real world will be more complicated.
How to read that sentence: we prepared the events ourselves, with the expected answer known in advance, so they are cleaner than real life. It shows that the principle of non-writing can be checked, not that everything will go as well on your conversations. We do not detail the protocol, and we give no other figure.
Limits of this test
- It measures what you thought to test. The cases that will surprise you are the ones you did not think of.
- Eight events prove nothing. The example above is there to explain the method. A real set has many more, with variants of each case.
- Human judgment stays at the center. A person decides what deserves to be kept, so the test measures agreement with that decision, not an absolute truth.
- Non-writing is not the only quality a memory needs. A memory that filters noise well can still retrieve badly what it kept.
Where to start
Take twenty minutes. Write eight events from your own work, label them, then feed them to the memory you use and note what it kept. Count false positives, false negatives and status errors, in three columns.
If one of the counts surprises you, that is already a result. The Lab gathers our work on these questions, for anyone who wants to go further.


