Machine Testimony Conformance Census  ·  No. 1  ·  4 September 2026
Eight systems, read at pinned commits, against the Testimony Record specification

What Eight Agent Systems Record About What They Did

The first conformance census of the Testimony Record specification

Troy Clifford

Machine Testimony

---

---

1. What was measured

The specification ships a validator. It answers one question, which is whether a given record conforms, and it cannot answer the question this census asks. Point it at a framework that has never heard of the specification and it reports nothing, for the same reason a spellchecker reports nothing about a blank page. Running it across eight projects would have produced eight identical non-answers.

So the requirements were rewritten as questions about what a system keeps. Instead of asking whether it writes a belief entry with an evidence array, the question became whether the source a stored fact came from can be recovered from the store at all.

That distinction matters more than it sounds. A system holding the information in some private shape is a translation away from conformance, perhaps a day's work. A system that never captured the information cannot get there at any price, because the thing itself is gone.

Twenty requirements came out of the four levels, spread five, five, seven and three. Each subject declares what it is in the business of doing and is scored only on requirements falling inside it. A vector store is not an approval gate, so it is not marked down for lacking one.

2. What this does not produce

No totals. No percentages. No ordering of one system against another, and the tooling cannot emit one even if somebody asked it to.

These systems are not attempting the same job. A single number laid across them would be read as a league table inside a week, and after that nobody would read anything else on the page.

What comes out instead is a map of gaps: for each system, the highest level its existing capabilities already satisfy, and above that, the specific facts it does not keep together with where that was checked. A level counts as reached only when nothing in it is missing. That is a strict rule with an unfortunate side effect, which is that a system meeting four requirements out of five shows no level at all. Table 2 gives the per-level counts so the result reads correctly.

3. The assessment

Tables 1.1 to 1.4 give the verdict on every requirement, one table per conformance level. Throughout: "yes", the system keeps this fact; "part", partially, or not durably; "no", it does not keep it; "n/a", outside what the system claims to do; "?", kept somewhere that could not be read, so not settled.

Table 1.1. TR-1 Recorded. The record exists and is append-only.

Table 1.2. TR-2 Explained. Every belief resolves to its evidence, and disagreements survive.

Table 1.3. TR-3 Gated. Actions carry a verdict, and approvals carry a name.

Table 1.4. TR-4 Verifiable. The record can be shown not to have changed.

Table 2. Requirements met, out of those that apply to what each system claims to do. Never summed across levels.

OMEM is the specification's reference implementation and was written by the author of these questions. Its row proves nothing. It appears here because the questions ought to be answered about the system that produced them before they are put to anybody else, and because a reader who doubts a question can open the source behind every answer in that column. Section 7 sets out the problem in more detail.

4. Three findings

Nobody records who approved

Four of the eight systems will stop a consequential action and wait for a person to decide. None writes down who that person was.

The OpenAI Agents SDK's approve() takes no approver argument. The only thing in its approval path carrying the word identity turns out to identify the tool call, by server label and request id, rather than anybody human. CrewAI's request_human_input reads a line from a console and hands back the text. AutoGen's ApprovalResponse insists on a reason, which is more than most of them manage, and has nowhere to put a name. LangGraph resumes a paused graph with Command(resume=...), an arbitrary value from whoever happens to hold the thread.

The pause is real everywhere. The attribution is missing everywhere. A deployment that approves programmatically leaves a record indistinguishable from one where somebody read the arguments and thought about them, which is the whole problem for any team that has to evidence human oversight rather than merely perform it.

Letta Code is the exception, and it is recorded as undetermined rather than guessed. It carries an acting-user identity that exists specifically to name the human who started a request instead of the credential that spawned the runtime, and the server validates it before honouring it. Whether that identity survives as far as the approval record could not be established from the harness, because the record lives elsewhere.

Nobody records that data was destroyed

Absent in five systems, partial in two.

All eight have a deletion path somebody will reach for when a subject asks to be erased, and afterwards the store looks much like one where the data never arrived.

The two partial cases fail in opposite directions, which is what makes them worth reading. mem0 writes a history row when a memory is deleted, which is the right instinct, and writes the deleted text into that row, so content erased from the vector store is still sitting in the local history database. Letta Code records a deletion as a git commit, dated and attributed, which as a trace is about as good as it gets. The content then stays in history, and on every mirror the commit was pushed to, so honouring a real erasure request means rewriting the history that made the record worth having.

Almost nobody can show a record did not change

The fourth level is absent outright in six of the eight.

Where history survives, it survives because the code behaved. That is a weaker claim than it appears, since it says nothing about whether anybody went around the code.

Two systems are closer than their scores suggest. LangGraph's checkpointer already links every checkpoint to its parent, which is the shape of a hash chain with identifiers standing in for digests. Carrying a digest over each checkpoint and its parent would make that chain verifiable without touching the write path. Letta Code keeps memory in a git repository, so its history is content-addressed without anybody having designed it that way, and the post-commit push mirror is most of an external anchor already. Commit signing is switched off deliberately, for a reason that holds up: the harness-managed committer identities have no key, and a global signing setting would break memory initialisation outright.

5. What each system would need

Graphiti and LangGraph both meet four of the five requirements at the first level and fail only on recording destruction. One change each.

Neither got there by aiming at this specification. Graphiti is bi-temporal, so a contradicted fact is stamped invalid instead of deleted and facts point back at the episodes they were pulled from. LangGraph needed append-only checkpoints for resumption and time travel, and append-only is append-only whatever the motive. Two independent designs arriving at the same property is better evidence for the property than any argument in the specification.

mem0 and CrewAI both run a model over existing memories to decide whether to keep, update or delete each one. mem0 keeps the old value in a separate history database. CrewAI keeps nothing. Its most tractable finding is small and specific: a blocked tool call builds a refusal reason and throws it away two frames later, so the model receives a fixed string naming the tool instead of the cause.

The OpenAI Agents SDK models the proposal side more carefully than anything else here, down to a ledger recording whether each call executed, and then leaves durability to whatever application embeds it. AutoGen has the best-structured approval request of the eight and returns a refusal as exit code 1, the same shape an execution error takes, so anyone counting failed executions afterwards counts the refusals among them.

6. How the instrument refuses

A census of other people's software can do real damage, and the people in it did not ask to be in it. Being careful is not a defence, because nobody can audit care. The rules below are enforced by the tooling on every assessment, and a file breaking one of them cannot be published.

Every verdict cites evidence. That includes a verdict of absent, which has to carry at least one record of where the assessor looked and came up empty. This is the rule that matters most. Saying a system lacks something without saying where you looked is an accusation dressed as a measurement, and it is the failure this exercise was always most likely to commit.

No applicable requirement may be left out, since a silent omission is how an unflattering result quietly disappears. Scope cannot be used as an escape hatch in either direction: a subject cannot mark a requirement irrelevant inside a business it claims, and cannot be scored on a capability it never offered. A requirement whose answer sits somewhere the assessor cannot read is recorded as undetermined, which blocks a level exactly as absence does, because a level awarded on unchecked facts looks identical to one that was verified. Partial does not clear a level either.

These rules caught the author twice. The CrewAI assessment was refused outright for recording an absence with no repeatable search behind it. The Letta Code assessment could not be finished at all until the undetermined verdict existed, because both of the available answers were guesses.

7. Conflict of interest

OMEM is the reference implementation of the specification these questions come from, and the same person wrote both. It scores well here in the way a dictionary's author spells well. Read its row as carrying no evidential weight.

It is included for two reasons. The questions ought to be put to the system that produced them before they are put to anybody else. And a reader who suspects a question is unfair can open the source behind every answer in that column and check.

That exercise was not a formality. It turned up three problems, two of them the author's fault. OMEM's own record export was emitting no evidence entries at all, and passed the second conformance level only because an empty list satisfies the check without meaning anything. The first draft of the rubric scored a lawful response to an erasure request as a first-level failure, which would have marked down every system deployed in the European Union for obeying the law. And it scored a system that never resolves contradictions as failing the second level, when the specification records resolution only if it happens, and never resolving is the more careful design. All three were fixed before any other system was read.

8. What a verdict is a claim about

Every verdict here is about one named commit. Not about a project.

The claim is not that a system does not record something. It is that at a stated forty character object identifier it did not, with the file and the line given. Each subject pins that identifier and the tooling rejects an abbreviation, so any reader can check out the exact tree a citation refers to.

It follows that a project shipping a missing capability tomorrow has not made anything here false. The assessment was of a commit that still exists and still reads the same way. When a system changes, the response is a new assessment with a new date sitting beside the old one, never an edit to it. That is what this specification asks of everybody else, and the census does not get an exemption from its own rule.

Each requirement in each assessment carries an identifier of the form MTC-2026-09-04-mem0-R1.5. It names the observation and not the state of the world, so it survives the behaviour it describes being changed. The manifest deposited alongside this document fixes the digest of every assessment and of the questions they were scored against, since unchanged answers to changed questions are not the same assessment.

9. Reproduction

Everything is in the reference implementation's repository under benchmarks/census. The assessments are data. The rules that reject a careless one are code. Both are checked on every commit.

git clone https://github.com/troybrandonc-bit/Omem
cd Omem

python3 benchmarks/census/run.py            # the report
python3 benchmarks/census/run.py --check    # validate every assessment
python3 benchmarks/census/manifest.py --check
python3 server/tests_census.py              # the refusal rules, tested

One finding is demonstrated by execution rather than by reading.

Every other verdict came from reading source, which is honest and weaker than it sounds, and most of them concern what a system does not record, which running it cannot establish either way. The retention of deleted memory text is different on both counts. It asserts that something is written, and it concerns data somebody asked to have erased. So it is checked by running mem0's own storage layer, with no network, no model and no credentials.

git clone https://github.com/mem0ai/mem0
git -C mem0 checkout 9a7924befd7026e41e445ba809370009e5e985a6
python3 benchmarks/census/verify/mem0_delete_retains_text.py --repo ./mem0

Corrections

Nothing here is self-reported and nothing is taken on trust in either direction. Every claim cites a file and a line at a pinned commit, or a search that can be repeated, so a wrong verdict can be demonstrated wrong rather than argued about.

If a verdict about your system is wrong, the fix is a pull request against its assessment file, or an email naming the requirement and where to look. A correction that lands changes the file, the report and the assessment date. There is no fee, no membership, and no requirement to use any particular software.

If your system is not here and you would like it to be, the same applies. Absence from this document is not a judgement. It means nobody has done the reading yet.

---

Notes

1. Assessed commits are recorded in full in each assessment file and in the published manifest. Systems were read in the order they were obtained, and no result was known in advance of reading.
2. The specification imposes one duty specific to derivation, which is that a fact a system inferred can be told apart from one it was told. No further requirements were invented to fill that column out. Every question here has to trace back to a published conformance level or the assessment stops being checkable.
3. A dated publication establishes when an observation was made. It does not oblige anybody to credit it. Attribution is reliably produced by coordinated disclosure, where the reporting process names the reporter, and that is not what happened here. These findings were published rather than sent.

Census text CC BY 4.0. Schema and tools MIT. Machine Testimony is a research programme, not a registered institute or a certification body. Operated by Garnet Taurus Ltd.

Also readable at https://machinetestimony.org/census/2026-09/

Machine Testimony