Assess a system yourself

The register is one person's reading. That is the weakest thing about it, and the fix is not for that person to read more carefully. It is for the instrument to be something other people run.

So it is not reserved. Twenty requirements, a harness that refuses an assessment nobody can check, and a method written down. The rubric is CC BY 4.0, the tooling is MIT, and commercial use is expected rather than tolerated. There is no fee, no licence to sign, no certification body, no partner tier and nothing to buy.

If you audit AI systems, this is billable work

You already have clients asking what their agent stack records, what they will have to show under Article 12, and whether a supplier's claims about human oversight survive contact with the source. Those questions take a repeatable instrument, and building one is months.

Take this one. Run it against a client's supplier, charge them your rate, and keep the deliverable. Nothing here asks you to publish anything, credit anybody beyond the licence, or tell this project it happened.

git clone https://github.com/troybrandonc-bit/machine-testimony
python3 census/run.py                    # the current table
python3 census/run.py --for langgraph    # one subject, in prose
python3 census/run.py --check            # do these assessments hold up
cp census/TEMPLATE.json census/subjects/their-system.json

The template carries all twenty questions with the bar for present beside each one, so the work is reading source and writing down what you found.

The rule the whole thing rests on

Every verdict cites evidence, including a verdict of absence. Saying a system does not record something, without saying where you looked, is an accusation wearing a measurement's clothes. An absent or undetermined verdict must carry evidence of kind searched, and the harness rejects the file if it does not.

That is the difference between this and an opinion piece about other people's software, and it is the reason an assessment run with it can be handed to the party it is about.

The five verdicts

  • present and partial, each with a stated bar in the rubric rather than left to judgement.
  • absent, which cites where you looked.
  • undetermined, for when you genuinely cannot see. It exists because one system's approval record lives in a server that is not in the repository, and the choice was otherwise between calling a capability absent without looking and calling it present without looking. It does not clear a level, and it is a better answer than a guess.
  • not_applicable, because a system that takes no actions is not marked down for having no approval gate.

Pin the commit

A full 40-character object id, not an abbreviation. An assessment of "the latest version" is false as soon as the software moves, and an abbreviated id leaves room to argue about which tree was read. The harness refuses both.

What to do with the result

Three options, and no obligation.

  1. Keep it. A client deliverable that never leaves their building is a perfectly good use of this.
  2. Publish it yourself. CC BY asks for attribution to the rubric, not for permission. The assessment is yours.
  3. Put it on the register. A pull request against census/subjects/. It has to pass run.py --check, the same bar every row already there had to clear, and assessed_by carries your name, which the register shows.

Disagreeing with a row, including mine

A second assessor contradicting an existing verdict is more useful here than a new subject. Point at the code: open a pull request naming the requirement and where to look. If it lands, the file changes, the register changes, and the date on the row changes.

Arguing with me is not the remedy and does not work. This register already carries corrections that arrived because somebody said I had read something wrong.

Conflicts of interest

If you assess something you built, sell, or are paid by, say so in the file, and expect the row to be read as carrying no weight. That is what this project does with its own row.

If a client pays you to assess their own system, a reader is right to discount it, for the same reason a credit rating paid for by the issuer is worth less than one paid for by the buyer. The assessments worth most are the ones commissioned by whoever has to rely on the answer.

What this is not

Not a certification, an accreditation, or a mark. Nobody is entitled to call themselves conformant because a file says so, and no system is non-compliant with any law because a row says absent. The EU AI Act's obligations run through Articles 12, 13 and 14, and presumption of conformity comes from the harmonised standards, not from this.

What it gives you is a repeatable reading of what a system actually records, with every answer traceable to a line of somebody's source. That is worth money to whoever has to rely on it, which is why you should feel free to charge them for it.