A small public lab · Joshua Bauer, Ottawa

When an AI agent tells you it's done, how do you know it's true?

An AI agent is an AI that does jobs for you, like writing code or running a task, and then reports back. It often says "done". I check whether that is true, using a written record the AI can't change.

Before each test, I write down what I expect and lock it with a time stamp. I call that a "seal", and it means I can't change my prediction later. Every result links to its evidence.

Work with meCheck the claims

Recent results

  • The sealed studies of 3 to 7 October 2026 are public as one record with a permanent address (a DOI): the designs, answer keys, raw results, scoring programs and seals. The research record, doi.org/10.5281/zenodo.23238531
  • A ticket that names only the check took never-ran "shown" to 0 of 16 for each of four small models. The result and its seals
  • A coat-check ticket before the work cut false "done" from 16 to 7 of 36 jobs. The result and its seals
  • The checker, with a local model reading the records, passed a fresh sealed exam: 141 of 150 (94%), exactly at the pass bar, with no planted failure called a success. A local model also found 82% of real agents' completion claims, against 27% for the rule-based version. The results and seals
  • The planted-fault bench rejected two rule-based checkers before either judged an agent. The write-up and seals
  • The method was named the Sonny Test and archived with a DOI. doi.org/10.5281/zenodo.23117475
  • Receipt Desk released: every status (done, failed or not shown) cites the line it copied from the record. The post
  • It Quoted the Failure: under a plain status prompt, all 35 replies that called a failed check "done" quoted the failing line. The write-up

Every result, with its seals →

Projects

  • Public research record The sealed studies of 3 to 7 October 2026, with their answer keys, raw results and seals, and a DOI.
  • The coat-check tests Like a coat-check ticket, a note written before the work says what check will show it is done. Tested on the Kaggle benchmark and on 12 coding jobs.
  • The Sonny Test The method, as one test, with its sealed record.
  • Swarm receipts A test bench for swarm oversight tools, and the two baseline checkers it caught.
  • It Quoted the Failure Two kinds of false "done": 48 logs, predictions sealed before the first run.
  • Receipt Desk Done, failed, or not shown, with the exact line it copied from the record.
  • The Watched Check Six AI coding sessions sharing one computer, and where the method took shape.
  • StatusEvidence scorer The check as a scorer, offered to Braintrust's open-source evaluation library.

Every project, with its date, status and links →

Start here

  • Work with me Bring me a problem worth solving. I check AI's work, build with AI, design tests that hold up, and explain AI in plain words.
  • Check the claims Each claim on this site, linked to the record that settles it.
  • The Sonny Test My method: a check counts only if the AI being tested can't change the record it reads, and the check has already caught a mistake I planted on purpose.
  • Barber Shop Corner Questions about AI from people who don't work in it, answered in plain words. Ask your own.