What I check before I believe an agent

I can’t read code. I run servers, a backup system, a few apps on my Mac and phone, and a wiki that an AI model keeps up to date. Agents wrote all of it. I said what I wanted and they built it.

So when an agent reports back like this:

Done. All tests pass.

I can’t open a file and look. I have a short list of checks that work without reading anything. This post is that list, plus the October week when one of the checks proved less than I thought.

A pass only covers its own stage

The build passed. The command returned OK. I ran it by hand and it worked. Each of those is a true statement about one stage. None of them says whether the file exists, whether the schedule fires or whether the alert reaches me. I treat every stage as unchecked until I’ve looked at it directly.

Most of what follows comes from that.

The agent that builds doesn’t review

The builder and the reviewer are different agents, running on different models. The reviewer reads what was made, not the builder’s summary of it, then verifies and fixes.

Reviews run in rounds, and fixes get reviewed too

One round of review finds the easy problems. A small Mac menu tool went through four read-only adversarial rounds and an independent review, which found 12 issues, all fixed. A chat assistant went through three independent reviews.

The 10 regressions are the reason fixes go back to review. A fix is new code that nobody has looked at yet.

Findings get a number, and the number decides

Reviewers rate each finding from 1 to 10. The score decides what happens next.

Without a cutoff every review turns into a rewrite. With one, the small stuff is still on record and I can find it later.

Bigger builds start with a written design

For anything larger than one tool, the agents first commit a design file. It freezes the interface the two builders share and gives each builder a set of files only that builder may edit. Acceptance gates come after: local, then live, then on a physical phone. A gate that hasn’t run is listed as not run.

“OK” proves the command ran

This is the check that failed on me.

After any automatic or bulk action I ask the agent to read back what it made: the file, its size, its timestamp, the row count. Then I compare that with what I asked for. The agent can do this without me reading anything, and the comparison is simple enough for me to do.

Before you tell me this is done, prove it from the result, not from the command.

1. Find what the task actually produced: the file, the records, the page or the message.
2. Show me its name or location, its size, when it was created, and how many rows, items or lines it has.
3. Open it and show me a short sample of what's inside.
4. Compare that with what I asked for, point by point, and say which points match and which don't.
5. If anything is empty, zero, older than today or missing, say so in the first line. Don't call it done.

"The command finished" or "exit code 0" is not proof.

$ ./nightly-dump.sh ; echo "exit code: $?"
exit code: 0

$ ls -l /backups/nightly/
-rw-r--r--  1 backup  backup  0 Oct  9 03:00 db-10-09.sql.gz

That block is an illustration of the pattern, not the real log. The paths and names are invented. Exit code 0 means the script believed it finished. The file is 0 bytes.

The same pattern turned up again in other places.

An alert is untested until it has reached me

On 14 October the chat and mail channels my alerts depend on were both failing to deliver. A rule with nobody able to receive its alerts is as silent as no rule. The same day a refresh job failed, and nothing alerted, because the list of failed timers left out the refresh and backup jobs.

Delivery was fixed, the jobs went onto the watch list, and by 2 November the alert mail was arriving, often too often. The rule I keep: a scheduled job should fail loudly and sit on the watch list. Not every job does yet, and I haven’t made one fail on purpose to test the path end to end. That test is next.

The first unattended run is a separate check

A manual run exercises the command. The timer, the clock, the network and that night’s data stay untested until the job fires on its own. On 4 October a scheduled run failed after a manual repair had passed. On 17 October a task meant to run every 30 minutes turned out to have been running hourly for a while.

I write down when the first unattended run will happen and what I expect to see. The next morning I look.

Briefs and reports get checked too

A brief I passed to an agent said about a quarter of the rows in a dataset were inconsistent. The cleaning report counted 30%. Numbers inherited from a brief get recomputed before anyone builds on them.

When one agent reports a verdict on another agent’s work, I label it as reported. Two repositories had review files that disagreed with the “ship” verdict reported for them.

My own test tools failed quietly as well

Branded Chrome on my Mac exits about 30 seconds after starting headless, so browser tests looked like crashes. Playwright’s headless shell doesn’t, and I use that. Tests of one native tool wrote into the installed app’s learned state, so they now run in isolation. When a workflow agent stalls, it can’t take a message mid-run, so I read its transcript and fix the shared environment.