Two apps, one night, zero users

At the end of October I told my agents, “Don’t wait for anything on me. Gonna sleep.” Then I slept.

Within two days there were two apps. The dinner chooser took one night. The bot took about two. Between them roughly 1,000 automated checks were green: 857 unit and 27 end-to-end tests on one, 71 and 45 on the other. A separate reviewer model had read both. Nobody had used either of them for real yet.

A green check proves the check ran. I keep relearning that.

The setup

One model designs the thing. Two builders from different vendors build it in parallel, each allowed to edit only its own files. A reviewer model reads the result and rates every problem from 1 to 10. I tell the pipeline once, then I go to bed.

App one: dinner chooser

Two phones swipe through dinner options. The first one both people like is dinner. Small, boring, exactly what I wanted to test the pipeline on.

It got three reviews. The first two found things the tests had missed. The third found one 3 out of 10 and cleared it.

Then my own rule undid the third review’s clean result. I’d told the agents that nothing opens a browser window on my screen, so the builder replaced the setup handoff with a local page. That change went in with no reviewer. It’s marked “unreviewed” in the log, and it stays that way until someone looks.

The checks at the pause: 71 passing, plus 45 browser tests.

The thing no test could decide

An agent had written “AWS, not Cloudflare” into the requirements. It told me later it had taken an offhand remark of mine about a different project as a rule, and it built to it without asking.

I asked why it was AWS. The answer, in cost terms: AWS runs about 1 to 5 cents a month for this, with no hard ceiling. A stranger flooding the address gets billed to me. Cloudflare’s free plan costs nothing and just stops answering if it’s flooded, but it needs the server side ported and another review round.

I said, “The difference is the cost of course. Don’t act yet.” Two minutes later: “Drop it for now. Remind me to come back to the ideas we had on this session on Friday. I’m gonna deal with the Telegram bot during the week. Start a new session on that. This one has became messy.”

My AWS login had expired too. Nobody signed in. The app has never been deployed, so zero people have swiped.

App two: voice-note bot

You send a voice note to a Telegram bot. It turns it into tasks in my to-do app. When I forward a message, it figures out what the message is asking me to do.

Eight reviews before it was fully live, counting the add-on for speech. Findings rated 5 or higher got fixed and went back to review. When the third review of the final design began, the suite stood at 857 unit tests and 27 end-to-end tests.

It broke on day one anyway. Three times.

The second break was the to-do app’s API. It leaves two fields out for personal projects, and the agents had assumed they’d always be there. The fix was small.

The third one wasn’t in the code. Every call to the AI model service from my cloud account was refused. In one region the message was “Operation not allowed.” In another it said “Your account is currently being verified.” Amazon’s own model failed the same way, and so did the console playground. The same new account was also capped at 5 functions running at once. Nothing to fix in code. The only route left was a support case.

That’s 509 passing tests and a bot that couldn’t turn one voice note into tasks.

What I decided

I moved the AI step to my laptop. I said, “make sure it’ll use my Claude on my laptop. Not the cloud Claude.” The bot now catches the voice note in the cloud, and a small helper on the Mac does the thinking, and since 1 November the speech-to-text too, when the lid is open. The cloud model service is dropped.

On 1 November a voice note went all the way through and the transcript was good. I’d asked for the larger speech model. The agents tested it against the smaller one and kept the smaller one. The larger model silently skipped the first 20 seconds of one note in three of four settings. They told me why, and the test results backed them.

So the bot has a user, and it’s me. The chooser has none. The title lies by one app.

The 1 to 10 rule

I gave the reviewers one rule: “Next time when [the reviewer]‘s going for tests, ask it to rate the severity of the bugs from 1 to 10. Anything below 5 can be logged and deferred.”

Rule: rate every problem from 1 to 10, and fix 5 and above.

When you review or test anything:
- List every problem you find, with a severity from 1 (cosmetic) to 10 (data loss, security hole or the main feature broken).
- Fix everything rated 5 or higher, then review again.
- Problems rated 1 to 4 go in a list for later. Don't fix them now unless I ask.
- Before you call it done, try it the way a real user would: on a real device, with a test account, starting from nothing. Ask me before you touch a real account or real data. If you can't do that yourself, tell me exactly what to try.

The rule exists so a review doesn’t turn into a rewrite. The chooser’s log now has seven findings at 3 or 4 that I’ve never looked at. One of them says a phone with a fast clock can’t pair. It’s rated 4, and it stays.

I think the cutoff is right and the deferred list is debt. Hit me if I’m wrong.

Reviewers read code. The hosting choice and the unverified cloud account weren’t in any code, so no reviewer rated them.