It Worked Once
  1. Posts
  2. The app I can't read
  3. Entry 016

The app I can't read

A client app with a real database, real logins and real financial data, built and run by agents for someone who can't read a line of it.

Filed
Reading
4 min
Outcome
Fixed
Series
The app I can't read

A client’s customers sign in to an app to see their portfolios. I build it and look after it for the client. Agents wrote every part of it, and I can’t read any part of it.

Clients log in with a user ID and a password, or a code sent by text message. Managers have their own accounts and can see more. Each night a job pulls portfolio data from the broker’s system, checks it and publishes it. A good run ends with a line like status=ok portfolios=51, which is the kind of line I can read. Underneath sits a Postgres database with 30 migrations. On 3 November, at version 1.4.0, a restore of the backup showed 48 tables and 30 migrations.

Agents built the backend, the frontend, the database changes, the nightly job and the security rules. Other agents run the releases, the backups and the monitoring. I wrote none of it.

What I’m responsible for

The customers trust the numbers on that screen. If one client sees another client’s portfolio, or sees yesterday’s figures labelled as today’s, that is my problem. “The agent did it” doesn’t help anyone.

I can’t judge whether the code is good. I can observe behaviour, so each thing I watch has to tie to something that matters.

Tests come first

Two agent skills by Matt Pocock are installed for my agents: one for test-first development (tdd) and one for diagnosing bugs (diagnosing-bugs). On 6 October I had 35 older copies of skills, piled up in another tool’s folder, moved aside so only one set is live.

Test-first means the test is written and seen failing before the fix. For me that’s useful because a failing test is a sentence I can read. “A client account must not see portfolios outside its own client” is a rule I can agree or disagree with, even if I never open the test.

The diagnosis skill does the opposite job. When something breaks, the agent has to reproduce it and show evidence before it changes anything. It stops an agent from fixing a guess.

The suites grow with the app. At the 1.1.0 merge the API had 475 tests and the web side 586.

A different agent reviews

The agent that builds a change doesn’t review it. A second agent reads what was made, tries to break it and rates each finding from 1 to 10. Findings of 5 or more get fixed. The rest go on a dated list. I explained that method in entry 001, and it applies here unchanged.

Eight gates

A release has to pass eight steps, in order.

Eight release gates in a row, with the test and restore numbers beside them, and a red stamp for the 59-portfolio incident.

Figure: the eight release gates, and the 59-portfolio incident of 26 September.

  1. A source gate on a clean tree: unit suites, builds, configuration checks.
  2. Tests against a real, disposable database, plus a browser run on the built app.
  3. A backup and an isolated restore, before and after, with table and migration counts.
  4. An image built for the exact change, checked by its ID.
  5. A guarded switch that replaces only what changed, checks health and rolls back by itself.
  6. A parity and drift check: the files on the server match the repository, and the server’s settings match the committed baseline.
  7. A public check: health returns 200, an anonymous request for account data returns 401, and the served page contains the new version string.
  8. A browser check of the real sign-in flows.

Then the release gets a tag. I can’t tell you how any of these scripts work. I can tell you what each one prints, and I look at that output.

Read-backs on the data

The restore tests are the part I trust most. A backup that was never restored is a hope. After each backup the agents restore it into a throwaway database with no network and compare counts: tables, migrations, rows.

The nightly job gets the same treatment. If a symbol in the source data has no label, the publication stops and a critical alert names the symbol. Clients stay on the older data and I get told why. Since 2 November a failed run is retried twice, 20 minutes apart, and the failure alert waits an hour so that a retry that works sends no mail.

Where this falls short

The gates grew out of things that went wrong, and they don’t catch everything.

The checks of that time covered the login and the data, and none covered a grant like that one. The fix was a rule that access is decided by scope and fails closed. That is the first story in this series.

What comes next

This series takes the app one incident at a time. The 59 portfolios come first. After that, the nightly job and the symbol that stopped it, a backup that broke after migration 20, and a release that rolled itself back.

Revision record

First published.
Fact-checked: login throttling has been live since 19 October (0.6.3); sign-in description corrected.

Related entries