A data fix quietly rewrote the FAQ

I build and look after an app for a client. One figure in it is called “credit”. On 3 and 4 October, agents fixed how that figure is calculated. I approved both fixes.

Each fix also changed two pieces of text: the tooltip next to the figure, and the FAQ answer to “What is credit?”. Both were rewritten to describe the new calculation. Nobody asked me about the wording, and I didn’t see the change.

On 24 October, launch day, I was sending the agent a list of text fixes for the app. I had already told it the tooltip’s definition had changed without my permission. Then I found the FAQ answer had changed too, and wrote: “The FAQ answer has changed without my permission somewhere in the releases. Why that happened? Fix that:”

Why the agent did it

The agent changed a calculation. Afterwards the old text described something the app no longer did. So it rewrote the text. I can’t see its reasoning, but that’s the likely chain: from where it sat, keeping the app consistent was part of the job.

It was the wrong call, because the text clients read in a financial app isn’t the agent’s to write. It’s mine. Whether a sentence is accurate and whether I approved it are two separate questions, and the agent only asked the first.

Nothing stopped it either. No test held the FAQ answer, so changing it broke nothing. The release passed.

What it cost

The text clients read was wrong on a money term, and I didn’t know until launch day. Putting it right took an extra release, 1.2.3, on the day itself.

I also had to check whether other text had moved. The agent compared the app against the wording I had approved and found no other unapproved changes to the FAQ or the definitions.

The second safeguard

Telling agents the rule is one layer. The other is a test that holds the approved text. It’s a small check that compares what the app shows against the exact sentence I approved. If a later change alters the sentence, the check fails and the release stops. The FAQ answer has one now.

Rule: customer-facing text changes only when I give you the new text.

Customer-facing text means anything a customer can read in the product: labels, tooltips, FAQ answers, definitions, hints, error messages, emails, button text.

When a task changes how something is calculated or stored:
1. Do the task. Do not edit customer-facing text as part of it, even if the text now looks wrong or out of date.
2. Before you finish, compare the text a customer sees against what it said before you started. List every string that changed. If none changed, say so.
3. If the old text no longer matches the new behaviour, do not fix it. Tell me which text, where it appears, what it says now, and the wording you would propose. Then wait.
4. Only change customer-facing text when I paste the exact new wording in this conversation. Use my wording exactly.
5. If a test holds that text, do not edit the test to make a change pass. Tell me instead.
6. When you report the work, finish with one line: "Customer-facing text changed: none" or the list of what changed and where my wording came from.

I can’t read the code, but I can read a list of sentences. That’s what step 2 gives me.