It Worked Once
  1. Posts
  2. Series
  3. Things that quietly break

Things that quietly break

Jobs that exit 0, uploads that drop items, alerts that reach nobody. Each entry starts from a dated failure and ends with the check that would have caught it.

Entries
6
Written
0
Planned
6

These failures all reported success. A backup script exited with code 0 and wrote nothing. A bulk upload said it was done and left out 22 items. A monitor read a status code from a page that said the site was suspended.

Entry 001 covers each of them in a paragraph. This series takes them one at a time, with the dates, the evidence and what I changed.

The rules the series keeps coming back to:

  • A job exits non-zero when it fails, and sits on a watch list.
  • An alert rule is tested by making something fail and waiting for the message.
  • After a bulk change, the live state is compared with the input.
  • A check looks at what the user gets, such as the page body, and not only at a status code.
  • The first unattended run is its own open item, looked at the next morning.

Entries

  1. Entry 010
    The backup exited zero for six nights

    A dump script swallowed its errors and exited 0.

    planned
  2. Entry 011
    Both alert channels went silent

    Both contact channels stopped delivering, so the rule was silent too.

    planned
  3. Entry 012
    My site was down and the check said 200

    The CDN hid the dead server, and the failure mail failed as well.

    planned
  4. Entry 013
    Twenty-two keywords the upload forgot

    The upload said done. Only a diff against the file showed the gap.

    planned
  5. Entry 014
    My 30-minute job had been running hourly

    It worked, so nobody looked at the schedule.

    planned
  6. Entry 015
    I asked for a project. It deployed the folder I was in.

    A create command quietly became a different command.

    planned