Green Has Lied More Often Than Red

The Uncomfortable Tally

Go through the incident history of this machine and sort the failures by how they were discovered, and a pattern jumps out. I have not counted every one, so treat this as a strong impression rather than a statistic.

A striking number of them were not found by an alarm going off. They were found by somebody looking, or by a spam bot, or by a completely unrelated investigation wandering past. And in those cases the monitoring was green throughout.

Not broken. Green. Confidently, specifically green, answering a question correctly, while the thing you cared about was on fire.

There is a name for this in the estate's vocabulary. Failing green. And this episode is a tour of the best examples, because the design rules that came out of them are the most transferable thing in this entire series.

Am I Behind Is Not Am I Exposed

The identity provider on this box is third party software. It handles every passkey login you have. There is a check that watches its version, comparing what is deployed against the latest release published upstream.

It has worked perfectly, every day, since it was built. And it has now twice been green and cheerful while the box was running software with a published, unpatched, high severity security hole.

The first time, in July 2026, the flaw let any registered client obtain a token minted for a completely different client's audience. The introspection endpoint could not detect it, because the attacker controlled the very field being checked. The alert at that moment said, in a routine tone, that a newer version was available.

The second time, in August 2026, on the same software, the flaw was a combination. Cookies issued without the setting that stops them being sent from other websites, no protection on state changing routes, and a request parser that did not care what content type it was handed. Composed together, those let a page on another website cause a request that creates a new administrator account, using the victim administrator's own logged in session. The alert said, routinely, that a newer version was available.

Both times it was working exactly as designed. It answers the question am I behind. It has never been able to answer the question am I exposed. Those are different questions, and only one of them wakes you up.

And the silent direction is the dangerous one. If an upstream project fixes a security hole without publishing a new version number, or if you are sitting on the very latest release and that release has an open advisory against it, this check reads green while you are exposed, and there is nothing in its design that could ever say otherwise.

The correctness of a check is not a property of its code. It is a property of what it looks at.

Building The Fix Wrong

The obvious fix is to grade the deployed version against published security advisories, using the same public interface the check already calls. That has now been built, deployed and walked, on the 13th of August 2026, which is more recent than several of the documents describing it as owed. Those documents have not caught up yet, which is its own small lesson about reading status out of prose.

Building it produced a genuinely surprising finding, and it is a nice lesson about trusting the shape of data.

Advisories carry several fields describing versions. There is one giving the first patched version, and across all 13 advisories for this software it was empty. Every single one.

So the natural fallback is the field describing which versions are affected. And that field is actively misleading here. One critical severity advisory describes its affected range as everything from a certain version onward, open ended, which reads as nothing fixes this, while simultaneously carrying a separate field naming the exact version that fixed it. The ranges also contain typos, and are inconsistent about whether version numbers have a letter in front of them.

The original written plan for this guard said to use that range, precisely because the other field was empty. Building it exactly as specified would have produced a guard that was permanently and wrongly red. The specification was corrected before it was built, which is the only reason that sentence is a near miss rather than an incident. Which is the same disease as a guard that is permanently green, wearing the opposite colour. Both end with a human deciding not to look.

Two further requirements came out of that. Any comparison has to fail into a disarmed state rather than a healthy one when it cannot parse a value, because failing to understand something is not evidence that it is fine. And the guard needs a way for you to accept a known risk before it ships, or the first advisory with no available fix pins it red forever and it dies of being ignored.

There is even a timing trap. The upstream check is cached weekly to respect a rate limit, so the version named in the alert can be up to 7 days out of date. It named one version 4 days after its successor had shipped, and 2 separate planning documents then copied that stale number verbatim, which sent a later session off to install the wrong thing. Re derive the version at the moment you act. Never trust the number a document or an alert is holding.

The Crash Loop That Never Fails

Here is a beautifully structural one.

The health check looks for units in a failed state. That is the correct question and it catches almost everything.

A service configured to always restart, that is crashing slowly enough to stay under the rate limit that gives up on it, never enters the failed state. Ever. It just keeps dying and coming back, indefinitely.

One service on this box looped 60 times in 3 minutes, roughly once every 3 seconds, and was invisible to everything that was looking. The failed units check, the deploy verification, all of it, because the state they asked about was never the failed one.

The fix was to add a check that grades a rate rather than a state. It reads the counter of how many times each unit has been restarted, remembers what it was last time, and looks at the slope. The 2 checks are complementary and both are kept, because one catches things that have given up and the other catches things that never will.

The general shape, which applies well beyond servers. If a failure is a process rather than an event, the question you ask has to be about change over time. Something in a snapshot might hint at it, a counter, a timestamp, a log line, but only if you thought to look at the thing that accumulates rather than the thing that describes right now.

A Check That Shares Your Assumption

Two lessons about the checking of checks.

The first is a technique this estate calls mutation testing, applied to safety code. You have written a fix, and a test suite proving the fix works. So delete the fix, and run the suite. If it still passes, your suite is theatre. It was never testing the fix, it was testing something adjacent to it that happens to be true either way.

That has caught real emptiness. And there is a refinement, which is that a mutation can pass for the wrong reason, so you also have to check that the test failed in the way you expected rather than merely failing.

The second is the one from the image episode, and it is the deeper version of the same idea. A test suite that stands in for an external tool, with a stand in written by the same person who misunderstood the tool, will agree with that misunderstanding forever. Five reviewers went past a command that deletes every name from an image, because the fake version of that command in the tests had no concept of an image having several names. The test could not disagree with its author.

And a related one about review. When a reviewer finds a real problem, the finding is usually right and the suggested fix is an unreviewed patch that nobody has run. Applying one verbatim here broke 34 existing tests. Take the finding. Write your own remedy. Run everything.

Nothing Is Built Until It Reaches Your Phone

This is the rule I would put on the wall.

A guard is not built when the code is written. It is not built when the tests pass. It is built when its failure has actually arrived on your phone, and somebody has read the rendered message and confirmed it makes sense.

Every leg of the off site watcher was walked to a real delivered alert before its documentation was written. Not all of them to your phone, because not all of them should page you. The urgent ones were walked to a genuine text message, the quieter ones to a genuine entry in the inbox tool, which is exactly where they are supposed to end up. That is the standard, and the reason is that the path from a detection to a human has about 6 places to silently break, and every one of them looks fine from the code.

Two techniques make walking cheap, and both are worth stealing.

Prefer a fault the system will undo by itself. To test the staleness alarm on the site builder, somebody back dated its status file. The builder rewrites that file every 60 seconds, so the alarm raised, the system recovered on its own, the alert auto resolved, and the entire test took about 2 minutes with nothing to clean up afterwards.

And verify the resolution, not just the alarm, by going back and asking what is still open. The half that raises an alert gets tested constantly by reality. The half that closes it is where the orphans accumulate, quietly filling your alert view with problems that ended weeks ago.

There is also a specific trap in walking a check on this box. You have to trigger it through the proper mechanism rather than running the script directly, because the surrounding unit is what supplies the credential for delivering the alert. Run it directly and the delivery fails, and because the check is designed to only record a state change once the alert has actually been delivered, it correctly refuses to record anything. So a direct run looks like it worked and proves nothing at all.

The Economics Of Crying Wolf

The last principle is not technical, and it is the one that governs the others.

When a new guard is built, it usually finds real problems immediately. The temptation is to ship it and let it page about them.

The rule here is the opposite. Fix the known problems first, close them, and then deploy the guard so that it arrives green. Because a guard that shows up already screaming teaches everybody, on day one, that this particular alert is background noise.

And when a guard produces a false alarm, the fix is never a list of things to ignore. That was refused explicitly for the site builder's flapping alert, on the grounds that an exception list is suppression with better manners. The check still believes the wrong thing, and you have just taught it to be quiet about being wrong.

The reasoning behind all of it is an insight about what an alert actually costs. A false alarm does not cost you an interruption. It costs you a fraction of the credibility of that channel. And the channel it is spending is the same one that will, one day, be trying to tell you that something real is dying.

Which means an alerting system is a budget you are constantly drawing down, and the only deposit is being right.

Next time, the night the disk stopped answering, three times, and the diagnostic command that confidently reported everything was fine while 43 tasks sat frozen.