The Alert That Travelled Through The Thing That Was Down

Thirty Minutes Of Nobody Noticing

On the 8th of July 2026, popcorn2 ran out of memory. The kernel stayed alive. The SSH daemon wedged. All 34 public web addresses returned nothing at all, for about half an hour.

The off site watcher, the machine at your house whose entire purpose is to notice exactly this, saw nothing and said nothing. Not because it was broken. Because it was working precisely as it had been built.

Detection was you, manually, because you happened to be at the keyboard.

This episode is about what was wrong with that watcher, and about the 3 layers of watching that now exist across 4 machines in 3 countries, each one specifically forbidden from vouching for itself.

The Original Sin

Start with the previous server, because it had the purest possible version of the mistake.

Its monitoring was a loop running inside the statistics process. When it found a problem, it sent a message to the public address of the inbox tool.

Two consequences, both fatal, both obvious in hindsight.

The alert saying the web server is down travelled through the web server. The alert saying the inbox tool is down was sent to the inbox tool. Every failure that mattered enough to alert about was, by construction, a failure that ate the alert.

And nothing anywhere checked that the monitoring loop was still going round. If the process holding it died, or the loop simply stopped, the silence was indistinguishable from everything being fine. Which is not a subtle failure mode, it is the default one. Silence is what a healthy system and a dead system both produce.

Out of that come the 2 rules that shape everything on this box. A monitor must never route its own alert through the thing it monitors. And a monitor must have a dead man's switch, which means it must be continuously proving it is alive to somebody else, so that dying is loud.

Layer One, On The Box

The machine checks itself with a program that runs on a timer and has, at the last count, around 25 separate legs.

I want to be careful with that number, for a reason that is about to become funny. The reference document says 19. It also carries a note saying that this number must be counted by running the thing and never read off that line, because the documentation once said 15 while the box was actually running 19. When I checked the code, it was well past 19 again. The warning label correctly predicted its own failure, twice, which is the most honest piece of documentation on this machine.

The legs, roughly.

Backup freshness. Whether the reconciler is still running. Disk, memory, swap. Whether the broker is answering. Failed units, and separately, units that are crash looping. Database connections. The state of the private network tunnel. The firewall. Whether telemetry is still arriving. Whether the restore drill has run recently. The image mirror ledger. Whether the disaster recovery manifest has drifted. Whether the off site watcher is alive. Clock accuracy. The identity provider's version. And whether the derived metrics layer is complete.

Enumerations in documents rot by undercounting, silently, because adding something is a change to reality and updating the list is a separate act somebody has to remember. Nobody ever forgets to add the check. Everybody forgets to add the line.

Where those alerts go matters too. They land in a signals view, not in your idea inbox. That split exists because machine generated alerts had become 558 of 1779 items in the inbox, which was by a factor of 3 the single biggest reason your actual thoughts were buried in there.

Signals behave differently. Each one has a stable identity, so a recurring fault updates one entry instead of filing a new one every time, and recovery closes it rather than posting a second message.

And the identity has to be genuinely stable, which caught 2 producers. One put a restart counter in its key, so every occurrence looked new, and it buried 78 out of 103 sampled items. Another put the age of the problem in the message, and filed 10 items in 90 minutes for what was actually 2 faults. The rule now is that the key names what is broken, and everything that changes goes in the title.

Layer Two, In Your House

The box cannot report its own death, so a machine at home watches it, and alerts by sending a text message through your home router directly. That path touches the VPS at no point, which is the entire reason it exists.

Which brings us back to the 30 minutes nobody noticed.

The old probe opened a TCP connection to the SSH port. If it connected, the box was up.

Here is what is wrong with that, and it is genuinely one of the best pieces of systems knowledge in this whole series. Completing a TCP connection is the kernel's job, not the application's. The application asks the kernel to listen, and from then on the kernel finishes handshakes and stacks the finished connections in a queue, handing them over whenever the application gets around to asking for one. So if the application is wedged, stuck, or starved of memory, the kernel keeps accepting into a queue nobody is emptying. That queue does have a limit, so eventually connections start failing, but the window where everything looks fine is long.

Which means the strongest alerting channel in the estate reported everything was up, throughout an outage where nothing worked.

The fix is beautiful in its cheapness. In SSH, both sides announce themselves with an identification string, and in practice a healthy server sends its own the moment you connect, without being asked. So simply reading, after connecting, is a free and credential free proof that the server application actually reached the point of writing bytes to you.

That gives 3 states instead of 2. Dark, meaning no connection at all, the box or its network is gone. Up, meaning the greeting arrived. And a third state that did not exist before that July morning, called wedged, meaning the connection was accepted and no greeting ever came. Kernel alive, userspace starved. That is the state the outage lived in, and it was invisible.

There is even a small correctness detail worth stealing, and it is why it reads in a loop against one deadline rather than reading once. TCP does not preserve the boundaries the sender wrote in. A perfectly healthy greeting can arrive as 2 characters and then the rest, and a naive single read would see a fragment and call the box wedged.

Watching The Watchers

The rebuilt watcher runs every 2 minutes and alerts only on transitions. One message when something breaks, one when it recovers, never a repeat. Because an alerting system that repeats is an alerting system you will filter.

Its legs cover the obvious things, box liveness and a hard breach rule when several endpoints across several groups fail at once. But 4 of them exist specifically to watch the watching, and each is a good idea.

There is a vantage check, where the home machine asks whether it can see the internet at all, because a home network outage would otherwise look exactly like the entire cloud disappearing.

There is a canary on the text message path itself. Every cycle it checks that the router is reachable and its interface is answering, without sending anything, because sending yourself a message every 2 minutes forever is not a monitoring strategy. If that path is dead, the alert about it obviously cannot be a text message, so it goes to the inbox tool instead. And once a month it sends a genuine test message, because a reachable router is a strong hint and only an actual delivered message is proof.

There is a liveness check on the box's own self check timer, done by reading the age of a file the box touches on every successful run. No amount of probing web addresses can tell you that a scheduled task has stopped running. Only its silence can, and only if somebody is listening for it.

And there is a check on the inbox tool itself, which is the destination for every alert that is not urgent enough for a text message. If that is down, everything non urgent is being written into a void, so that one specifically escalates to a text message.

There was also a structural fix to the grouping. All the endpoints that depend on the database used to sit together in one big group of 18, and the hard breach rule requires failures across multiple groups. Which meant that a complete database outage, one of the worst things that can happen, was mathematically incapable of triggering the loudest alarm. Splitting that one group in 2 fixed it, leaving 4 groups in total.

Layer Three, The Square Nobody Owned

Draw the grid. Box up, home machine up, nothing to report. Box down, home machine up, the home machine alerts. Box up, home machine down, the box notices and alerts.

Box down and home machine down. Nobody. Total silence.

That is not a fantasy scenario. It is a cloud region incident on the same night as a power cut at your house, and the power cut takes out both the home machine and the router that is its pager.

That square is watched by a third layer, running on the 2 nameservers you operate for another project, one in Amsterdam and one in Warsaw. They are the only always on hardware you have that is neither in that cloud region nor in your house, and they run on local storage rather than the networked kind that failed in the incident we are about to cover.

But be careful about what they cover, because this is the most honest thing in the whole watcher design. Row 3 has 2 halves. If your home machine dies while the house is fine, they detect it and they page you through the home router, which is a completely independent path. That is the likelier half and it is covered.

If the house itself goes dark, power cut, everything off, they will detect the problem perfectly and have nowhere to send it. The router that is their pager is in the dark house. There is a channel that would work in that case, a phone notification service, and it is written, and it is switched off, because you looked at what it cost and declined. That is recorded in the documentation as a deliberate decision rather than an oversight, with the date, and the code is left in place so that re enabling it is one setting.

I find that more reassuring than if it claimed full coverage. An uncovered case you have named and priced is a risk. An uncovered case you believe is covered is a surprise.

The documentation also contains a rule I would frame. This layer owns its square and nothing else, and it must never grow into a second home watcher. Duplicating a leg that another watcher already owns buys nothing, and costs a double page on the worst night of the year.

The Reboot That Switched Off The Guards

One last story, and it is the one that proves the layering is not paranoia.

In Linux, starting a service and enabling a service are 2 different things. Starting runs it now. Enabling installs the links that cause it to be brought up as part of normal boot. Start without enable is, as the manual here puts it, rent until reboot.

Four timers had been started but never enabled, months earlier, and nobody noticed because the box had not rebooted in all that time. Then it rebooted, after a storage incident, and all 4 quietly did not come back.

Two of them were the reconciler and the self check. The guards themselves. The box flew blind until somebody re enabled them by hand.

And there is a genuinely uncomfortable observation attached. There is a check for exactly this class of problem, which reports timers that are not persistently enabled. It works. It cannot help, because it only runs while the thing running it is running, and the thing running it was one of the 4.

No system can prove its own liveness. That is not an engineering shortcoming, it is closer to a law. It is the entire reason this box has 3 machines in 3 buildings, each one required to keep proving to somebody else that it is still here.

Next time, the deeper question. Why almost every guard on this machine has, at some point, told a confident and complete lie.