Eight Hundred Megabytes A Second, And Nothing Was Moving

The Disk Stopped Answering

At quarter past eight in the evening, universal time, on the 21st of July 2026, which is quarter past ten at night where you were, the storage cluster your server's disk lives on lost a node.

Not your disk. Not your machine. A piece of the provider's infrastructure, several layers below anything you own or configure.

What happened next is the most instructive outage in this entire estate, because almost nothing about it behaved the way a broken server is supposed to behave, and because the single most confident diagnostic anyone ran during it was completely wrong.

What A Dead Disk Looks Like From Above

The machine did not crash. That is the first thing to understand and the reason everything else was confusing.

The processor was fine. Memory was fine. The network was fine. The kernel was running. Programs that were already loaded and only needed to think kept thinking perfectly.

Anything that wrote to the disk the ordinary way stopped, and stayed stopped.

Not slowly. Not with an error. It just did not come back, and it could not be interrupted. There is a process state in Linux for this, usually written as D, standing for uninterruptible sleep. A process in that state is sitting inside a kernel call, waiting, and a signal cannot pull it out. You cannot kill it. Not as yourself, not as root. It will come out when whatever it is waiting for finishes or fails, and your kill lands the instant it does. If that never happens, it never happens.

So the load average climbed past 30 on an idle machine, which sounds impossible until you know what it counts. Linux load average is not a measure of processor use. It counts everything waiting to run plus everything stuck in that uninterruptible state. So a machine doing absolutely nothing, with 30 things frozen on a dead disk, reports a load of 30. It was measuring the pile up, not the work.

The system log daemon became unkillable. Anything that wanted to write anything joined the queue. And eventually, because logging in writes things, even logging in stopped working.

The Pager Earned Its Keep

Here is the good news, and it is the reason the previous episode matters.

Nine minutes in, your phone received a text message about a hard breach across multiple service groups. At 15 minutes, another one saying the inbox tool was down. At 27 minutes, a third saying the box was wedged.

Wedged. That is the third state invented after the July outage where the watcher saw nothing. Kernel alive, userspace starved. Diagnosed correctly, from a machine in your house, over a channel that never touched the failing server, on the first night it was ever needed for real.

It also told you something the box itself could never have reported. A machine whose every write hangs cannot log, cannot alert, cannot even complete the act of noticing. Everything it might have said about its own condition was in the same queue as everything else.

The Command That Said Everything Was Fine

Now the part I would teach in a class.

At some point during the night you want to know whether the storage has recovered. So you write a test file to the disk and see how fast it goes.

Somebody did, using a flag that tells the system to skip the page cache rather than writing through it. That is a completely standard way to measure disk speed, and it exists precisely so that you are not accidentally measuring memory.

It reported 862 megabytes per second.

At that exact moment, 43 tasks were sitting frozen in that uninterruptible state, waiting on the kernel's flush workers. The storage was thoroughly dead. And the measurement produced a confident, specific, plausible number saying it was healthy, which very nearly closed the incident.

The reason is exact. Skipping the cache also skips the writeback path, the machinery that takes cached data and gets it onto the device in the background. And the writeback path was precisely the thing that had stalled. The test carefully avoided the only broken part of the system, and then reported on everything else.

The honest probe is the opposite of the clever one. Write normally, through the cache, then explicitly ask the system to flush it to disk, and put a timeout around the whole thing. If the storage is stalled, it hangs, and the timeout expires and tells you so. If it has recovered, it finishes in about a tenth of a second. Confirm by counting how many processes are in that frozen state, where the answer you want is zero. And note the honest limitation of the timeout, which is that it can report the hang but cannot actually rescue a process already stuck in the kernel. It gives you the answer, not your process back.

The general lesson is one I keep meeting in this material. An optimisation and a diagnostic want opposite things. The fastest path to an answer is very often the path that skips the thing you were asking about.

It Happened Three Times

That night was not one outage. It was 3.

The first, at quarter past eight, took the box down hard for about 90 minutes. The second came just after 2 in the morning. The third around 10 past 3.

And the third one happened after the provider had posted that the situation was stable and downgraded the incident to monitoring.

Which produces a rule with a slightly cynical edge and it is correct. A provider's status page describes their view of their fleet. It is not a statement about your machine. Verify your own box with your own probe, and do not grade your recovery against somebody else's timeline.

There is a companion rule for the middle of it, which is about resisting the urge to act. When the signature says the storage backend is dead, restarting services accomplishes nothing, because they will restart into the same dead storage. And forcibly stopping and starting the machine mid incident means re attaching through the same crashed cluster, with a real chance of getting stuck in a stopping state that you then cannot exit.

The correct move during a provider side storage failure is largely to wait, and to spend the waiting confirming it is genuinely provider side rather than something you did.

Recovery Is Three Different Events

The storage came back. The box was not back. That gap contained 3 separate aftershocks, and each one is its own lesson.

The first. Podman's rootless setup came out of it corrupted. Containers refused to start with an error about not being able to join an existing user namespace, and the official repair command for that exact situation failed with the identical error. The path that worked was one clean reboot, which beat 16 rounds of per account surgery.

The second, and this is the sharp one. Anything that was merely blocked on the disk returned by itself when the disk returned. Anything the service manager had given up on did not.

Services are configured to retry, but they retry a limited number of times before deciding the thing is genuinely broken and staying down. During hours of hung storage, several burned through that budget and stayed down.

And they did not show up in the obvious sweep, for the reason from episode 1. Every service account has its own separate service manager. Asking one of them for failed units tells you nothing whatsoever about the other 20. The failure was perfectly visible, in a place nobody was looking.

The identity provider was one of them. The service that a great many of your logins depend on, sitting at an error, while the other 40 public addresses were serving perfectly. From any dashboard, the box looked entirely recovered.

The instruction that came out of that is 7 words. After the disk passes, sweep the edges. Not the units. The actual public addresses, from outside, because a service that is down inside a manager nobody is querying will not raise its hand.

The third aftershock we met 5 episodes ago. The hard kill tore the final record being written into the telemetry store in half, and the query engine refused to read the entire file, so one of the health checks went dark for 7 hours with a message that actively argued it would fix itself.

And there is a fourth, from the reboot just before midnight in the middle of that night. That reboot is the one that silently switched off 4 timers that had been started but never enabled, including the 2 that constitute the box's own self monitoring. It also did not end anything. Two more stalls came after it.

What Was Lost, And What Was Learned

One genuine gap was found that night. When an alert was raised while the inbox tool was down, the delivery failed, and the system marked it as having been alerted anyway. So it was never sent again. The alert about the outage was consumed by the outage.

That one has since been fixed, in August 2026, for both the raising and the resolving half, with bounded retries. Which is worth saying out loud because the documents that recorded it as an open hole are still sitting there describing it as owed. Several of the gaps in this series have quietly closed since somebody wrote them down.

But the thing I keep returning to about this night is the structure of the failure rather than the failure itself.

Everything that broke was, from the machine's own point of view, invisible. It could not log the problem, because logging writes to disk. It could not alert, because alerting requires running code that eventually touches storage. It could not even reliably let you in to look. The only reason anybody knew within 9 minutes is that a completely separate machine in a completely separate building was asking a question that required no cooperation from the patient.

And the only reason the recovery was correct is that somebody eventually distrusted a measurement that agreed with what they wanted to believe.

Those 2 things, an outside observer and a suspicion of convenient answers, are most of what operational competence actually consists of. The rest is knowing which flag to leave off the command.

Next time, the perimeter. Firewalls that can lock you out permanently, a magic address inside every cloud that will hand out the keys, and a door that was examined carefully and deliberately left unlocked.