The Door We Looked At And Left Unlocked

Three Roads

There is a sentence in the security notes for this machine that I think is the most honest thing in the whole estate.

Per service accounts close exactly one of the 3 roads to root.

Not the roads. One road. And then it lists the other 2, by name, with an assessment of each, including one that was examined in detail and deliberately left open, with the date of the decision and the reasoning written next to it.

That is what a real security posture looks like, as opposed to a security posture that is mostly a list of things somebody switched on.

The Wall That Can Lock You Out Forever

Start with the firewall, because it is the one piece of this system that can destroy your access to the machine in a single command.

The rule is default deny on the way in. Nothing may arrive unless it is explicitly allowed, and the allowed list is short. Secure shell. Web traffic on the usual 2 ports, plus the newer web protocol which arrives over a different transport. The private tunnel. And the protocol a machine uses to get its own network address.

Everything else is dropped without a reply.

Now consider what a mistake costs. This machine is in a datacentre in France. There is no keyboard. The normal way in is the very thing a wrong rule would block. Get the secure shell line wrong and you have locked yourself out of your own production infrastructure, and the way back involves the provider's console and a genuinely unpleasant evening.

So firewall changes go through a pattern I would use for any risky remote change. Before applying, arm an automatic rollback on a 5 minute timer. Then apply. Then prove you can still log in, freshly, as a new connection rather than the one you already have open. Only once that is proven do you cancel the rollback and make the change permanent.

If anything goes wrong, including you being unable to reach the box to cancel anything, the machine undoes it by itself in 5 minutes and you have lost nothing.

There is a second trap in the same area. The natural way to load a new rule set is to clear everything and start fresh. Doing that here would also delete the rules podman maintains for container networking, which it created at runtime and which nothing would restore. So the configuration deliberately replaces only its own section and leaves the rest untouched. A clean slate is not always cleaner.

The Magic Address Inside Every Cloud

This one is genuinely interesting because it is a hazard that exists on every cloud server in the world, and most people running one have never thought about it.

Cloud instances need to learn things about themselves at boot. What their address is, what configuration they were created with, which keys are supposed to be able to log in. That is provided by a metadata service, reachable at a fixed internal address that is the same on every machine at that provider.

It is not authenticated. Anything running on the machine that can make a network request can read it, because the fact of running on the machine is treated as proof enough.

Now walk the road. A service on this box gets compromised, in the ordinary way, through a bug in a web application. The attacker cannot become root, because of everything in episode 1. But they can make network requests, from inside the container, as that service's account. So they ask the metadata service, and it tells them about the account's key set and the configuration the machine was built with.

That is a path from a bug in a photo booth to information about how the machine authenticates its administrator.

It was proven reachable from a service account here, on both the older and newer network protocols, which is a nice detail because blocking only the obvious address would have left the other one wide open.

So there is now a rule on the way out, blocking that destination on both the older and newer address formats, for every account except root. It is deliberately surgical. This is not a general restriction on what services can reach on the internet, which would break half the fleet. One service, blocked on its 2 addresses, with one exempt account.

The exception is root, and the reason is precise. At every boot, the machine fetches its key list and its configuration from that same service, as root, and the firewall rules load before that happens. Block it for root and you have not hardened the machine, you have broken its ability to start.

The Door That Was Left Open On Purpose

That metadata path is the third of the 3 roads, and it is the one that is now closed for everything except root.

Which brings us to the second road, and the decision I opened with.

Root's list of accepted login keys on this box is wiped and regenerated at every single boot, from the set of keys registered in your cloud provider account. Which means access to the machine ultimately depends on 2 things. Holding an enrolled key, and holding the provider account.

That is a real road. Somebody with your provider login can add a key and be root on your server after a reboot, with no exploit involved.

It was examined and deliberately kept, and the reasoning is exactly right. That path is the break glass procedure. The restore runbook explicitly assumes that if every local credential is lost, a provider console login can mint new access. Closing that road, by blocking metadata for root or disabling the key regeneration, would sever the recovery leg that exists for the worst day.

So the control that actually guards it is not on the machine at all. It is custody of the provider account, its password and its second factor, which is escrowed on the paper key alongside everything else.

Considered, declined to block, recorded with a date. That is a complete and defensible answer, and it is enormously better than the same door being open because nobody looked.

And then, outside the 3 roads entirely, there is the residual that nobody can close. Every container on this box shares the host's kernel, so a bug in the kernel is a bug under all of them at once. And the database and the web server sit close to root by their nature. Those are named rather than solved, and the response is not to pretend otherwise but to keep genuinely irreversible data, the health records in particular, somewhere off this machine entirely.

One Cookie, Every Subdomain

Here is a consequence of your own architecture that is easy to miss and slightly alarming.

Your tools share a single sign on. You log in once with a passkey and everything under your main domain accepts you. The way that works is a cookie scoped to the whole domain rather than to one host.

Which means the browser sends that cookie to every single subdomain.

The cookie is marked so that page scripts cannot read its value, which is the right setting and blocks the crudest attack. But it does not make the problem go away, it changes its shape. A scripting hole on the least important tool you own lets an attacker make requests as you, with the cookie attached automatically, which is most of what reading it would have bought them. And if somebody takes over an abandoned subdomain you forgot about, the browser will helpfully deliver your session cookie straight to their server, because as far as it is concerned that host is part of the family.

The strongest available protection for cookies, the one that hard binds a cookie to exactly one host, is unavailable by definition, because it requires the cookie to have no domain scope at all and the whole point here is that it does.

So the consequence is stated plainly in the notes and it is a genuinely non obvious one. Subdomain hygiene, deleting DNS records for things that no longer exist, is a first class authentication control on this estate. Not tidiness. Not cosmetics. It is part of the login system, because every name that resolves is a potential place to steal the cookie from.

The Zone That Lies To You

Which leads to the last thing, and it is a small horror story about DNS.

One of your domains delegates its authority to one provider. A different provider still holds a zone file for the same domain, left over from years ago. That zone is not authoritative, so ordinary lookups never reach it. It shapes nothing about how the domain actually resolves today.

It also still contains a record pointing at the address of the old server. The one that was destroyed in July 2026, whose addresses were released back to the provider.

Released cloud addresses get handed out to other customers. So that inert, harmless, invisible record is a loaded one. Point the domain's delegation at the provider holding the stale copy, which is a completely reasonable thing for a future person to do while consolidating things, and it becomes a live record aimed at whoever holds that address now.

And the tooling makes it worse in the friendliest possible way. Ask your own DNS tool to list the records for that domain and it shows them to you, formatted neatly, looking exactly like real records. It does print a warning line saying this provider is not authoritative here and what follows is a stale copy. That line is very easy to skim past when the list below it looks completely normal.

Two rules came out of it. Never conclude that DNS is clean from one provider's control panel. Confirm by querying the servers that are actually authoritative, which is what finally settled it.

And when you retire a machine, search every domain you own for its address rather than working from the list somebody handed you. That day produced 2 separate misses, this record and 3 extra shell aliases that a runbook had confidently counted as 5. Both came from trusting an enumeration that had been prepared earlier instead of deriving the set again from the world.

Which is, one last time, the thesis of this whole series. Almost every failure here was a document, a tool, or a check answering a question correctly, while somebody read it as the answer to a slightly different question.

Next time, the last piece. How a website gets published on this box without your Mac being involved at all, and why that took a new word in the registry.