A Name Is Not A Thing: How Half The Fleet Became Unrestorable

Where Your Software Actually Lives

You have backups of your data. Databases, files, state directories, all of it uploaded nightly to a bucket a compromised server cannot delete from.

Now imagine the box is gone. Genuinely gone, provider incident, disk destroyed, nothing left. You have all your data. What do you run it with.

The applications on popcorn2 are container images, and those images were built in the per account image stores on that machine. They exist there. And, until somebody thought about this properly, nowhere else at all. Every piece of software you own, in one place, on the machine specifically designed to be replaceable.

That gap is the subject of this episode, along with the fix, the way the fix broke, and the single most important distinction in the whole system. The difference between a name and a thing.

The Mirror

The fix is a private container registry at the cloud provider. Every image the box runs is pushed there, so a rebuild pulls finished software rather than reconstructing it from source on a machine that does not exist yet.

Two categories go up. Your own applications, pushed as part of every build, so a new service is mirrored from birth rather than from whenever somebody remembers. That push is deliberately best effort. If it fails, the deploy is not blocked, because refusing to ship a fix because a mirror was unreachable would be the wrong trade. The failure is recorded loudly instead, and a separate check keeps failing until somebody re mirrors it. So a service can be live and briefly unmirrored, on purpose, visibly.

The second category is the third party images the platform depends on, the database, the collector, the identity provider and a few others, mirrored as insurance against an upstream project deleting or moving a version you depend on. That happens more than you would like.

The credentials are the elegant part. The box holds 2, one that can only push and one that can only pull. Neither can delete anything. So from the box's point of view that registry only ever grows, and that property is exactly what turns a mirror into a backup.

The deletion credential deliberately does not live on the machine. When old images genuinely need cleaning up, that is run from the Mac, by a human, with a preview by default.

One honest caveat, written plainly in the documentation. The registry is in the same cloud account as the box and the backup bucket. So it defends against losing the machine. It does not defend against losing the account, and pretending otherwise would be the sort of comfortable lie that gets discovered at the worst moment. A separate copy of the crown jewels sits at your house for exactly that scenario.

Digests, Tags, And The Thing That Bit

Now the concept, because everything after this depends on it.

An image can be referred to in 2 ways.

A tag is a name. Something like capture colon wave 4. Names are convenient and names move. That is what they are for. You build a new version, you put the same name on it, and everything that refers to that name now gets the new thing.

A digest is a cryptographic fingerprint. It is an honest one. If the bytes are identical, the digest is identical.

The catch is which bytes. The digest people quote covers the manifest, the little document describing how an image is assembled. And pushing an image to a registry can repackage that description, same software, same layers, different wrapping, and therefore a different fingerprint.

That caused 2 separate false alarms here, both reading like something serious. Warnings that the registry appeared to hold a different build from the one running, that disaster recovery would restore something unvalidated. Both times, on investigation, exactly the same build.

The identity that survived the repackaging is deeper in. Every image has a configuration object, describing the environment and the uncompressed layers, and that object has its own identifier which does not shift when the outer wrapping is rebuilt. So that is the comparator this box now uses everywhere.

There is a line in the operations manual that reads like a small philosophical statement, and it is the whole lesson. Digest is not build. The fingerprint was answering a question about packaging when everyone was asking a question about software.

The Day The Provider Deleted The Fleet

Here is why that mattered, in the form of a real emergency.

The original design froze a record of which images were running, and it recorded them by tag. Restoring meant pulling those tags back down.

Then something pushed to one of those tags from outside the box. Perfectly legitimate. The tag moved to the new image, exactly as tags do.

Which left the image that was actually running with no name pointing at it. And a registry, quite reasonably, garbage collects images that nothing refers to. So it was deleted.

When this was measured, half the fleet was unrestorable. Not degraded. Gone from the only off machine copy, while every dashboard was green and the backup was running perfectly every night, because backups cover data and this was software.

The redesign is simple once you have the concept. Every build now also gets pushed under an additional tag derived from that stable configuration identifier. It is still an ordinary registry tag, technically movable like any other. What makes it effectively immutable is that its name is derived from the content, so a different build computes a different name and has no reason to ever land on this one, and the credentials on the box cannot delete it. So the running image always keeps at least one reference pointing at it, and never becomes the orphan that gets collected.

The Frozen List, And Why Being Out Of Date Blocks Everything

Alongside that sits a manifest, which is a frozen list of exactly which image every running service was using at a moment in time, recorded by that stable identifier.

It plays 2 roles. During a restore it says which images to pull, so you get back the builds that were actually running rather than whatever the labels happen to point at now. And it verifies, by checking that what arrived is what was recorded.

Which produces a consequence worth being precise about, because the documentation on this box is currently a little behind itself.

Under the old tag based chain, a stale row was a blocked restore. The verification compared things that no longer lined up and hard stopped. That was measured once at 12 rows out of 32, meaning a genuine disaster would have halted on 12 images.

Under the current chain, a stale row is quieter and in some ways worse. Because the manifest now names an immutable reference to a specific build, a stale row restores that build, successfully, verified, green. It just happens to be the version you were running before the last deploy. You do not get a failure. You get an older service, silently, at the exact moment you are least able to notice.

So every deploy that rebuilds an image owes a refresh of that list, and the deploy procedure now runs the detector at the end to prove whether one is owed rather than relying on anybody remembering.

There is a small trap in doing that refresh, which has now caught people twice, including once after it had already been written up. The command that produces the new list prints the entire list to the screen, and prints its friendly success message at the very end. So if you pipe it into something that shows you the last few lines, you see the success message, everything exits cleanly, and the list itself has been silently thrown away. The only proof it worked is looking at the difference in the file afterwards and seeing the identifier actually change.

Two Clocks

The check on all this used to be one thing and is now deliberately 2, and the reasoning is a nice piece of design thinking.

There are 2 questions. Does the frozen list still match what is running. And is every image in it actually retrievable from the registry.

The first question goes stale instantly, the moment anything is deployed. The second needs the network and is slow.

Bundled together, they ran weekly, which meant up to 7 days could pass with genuine drift reading as green. So they were split. The local question is measured at 8 seconds across the whole box, so it now runs every monitoring cycle, and detection went from about a week to one cycle. The network question stays weekly.

And then the part that makes it safe. The 2 answers must never be readable as each other. If a fast local check could satisfy the slow network check, the entire question of whether your disaster recovery is retrievable would go permanently blind, which is a far worse failure than the one being fixed.

So there are 3 independent defences. The 2 checks write to separate files. The tool refuses to run at all if those 2 paths ever resolve to the same place. And each record identifies which mode produced it, graded against a strict list, so anything unexpected fails closed rather than being interpreted generously.

The Cleanup That Nearly Deleted The Only Copy

Immutable tags solved disaster recovery and created a hoarding problem, because nothing ever removes them and a named image is invisible to the ordinary cleanup that removes unnamed ones. One account had accumulated 166 of these tags and 20 gigabytes. The registry had 192, of which 33 were actually in use.

Writing the tool to clean that up produced 3 refusals that are each worth more than the tool.

The first. If a repository has no accountable entries at all, skip it entirely and report it as unaccounted, rather than concluding everything in it is garbage. The static site builder is why. It appears in no enumeration list, has no manifest row, and is a batch job so is never running. The naive rule would have deleted its only off machine copy, of a thing that is genuinely live and publishing a website.

The second. A tag created after the keep list was generated is unknown, not superseded. The list's authority stops at the world as it was when it was made, and anything newer is outside its knowledge rather than beneath its notice.

The third. If a running container refers to its image in a way that cannot be traced back to a repository, refuse the whole run, because that entry cannot be filed and the real repository would then look unused.

And then there is the footgun that 5 review rounds missed.

The command to remove a single tag from an image, if you give it no tag to remove, removes every name the image has. Which means removing one immutable tag also strips the ordinary name, and the ordinary name is what the service's configuration resolves to, so the service can no longer start.

Five reviewers read that line and none caught it, and the reason is the best part. The test suite had a stand in for the command that logged what it was asked and had no model of image identity at all. It encoded the author's assumption about how the command behaves, so it could never contradict it. A test that shares your misunderstanding will agree with you every single time.

The rule written afterwards is one sentence. When a stand in represents an external contract, the contract is what needs checking, not the stand in.

Next time, who is watching all of this. Three machines, in 3 buildings, each one refusing to report on itself.