Two Files, One Word, No Backup

A Service That Believed It Was Safe

On the 11th of August 2026, a new service went live on popcorn2. It holds the advertising bookings for the newspaper's autumn season. As far as anybody knows, it is the only record of them.

Its configuration contained a line saying db equals true, and a comment underneath, written in good faith, explaining that this line meant the service was automatically backed up from birth.

A day later it was still completely absent from the backup.

Nothing failed. No alarm fired. The backup ran perfectly every night and reported success, because it had faithfully backed up everything it knew about, and it had never been told about this.

How The Backup Decides What To Save

The nightly backup runs as root at just after 3 in the morning, box time, which is universal time, and its core is registry driven. That phrase carries the whole design.

For the important part it does not have a list. It reads the registry, the live record of what exists on this box, and for each service it finds it asks 2 questions. Have you declared any directories that hold state. Have you declared a database. There are still hard coded lists alongside that, for the SQLite databases and a handful of upload directories and configuration files, and those are exactly the part that has caused trouble.

Then it does the appropriate thing for each. Postgres databases get dumped. State directories get archived. The SQLite databases get copied using SQLite's own online backup mechanism rather than a plain file copy, because a plain copy of a database being written to can miss changes that have not landed in the main file yet, and the online backup guarantees you a consistent snapshot instead. The registry itself gets saved, and so does the whole floor configuration including the encrypted secrets.

Two things it never takes. The private key from the secrets episode, and any rendered plaintext secret. Those are excluded structurally, so that the backup, if it ever leaked, is a pile of encrypted material rather than a set of working credentials.

The word for all this is born backed. Declare it and you are covered, with no separate step to remember.

The Two Words Spelled Db

So back to the booking service, and the trap, which is genuinely nasty.

There are 2 files. A service repository authors a blueprint, saying what it wants deployed. And there is the live registry, which is what the backup reads.

Both of them have a thing called db, and they do completely unrelated jobs.

In the blueprint, db equals true means mount the database socket directory into this container. That is all it does. It grants the ability to connect. It is a plumbing instruction.

In the live registry, there is a small sub table with 4 keys describing an actual database, and that sub table is the only thing that gets a Postgres database discovered and dumped.

Same word. Two files. No relationship. And crucially, writing the first one does not create the second one. A service repository authors the blueprint and never opens the registry, so somebody who has just written db equals true has connected to a database and declared exactly nothing about backing it up.

The fix has 2 halves and neither is a document. The provisioning script now writes the registry entry as part of creating a database, so the ordinary path produces both. And the monitoring now flags any database that exists on the box and is claimed by nobody, so an unbacked database becomes drift rather than silence.

The Trap In The Other Direction

Now the part that is more interesting, because it is the inverse mistake and it very nearly caused a false emergency.

Having learned that a state declaration is what produces a backup, the natural next move is to audit. Go through every service, check which ones have declared state, and raise the alarm about the ones that have not.

That produces wrong answers, and one of them was nearly filed as a top priority incident about the identity provider's database being unprotected.

Because the backup script also carries a small hand written map, from before the registry driven design existed, covering 3 legacy databases that declare nothing at all. The identity provider is one of them. Its database is backed up nightly and has been for a long time. It just does not appear in the registry as declaring anything, because it predates the mechanism.

So the rule, and it is one of the sharpest in the whole estate, is this. Do not answer the question of whether something is backed up by reading the configuration that is supposed to cause backups. Read what the backup actually did. It writes a journal of every leg it uploaded, every night. That journal is the truth, and it is the only artifact that describes the world rather than describing an intention.

The general form of that is worth carrying. A rule that produces coverage is not the same as coverage, and reading the rule tells you what should have happened.

The Nightly That Backed Up Nothing And Reported Success

Two more failure modes in this area, both of the same species, and the species has a name in this estate. Failing green.

The first. The registry was being updated, and the disk filled up in the middle of the write. The old approach truncated the file and wrote the new content, so what remained was a registry cut off partway through.

Every reader of that file did exactly what it was told. It looked for the table describing databases, and did not find one, and concluded correctly that nothing was declared. So the nightly backup backed up nothing at all. It logged no failure, because nothing failed. It stamped a successful run, which kept the dead man's switch quiet, and everything downstream reported healthy.

The fix is unglamorous and correct. Every writer of that registry now writes to a temporary file, flushes it to disk, sets the ownership, and then atomically replaces the original, with a checksum comparison so that a half write can never become the live file. Either the whole new version is there or the whole old one is.

The second is subtler. A state directory can quietly become a link pointing somewhere else, without any deploy happening at all. The most likely path is entirely reasonable. The disk gets tight, somebody moves a fat directory somewhere roomier and leaves a link behind so everything keeps working.

The archiving library, by default, does not follow links. It archives the link itself. So the backup would faithfully save a tiny pointer, containing zero bytes of your data, and report success.

The interesting part is where the check for this was put. Not in the deploy path, where you might naturally put it, because this can happen with no deploy involved. It lives inside the function that actually writes the archive. The thing that does the work is the thing that checks, because it is the only place that sees the world at the moment it matters.

Where It Goes, And Why Ransomware Cannot Reach It

The destination deserves a moment.

The backups go to a dedicated storage bucket, in a separate, isolated project at the cloud provider. Not the same project as the server. The credential the box holds for it cannot delete anything, and the bucket keeps previous versions of everything written.

Think about what that means in a bad scenario. Somebody gets into the box. They have root, they have everything. They cannot simply delete the backups, because the only credential present on the machine cannot delete, and overwriting leaves the previous version recoverable. That is a very large obstacle rather than an absolute one. Old versions do eventually expire on a schedule, so an attacker who held the machine quietly for long enough could still grind away at it. What it defeats is the actual threat, which is the fast destructive one.

The isolation is the entire point of the arrangement, and the documentation says explicitly that it must not be simplified back to sharing a bucket with everything else, which is exactly the sort of tidy up that looks like good housekeeping and quietly removes the property that makes it a backup rather than a copy.

Proving It, Which Almost Nobody Does

The last piece is the one most estates skip forever, and it is the one that decides whether any of the above was real.

A restore drill runs weekly, automatically. And in July 2026 a genuine full restore was performed, end to end, using only artifacts from off the box. The storage bucket, the image registry, the git repositories, and the escrowed key.

The result was 5 Postgres databases restored with matching row counts and the correct Swedish text sorting rules, 6 SQLite databases, 15 state directories, 11 static sites, everything integrity checked. The escrowed key decrypted what it needed to. Every image was retrievable. And the identity provider came up and served logins on the restored database and the restored key, which as we saw 2 episodes ago is the pair that has to arrive together or you have nothing.

There is one procedural note attached to that runbook, which is that you execute from its detailed section and not from its overview. The overview reads like a summary of the steps and quietly omits several that are load bearing. That is written at the top of the document in bold, which is the correct place for a warning about the document itself.

Next time, the images. Where they live, why losing this box would have lost half of them, and the difference between a name and a thing.