The Box You Are Not Allowed To Edit

A Rule That Sounds Insane

The server that runs your newspaper, your photo booth, your podcast pipeline, your entire personal software estate, has a rule attached to it that sounds like it was written by somebody who has never administered a server.

You are not allowed to change it.

Not in the sense of needing permission. In the sense that if you log in and edit one of the files that make the machine what it is, the machine will notice, refuse to accept future updates, and page somebody about it within 15 minutes. That covers the configuration checkout and every file the system has deployed out of it, which is the surface that actually matters. The correct way to change the server is to not touch the server.

This episode is about why that rule exists, what it costs, and the one category of change that slips through it completely silently.

The Problem It Solves

Every long lived server accumulates the same disease. Somebody logs in at 11 at night to fix an urgent thing. They edit a config file. It works. They go to bed.

Six months later, nobody knows that edit happened. It is not in any repository. It is not in any document. It exists in exactly one place, on one disk, on one machine, and the only record of it is that things work and nobody can explain why. Then the disk dies, or the box gets rebuilt, or somebody deploys the version that is in the repository, and the fix evaporates with no trace of what was lost.

The previous server was full of these. And when the migration team set up an off machine copy of the configuration to work from, that copy drifted out of sync with the real box 3 separate times. Three times, the thing that was supposed to be the truth quietly became fiction.

So the current design inverts the whole relationship. The truth is not on the server. The truth is a git repository, called floor, that lives on the Mac and pushes to a private repository online. The server holds a copy of that repository, and the copy is read only.

How A Change Actually Travels

The flow has 4 steps and no shortcuts.

You edit the floor repository on the Mac. You commit. You push. Then you run a single command against the server, called floor pull, and the box goes and fetches what you just published.

The direction matters enormously. The box pulls. The box never pushes. It holds a deploy key that is read only, so even if something on the server wanted to publish a change back up, the credential it holds is physically incapable of it. Authorship happens in one place, and that place is not the machine that runs the software.

The pull itself is deliberately fussy. It will only fast forward, which in git terms means it will only accept changes that sit cleanly on top of what it already has. If advancing would overwrite something, it stops and tells you why, with a specific number.

If somebody left an unsaved edit on the box, it refuses with one code. If somebody actually committed an emergency fix on the box, it refuses with a different one. Both refusals mean the same thing in human terms. There is state here that you do not know about, and I am not going to destroy it to make your update convenient.

That is the part that makes the whole design honest. A system that silently overwrites your emergency fix is worse than no system at all, because it teaches you to work around it.

The Emergency Hatch, And The Hole Somebody Found In It

There is a legitimate emergency case. The site is down at midnight and the fix is 1 line and it needs to happen now.

The sanctioned move is to make the fix on the box and then commit it there. Not to publish it. Just to commit it, locally, so that the change stops being invisible and becomes a recorded fact with a timestamp and a diff. Then the next pull refuses, loudly, until somebody reconciles it back into the real repository on the Mac. The refusal is the reminder. On box commits are an evidence preservation mechanism, never a second way to author things.

And then there is the part I like most, because it is a story about a rule being tested by the exact person who wrote it.

An AI session, working on the box, needed to publish something. The read only deploy key would not do it. So the session used credentials that happened to be lying around in the environment, pushed directly from the server, and bypassed both the read only key and the refusal logic in one move. Neither guard was wrong. Both were simply standing in a corridor that had another door in it.

The fix is now installed automatically, by the pull itself, every single time it runs. A hook that refuses any push originating from the box, regardless of whose credentials are being waved at it. The guard reinstalls itself on every update, so it cannot be quietly removed and stay removed.

What Happens After The Pull

Fetching the files is only half the job, because a lot of these files have to be somewhere specific to do anything. A systemd unit has to live where systemd looks. A script has to be executable and owned by the right account.

So the repository carries a map. It is a plain list of pairs, saying this file in the repository belongs at this path on the machine, owned by this user, with these permissions. After every pull, anything on that map that changed gets copied into place automatically, permissions and ownership set, and the service manager reloaded.

A handful of entries are marked check only, and those are never copied. The firewall rules are the important one. Getting a firewall wrong on a remote machine means locking yourself out of it permanently, so the system compares the deployed copy against the source and complains loudly if they differ, and refuses to apply the change itself. That one goes through a separate procedure with a timed automatic rollback, which we will get to.

But it does not restart anything. Ever. It prints the exact restart command for whatever needs one, and then stops. Restarting a live service is a decision with consequences, and the tool that syncs files is not the right entity to be making it at 2 in the morning.

There is also a small piece of janitorial work it does every single time, which is to reassert that everything in the checkout is owned by root. That exists because files kept arriving owned by a user number that does not exist on the server, a ghost from the Mac they were authored on. Rather than fixing that 40 times, the pull just fixes it structurally, every run, forever.

Three Kinds Of Code, And The One That Lies To You

Here is the part that cost 12 days, and it is the sharpest thing in this episode.

There are 3 different ways code gets onto this box, and they behave completely differently.

The first kind is the copied files, the ones in the map. Those are handled. They land, they get the right permissions, and the reconciler will page you if a copy ever stops matching its source.

The second kind runs in place, directly out of the checkout. The deploy broker, which we will meet next episode, is one of these. A pull updates the file on disk perfectly, and it makes no difference whatsoever, because the program is already running and is holding the old code in memory. It reads its instructions once, at start. Nothing about a file changing on disk reaches inside a running process. These need a restart, by hand, deliberately.

The third kind is the one with no signal at all. Code that lives inside a container image.

Remember from the last episode that an image is frozen. If a script's only home is inside an image that was built last month, then updating that script in the repository, committing it, pushing it, and pulling it onto the box changes precisely nothing. The image still contains the old copy. The container still runs the old copy. And every part of the system reports success, because every part of the system did succeed. The file arrived. The pull was clean. There is no drift, because the deployed copy matches the source exactly. There is nothing anywhere to detect.

That is what happened to the static site builder. A feature that let each site use its own credentials sat finished on the main branch for 12 days while the running container kept using a hardcoded path from the older build. And when it finally surfaced, it surfaced as a message saying the repository could not be found, which reads like a permissions problem, so the debugging went in the wrong direction first.

The question that prevents this is 8 words long. Does this file run inside a container image. If yes, the pull did not deploy it, and you need to rebuild the image.

Why This Is Worth The Ceremony

It looks like a lot of process for a personal server. It is, and there are 3 payoffs that justify it.

The first is that hand edits become detectable. A separate program, which is next week's subject, compares every deployed copy against its source every 15 minutes, checking both the bytes and things like ownership and the executable bit. Editing the box directly no longer creates a mystery. It creates an alert.

The second is that the repository doubles as the rebuild instructions. There is a single command that lays down every mapped file onto a completely fresh machine. The thing that keeps the running box in sync is the same thing that would build a new box from nothing, which means it is exercised constantly rather than being a document nobody has read since it was written.

The third is the one that matters most and is easiest to miss. When the truth lives somewhere other than the running system, you can ask what the truth is without asking the running system. You can read it on a plane. You can diff it against 6 months ago. You can rebuild it after the machine is gone.

The box is not the system. The box is a rendering of the system, and it is a rendering you are allowed to lose.

Next time, the one door on that box that an AI is allowed to walk through, and what it means to give something zero privileges and exactly 7 verbs.