Zero Privilege And Seven Verbs: The Broker

An Operator With No Powers

There is an account on popcorn2 whose entire job is to operate the machine, and it cannot do anything.

It has no administrator rights of any kind. It cannot run containers. It cannot read the secrets. It cannot install anything, or restart anything, or become anybody else. It is an ordinary unprivileged user, so it can of course look at whatever the machine leaves readable to everyone, but if it tries to do any actual piece of server administration, every single command is refused.

That account is the one an AI uses. And the design question that produced it is a good one. If you are going to let a language model operate production infrastructure, how much of the machine do you hand it?

The answer this box arrived at was, none of it. Not a reduced set. None. And then, separately, one door.

The Door

The door is a socket. In Unix terms it is a special file that two programs can talk through, and it is owned by root with permission granted to exactly one group. On one side sits a small program that does hold real privilege. On the other side sits the operator account with none.

To do anything at all, the operator sends a short message through that door. The message contains 2 things. A verb, and the name of a service. That is the complete vocabulary. There is no field for extra arguments, no way to pass a command through, no escape hatch where a cleverly worded string becomes an instruction.

There are 7 verbs in total. Deploy. Restart. Stop. Logs. Render a secret. Reconcile. Decommission. That is the entire privileged surface area of the artificial intelligence on this machine.

And there is a detail about how the door checks who is knocking that is worth pausing on, because it is the difference between real security and the appearance of it.

The Kernel Decides Who You Are

The naive way to check identity is to ask. The client says who it is, and the server believes it. Almost every web system on earth works this way, wrapped in enough cryptography to make the claim hard to forge.

The broker does not ask. When a connection arrives on that socket, it asks the kernel directly which process is on the other end, and therefore which user owns it. The kernel knows this for certain, because the kernel is the thing that created both processes. There is no claim to forge because nobody was ever asked to make one.

That property, identity established by the operating system rather than asserted by the caller, is one of the genuinely nice things about doing this over a local socket instead of over a network. It is worth being precise about what it proves. It proves which local process is on the other end of this connection and which account owns it. It does not prove who is driving that process. But combined with the fact that only one group can reach the socket at all, that is a much stronger foundation than a claim in a message.

On top of that identity sits authorisation, and it is per verb rather than one flat list. Deploying is checked against the catalog of what may be deployed. Restarting, stopping, reading logs and decommissioning are checked against the registry of what actually exists, so you can only operate on live services. Rendering a secret authorises itself from the existence of the encrypted file and a matching service account. Reconcile takes no service at all. And in every case an unknown name does not produce a guess or a best effort. It produces a refusal.

The Catalog That Root Writes And The AI Cannot

Here is the constraint that does the most work, and it took me a moment to appreciate how sharp it is.

When the operator says deploy the hotel bot, it does not say what the hotel bot is. It cannot. There is a file in the floor repository called the deploy intent catalog, and that file is the only thing that answers the question. Which image. Which port inside the container. Which port on the host. Which trust zone. Which sensitivity ring. Whether it gets a public web address. Whether it needs a secret, and if so which key names. Which directories it may mount.

That file lives in git and is owned by root, which means the only way to change it is through the authoring path from episode 2, on the Mac, by whoever holds that repository. The operator has no path to write it at all. So the operator can name a service, and it can never choose the measurements of one. It cannot invent a new service. It cannot ask for a container that mounts the whole disk, because mounting the whole disk is not a thing anyone can request. It is a thing that either is or is not written in a catalog it cannot reach.

There is also a subtle separation here that took a real bug to discover. The catalog of what may be deployed is deliberately kept apart from the record of what is actually running. They sound like the same list and they are not. If you merged them, then writing down that a service is deployable would immediately create a record saying it exists, and the monitoring system, which we get to next episode, would start reporting that a declared service is missing. So intent and reality live in 2 separate files. Deploying copies selected values from the first into the second, and the monitoring compares them, but declaring something deployable never by itself creates a claim that it exists. What might exist, and what does exist, stay 2 different questions.

Where The Design Meets The Real World

So that is the architecture, and it is genuinely good. Now let us talk about all the ways it has lied.

The first one is my favourite because the lie is technically true.

You rebuild a service. New code, new image, everything correct. You tell the broker to deploy it. The broker thinks for a moment and reports success. And the website continues serving the old version, indefinitely, forever, with everything green.

Here is why. The broker's job is to make declared state match reality. It checks the unit file, the web address configuration, and the registry row. When you rebuild an image and keep the same label on it, none of those 3 things change. The unit file still points at the same label. The web configuration is unchanged. The registry row is unchanged. So the broker looks at all 3, correctly concludes that everything already matches what was declared, and does nothing at all. Which is exactly right, and completely useless.

Its success message means configuration has converged. Not that your change is live. Those are different sentences, and one site served a 9 day old build because everyone read the first as the second. What makes it genuinely nasty is that running the deploy again also does nothing, which reinforces the belief that everything must be fine.

There is a second flavour that is worse, because it happens even when the deploy does change something. A network setting was added to a service. The broker rewrote the unit file correctly, and the new setting was visibly there on disk. The running container had been up for 2 hours and knew nothing about it, because a container reads its configuration once, when it is created. Changing the recipe does not change the cake.

The Exit Code Is Wrong In Both Directions

The restart verb has its own problem, and it is a beautiful little piece of concurrency comedy.

When a container publishes a port, there is a small helper process holding that port on the host's behalf. When the container stops, that helper releases the port, but it does so asynchronously, on its own schedule, a moment later. The broker's restart tears the old container down and immediately starts the new one, which tries to grab a port the old one has not quite let go of yet. The result is a failure message about the address already being in use.

And then the service manager, which is configured to always retry, tries again a few seconds later, and it works. So the service is up and healthy, and the broker has already returned a failure.

Put that next to the previous problem and you get the actual lesson of this episode. The broker reports success when nothing happened, and reports failure when everything worked. Its exit code is unreliable in both directions.

Which is why every guard on this box, and there are many, was built on a single principle. Assert state, never a verb. Do not ask whether the deploy said it worked. Ask what is actually running, and compare it to what should be. The output of a command is a story about the world. The world is available for inspection directly.

The Registry That Only Wrote Once

The last one is the most expensive, and it is a lesson about where you put a check.

Services declare which directories hold data worth keeping. That declaration is what the nightly backup reads. Declare it, and you are backed up from that moment. It was described, correctly, as coverage by construction.

Except the code that wrote that declaration only ran when the service was first registered. It checked whether the row existed, and if it did, it left it alone. So the declaration was written once, at birth, and never again. A service that gained an important directory later, after it had already been deployed, never got that directory into the record, and no redeploy would fix it, because redeploying skipped the write.

One service kept its authentication tokens in exactly such a directory. Those tokens sat unbacked for 46 hours across 223 consecutive monitoring runs that were all reporting a problem, while the nightly backup reported complete success every single night. The backup was not broken. It faithfully backed up everything it had been told about, and it had not been told.

The fix was structural rather than clever. One function is now the only thing in the entire system permitted to write that declaration, and it runs on every deploy rather than only on the first one. So the ordinary act of deploying a service repairs the record.

And the deeper lesson, which shows up throughout this estate, came from the first attempt at fixing it. The first attempt added careful validation in the high level code. But a shell script further down had already written the value before that validation ever ran. So a bad entry got written anyway, and the later check could only complain about something it was too late to prevent.

If a guard runs after the write, it is not a guard. It is a witness. Put the check where the writing happens, and make sure that place is the only place that writes.

What The Broker Is Really For

It is tempting to see all of this as automation. It is not automation. Deploying by hand would be faster.

What the broker actually produces is a boundary with a known shape. There is a precise, short, written answer to the question of what an AI can do to this machine, and the answer is 7 verbs against services that a human wrote into a catalog. Everything outside that is not merely discouraged. It is unreachable.

That is worth more than convenience, and it is worth the fact that the tool occasionally lies to you about whether it worked. You can build detectors for lies. You cannot build a detector for an unbounded blast radius.

Next time, the file format that runs every service on this box, and the reason a directive that means one thing in the manual means something completely different here.