The Anti-Join: What Is Supposed To Exist Versus What Does

Two Lists

There is a program on popcorn2 that wakes up every 15 minutes, does one conceptual thing, and goes back to sleep.

It builds 2 lists. The first is what is supposed to exist. The second is what actually does. Then it compares them, in both directions, and reports anything that appears in one and not the other. Not everything on the machine, to be precise. A deliberately bounded set of things it knows how to ask about, which is a distinction that turns out to matter.

In database language, finding the rows on one side with no match on the other is called an anti join, which is a lovely name for a suspicious activity. In practice it is the closest thing this machine has to a conscience.

The List Of What Should Exist

The first list comes from a file called the registry. It is the declarative record of every service on the box. Entries come in kinds, an ordinary container service looks different from a static site or a batch job, and most of the fields are optional. But the vocabulary is shared. Which Linux user owns it. Which container. Which port it listens on. Which public web addresses it answers to. Whether it reports telemetry. Which secret keys it is allowed to hold. Which directories hold data worth keeping.

Two details about that file matter more than they look.

The first is that there is no separate port table anywhere. The registry is the port table. When you need to allocate a port for a new service, you look at what the registry already claims and what is actually listening, and you take an unused one. There is no second document that could disagree with the first, because there is no second document.

The second is where it lives. It is not in the floor repository with everything else. It sits in a state directory, deliberately outside the code tree, and the reason is that it once was inside, and a synchronisation job overwrote it with an older copy. The system correctly noticed and reported everything as unrecognised, which is the right response and not a pleasant morning. Live state and shipped code do not belong in the same place, because deployment tools are allowed to overwrite shipped code.

Adding a service to that file is what the estate calls being born covered. The moment the entry exists, the monitoring includes it, and each further thing it declares brings its own coverage with it. Declare a web address and the edge configuration is rendered and watched. Declare a state directory or a database and the nightly backup picks it up. What the phrase does not mean, and this has cost real money, is that a bare entry produces all of it. Coverage follows declarations. It is a consequence of saying what you have, not merely of existing.

The List Of What Actually Exists

The second list is gathered by going and looking. What containers are running, per account, because as we established there is no global view. What ports are listening. What web addresses the reverse proxy is actually serving, and whether their certificates are healthy. And, for a specific named set of scheduled jobs rather than every timer on the machine, whether those are properly enabled.

Then the comparison runs both ways, and both directions catch a different class of problem.

Going from declared to actual answers the question, is everything that should be here, here. A service in the registry with no running container. A declared web address that is not being served. A certificate quietly running down.

Going the other way answers the more interesting question, which is what is here that nobody declared. Something listening on a port that no entry claims. A web address being served that appears in no record. Those get labelled rogue, and rogue is the word that means somebody did something outside the system, or something was left behind by a cleanup that did not finish.

And there is a third direction, added later, which asks the generator from the last episode what a reboot would produce, and compares that. Because a retired service that is not running today is not rogue today. It is rogue at the next restart, and by then nobody will connect the two.

On top of all of that, the same program checks the floor repository itself. Is there an uncommitted edit sitting on the box. Does every file that the deployment map says should have been copied somewhere still match its source, in both content and in things like ownership and the executable bit. And is the box behind, with some commit having waited more than 7 days to be deployed.

Those 3 do not all mean the same thing, which is worth being precise about. An uncommitted edit means somebody worked directly on a machine that is supposed to be a read only copy. A mismatched copy might mean that too, or it might mean a reinstall silently failed. And being behind means somebody committed on the Mac and never deployed it. Three different stories, one alarm, and the alarm is right to fire for all of them.

What It Does With A Problem

Here is the part I find genuinely elegant.

When it finds drift, it does not send an alert. It withholds a stamp.

On a successful clean run it touches a file, recording that it completed and found nothing. Something else entirely, running on a completely different machine in a different building, watches the age of that stamp. If the stamp stops being refreshed, that watcher starts making noise.

Think about what that buys. A program that alerts when it finds a problem cannot alert when it is dead. A program that must keep proving it is fine, to somebody else, fails loudly by going quiet. It is a dead man's switch, and the entire monitoring philosophy of this box is built out of them. There is a whole episode coming on the layers of that.

The Container That Lived For Five Seconds

Now the incident, because this design has one specific weakness and it found it.

There is a static site publisher on this box. Every 60 seconds a timer fires, a container starts, it checks whether the website's source has changed, it builds if needed, and it exits. It lives for about 5 seconds and then it is gone. It is a batch job, not a service.

The reconciler samples what is running every 15 minutes. And roughly 20 percent of the time, that sample landed inside the 5 second window when the builder was alive. It saw a running container that appeared in no registry entry, correctly applied its rule, and declared a rogue.

So several times a day, a genuine action level alert fired about something that was completely fine.

Here is why that mattered, and it is not the reason you would first assume. The dead man's switch was never in danger, because clean runs kept interleaving and the stamp kept refreshing. Nothing broke. The cost was entirely in credibility. The same alerting channel that says a real service has died was now crying wolf several times a day. Train people to ignore that channel and you have not degraded your monitoring, you have destroyed it, while every dashboard still says the monitoring exists.

The Fix, And The Fix That Was Rejected

The obvious fix is to add the builder's name to a list of things to ignore. That fix was explicitly refused, and the reasoning is the best part of this whole story.

A name exemption is not a fix. It is suppression with better manners. The check still believes the thing is wrong, and you have simply told it to be quiet about that specific case. Nothing has learned anything.

The second tempting fix was subtler and is actively worse. Ignore any container younger than a few seconds. That sounds surgical, and it is a hole shaped exactly like the thing you are trying to catch. The reconciler samples every 15 minutes. A genuine intruder, or a genuine leftover, is never going to be caught in its first few seconds of life. So the rule would let through practically nothing except the false alarm, while quietly widening the door for precisely the class of thing the check exists to find.

What was actually built was a new word in the registry. A batch job can now declare itself as a batch job. Not an exception to the rule, a category the rule understands.

And then the important part. That new category earns its coverage rather than merely being excused from the old one. A declared batch job gets checked in a way that suits it, by grading its timer rather than its container. Is that timer permanently enabled, and is it active. Which catches something nothing on this box could see before, which is a scheduled job that was silently switched off. It does not fail. It does not crash. It simply stops happening, and everything reports green forever.

There were 2 further traps in building even that small thing, and they are both instructive. You have to check the timer and never the service, because a timer triggered service legitimately has no automatic start configuration, so asserting that it was enabled would produce a permanent false alarm. And the registration has to happen when the job is actually switched on, not when the code ships, or a freshly built machine drifts the instant it is set up.

The Redirect That Had To Stay Declared

One more, because it is the perfect example of a tidy up that breaks things.

A web address had stopped serving content and become a simple redirect to somewhere else. Reasonable evolution. And since it no longer served anything, somebody could very sensibly conclude that its entry in the registry was now clutter, and remove it.

That breaks the reconciler, immediately. The comparison keys on what is served, not on what it does. A redirect is still a served address. So the moment the declaration goes, the address becomes an unrecognised thing being served by the machine, which is precisely the definition of rogue.

The declaration is load bearing exactly because the block does nothing. Which is a genuinely counterintuitive sentence, and it is why both live examples of this now carry their reason written directly inside the registry file, so the next person to tidy up reads the explanation before making the same correct looking mistake.

That is the recurring shape of this whole system, really. Almost every rule that looks arbitrary is a scar. The way to work with it is not to memorise the rules. It is to know that each one is a healed wound, and to go and read what cut it.

Next time, where the passwords live, and why the encrypted versions of every secret on this machine are deliberately committed to a git repository.