A Unit That Is Not A Unit: The Quadlet Trap
The File That Looks Familiar And Is Not
Every containerised service on popcorn2 is described by a small text file. Not the static websites, which Caddy simply serves, and not the root side jobs, but everything that runs as a container. The file has sections in square brackets and settings written as a name, an equals sign, and a value. It is the format you have seen a hundred times if you have ever looked at how Linux starts things.
And that familiarity is a trap that has now bitten twice in the same year, on 2 different services, in exactly the same way. Once during a bring up, where the thing simply would not start, and once on a live service.
Because those files are not what they appear to be. They are called quadlets, and understanding the difference between what they look like and what they are is the difference between a service starting and a service silently refusing to.
What Is Actually Happening
The service manager on Linux, systemd, does not run these containers itself. It has its own container mechanism for other purposes, but the containers on this box are podman's business.
So there is a translator. When the system starts, or when you tell it to reload, a small program walks through the quadlet files and turns each one into a real systemd unit. The unit it produces does something extremely boring. It runs podman, with a long list of command line flags.
That is the whole mechanism. You write a file that describes a container in a systemd shaped language, and a generator converts your description into a very long podman command.
Which means the settings inside a container section are not systemd settings at all. They are podman flags wearing systemd clothes. And the moment you assume otherwise, you are reasoning about the wrong software.
The Dash
Here is the specific one, and it is beautiful in the way that only a genuinely subtle bug can be.
In systemd, when you point a unit at a file of environment variables, you can put a dash in front of the path. The dash has a precise, well documented, universally known meaning. It means this file is optional, and if it is missing, carry on without complaint.
Somebody wrote exactly that, for exactly that reason. There was an optional secret feeding an optional feature, and the sensible thing was to say load this if it exists.
The quadlet generator does not implement systemd's dash. It has never claimed to. It takes the whole string, dash and all, and treats it as a path. Worse, because the string now begins with a dash rather than a slash, it is no longer an absolute path, so the generator resolves it relative to the directory the quadlet files live in. The command that comes out the other end points at a file inside a directory named dash, inside the quadlet folder, which has never existed and never will.
Podman cannot open it. Podman refuses to start. The container does not run.
And the generator reports success. It exits cleanly. Every check of the form does this file generate correctly passes green, because the file generated perfectly. It generated a perfectly formed command to do the wrong thing.
The service this happened to was the telemetry collector, which is the single point through which every other service on the box reports what it is doing. So an optional secret, for an optional feature, took down the entire observability plane of the machine, and did it in a way where the obvious diagnostic said everything was fine.
Two Rules That Came Out Of It
The first rule is that there is no such thing as an optional environment file here. Not because it is forbidden, but because the mechanism does not exist. So the file always has to exist. An empty file is accepted without complaint. A missing one is fatal. Both of those were tested rather than assumed, which is the whole spirit of this next part.
The second rule is the general one, and it is now written in several places across this estate in slightly different words. Ask the generator. Do not reason about it.
You can run the translator by hand, in a scratch directory, and have it print the podman command it would produce. It takes 5 seconds. You then read, with your own eyes, the actual flags. Not what the documentation implies. Not what the directive means in the other system that uses the same syntax. The literal command.
Every time this has been done, it has been faster than the argument about what the setting should mean.
Names, And Why A Naive Scan Pages You At Night
There is a second reason to ask the generator instead of reading the files, and it is quieter but it caused real trouble.
A quadlet file has a filename. It also, optionally, contains a setting for what the container should be called. Those are 2 different namespaces, and they do not have to match. A file called foo produces a service called foo, which may be running a container called bar.
The registry of what should exist on this box records container names. A program that tried to check the registry against the files on disk would be comparing 2 different kinds of name, would find mismatches everywhere, and would report drift. Drift, on this box, means an alert. So a naive comparison does not produce a wrong answer quietly. It produces false alarms loudly, at 3 in the morning.
The generator hands you both names at once, plus the type of thing, plus where it came from. Which is why every checking tool on this box now asks it rather than reading files.
It also, for free, ignores the 13 inert leftovers scattered around, files with backup style suffixes that a plain directory scan would confidently report as unknown services. And it searches in places a hand written scan forgets, because there is more than one directory these can live in.
There is one last landmine in this area that I enjoy purely as a piece of engineering comedy. The registry has a field literally named uid, which in Unix means a numeric user identity. In this registry it contains a username, as text. A checking tool that dutifully converted usernames to numbers to compare them found that nothing matched anything, and cheerfully reported that the entire fleet was unaccounted for.
The Files That Are Waiting For A Reboot
Now the structural problem, which is genuinely interesting.
Monitoring on this box mostly works by asking what is running. That is the honest question, and it catches almost everything. But it cannot see a quadlet file for a service that was retired, because nothing is running, so there is nothing to notice.
That file is not harmless. Of the roughly 51 quadlet files on the box, 43 of them are configured to start automatically at boot. So a retired service whose file was left behind is not dead. It is asleep. And it wakes up the next time the machine restarts, possibly onto a port that something else now owns. That very nearly happened during a component swap, with 2 things reaching for the same port.
The fix was to ask a different question. Instead of what is running, ask the generator what a boot would produce, and compare that against the registry. It is the difference between checking who is in the building and checking who has a key.
Two things still produce these orphans, and it is worth knowing why neither was simply fixed.
The tool that copies files into place never deletes a file that has been removed from its list. That is deliberate, and I think correct. This box is a read only copy of a repository, and giving an automatic sync tool the authority to delete live system files means that one truncated configuration file becomes an outage. So it detects and reports rather than deleting, and removal is a deliberate human act.
The decommission verb removes one file. For a service built as a group of containers, which the photo booth is, that single file does not exist, and all 6 of its files would be left behind while the registry entry disappeared. That one is still unfixed. It is simply now visible, which on this box is considered an acceptable resting state for a known problem.
The Word That Saves Your Cleanup Code
One last thing about containers, which does not fit anywhere else and is too good to leave out.
When a container starts, it runs one command. Very often that command is a shell, which then runs your actual program, because you need a shell to expand a variable or join 2 things together.
The consequence is that the shell is process number 1 inside the container, and your program is process number 2. And when the system asks a container to stop, it sends a polite termination signal to process number 1. The shell, which has no instructions about what to do with it, discards it. Then 10 seconds later, the system gives up on politeness and kills everything.
Which means every shutdown routine in your application is skipped. Every cleanup step. Every close this connection properly, every mark this job as finished, every let the other end know we are going.
There is a single word that fixes it. If the shell is told to exec the program rather than run it, the shell replaces itself with your program entirely, and your program becomes process number 1, and receives the signal.
This was found on a service that talks to quantum computing hardware, where the skipped cleanup was the step that ends a live session on a platform billing three thousand euros an hour. The tell, across the whole fleet, is beautifully simple. If a service takes about 10 seconds to stop, it is doing this.
Next time, the file that says what is supposed to exist on this box, and the program that spends every 15 minutes checking whether reality agrees.