Nobody Home: Thirty-Eight Services You Cannot List
The Question That Starts Everything
Here is a fact about your server that sounds like a bug report. There are roughly 38 services running on popcorn2 right now, and there is no single command that will list them for you. None. You can ask the box what is running and it will tell you almost nothing.
That is not a gap. That is the entire security design of the machine, showing through. And to understand why, we have to start further back than you might expect, because everything else in this series sits on top of it.
What A Container Actually Is
Forget containers for a second. Think about what running software on a server used to mean, and still means on most servers in the world.
You get a machine. You install Python on it, and a web server, and a database client, and some image library that the photo tool needs. Then you install a second application, and it wants a different version of that same image library. Now you have a negotiation. Every application on the box shares one filesystem, one set of installed packages, one set of users. They all live in the same apartment and argue about the thermostat.
A container image is the answer to that, and the trick is embarrassingly simple. You take a complete, finished, working filesystem, everything the application needs down to the system libraries, and you freeze it. Not a backup. A snapshot that is designed to be run.
The most useful mental picture is a photograph of a computer's disk taken at the exact moment everything was installed correctly and nothing had happened yet. The image is that photograph. It is read only. It never changes. When you want to run the software, you do not restore the photograph. A thin writable layer is stacked on top of it, and the program inside is started. Nothing boots. That combination of a frozen image, a scratch layer, and a running program is a container.
That distinction is the one to hold onto, because it comes back in almost every episode. The image is the frozen thing. The container is one running copy of it. You can start 5 containers from the same image and by default none of them shares state with the others, though you can absolutely wire them together on purpose if you want to.
Why This Is Cheap, And Why It Is Not A Virtual Machine
The obvious objection is that this sounds enormous. A full filesystem per application, on a box with 8 gigabytes of memory and about 93 gigabytes of disk.
Two things make it cheap. The first is that images are built in layers, and layers are shared. If 10 services start from the same Debian base, that base is stored once and all 10 point at it. Each one only stores its own differences. There is a wrinkle on this particular box, which we will come to, where that sharing does not reach as far as you would hope.
The second is that a container is not a virtual machine, and this is the part people get wrong for years. A virtual machine virtualises a whole computer. It runs its own kernel, boots its own operating system, and believes it has its own hardware. A container does none of that. A container is one or more ordinary processes running on the host's own kernel, with a set of lies told to them about what filesystem they can see, what other processes exist, and what network they are on.
That is most of what it is. It is a process wearing a blindfold. It starts in under a second because there is no operating system to boot. It shares the kernel with everything else on the box, which is the source of both its speed and, later, some of its sharper edges.
And there is one more piece of vocabulary that will matter enormously by the time we get to disaster recovery. When you refer to an image, you usually refer to it by a tag, something like capture colon wave 4. A tag is a sticky label. It can be peeled off one set of bytes and stuck onto another. The bytes themselves have a separate, permanent identity. Tag and identity are two different things, and an entire episode of this series exists because that distinction was learned the hard way.
The Daemon Problem, And Why The Box Runs Podman
Now, the thing that actually runs containers.
The famous one is Docker, and conventionally Docker works like this. There is a background program running as root, permanently, called the daemon. When you want to run a container, you do not run it. You send a polite request to the daemon, and the daemon, which is root, runs it for you. Docker can be run in a rootless mode these days, but the classic arrangement is the one almost everybody has.
Read that again, because the security consequence is brutal and it is not hypothetical. Anything that can talk to the Docker daemon can ask it to start a container that mounts the whole host filesystem. Which means anything that can talk to the daemon is, for all practical purposes, root on that machine. Access to Docker is not a lesser privilege than root. It is root with extra steps.
Popcorn2 does not run Docker. It runs Podman, and Podman's central idea is that there is no daemon. Nothing privileged sits in the middle of starting a container. The container is started by the ordinary unprivileged user that owns it, runs under that user's own service manager, and inherits that user's limits rather than root's.
Root is still involved on this box, to be clear. Root creates those service accounts and installs their unit files, and there is a privileged broker that performs deployments. What is gone is the permanently running privileged thing that anybody can ask to run anything.
That property is what makes the next decision possible.
A Linux User Per Trust Domain, And Why That Is The Whole Point
On popcorn2, services run under service accounts whose names all start with svc, and the unit of separation is not the service. It is the trust domain.
Most services get their own account outright. Koma has one. The photo booth, the hotel bot, the print service, the newspaper's live feed, each is its own account. But where several services genuinely are one system, they share. The 11 PärKit tools, Capture and Time and Director and Stats and the rest, all run under a single account, because they are limbs of one organism rather than 11 independent products. The podcast feed and its builder share another.
Those accounts have a property called lingering, which means their personal service manager keeps running whether or not anybody is logged in. Nobody is ever logged in. The manager runs anyway, which is what allows a service to survive a reboot without a human present.
Each account owns its own containers, its own home directory, its own rendered secrets, and nothing else. That account is the trust domain, and the phrase is deliberate. If somebody finds a hole in the hotel booking service and gets code running inside its container, what they have obtained is the ability to act as the hotel booking service's account. They can read the hotel booking service's files. They cannot read the newspaper's database password, because that file belongs to a different account and the filesystem says no.
Underneath, this is done with something called user namespaces, and the mechanism is worth 30 seconds because it is genuinely clever. Inside the container, the application often believes it is root. It has to, because a lot of software refuses to start otherwise. But that root is a fiction maintained by a mapping. In the ordinary shape here, the container's root maps to the service account's own perfectly unprivileged host identity, and every other identity inside the container maps into a block of 65536 subordinate numbers assigned to that account. So the process thinks it is user 0, all powerful, and the kernel sees an ordinary account that owns nothing outside its own home.
The container is not lying to the kernel. The kernel is lying to the container. That is rootless, and it is why a container escape on popcorn2 is a bad afternoon rather than a total loss.
The Prices You Pay For That
Every good design decision buys you something and charges you something, and this one charges in inconvenience.
The first bill is the one we opened with. Because containers belong to accounts, and there is no daemon holding a global list, there is no global list. Asking the box what is running gives you nothing useful. You have to ask each account separately, one at a time. The box has a small helper for exactly this, and the muscle memory of everyone who works on it is now per account, not per box. The absence of a fleet wide view is not an oversight. It is the isolation being real.
There is a storage cost hiding in the same place. Each account has its own image store, so that shared Debian base I mentioned earlier is shared within one account and duplicated across accounts. Twenty separate copies of the common layers, not one. That is part of why disk usage on this box needs occasional attention, and why an entire later episode is about pruning old images.
The second bill is that isolation applies between services too, in ways that surprise people. Two containers owned by two different accounts do not share a network or a loopback address. Podman's own maintainers have said plainly that shared container networks across users are impossible by design. They still share the host's kernel, and they can absolutely talk to each other, but only by going out to the host and back in like any other client. There is an entire later episode about exactly that, and about the 18 days a paid service quietly failed to send a single notification because nobody knew the route existed.
The third bill is subtler and it is a lesson about false silence. Because Postgres also runs rootless in a container on this box, there is no system user called postgres. There has not been for a long time. Which means the reflex command that every server administrator in the world types, the one that runs a database query as the postgres user, does not fail loudly here. It writes an error onto a channel you are probably not reading, and it returns an empty result.
An empty result reads exactly like the answer to a real question. It reads like the table does not exist. It reads like the data is not there. During the migration, that almost produced the wrong conclusion about whether data had survived a move, which is the single worst moment to receive a confident false negative. The rule that came out of it is one you will hear again in this series in a dozen costumes. An absence has to be proven. It can never be inferred from a command that went quiet.
What Rootless Does Not Buy You
One last honesty check, because it would be easy to walk away from this episode believing the box is safe now.
Per domain service accounts close exactly one road to root, the one that runs through the container runtime. There are others. There is a hosting provider console that can hand out access to the machine. There is a metadata service inside the cloud network that will, if you ask it nicely from the right place, describe how to get in. There are bugs in the kernel itself, which every container on the box shares by definition. Postgres and the web server sit close to root by their nature.
Rootless is a wall, and it is a real one, but it is one wall. Every container on this box additionally runs with new privileges disabled and with essentially all of its special powers stripped away at start, with a handful of deliberate, documented exceptions where a piece of third party software would not run otherwise.
That layering, wall behind wall behind wall, each one honest about what it does not cover, is the design signature of this entire machine. Once you see it in the container layer, you will see it in the secrets, in the backups, in the alerting, and in the disaster recovery.
Next time, the strangest rule on the box. You are not allowed to edit it.