Eighteen Days Of Silence: Two Services On One Machine

The Easiest Call In The World

Your mining maps service takes an invoice. When it does, it is supposed to send you a note, into your own inbox tool, so you know money happened. Both of those programs run on the same computer, roughly 2 centimetres apart in the same rack.

That notification would have failed every single time, and nobody found out for 18 days.

Not because it was intermittent. Because in those 18 days, not one invoice was submitted. The code path simply never ran. It sat there, broken from the day it went live, waiting for a customer.

What finally exercised it was a spam bot. A junk request hit the handler, the notification attempt failed with a connection error, the service reported itself degraded, and the monitoring fired correctly. Which is a genuinely funny outcome and also a slightly horrifying one, because the alternative was a real order arriving first, being quietly saved to the database, and never once surfacing to you.

A failure that only appears when a customer arrives is not a failure you can rely on finding.

Why The Obvious Thing Does Not Work

There are 2 obvious ways for one program to call another on the same machine, and on this box both of them fail. Understanding why is the entire episode.

The first obvious route is the loopback address. The universal shorthand for this computer, right here. Every service on this box publishes its port there, and from a shell on the machine it works perfectly.

From inside a container it does not, and the reason goes back to episode 1. A rootless container has its own network namespace. Its loopback is its own loopback. When code inside a container asks for this computer, right here, the answer it gets is the container, not the machine. The host's port is simply not present in that world.

And here is the part that surprised everyone, including the documentation. This is true even between 2 services owned by the same Linux user. There is no same user exemption. It was measured directly. From the host, the inbox tool's port answers correctly. From inside 2 different containers belonging to the same trust domain as the inbox tool, the identical address produces a connection error.

So the second obvious route is to just use the public web address. Go out to the internet and come back, like anybody else would.

That fails too, and it fails in a more interesting way. The networking helper that gives these containers their connectivity copies the host machine's public address into the container. Which means when your code resolves the public name and gets the box's public address, the packet does not leave. It turns around and arrives back inside your own container, which is not serving that website. This is called hairpinning, and the result is another connection error.

Two obvious routes, 2 connection errors, and the error message is the same one you would get from a service that was down.

The Route That Was There All Along

There is a correct route, and it has existed since early July 2026.

You add one line to the service's entry in the deploy catalog, saying that a particular hostname should resolve to a particular fixed address. That address is the default the networking helper on this box uses to mean the host machine, reachable from inside a container, and pointed at the host's real network rather than its loopback. It is a property of this setup rather than a universal container address, which is one reason nobody found it by guessing.

And on the host's real network, Caddy is listening on every address, ready to serve. So the call goes out of your container, to the host, into the proxy, which presents a genuine certificate for the name you asked for, and routes it down to the target service exactly as it would for a visitor in Stockholm.

The doctrine that came out of it is 4 words. Service to service is external. Always. Same machine, same user, does not matter. You use the public address, the real certificate, and your own authentication token, exactly like any client on the internet.

The one accommodation is that single pinned line, and there is a crucial thing about it. It is routing metadata, not permission. It tells your container where to send the packet. It grants nothing. Any container could reach that address anyway. What actually controls access is the token in your request, which is why every consumer gets its own dedicated, individually revocable token rather than sharing one.

The Documentation Was Wrong Twice

This is the part I would keep if I could only keep one thing.

The route existed and was in production, crossing between different trust domains, on several services at once. Then a session went looking for it, probed an address that was off by one from the correct one, found nothing, and concluded that no such route existed. That conclusion was written into the canonical documentation, stating that only the database ever crosses trust domains.

The pattern the document said did not exist had been running in production for 13 days at that moment.

Then it got worse in the opposite direction. A later version of the same document claimed there was an exemption for services under the same user, and cited a specific service as living proof that it just worked.

Both halves were false. The loopback does not work from any container. And the service cited as proof was, at that exact moment, the single silently broken lane on the entire box. Its notifications had never once been delivered, from the day it replaced a daily text message. It was being used as evidence that everything was fine while being the only thing that was not.

The rule written afterwards is short. Before recording that something does not exist on this box, go and search the code that generates the configuration for the mechanism. A negative claim in documentation is a claim, and it needs measurement exactly as much as a positive one does. Arguably more, because a negative gets quoted for years without anyone ever having a reason to retest it.

How To Recognise It In Ninety Seconds

The diagnostic is clean, and worth knowing because this will happen again.

A connection refused, on the box, against one of your own public addresses, is a missing pin. It is almost never anything else.

It is not a bad token, because a missing or wrong token produces an authentication failure, not a refused connection. In fact a refused connection is proof that your code ran and tried, which is more than a silent failure gives you.

It is not the target being down, and you can check that in 5 seconds from your laptop, where it will be perfectly fine. That asymmetry is the signature. The public address works from everywhere on earth except from inside the box that hosts it, which is why the problem never appears in testing and always appears after deployment.

There is even a trap in the act of testing this. If you go probing that host address directly, you have to set the name at the encryption layer, not just in the request header. Without it, the proxy answers with a low level cryptographic error, which reads convincingly as this address does not serve that site. A measurement was recorded that way once, and it was wrong.

The Guard, And Why It Reads A File Nobody Talks About

Once you have a rule this important, you want something checking it, because the pins went missing on 3 separate services in 5 days.

And the way they went missing is the reason this box now has a doctrine about not trusting tools.

The deploy reported success and correctly wrote the new pin into the configuration file. The running container had no pin at all, and had been running for 13 days. Then a restart was issued, and the restart reported failure, while the underlying retry had actually succeeded and the service was fine.

Success meaning nothing happened. Failure meaning it worked. That is the same broker behaviour from episode 3, in the wild, on the same afternoon.

So the guard that checks pins does not ask the deploy tool anything, and it does not read the configuration file either, because the configuration file was correct while the running container was not. It reads the actual hosts file that podman generates inside a container, which is written at the moment the container is created, from what it was genuinely created with. That is the single artifact that distinguishes the configuration was updated from the container was restarted onto it.

The first version of that guard used a much more natural approach, running a lookup command inside each container. It was abandoned at the first live run, because 2 of the containers on this box are built with nothing in them at all. No shell, no tools, just the application. Which is excellent security practice and means you cannot run anything inside them to ask a question. The guard would have left 2 perfectly correct pins permanently unprovable, and therefore been permanently red.

There are 4 more things on this box that a naive version of that check gets wrong. Backup copies of configuration files that nothing reads, which look like pins and are dead. A commented out line, likewise. A legal alternative form of the setting that looks like a mistake and is not. And pins declared once at the group level for a set of containers, which are entirely live, but do not appear in each member's own configuration, so checking members individually finds nothing while the pin is working perfectly.

And the last subtlety is the best one. Six different services declare a pin to the same hostname. If the check simply gathered all the pins into one pile and confirmed the hostname was present, then 5 healthy services would happily vouch for the sixth one that had lost it. So the check is keyed on each individual container, not on the set of pins. A collection cannot prove that nothing is missing from it, which is a sentence worth carrying into every audit you ever design.

Next time, the database. One socket, no network port, and the upgrade that silently gives you a brand new empty server.