Killed 1.4 Seconds From The Finish: Certificates And The Front Door

One Door For Forty Addresses

Popcorn2 serves something like 40 different web addresses. Newspapers, maps, a photo booth, a podcast feed, a shop placeholder, a handful of tools that only you use.

There is exactly one program on that machine listening for web traffic from the internet. Every request in the world arrives at it, and it decides, by looking at which name the visitor asked for, which internal service should answer. The services themselves are listening too, but only on the machine's internal address, where nobody outside can reach them.

That program is called Caddy, and this episode is about what it does, and about 5 separate ways the front door has bitten, including one where a safety check destroyed the very thing it was checking for, with 1.4 seconds left on the clock.

Why There Is Only One

The arrangement is called a reverse proxy, and there are 3 reasons it exists rather than each service simply answering the internet itself.

The first is mechanical. Two programs cannot both claim the same port on the same address. There are tricks to share one deliberately, but as a rule there is one port 443 for secure web traffic on the public address, and 40 services cannot each have it.

The second is that certificates are a chore, and doing a chore 40 times is 40 chances to forget. Concentrating them in one place means one program is responsible for all of them and knows about all of them.

The third is the important one. Because a proxy stands in front, the actual services do not need to be reachable from the internet at all. Every service on this box publishes its port on the machine's own loopback address, which is the network equivalent of a door that only opens inward. Nothing outside can dial them directly. The only path in is through the proxy, which means there is exactly one place where untrusted traffic enters, one place where certificates are handled, and one place to reason about ingress. The applications behind it still handle real requests from real strangers, so they still need to be careful. What has been removed is 40 separate front doors.

Certificates Without A Chore

Caddy's headline feature is that it obtains and renews certificates by itself, with no scheduled job and no reminder in a calendar.

The mechanism underneath is a protocol where the certificate authority, in this case Let's Encrypt, needs proof that you actually control the domain before it will vouch for you. So it sets a small challenge. There are a few forms it can take, and the one usually in play here is that you place a specific value at a specific location on the plain unencrypted port, and then it comes and looks. If the value is there, control is proven, and a certificate is issued that is good for about 90 days.

Ninety days is short on purpose. It forces automation, because no human will reliably do a thing 4 times a year for 40 domains. Caddy renews well before expiry, quietly, forever. There is no certbot on this box and no renewal cron job, because there is nothing to schedule.

Adding a new address is one line in the deploy catalog, which the broker turns into a proxy configuration. And that is where the trouble starts, because certificate issuance takes time, and time is the thing automation is worst at waiting for.

The Rollback That Caused The Failure It Detected

A new domain was being added. Sensible engineering instinct, entirely correct in spirit. Add the address, then verify it works, and if it does not, roll back automatically so a broken change never lingers.

The verification slept 3 seconds, then checked whether the new address answered securely.

It did not, because obtaining a certificate takes somewhere between 5 and 60 seconds. So the check saw a failure, concluded the change was bad, and rolled back, which meant rewriting the configuration and telling Caddy to reload it.

That reload cancelled the certificate order that was in progress.

The log tells the story with painful clarity. First, a line saying the authorisation had been finalised and was valid, meaning the hard part was over and the certificate authority had agreed. Then, moments later, a line saying the certificate chain download had been cancelled.

It was killed 1.4 seconds from completion, by a 3 second sleep, by a safety check. And the check then reported the exact absence it had just created, which pointed the investigation straight at the configuration, which had been correct the entire time.

The Shape That Works

The corrected pattern is 3 rules and they generalise well beyond certificates.

Check the existing thing first. If adding something broke what was already working, that is a genuine emergency and rolling back immediately is right. That is the only condition that justifies an automatic rollback.

For the new thing, poll instead of sleeping. Ask repeatedly for up to a few minutes rather than waiting a fixed guess and asking once. A fixed sleep is a bet about how long something takes, and you will lose that bet on the worst possible day.

And if it is still not up when you run out of patience, leave it alone and say so. Caddy retries on its own schedule and converges by itself. Rolling back does not fix a slow start, it guarantees a failure. A new address was serving nothing at all a minute ago, so slow is not the same as broken.

There is a related rule about reloads generally. Batch every change into a single reload rather than reloading after each one. A storm of reloads cancels in flight certificate work and can burn through the authority's limit on failed attempts per hour, at which point the system falls back to a different provider and you now have 2 problems.

And a final one, learned in the same incident. If you roll back the proxy configuration, you must also roll back the registry entry, or you have created exactly the drift the previous episode was about, declaring an address that nothing serves. Revert both, or neither.

The Log File That Was Lying About Its Own Name

This one is my favourite in the whole series, because nothing was broken and everything was wrong.

There is a catch all block in the proxy configuration whose job is to handle requests for names the box does not recognise. Somebody knocks asking for a domain that is not ours, we answer with a not found, and we log it, because who is knocking is genuinely interesting information.

That log fed a panel in the statistics tool titled who is knocking, listing the mystery hosts.

The panel was reporting roughly 20 times too many, and it was listing your own domains as bogus hosts. Your newspaper's live site. Your personal homepage. All of them, flagged as suspicious unknown traffic, while being served perfectly to real visitors the entire time.

Here is why. The catch all block has no hostname attached to it, because its entire purpose is to catch things that match no hostname. And Caddy attaches a log setting to the hostnames of the block it appears in. With no hostname to attach to, it quietly became the default logger for the entire port.

Which meant every one of your legitimate domains, redirecting visitors from the insecure port to the secure one, wrote its redirect into the catch all log too. Over 7 days that was 44465 redirects against 2258 genuine unknown host errors. The file named for unmatched traffic contained all traffic.

The comment in the script that created it reasoned correctly about how routing precedence works, and then silently assumed that logging follows routing. Those are 2 different layers of the same program.

The rule that came out of it is one line and it is the best diagnostic advice in this entire series. Before you trust anything derived from a log file, deliberately produce a case that should be excluded from it, and then go and look for it in the file. A filename is a claim about what a log contains. It is never evidence.

Fourteen Seconds

One more, short, and slightly chilling.

A new domain was set up, published nowhere, linked from nothing, mentioned to nobody. Fourteen seconds after its certificate was issued, it received its first unsolicited visit, from 2 unrelated addresses pretending to be a desktop browser.

The reason is that certificate transparency is public by design. Publicly trusted certificates get submitted to public append only logs, so that a fraudulently issued certificate for your bank cannot hide. That transparency is a genuine security win and it is also a real time feed of hostnames, which people scan continuously. Anything named in a logged certificate is visible within seconds of issuance.

So obscurity is not available. The moment a name has a certificate, it is known. Not linked, not indexed, but known, and being probed within seconds. Anything you put behind a name is protected by its authentication or it is not protected.

The Domain With A Swedish Letter

Last one, because it is a lovely example of the tools being wrong rather than the system.

There is an alias domain containing an ä. Domain names with non English letters are encoded into a plain ASCII form behind the scenes, an ugly string beginning with x n dash dash, and that encoded form is what the network actually uses.

Two things came out of setting it up. The first is that you should never use a registrar's built in domain forwarding for this. Theirs answers with a temporary redirect, which tells search engines to keep indexing the alias, the opposite of what an alias is usually for, and it discards the path entirely, so a link to a specific page lands on the homepage. A redirect configured in Caddy is permanent, keeps the path and the query, and can carry its own log so the question of whether anyone uses the alias is answerable.

The second is that the command line tool everyone reaches for to test a website, on this Mac, has no support for international domain names at all. It fails on the ä form with an error that reads exactly like a broken certificate. Which means the obvious test says the thing is broken while every real browser loads it perfectly.

The technique for that is to use the encoded form as a control. If the ugly ASCII version works and the pretty version fails, the variable is your client, not your server. And for anything user facing, verify in an actual browser, because that is what your visitors are using.

Next time, the thing everyone gets wrong exactly once. What happens when 2 services on the same machine try to talk to each other.