A Database With No Address: Postgres On Popcorn2

Try To Attack It

Here is a challenge. You are on the internet, and you want to attack the database on popcorn2. It holds the newspaper, the client work, the podcast pipeline, the location history, and most of the rest of PärKit, though plenty of smaller things on the box keep their data elsewhere.

You cannot get to it. Not because the password is strong. Because there is nothing to connect to. From the public internet there is no port anywhere that will accept a database connection to that machine. Postgres itself is not listening on the network at all. There is exactly one exception, over a private encrypted tunnel, and we will get to it.

This episode is about how that works, what it costs, and the upgrade path that will silently hand you a brand new empty database and report success.

The Front Door Is A File

Databases usually listen on a network port. That is how the world talks to them, and it is why database ports are among the most scanned things on the internet.

Postgres also supports a second way in, which is older and gets far less attention. A Unix domain socket. It looks like a file on disk, and 2 programs on the same machine can talk through it. It never touches the network stack at all. Postgres actually creates one of these by default on any Unix system, so this is not exotic. What is unusual is having nothing else.

On popcorn2 that socket is the only door. The setting that tells Postgres which network addresses to listen on is deliberately empty, so it listens on none of them. What exists instead is a socket file, sitting in a directory on the host.

The consequence is worth savouring. Access to that database is now a filesystem permission question rather than a network question. To connect, you must first be able to see that file. Which means the entire population of the internet is excluded, not by a rule that could be misconfigured, but by the absence of anywhere to knock.

And Postgres itself runs the same way everything else does on this box, as a rootless container under its own Linux user. It is version 18, with the vector extension available for the AI features that need it.

How A Service Gets In

If the door is a file, then letting a service through is a matter of showing it the file.

That is one line in the deploy catalog. A service says it needs the database, and the broker mounts the directory holding that socket into the container. There is a small detail here that matters and turns up more than once on this box. It mounts the directory, not the socket. Sockets get recreated when the database restarts, and a mount of the file itself would go stale and point at something that no longer exists.

That mount grants connectivity, and only connectivity. It is not authentication. The service still needs a username and password in its connection string, exactly as it would over a network. What has been removed is the ability of anybody outside the box to even attempt it.

One Database For A Whole Suite, On Purpose

There are 6 Postgres databases on the box. One for the newspaper's live feed, one for the client tool, one for the podcast pipeline, one for the party service, one for the newspaper's advertising deadlines, and one shared one for the entire PärKit suite.

That last one deserves defending, because it looks exactly like a mistake. A dozen different tools, most of them connecting with the same credentials, most of them able to read each other's tables. Not all of them, as it happens. The natural language query tool and the connector for outside AI clients both use their own deliberately narrowed roles, because both are places where something outside the suite gets to influence what gets asked.

It is deliberate, and the reasoning is that PärKit is not 11 applications. It is one organism that happens to be deployed as 11 processes. Capture and Time and Director and Stats are not independent products with a business relationship. They are limbs. A limb reading another limb's tables is not a boundary violation, it is the point of having a shared body.

So the boundary is placed around the suite rather than between its parts. And the rule that goes with it is absolute. Services that are not part of the suite, client work, experiments, anything with a customer attached, never receive that credential.

That was actually measured rather than assumed, which I appreciate. Using the suite's credential, an attempt to read into every neighbouring application's database was denied uniformly. Zero tables readable, everywhere. The perimeter was found to be somewhat ad hoc, assembled from defaults rather than designed, with only one database having had its public access explicitly revoked, but there was no path from that credential to escalating into anything else.

Guard By Sensitivity, Not By Location

The policy on top of that is one line and it is the right line. Open by default, guarded by sensitivity, never by zone.

The default across the whole estate is that data is readable across services. That is encouraged rather than tolerated, because the interesting things this estate does come from joining data that would normally sit in separate silos.

Then a small enclave gets a genuinely hard boundary. Not a revoked grant, which is one setting somebody can toggle back, but a separate credential that does not exist in the shared pool at all. There is still plenty that could be configured wrongly, sockets and roles and secret distribution, but reaching that data is no longer one flipped permission away. Alongside it, a deliberately narrow read path, a role that can see a specific short list of tables and nothing else.

The membership list of that enclave has a detail I want to point out, because it is a small piece of intellectual honesty that most documentation lacks. It is split into 2 groups. The ones you have explicitly named as sensitive, the client tool and the ecommerce data. And a second group marked clearly as inferred by an AI and not confirmed by you, which includes the health tracking, the payroll data, and the bookings.

The instruction attached is that no expensive hard boundary gets built for an unconfirmed member without your word. Which means the document knows the difference between something it observed and something it guessed, and says so in the text. That distinction is one of your own standing complaints about how AI reports things, and it is nice to see it enforced in the artifact itself.

The One Exception, Over A Private Tunnel

There is exactly one way that database is reachable from off the box, and it is the only service on the machine published onto anything other than loopback.

Your sister's server, a separate machine at the same provider, runs an application that needs this database. So there is a WireGuard tunnel between the machines. WireGuard is an encrypted private network where every peer is identified by a cryptographic key, and each key is bound to the specific tunnel addresses it is allowed to use. Traffic from anything without a recognised key is not refused so much as ignored entirely. Inside that tunnel, and only inside it, the database is published on a private address.

Three peers are in that network. Popcorn2, the router at your house, and your sister's box. That published database address is the only service on the whole machine reachable on anything other than loopback, and the documentation is emphatic that it should stay the only one.

Every exception to an architectural rule makes the next exception easier to argue for, and the rule here is genuinely doing security work.

The Upgrade That Gives You Nothing

Now the trap, and it is a good one because everything about it looks successful.

On this box, because of how the database image is arranged, the data lives in a directory named after the major version. Version 18's files sit in a path with 18 in it.

Suppose you upgrade the way you upgrade everything else in a container world, by changing the version label on the image and restarting. For a minor update, 18 point 4 to 18 point 5, that is the right shape and it is the normal thing to do, release notes permitting.

Do it across a major version and here is what happens. The new database starts. It looks in the directory for its own version, which is a different directory. It finds nothing there. So it does what a database does when it finds no data, which is to initialise a brand new empty one. It starts up, healthy, reports success, and serves an empty database.

Your old data is not deleted. It is sitting in the neighbouring directory, untouched, ignored, with no error anywhere to indicate that anything was skipped. The service is up. The monitoring is green. The tables are gone.

A major upgrade has to be deliberate. Either export everything, create the new database, and load it back in, or run the dedicated upgrade tool that migrates a cluster in place. Both are supported and either is fine. What is not fine is changing a label and restarting, because the failure mode there is not a crash. It is a fresh start.

Two Small Structural Facts

Two closing details that explain why the services on this box behave the way they do.

The first is that there is no dependency ordering between services here. Normally a service can declare that it must start after the database. On this box every service runs under a different Linux user, each with its own independent service manager, and those managers cannot express ordering across the boundary between them.

So the database might not be up when an application starts. Which means every application is expected to tolerate that and retry with backoff until the database appears. That sounds like a limitation and it is quietly a strength, because it pushes the whole fleet towards surviving the database going away and coming back, which is a property most estates only discover they lack during an incident. Whether every single service on the box genuinely handles it is a different question, and not one anybody has proven.

The second is a small piece of housekeeping wisdom. There is no shared Redis on this box. There used to be an entry for one in the documentation, and services listed it as a dependency, and during the migration it was carefully accounted for twice before somebody actually checked and found that nothing had ever used it. It was a phantom, propagated by documents citing documents. The photo booth does run its own Redis inside its own group of containers, which is a completely different thing and is genuinely used.

The rule that came out of it is one of the shortest in the manual. Verify that declared dependencies are real. Check the code, check the environment file, check what the process actually opens. Do not check the table that lists what things are supposed to need, because that table is where phantoms live.

Next time, telemetry. What every service on this box reports about itself, where it goes, and the query that took the whole machine down.