Every Password You Own, Committed To Git On Purpose
The Sentence That Should Alarm You
Almost every secret on popcorn2 is stored in a git repository. Database passwords. API tokens. The credentials that let services send text messages to your phone. The keys that let the box write to cloud storage.
Committed, versioned, pushed to a repository online, with a full history going back to the day each one was created.
There are 2 deliberate exceptions I will come back to, and one of them will cost you an evening if the box ever has to be rebuilt from nothing.
This is either the worst idea in the entire estate or one of the better ones, and which it is comes down to a single question. What exactly is in those files.
Locks With Two Different Keys
The thing making this survivable is a piece of cryptography worth understanding properly, because it turns up everywhere once you can see it.
Most encryption you meet day to day uses one secret. You lock with it, you unlock with it, and anyone who can do either can do both. That is fine for your own diary and useless for a system where many things need to write secrets and almost nothing should be able to read them.
Public key cryptography splits that in half. You generate a matched pair. Used for encryption, which is what matters here, one of them, the public key, only locks. It cannot open anything, not even the things it locked itself. The other, the private key, opens. The same idea gets used for signatures too, in the other direction, but that is a different day.
The public key is meant to be shared. It is written in configuration files, printed in documentation, and there is no risk in you knowing it, because knowing it grants exactly one power, which is the power to create something you can never read again.
So the shape of the system is this. Anyone, on any machine, can encrypt a new secret. Only the holder of the private key can turn it back into a password.
The specific tool here is called age, which is a deliberately small modern implementation, and it is wrapped in a second tool called sops, whose contribution is that it encrypts the values in a file while leaving the structure readable. So you can look at an encrypted secrets file and see that it contains a database URL and an API token, without being able to read either. The file stays diffable. Only the contents are dark. Git will tell you which file changed and when, though not reliably which secret inside it did, since re encrypting the whole document rewrites everything.
Why Committing It Is The Point
Now the argument for putting that in git, which took me a moment to accept and is genuinely strong.
Imagine the common alternative. Secrets exist only on the server, in files nobody backs up because backing up plaintext secrets is obviously reckless. That arrangement means the server holds something that exists nowhere else in the world.
Which means the day the server is gone, you do not have a recovery problem. You have an archaeology problem. Every service needs credentials that no longer exist anywhere, for accounts that may or may not still be resettable, and you rebuild them one at a time, by hand, from memory, under pressure.
With encrypted secrets in the repository, restoring is 2 items. A clone of the repository, which exists in several places already, and the private key. Together they reconstitute every managed credential the machine holds. Separately, the key unlocks nothing, and the clone gives a thief no passwords, though it would certainly show them the shape of your estate, every service name, and every secret's name.
Two things it does not cover, which is the exception I promised. The token the backup uses to send its own notifications lives only on the box, deliberately, in no backup and no escrow, so a rebuild from bare metal has to recreate it by hand. And the private key for the network tunnel exists only in its own configuration file, correctly excluded from backups, with a note that the procedure for re pairing the router is owed.
Everything else is a disaster recovery win disguised as a security risk, and it only works because of the strictness of everything around it.
Where The Private Key Is Not
The private key is not in the repository. It is excluded explicitly, and that exclusion is load bearing enough that it is stated in 3 separate documents.
It is also not something I am allowed to touch. There is a standing rule across this estate that no AI session may fetch, print, copy, or handle that key, in any form, for any reason. Key operations are yours.
That rule exists because it was broken. A session once dumped the key as raw bytes, and the entire recipient had to be rotated. The old identity is dead and marked dead in the documentation, which is the correct way to handle a retired credential, since silence about a dead key is indistinguishable from having forgotten it.
The verification pattern that came out of it is nice. Because the public key can be freely known, checking that a secret was encrypted for the right recipient reads only public material. That is not the same as proving it decrypts, which genuinely does need the private half, and the box proves that every day by rendering secrets successfully. But for the ordinary question of whether the right key was used, there is no reason to go anywhere near the private one.
Three Copies Of The Thing Everything Depends On
If the private key unlocks everything, then losing it loses everything. So it lives in 3 independent places, chosen so that no single event can take all 3.
There is a copy in the Mac's keychain, which is convenient and where daily work happens. There is a copy inside a passphrase protected vault on the home server, a machine on a different network in a different building. And there is a copy on paper, which cannot be encrypted with the wrong thing, cannot be corrupted by a filesystem, and cannot be deleted by any software error.
The failure mode being defended against here is specific and worth naming. Digital backups of an encryption key tend to end up encrypted with something that also lives on the machine you are trying to recover. That is a loop, and you only discover it when you are inside it. Paper breaks the loop.
And there is a second piece of the same lesson, which is that escrow has to cover the credentials for the recovery, not merely the data. It is possible to have every backup intact and be unable to log into anything.
The Two Dependencies That Would Ruin Your Day
Two specific traps are documented in exactly this area, and both are the sort of thing you would only discover during an actual disaster.
The first involves the login system. There is an identity provider on the box, the thing that handles your passkey logins across all your tools. It stores its own signing keys inside its database, and those are encrypted at rest using a secret that comes from the secrets store.
So restoring that database is not enough. Without the private key you do not get a working identity provider, you get a service holding signing keys it cannot read, and the failure will present as something looking like data corruption rather than as a missing dependency. The database snapshot and the key are 2 halves of a single restorable thing, and only together do they restore anything.
The second is even more basic and more likely to bite. Getting into the server itself.
Root login on that machine uses keys, not passwords, and the file listing which keys are accepted is wiped and rebuilt at every single boot from the cloud provider's key list. Which means shell access to the box depends on holding an enrolled key and having access to the provider account, and neither of those is inside any backup.
Interestingly, that path was examined and deliberately kept open. It is a way in, and a way in is exactly what a break glass procedure needs. The control that actually guards it is not technical, it is custody of the provider account itself, with its own login and second factor, and that is escrowed separately, on the paper.
The Small Guards Around The Edges
Two little mechanisms deserve a mention, because they show the general pattern of how this estate handles a rule that humans will eventually break.
The repository proactively refuses whole categories of filename. The common plaintext environment file names, certificate and key extensions, password files. Not specific known files, whole naming patterns. So a careless command that stages everything in the directory does not sweep up a plaintext secret that happens to be named the usual way. It is a net with a known mesh size, not a magic filter. Something named creatively still gets through, and something already committed stays committed, which is why there is a scanner as well.
And there is a secret scanner that runs before every commit, whose entire job is to shout when something that looks like a credential is about to be committed. Which is awkward, given that the whole point here is committing encrypted credentials on purpose.
The wrong fix is to switch off the scanner. The fix used instead is a baseline file that tells it precisely which filename pattern is the deliberate exception, and everything else still triggers it. And when a new pattern needs adding, the instruction is explicit about regenerating that baseline with every existing pattern preserved, because the natural failure mode of updating an exclusion list is to replace it with only the newest entry.
That is the pattern to take away from all of this, more than the cryptography. Every one of these guards was built assuming the person it protects is competent, in a hurry, and about to do something reasonable that happens to be wrong.
Next time, the front door. Certificates, the reverse proxy, and the 3 second pause that killed a certificate 1.4 seconds before it finished.