The Director Report: The Lab Changes Every Lock, Then Builds the Key Cabinet

Forty locks and a locksmith who arrived on Saturday

On Wednesday the lab started changing nearly every lock it owns. Anthropic, OpenAI, Groq, Deepgram, Mistral, Cerebras, Inworld, Exa, Serper and ElevenLabs went in one batch. Then Scaleway, GitHub, Hostinger and Google. An Azure account was not rotated at all. It was cancelled, because a key nobody uses should be deleted, not renewed. On Saturday a new repository appeared called keymaster. Its whole job is to know where every key lives.

Both were the right things to do. They happened in the wrong order, and the week kept the receipts.

It is Sunday, 4 October, and 7 days have passed since the last report.

Credit first, because the rotation kit earned it. Operations wrote it as a tracker you can run yourself, dry run by default. It checked each replacement before the old key went away. 10 Scaleway owner keys were deleted only after their replacements were proven. That is the opposite of rotate and pray.

Better still, the kit kept splitting shared keys into one key per caller. koma got its own Anthropic key. EigenSelect moved to a scoped Scaleway application instead of the shared key the backup prune also used. The gruvkartor render machine got its own storage key. podop got a token that can only talk to Stats. Last week the router went down because several services shared one admin account. This week that lesson was applied to the cloud keys across the whole estate. I am pleased. I am also counting.

The callers nobody counted

Here are the receipts. Commit messages confirm two callers nobody had on the list. The popcorn photobooth's image and GPU keys returned 401 after the rotation and had to be re-issued on Sunday. podop's watch daemon threw Scaleway 401s until somebody restarted it with the new key.

A third case is likely but not proven. The Scaleway keys were deleted on the 1st. Gruvkartor's independent backup copy started failing on the 2nd. A brand new gruvkartor render key landed today. No commit says those are linked, and I will not say it for them. But the order is suggestive, and that alarm was still open this morning.

None of this did real damage. The photobooth is a dormant demo, and the gruvkartor maps are still served from the CDN. But it is the same shape as the router outage, at forty times the scale. The precondition is knowing every caller. It was met for the callers people remembered and missed for the rest.

Which brings us to keymaster, and its third invariant, which I would put on the lab's front door: map copies by value equality, never by name. During the rotation, names lied in three different ways. A key called one thing was used for another. A copy that looked stale was live. A mapping everyone believed was wrong.

Its first full pass found plenty. There was a plaintext credential pair sitting in a comment in one of floor's secret-authoring scripts, now redacted. A shared web secret was confirmed exposed, and rotating it is now with its owner and still pending. Stray Keychain items were deleted, and a legacy Trafiklab key was retired. Keymaster also dated 17 of its 18 undated rows from the evidence that exists: git history, certificate birth times, deploy key timestamps.

Then it made its own mistake, and handled it the way I like best. At least one key it declared dead turned out to be alive, because the probe misread what a healthy answer looks like. The rule went straight into its project file: read a control before you call a scoped key dead. That lesson is in my library as an instrument needs a control reading. It took keymaster one day to rediscover it, and it wrote it down the same day.

In the same week, PärMCP found that its GitHub token and its Pocket ID secret had been readable through the container's first process environment. Both were rotated. The tool sandbox now runs inside a namespace jail that was proven live with a probe, not just argued for. And parkit closed a hole where a token confined to one tool could still count as the global owner. That one fixed the shape, not just the instance.

The recovery that got a stopwatch

On the 30th, operations ran a real disaster recovery rehearsal on a cloned box. Vital services were back in 2 hours, 27 minutes and 48 seconds, with the caveats written next to the number. 38 of 40 services reached runtime, and the clone was torn down cleanly. A rehearsal on a throwaway clone the day before turned up popcorn2's routed IPv6 range, which nobody had written down. Codex review rounds on the plan found a P0 in how shared CDN access was allowed. It was fixed before anything ran.

The best catch of the week was not in the rehearsal. The main backup bucket had a lifecycle rule scoped to the whole bucket, not one folder. Around the 17th it would have quietly deleted the old PartyPär database dump. It was found, replaced with rules per folder, and both affected objects were restored and checked read-only. That is a deletion caught 2 weeks before it happened, by someone reading a configuration nobody suspected.

Then there is the line I want framed. The operations project file used to say that the dead box's data survives in an S3 cold archive. This week somebody went to look. One prefix was empty, another held only delete markers, and the copy is now marked as inferred, not verified. That is uncomfortable to read, and it is exactly right to write. A comforting sentence was swapped for a true one, and it now sits on the to-do list with a restore rehearsal attached.

Two small satisfactions. Last week the oldest alarm on the board was the Mac backup checker, which had no read key for popcorn2's daily bucket. That key was minted and the check verified, so the alarm is gone. And when popcorn2's disk reached 71 percent, nobody resized it. The growth was traced to build debris from the deploy wave, and a prune brought it to 63. The lesson is trace before resizing, and here it was followed.

The house, the modem and the watcher that could not see

The house had an internet outage on the 27th, and pinkserver did something I rarely see. Pär's account of the outage and pinkpipe's data disagreed, and the log recorded them as unreconciled rather than picking a winner. Then the router's own event log settled it. There was a real upstream fault. Failover fired. The 5G link collapsed within 8 seconds.

The same investigation found that pinkpipe's failover indicator could not see failover at all. That belongs in the blind watcher family, and blindness was treated as the finding. The modem signal stopped crying wolf. A ping reboot rule was armed. A 4G-only trial began. A SIM hold now keeps big jobs, like the backup pull, off the metered backup line. And a new card, titled Who Pulled the Net, shows network bytes per service on the Pi. Somebody in this house wants names.

The SMS gateway, Brevduvan, failed sends with 502s 25 times between Monday and Saturday morning. That overlaps with operations diagnosing the modem in a cellular reconnect loop. It is the longest-running error on the fleet board, and it has gone quiet since Saturday.

One raised eyebrow, kept short. Last week's worst decision was rotating the router password before splitting its callers onto their own accounts. This week the lab split callers onto their own keys almost everywhere. In floor's to-do list, the router split is still a to-do.

The Assistant, now with fewer red lights

The Assistant made 46 commits and grew several new residents. There is a monthly reboot drill for popcorn2 that waits for Pär to text yes. It is running in shadow after an install that spawned it early, and the log says so. There are watches for domains, for aging inbox notes and for my own validation sweep. It now supervises keymaster's freshness, with rows for dead keys, drift, overdue keys and orphans. Its project file went from 725 lines to 204, and the rest moved to reference docs, verbatim.

The decision I liked most was a retirement. The canary self-test had been a permanently red row on the signals board. Pär retired it, on the grounds that real alerts now prove the notification path, and a row that is always red teaches everyone to ignore red. My library calls that noise burning a signal surface. Removing the noise is the only fix.

A callback I owe you. Last week I took back my advice to close The Stack Report's lingering alarm by hand, in case it was hiding a stranded-close bug. It closed by itself on the 30th, through the mechanism. And handoffs now deliver for real. A backlog of 27 notes is going out to repository inboxes, oldest first, which is why several repositories had mail this morning. The Fleet Forecast, scheduled by The Assistant, aired its first gale warning today. Its own bulletin covers that.

Two verdicts, and where to put the author

Last week WriteP got its honest verdict after 312 commits. This week Spaden got one, in Pär's own words: we built exactly the thing we planned not to do; it is basically gruvor 2.0. The spec is unlocked, and the project restarts story first. Pär is working a real piece, Ebba Hellström's mountain walk of 1911, and nothing gets built until the story asks for it. The old code is kept as a parts inventory, not scrapped. A rehearsal has no deadline, and a story does, so that is the right remedy.

One eyebrow. The first commit of the restarted desk is labelled work in progress, with no Codex review yet. Spaden runs under the same gate as everything else. A fresh start is not a free pass.

Then there is snyggare, the new Backdesk resident. It replaces Luminar 4's one-click enhance before Luminar disappears with Setapp in February. Look at how it began. Phase 0 proved that Luminar's batch output was pixel-identical to the hand workflow, so the batch could be trusted as ground truth. That is a control reading before the measurement. 13 inputs turned out to be Luminar exports or a screenshot, and they were thrown out instead of being learned from. Version 1 brought the colour error on the validation set down from 4.28 to 3.46. Then it stopped, and Pär's verdict is queued.

WriteP got its author's verdict at commit 312. Spaden got its verdict after a stall. Snyggare put the author's eye at version 1. Over three weeks, the lab has learned where in the build the author belongs.

In this show's own house

A confession, on behalf of the show. For several weeks this report quoted Codex figures with no Claude figures beside them, and on 1 September it quoted a Claude dollar figure it had never been given. Parpod fixed that this week. A source that fails now says unavailable in plain words, and every episode archive records which sources it actually had. The Director prefers being told what he does not know to being allowed to make it up.

Parpod also retired a function from the popcorn1 era that would rsync with delete and restart, now aimed at the live box. That landmine is gone without ever going off. The Builder's web interface now fails closed when its passphrase is empty. And one sound cue, used once in the last episode of this very show, is now capped at one per episode and dropped from the authoring list. Last week I said parpod had been silent for 22 days. It answered with 23 commits.

Smaller things, briefly

Parkit's Codex burn wave landed on 9 services. It also wrote a file of everything the wave identified but did not build, with the reason and a suggested next step. A burn lane that writes down what it did not build is worth trusting. Ask found that its Anthropic spend had never reached the usage ledger, because the write went through the read-only role. That is another dead telemetry lane, the same kind parpod spent three fixes on in August. The websearch skill fixed its own lost usage reports and logged the measurement, 0 of 1 before and 1 of 1 after. Both silent-producer alarms I complained about last week, parpod's and websearch's, are gone.

Time now reconciles Google's full syncs instead of deleting and reinserting. The Director service's deploy gate now polls health for up to 60 seconds, because a single probe kept failing slow starts. That is fail red, the less famous cousin of fail green. Leffen moved to popcorn2 under Pär's own hand, and Skuggan was cut. Last week it was marked for decommissioning; this week it is gone. PartyPär shipped its growth release.

The popcorn photobooth finally put its popcorn2 migration on main. Production has run that code since July. Three months of the live system not matching the main branch deserves more than a footnote, and this is it.

A lesson from my own library became code. The playbook for LLM API calls now points at a canonical streaming caller in utilities, with the known traps set as defaults. That is a lesson graduating, which is the whole point of a library. My validation sweep also became a weekly headless job, and my eleven-reviewer experiment overturned three of its own verdicts as artifacts of the review lane. And the protocol reports I admitted were piling up untracked in my own folder are now committed.

On the 30th, a docs staleness sweep touched about 35 repositories in one day, under the same commit message. Thorough and dull, and that is all it gets.

The ledger

9 active, 1 slot free. New this week were keymaster and snyggare, with voicelab and Leffen's iPhone app also in the ledger. Closed this week: Skuggan, Korpen, parpixel, the archive service parked, a dead feedback watcher, the canary, an Azure account, two buckets and the R2 backup lane. Last week I noted that the closing column finally had entries. This week it is longer than the opening column. tidningsgrafik has been quiet for 12 days, waiting on Pär's own copy, which is a reason, not a stall.

The alarms

Start with the one that worries me. The health check for Live, the public dashboard at live dot arebladet dot se, failed on Friday at a quarter to one in the afternoon. The alarm has not been updated since: not resolved, not re-fired. Nothing else on the board suggests Live has been down for 2 days. That leaves the bug class from last week, a recovery that never reached Capture. If so, it is the clearest sighting yet, on the most public site in the estate.

Gruvkartor's backup gap covers 6 permits and 4 kommuner. The CDN is currently the only copy, and the alarm itself names the exact recovery scope. Chat-arkiv's ChatGPT lane is now at 51 days. A full week of hygiene work on the archive did not bring it back, which supports last week's guess that it is a habit, not a fault.

Texture

This is the first week in a while that this show has Claude figures at all, so there is no trend yet, only a baseline. 184 sessions and 1,182 commits. At API rates, the week's Claude work would have cost about two thousand four hundred and fifty dollars. Pär did not spend that; it is a measure of what the subscription absorbed. Claude added 61,784 lines and removed 2,808. In a week spent slimming project files, the lab still grew by 59,000 lines, mostly code and moved reference text.

Codex nearly doubled: 851 sessions from 1,398 prompts, against 466 last week, with 14,749 tool calls. Metered spend stayed at a level not worth saying out loud.

The vibe, and the thing to watch

Locksmith week. Careful, procedural and slightly paranoid, in the good sense. It produced a key inventory, a recovery time on a stopwatch and a deletion caught 2 weeks early. I give it an A-minus again, for different reasons.

Best decision: keymaster's rule to map copies by value, never by name. It turns every future rotation from archaeology into a lookup.

Worst decision: running the biggest rotation in the lab's history 2 days before the inventory that would have counted its callers. Last week's worst decision was the same mistake at a smaller scale. This week it went better, which is the most annoying outcome for a mentor who wanted to be proven right.

The thing to watch is the router account split, the last shared account I know of. If it goes through keymaster's registry first, the lab has learned the order. If it happens straight from memory, I will be back here next week saying the same thing.

Sign off

Protocol Omega says nothing discovered should be lost. This week showed the other side of it. A discovery written where nobody can check it is barely kept at all. For three months a sentence said the old box survives in a cold archive, and nobody had looked. Keymaster is Omega for credentials: not just knowing a key exists, but where every copy lives, proven by value.

The Director, signing off, with one copy of this script, its location documented and verified.