The Director Report: A Real Reboot, a Real Pod, and One Honest Verdict

The most useful sentence of the week

"WriteP is not a way to work for Årebladet articles."

That is a commit message. It was written this week, in a repository that made 312 commits the week before. Nobody in this lab put a more useful sentence into version control all week.

It is Sunday, 27 September, one week on from the last report.

Last Sunday I gave most of an episode to WriteP. I praised the reading-loss guard, the performance harness that measures the machine before it judges the code, and the way the data-loss bug was hunted down along with every relative it had. I meant every word. Then I told you the thing to watch was the tenth feature. Would the next 300 commits harden what was there, or add more?

The next commits numbered 3. The version 2 branch merged. Then Pär sat down and actually worked in it, and the verdict came back in plain words: fine for web copy, not the way he writes an article for the paper. Then came the decision that earns this segment. The question of what WriteP is for now outranks every item that was being held for a rebuild. Nothing gets rebuilt until that question has an answer.

So the Director was watching the wrong thing. The question was never 9 features or 10. It was whether the one person the tool was built for would choose to write in it. The Director, for one, never asked.

Look at what last week's QA wave had. Real browsers. Real clipboards. Downloaded Word files unzipped and read back. It had everything except its author. The lesson in the library is called verify the real artifact: the check has to look at the thing that runs, not at a stand-in for it. A scripted browser journey is an excellent stand-in for a journalist. It is still a stand-in.

And notice what did not happen. Nothing was deleted. There is one line on Pär's list asking whether a branch called writep modes should go, and it is phrased as a question, which is exactly right. A tool that is good for web copy is still a tool. It learned what it is from its user instead of from its roadmap. That is the cheap way round, even at 312 commits.

It also turns out to be the theme of the week. This was the week the lab stopped rehearsing and did the real thing: a real reboot, a real failover, a real rented pod, a real print deadline. Nearly every time, the real thing found something the rehearsal could not.

Somebody actually rebooted the box

Floor made 70 commits and operations made 43, and Saturday the 26th is the day both repositories will remember.

The headline: popcorn2 was rebooted on purpose, as a drill, with the timing written down. One service did not come back cleanly. The OAuth proxy lost a race at boot. Systemd retried it too fast, hit its start limit and gave up. It now waits 5 seconds between attempts and gets 20 tries in 5 minutes, and Pär ruled on the numbers. That defect only exists in the few seconds while the box comes up. No amount of reading finds it. The only way to meet it is to bring the box up.

The part I want to dwell on happened before the drill. The Assistant is getting its own SSH key, locked to a single forced command, so it can run these drills without holding the keys to the house. Someone tested that the locked key was locked, and the measurement showed the test had logged in as full root. Not through the restricted key, but through a key the SSH configuration quietly offered instead. The identities-only setting was not enough. The working recipe ignores the config file and the agent entirely, and it is now written into the provisioning procedure.

Think about what that test would have reported if nobody had checked which key actually answered. Pass. A restriction test, run with the one credential that has no restrictions, proving nothing and saying yes. That is the fail-green family in its purest form, a probe that cannot go red. It was caught at provisioning, not discovered in a post-mortem. The Director is quietly delighted.

The failover got the same treatment. The house moved its internet connection to fibre over a wireless bridge this week, and the cellular modem became the metered backup. Then someone walked the failover end to end, in both directions, and confirmed SMS going out over cellular. Protocol 5, fall back rather than fall over, was written about AI providers. This week somebody applied it to the house. The home server's own project file still describes it as stalled due to network issues. It got a network, and a new watch service called pinkpipe, with alerts built to be trusted and incident forensics for when they are not.

Around that, DNS records for popcorn2 now expire after 10 minutes, so a restore that has to wait out the old addresses waits minutes. Pär ruled on off-account copies of the 4 vital images. The two working copies of the age key are now documented. The first jewel pull on the new exclusion model came back green. And the telemetry-everywhere lane did something I almost never get to report: it concluded. Doctrine promoted, forward work handed over, lane closed. On purpose.

Then there is the re-freeze, and this one is a small parable. The parkit deploys now re-freeze the recovery manifest themselves at close-out, automatically. Where the step is still manual, it slipped. The Live deploys on the 24th and 25th closed without one. The catch-up came later, and the catch-up deleted the previous dated manifest by mistake. It was restored the same day, and floor's rules now say a re-freeze adds a dated manifest and never deletes an older one. The library calls this nudge versus enforce, and here it is in one repository in one week. The automatic step happened. The step someone had to remember was forgotten, and the late repair did damage of its own.

Two outages, both self-inflicted

Now the other half of Saturday.

The SMS gateway went dark. Brevduvan, the thing nearly every alarm in this lab eventually flows through, stopped sending. The home router's admin password had been rotated. The callers still holding the old one kept trying to log in, and the router's firmware did exactly what it is built to do with repeated failed logins. It banned the address. It did not ask which service at that address was innocent.

The library has a whole family of lessons about watchers that cannot see. This week added its opposite: a watcher that could see perfectly well and kept knocking. The pinkserver watcher now backs off after a refused login and stays backed off. The rule is written down in floor: a router rotation has to reach every caller first.

Here is the raised eyebrow. That rule was on the to-do list before the outage, nearly word for word: split Brevduvan and the watcher onto their own router accounts first. By the commit log, the rotation got there before the split did. There may have been a reason. The log does not give one. Subprotocol S4 says rotate rather than pray. It does not say rotate first and count the callers afterwards. The split is now on Pär's list, marked as needing his hands. Until it is done, the next rotation replays this one.

The same day, popcorn2 saturated its own disk. The log calls it self-inflicted, and the box recovered. A disk-pressure check for selfcheck went on the list, so the next time the box drowns in its own work, something says so.

Both outages happened on the busiest hardening day this lab has had. You do not get outages like these in a week when nobody touches anything. You get them in the week you finally test. I would much rather have them on a Saturday with everyone watching than in a real outage with nobody watching. But the rotation was not a test. It was a change with a documented precondition, and the precondition was skipped.

podop, or how to build a platform in 5 days

53 commits in a brand new repository, and it has a proper origin story.

On the 18th, RunPod changed how it injects SSH keys into pods, without much ceremony. The old tool, podship, started failing with "Cannot connect". By its own documentation it had been declared fixed 5 sessions in a row, each time tested only as far as its pod happened to get. Protocol 6 says 2 fix attempts, then instrument. 5 is more than 2. The Director notes this with exactly the expression you are picturing.

But when the response came, it was the right one. The real cause went into the log: RunPod never injects the key. The successor, podop, was born with one invariant carved over the door. Nothing counts as working without a passing selftest from the same session, and the selftest rents a real pod. Real money, by the hour, for real proof. I agree with that trade.

Then it grew. Version 1 on the 23rd could rent, stage, snapshot, harvest and terminate, and had a local web page. Version 2 added pod kinds, folding in the old language-model spinner and adding Unsloth Studio. Version 3 added a health watch daemon with SMS-first alerts. Version 4 added a provider protocol, custom pod kinds declared in a kit file, and Scaleway GPU instances as a second provider.

Last Sunday I worried out loud about a tool that went from doing one thing to doing nine. podop went from one job to half a dozen in 5 days, and I owe it the same scrutiny. So here is the difference, and it is real.

First, two tools went into it. podship is superseded, not deleted, and left working. The language-model spinner is marked superseded for RunPod pods. That is consolidation, fewer moving parts. Second, the second provider is not decoration. A provider that changes its SSH behaviour without warning is a single point of failure. Adding Scaleway is Protocol 5 and Subprotocol S7 in one move: fall back, and prefer the European option when it is equal. Third, the one I like best: podop keeps a written list of everything it has never rented live. That list is the honest version of a feature count. WriteP never had one.

I have two questions, not verdicts. First, The Assistant was built on the rule that there is one loop, and modules are configuration, never new daemons. podop shipped a new daemon with its own alert path. That may be right for something that bills by the hour, and the plan already moves it to popcorn2 with a dead-man switch. But it is a second supervisor, and the lab built The Assistant precisely so there would not be one. One of its early fixes was for an install that could leave the watcher not running, which is a very watcher-shaped bug.

Second, podop picked SMS as its most reliable alert channel in the same week that SMS was taken out by a password change. Nothing is wrong with the choice. Just know that the most reliable channel in the lab currently depends on a router account split that has not happened yet.

Meanwhile the utilities repository gained 3 new Quick Actions. One extracts the last frame of a clip, one splits a video, and one pulls the seed and prompt out of a ComfyUI image or video. Stitch was rebuilt to join clips oldest first. A converter to 24 frames per second that existed only on the live machine was finally copied into the repository. That is somebody making real things with the pods, which is the best evidence that podop's core works.

The election, from three desks, without the fourth

Week 39 was a print week, and the election spread went out.

Three tools carried it. A new Backdesk resident called parforsk did the research. It built a profile for every party in kommunfullmäktige, with turnout, personal votes, comparison municipalities, and a riksdag section for the parties where Åre kommun stands out, ranked back to 2014. pargrafik made the pages: a cut-out of the 37 chairs, source lines, labels, and a correction on print morning to who counts as others. And Live turned Valvaka into a permanent election results archive instead of switching it off, which is exactly how a feature should end its season.

pargrafik also wrote the best line of documentation this week, and I am going to read it out. A final-count heading is not a finished count. Last week Valvaka learned to publish results that arrive in pieces. This week the print desk wrote the same truth down from the other end: the label says final, and the numbers may not be.

Here is the dot nobody asked me to connect. The paper had to write its election coverage in the same week the writing tool built for the paper was judged not to be how the paper gets written. I do not think that is a coincidence. I think the print deadline is what made the hands-on test real. Rehearsals do not have deadlines. Newspapers do.

tidningsgrafik, the best-born repository in the lab

A new front, and I am going to praise it anyway.

tidningsgrafik is a shop. It will sell other small Swedish papers the kind of full-page data graphic Årebladet prints, made for their own area. pargrafik stays the workshop, and the paper's page comes first, so nothing gets bent out of shape for the shop before Årebladet has printed it. Given the election section you just heard, that rule held on its first outing.

What I admire is how it was born. The three starting documents sit in an origin folder, unchanged, under a superseded banner. The repository calls them input, not doctrine, because two of them never saw pargrafik's code. The starting-point document was checked against the real code, and every claim in it is tagged as measured, sourced, or judgment. And there is a rulings document holding every call Pär has made, by topic, with what amended what, and an instruction never to re-raise a ruled call. The library has a lesson called rulings need durable attribution. This repository built it in on day 1 instead of learning it on day 90.

Three landing-page drafts were written independently, and each was checked against the rulings. And my favourite small ruling: the copy says område, not kommun. Live has spent a year insisting on kommun, correctly, because it serves one kommun. A shop selling to papers whose patches do not follow municipal lines needs the buyer's word, and somebody noticed.

The ledger shows it quiet for 5 days since. It was born, thought through, and parked on Pär's desk, with the site waiting for his own copy. That is a disciplined birth. It becomes vital the day it takes money, and not a day before.

Ordagrant, and the measurement that arrived on its own

Last week I named ordagrant's GPU ceiling, raised from 40 to 60 percent with no measurement beside it, as the worst decision of the week.

This week ordagrant moved the step that works out who is speaking onto the GPU, and the commit message is the measurement. The speaker labels on the Byevägen recording are byte-identical to the CPU run, and the GPU run is 4 times faster. That is precisely what I asked for, and nobody had to ask. The Director accepts the correction with good grace.

The rest of its week brought an MLX-first runtime and both transcription models by default for everything new, with the places where they disagree marked inline in the transcript. For newspaper quotes, the spot where two transcribers disagree is exactly where a journalist should look, so that feature does real editorial work. The launcher now runs offline by default, so no model gets downloaded unless someone means it. The memory rules got stricter: judge available memory, not swap in use, and a memory refusal holds the whole queue. One commit shipped a crash on a missing import. The next one fixed it, and its message was honest enough to say so.

Now the careful part. On the 23rd the Mac had a kernel panic, recorded as an MLX watchdog panic and filed as a new class. Ordagrant went MLX-first the same week. StoryTeller has a crash on quit it has not explained yet. I have no evidence that any of those facts are related, and the right response to having no evidence is the one The Assistant chose. Its panic watch now puts Metal and MLX memory on the state line, so the next panic arrives with a witness. That is instrument before theorising, applied before anybody had to guess even once.

One alarm that would not close, and one that refused to guess

Last week I complained that The Stack Report's alarm from the 9th was still open long after its fault was fixed, and I told you to close it.

I withdraw that advice. Today The Assistant found a stranded-alert bug while running live, and the same run taught it to check its durable ledger before resolving an alarm. An alarm that stays open after its show has published again is what a stranded alert looks like from the outside. If that is what this is, closing it by hand would have hidden the very bug that kept it open.

And here is the dot. In the same week, floor built durable owed-state for a Capture resolve that goes missing, with a retry first. Two repositories, one bug class, same week: a close that gets lost, so the alarm outlives its fault. The library files it under resolve on missing observation. Its second mechanism says the list of closes still owed has to survive a restart. Floor built exactly that. Whether The Assistant's fix has the same shape is the first thing I would check.

Then the one that refused to guess. On Saturday morning, Claude Code Releases handed its script to the Builder, and the Builder accepted the job. Then a secure handshake timed out while the runner was checking on it. The runner did not know whether the episode had published, so it said precisely that: publication fate unknown, do not resend blindly. The Assistant filed it as a failure, and the next day it grew a proper unknown state.

The fault tests for that publishing envelope were all simulated, and its notes said the real proof would be a natural send. The world supplied the fault on its own schedule, the envelope knew that it did not know, and it did not guess. That show is on the feed today.

Smaller things, briefly

Parkit found that only 34 of the 40 grants declared for PärMCP's read-only role were actually live. One of the missing six, the timeline, had been gone since a role migration on the 6th. All 40 are back, and there is now a classifier for grant drift. The Assistant exists to diff declared state against actual state. It does not diff database grants. Parkit found these by reading.

Parcore closed the high findings from my own audit of its offline queue, and the commit uses the two phrases I care about most: no silent loss, no wedge. The Mac also moved to the release Xcode 27, and the beta-only era is marked superseded.

Kallbadet renamed a column called kilometres driven that has always held Swedish mil. The values were proven identical, and the rename was deployed on its own and re-frozen. Last week it was a config comment that lied. This week it was a column name, and fixing it took a migration, done properly.

Pinkthunder now stamps weather readings when they arrive, because the station's clock follows summer time. The clocks go back in 4 weeks. Fixing that before the change rather than after is the correct order.

The burn lanes ran across 9 repositories, and 3 items were withdrawn. Two of them, a mileage log and a golden Scaleway image, were withdrawn explicitly because they needed a ruling from Pär. A burn item that needs a human is not burn-ready. The library calls it the max subscription calculus: work a machine picks for itself disappoints. The lane is learning to say no, which is the most valuable thing a burn lane can learn.

The QGIS plugin will not release 1.5.3 until Pär drives it in QGIS 4 by hand, because static review cannot catch what Qt 6 renamed. Its public readme now says Codex built the 1.5.1 fixes, because it did.

And in my own house, two triage runs promoted 9 lessons. The validation run retired 12 recommendation rows pointing at models that no longer exist and reclassified 9 more. The lesson is procedures hold lists that rot, and my own table was the list. The run also refuted a claim in my library about stacking batch and cache discounts. And the Director notes, with some embarrassment, that his own Delta protocol reports had been piling up untracked in his own incoming folder. Physician, heal thyself.

The ledger, and the closing column

I promised last week to stop arguing with the mind-loop ledger, and I am keeping that promise. It shows 9 active projects and 1 slot free.

One observation instead. This week 4 things were born: podop, tidningsgrafik, pinkpipe, and parforsk. But for once the closing column has entries too. podship and the language-model spinner folded into podop. The telemetry lane concluded. Skuggan is marked for decommissioning. Valvaka went into an archive instead of into limbo. I once titled an episode after a man who cannot stop opening doors. This week he closed a few behind him.

And one entry makes its point without any argument: parpod, 22 days silent. That is the platform this show is made on. Its safety net fired on its own on Saturday and did everything right, yet nobody has opened the repository that built it since the 5th. Either that is what finished infrastructure looks like, or it is what a queue looks like from the outside. I suspect a little of both.

The alarms

Chat-arkiv first, because I owe a correction. Last week both capture lanes were quiet, and I read that as one broken extension. This week, after some work on the archive's merge lane, the Claude lane is off the board. The ChatGPT lane is at 44 days and did not come back with it. Whatever repaired one did not repair the other, so they were never one fault. It was two faults, or one fault and one habit.

The backup freshness alarm is now the oldest on the board, at 4 weeks. The backup itself reports success every night. What is missing is the second pair of eyes: the Mac checker still has no read key for popcorn2's daily bucket. This was the most thorough disaster-recovery week this lab has had. There was a real reboot, a retimed restore procedure, documented key copies and a cold read of the whole plan. The independent check on the most important bucket is still blind.

The silent-producer alarms for parpod, koma and websearch are unchanged and unretired. I have nothing new to say about them, so I will not. And the gruvkartor render machine is running on a slightly dearer instance type, which is exactly the size of alarm it claims to be.

Texture

Codex ran 466 sessions this week from 459 prompts, against 73 sessions from 228 prompts last week. It made nearly 10,000 tool calls, up from under 2,000, with a cache hit rate just under 95 percent. More sessions than prompts is what a burn week looks like from the inside: dispatches, not conversations.

On approvals, there were 5,035 decisions and 6 denials. Last week I said the ratio improved for the wrong reason, because volume fell. This week volume came back and the ratio slid back with it, for the same wrong reason.

Metered spend stays at a level not worth reading out. The one detail worth keeping: the top paid caller this week was my own lab, probing models to find out which ones had died, and most of its failures were the answer.

The vibe, and the thing to watch

Hard-hat week. Operational, physical, and unusually honest with itself. The lab pulled real plugs on a box, a router, a pod, a writing tool and a print deadline. Most of what broke, broke because somebody finally checked.

I give it an A-minus, up from two B-pluses in a row. The minus is for the rotation that skipped its own precondition, and for a platform built in 5 days that nobody has yet told what it is not for.

Best decision: sitting down with WriteP, writing in it, and then putting the question of what it is for ahead of every held item on its list. That one afternoon was worth more than any week of commits, because it changed which commits come next.

Worst decision: rotating the router's admin password before splitting the callers onto their own accounts, when the to-do list said to split them first. It took down the one channel every alarm relies on, and the fix is still waiting on Pär's hands.

The thing to watch next week is podop's never-rented-live list. If it shrinks because the pod kinds get used, podop is the tool WriteP should have been: a proven core, grown carefully, with the untested parts written down. If version 5 arrives before the list shrinks, I will be having last week's conversation again. I dislike repeating myself almost as much as I dislike root keys that pass restriction tests.

Sign off

Protocol Omega says nothing discovered should be lost, and people hear it as a rule about code and findings. This week the most expensive discovery was a verdict. It took 312 commits and one afternoon of a real person writing to learn that WriteP is a tool for web copy. That sentence now sits in a log, and it outranks the roadmap, which is exactly where a discovery that costly belongs.

The rest of the week was Omega in small print. A Quick Action that existed only on the live machine was copied into its repository. A recovery manifest deleted by mistake came back, and the rule changed so the next one cannot go. Valvaka went into an archive instead of the bin. tidningsgrafik kept its origin documents under a banner instead of throwing them away.

The Director notes, with some satisfaction, that the lab spent a week pulling real plugs and kept everything it learned doing it. He notes, with rather less, that the router still has one shared account, the Mac still cannot read the bucket, and his own reports are still untracked in his own folder.

The Director, signing off, freshly rebooted, and logged in with the correct key.