The Query That Took Down The Server: Telemetry Without A Dashboard

No Dashboard, On Purpose

Most people, when they add observability to a system, install something with a web interface. A search box, some charts, a service map with animated lines. Jaeger, Grafana, the usual furniture.

Popcorn2 has none of that, deliberately, and the decision was yours. No Jaeger. No dashboard tool at all.

What it has instead is a pile of files and a query engine, and this episode is about why that turned out to be the better call, and about the afternoon a single query wedged the entire machine so hard that even logging in stopped working.

What A Service Reports About Itself

The vocabulary first, because it is small and it makes everything else readable.

When a service does a unit of work, handling a request, sending an email, generating an episode, it can emit a span. A span is a small structured record saying what the work was, when it started, how long it took, whether it succeeded, and a handful of labelled details. Spans link to each other, so a request that calls 3 other things produces a tree, and the whole tree is a trace.

The standard for this is called OpenTelemetry, and its real achievement is that it is vendor neutral. Your applications emit in one format and you can point that at any backend you like, or change your mind later without touching a line of application code. That decoupling is why the no dashboard decision was cheap rather than a commitment.

There is one contract every instrumented service on this box is expected to follow, and it is stricter than most estates bother with. Emit exactly one business outcome span, carrying an outcome label. Not a technical status. Did the actual job happen. Some services on the box are not instrumented at all, which is a known and deliberate state rather than a claim that everything reports.

And the alerting rule that goes with it is the important half. Alert on the rate of failed outcomes. Never on a request having returned a success code, and never on a span merely existing. A service can return 200 all day while doing nothing useful, and a healthy looking trace tree tells you the plumbing worked, not that the work happened.

Why It Travels Through A File

All of this data goes to one collector, running as its own user like everything else. And it gets there through a Unix socket, the same kind of file based door the database uses, mounted into the containers that report.

There is no network port for telemetry on this box, and the reason is a direct consequence of episode 1. Under rootless containers with this networking mode, a port bound to the loopback address is, from a container's perspective, effectively the machine's public address. So a telemetry ingestion port, which accepts anything anyone sends it and is not normally authenticated, would have been exposed in a way nobody intended.

The socket removes the question. There is no port. To send telemetry you must have the file mounted into your container, which means it must have been declared in the catalog by root.

A small implementation detail with real consequences. The collector has 2 ways of receiving data, and only one of them can listen on a socket at all. The HTTP one cannot. So the choice was made for us, and everything here speaks the other one. That is the sort of constraint you only discover by trying, and it is written down so nobody rediscovers it.

Files And A Query Engine

The collector writes everything out as JSON lines into a rolling store of about 4 gigabytes, split between traces, metrics, and logs. When it fills, the oldest gets evicted.

To read it, there is DuckDB. If you have not met it, DuckDB is a genuinely lovely piece of software. It is a full analytical database engine, in a single file, with no server and no daemon. You point it at files on disk and write SQL against them directly. Nothing is imported ahead of time and there is no schema to maintain. It works out the shape of the data while answering your question, and when the question is finished, nothing is left running.

Which means the query side of the observability stack is a directory of text files plus a program you invoke. There is a collector daemon, which is real software that can fail and needs updating like anything else. But there is no database server, no web interface, and no dashboard application sitting there waiting to be exposed to the internet.

Three good things fall out of that. Traces, metrics and logs sit in one store, so you can join across them, which the separate specialised tools make surprisingly hard. Access is plain SQL from a shell, which is exactly what an AI session working on this box actually wants. And the graduation path is already named. If volume or retention or query latency ever justifies it, the answer is a specific different database, and the trigger conditions are written down, so that decision is already made and just waiting.

Deriving Metrics From Traces You Already Have

There is a nice trick in the middle of this that saved a fleet wide rewrite.

The estate had a gap. Around 24 services were emitting traces, and exactly 1 was emitting metrics, because the shared setup code wired up tracing and not metering. Fixing that properly meant touching every service.

Instead, the collector was given a component that watches spans go past and derives metrics from them. Rate, errors, and duration, which in this world is universally abbreviated as RED. Every service that was already emitting traces got metrics for free, and not a single application was rebuilt.

There were 4 traps in getting that right and they are all worth knowing, because they are exactly the kind of thing that produces confidently wrong dashboards.

The fleet emits the older generation of naming conventions for web attributes. Every current document recommends the newer names. Use the newer names here and every label comes out empty, on every series, which looks like the data is broken rather than the query being wrong.

The success status is a text string in the derived metrics and a number on the raw spans. Two representations of the same concept, and mixing them up in the other direction had previously hidden 191 genuine errors.

The derived lane has to be filtered or it mostly measures itself. Health checks run about once a minute per service, and the web framework emits internal child spans for each request that were accounting for 43 of 99 series and triple counting every real request. The traces themselves stay entirely unfiltered, because those are the forensic record and you never want to have thinned the evidence.

And the limit on how many distinct series it will track is a hard drop, not an eviction. Past the limit, new things are simply refused. So a newly added service can silently fail to appear in the metrics forever while its traces keep landing perfectly. That is detectable precisely because the traces are unfiltered, and there is now a check comparing the 2 populations.

The Query That Killed The Box

Now the incident.

DuckDB, by default, will use up to about 80 percent of the machine's memory. On a laptop, sensible. On an 8 gigabyte server running 38 things, that number is the entire box.

Somebody ran a query across the whole trace store. Trace data in this format is deeply nested, so reading it involves expanding every record into many rows, and the memory use ballooned. The machine ran out, and the thing that died was the SSH daemon. Not the query. The ability to log in and stop the query.

Out of that came a set of rules that any timed job reading this store must follow, and they generalise beautifully to any resource hungry background task.

Use the operating system's memory ceiling, not the application's own setting. DuckDB's limit is soft accounting and it routinely overshoots roughly twofold. A kernel enforced ceiling means a runaway gets killed instantly, the job reports that it could not run, and the box survives. But do not set it too low either, because this workload cannot spill to disk below a certain size and simply errors out.

Bound the input rather than raising the ceiling. The active file grows continuously until it rotates, so a job that reads the whole thing works fine on Monday and dies on Thursday, becoming permanently broken. Read only the tail. Raising the cap just moves the cliff and couples your guard to the collector's rotation settings.

And treat an empty file as no evidence, never as a failure. A zero byte file is exactly what a rotation creates, momentarily, and DuckDB cannot infer a schema from nothing so it errors. Get that wrong and your guard raises a false alarm roughly daily, at whatever hour rotation happens.

Attest, Do Not Parse

One more, because it is the most reusable idea in this whole series.

The health check for the derived metrics has to know 3 values that live in the collector's configuration file. If the collector's config changes and the check does not, the check is comparing against a world that no longer exists and will report everything is fine forever.

So the first version parsed the configuration file. Just enough of it to pull out those 3 values.

Over 4 rounds of review, Codex found 4 separate ways that parser could report everything is in step while the collector had actually changed. Inline comments. Escaped quotes. Multi line text blocks. The order of an indicator character. Every single one of them failed in the same direction, which is the fatal direction, quietly reporting healthy.

The replacement does not parse anything. It takes a cryptographic digest of the entire configuration file, and of its own constants, and compares those. If either side moves by so much as one byte, the digest changes and the check disarms itself and says so.

It cannot tell you what changed. It can tell you, with certainty, that something did, and that is exactly the question it needed to answer.

The consequence is deliberate and slightly annoying. Editing a comment in that config file disarms the check. Which is correct. You changed the thing it depends on, so it wants you to look again and re confirm, rather than assuming a comment is harmless. A guard that must never report a false all clear is not allowed to be clever.

The Torn Line

Last one, quick, and it is a small masterpiece of a failure.

When the machine was hard killed during a storage incident, the very last line being written to the metrics file was cut in half. Thirty two kilobytes of a partial record.

DuckDB will not read the file. Not the broken line. The entire file, rejected, because it cannot make sense of it. So the check went dark for about 7 hours.

And here is the cruel part. The check's own message, written by somebody being helpful, said that a torn line self heals on the next write. That is doubtful even for a tear at the very end, since appending to a half written record just produces one longer broken record, and it is flatly false for a tear in the middle. So the message actively argued you out of investigating a real fault.

Repairing it has its own trap, which is that the collector holds that file open. Editing it in place with the usual tools creates a new file and leaves the collector writing into the old invisible one, which then no longer exists as far as anyone can see. So you stop the collector first, back up, remove the bad line, revalidate every remaining line, and start it again. And you find the bad line by parsing every line, not by looking, because reading a truncated view of a healthy record makes it look damaged too.

Next time, the backups. Including the day a service went live holding your only record of a season of advertising bookings, believing it was backed up, and being wrong for a specific and very sneaky reason.