systemd: The Most Hated Software You Cannot Live Without

The Script That Ran Forever

Before we talk about systemd, we need to talk about what came before it, because you cannot understand the fury unless you understand what was lost. Or, depending on your perspective, what was mercifully replaced.

For decades, every Linux system started the same way. The kernel loaded, initialised the hardware, and then handed control to a program called init, process number one, the first thing to run and the last thing to stop. Init would read a series of shell scripts, one for each service that needed to start, and run them in sequence. Start the network. Start the logging daemon. Start the web server. Start the database. One after another, in order, each waiting for the previous one to finish before the next could begin.

These scripts lived in a directory called etc slash init dot d, and they were just text files full of bash commands. If you wanted to know how Apache started, you opened the script and read it. If you wanted to change the startup order, you renamed files to change their sort position. It was transparent, predictable, and slow. On a server with dozens of services, booting could take minutes, because every service waited politely in line even if it had no reason to.

The transparency was real, though. A system administrator could trace every step of the boot process by reading shell scripts. There was no magic, no hidden state, no binary logs. If something went wrong, you added an echo statement to the script, rebooted, and read what it printed. The entire system was, in the truest sense of the word, human-readable. This mattered deeply to the people who ran servers for a living, and it is the thing they mourned most bitterly when it was taken away.

The Man from Guatemala City

Lennart Poettering was born in nineteen eighty in Guatemala City, grew up partly in Rio de Janeiro and partly in Hamburg, and ended up as a software engineer at Red Hat in Germany. Before systemd, he was already famous, or perhaps notorious, for PulseAudio, a sound server that eventually unified audio on Linux but went through years of being widely hated for breaking sound on people's machines in the process. If you used Linux in two thousand eight and your audio suddenly stopped working after an update, there is a reasonable chance Poettering's code was involved.

Poettering was brilliant, opinionated, and utterly uninterested in the Unix philosophy of doing one thing well. He looked at the Linux boot process and saw not elegant simplicity but a disorganised mess of shell scripts that could not express dependencies properly, could not start services in parallel, and could not automatically restart a crashed process. In twenty ten, he and another Red Hat engineer named Kay Sievers began building something new.

They called it systemd, with a lowercase s, and they prototyped it under the internal name BabyKit, because it was a process babysitter. The core idea was that services should be defined not as shell scripts but as declarative unit files. Instead of writing a bash script that checks for dependencies, sets environment variables, finds the correct binary, and runs it, you would write a simple text file that says: this service runs this binary, it needs the network to be up first, and if it crashes, restart it after five seconds.

Initially it was just an init system. It was just process number one. But it turns out that process number one needs to interact with a lot of other things to do its job properly.

That statement captures both the genius and the controversy. Poettering did not just replace init. He replaced init, cron, syslog, inetd, acpid, the console setup scripts, the hostname configuration, the locale configuration, the network configuration on some systems, and pieces of the login system. Each of these replacements had a rational justification. Each of them made someone furious.

The War of the Init Systems

The adoption of systemd across major Linux distributions was swift and bruising. Fedora adopted it first, which was unsurprising since Red Hat employed Poettering. Then Arch Linux switched. Then OpenSUSE. Then, most significantly, Debian.

The Debian vote in twenty fourteen was one of the most divisive moments in free software history. Debian's technical committee debated for months. The arguments were passionate and often personal. On one side, developers argued that systemd solved real problems: faster boot times, reliable dependency handling, and a consistent interface for managing services. On the other side, developers argued that systemd violated fundamental design principles by absorbing too many functions into a single project, creating a web of dependencies that made it nearly impossible to replace any single component.

The committee voted. systemd won. And then things got ugly. A group of Debian developers, unable to accept the decision, forked the entire distribution, creating Devuan, a version of Debian without systemd. The project still exists today. The debate spilled onto forums, mailing lists, and social media. Poettering began receiving personal attacks and, according to his own accounts, death threats.

We have been on the receiving end of mass hatred for some time now. The personal attacks have been extreme.

Linus Torvalds, who has never been shy about expressing displeasure, weighed in with a more nuanced position. He was not against systemd itself, but he was frustrated with the behaviour on both sides, maintainers who blocked patches, developers who refused to consider alternatives, and the general cultural rigidity that had transformed a technical disagreement into a tribal war.

In twenty twenty-two, Poettering left Red Hat and joined Microsoft, which raised eyebrows across the Linux community. Then in January twenty twenty-six, he left Microsoft to co-found a startup called Amutable, focused on cryptographic verification for Linux systems, alongside Christian Brauner, a Linux kernel maintainer, and Chris Kühl from the GNOME Foundation. The man who built the most used and most argued-about software in Linux had moved on, but his software had not. It runs on virtually every Linux server on the planet.

The Three AM Restart

Let's put the philosophy aside and talk about what systemd actually does for you, right now, on your Scaleway VPS. Every FastAPI application you deploy runs as a systemd service. When you wrote those unit files, the ones that say ExecStart and point to your uvicorn command, you were writing systemd configuration. When your app crashes at three in the morning because a database connection timed out, systemd notices that the process exited, waits the number of seconds you specified in RestartSec, and starts it again. Nobody had to be awake. Nobody had to log in and type a command.

With the old init scripts, a crashed service stayed crashed until a human noticed. There were tools like supervisord and monit that could restart things, but they were separate programs with their own configuration formats and their own quirks. systemd integrated this directly into process number one. If process one is watching your service, it will always be watching your service, because process one is the last thing to stop.

Here is something you might not have explored yet. systemd has a timer system that replaces cron. Instead of editing a crontab file, you create a timer unit that triggers a service unit. This might sound like extra work, but the advantage is that the timer and the service share the same logging, the same dependency system, and the same management tools. You can run systemctl list-timers and see every scheduled task on your system with its next run time, the last time it ran, and how long ago that was. Try getting that from cron.

Then there is socket activation, the feature Poettering was most excited about in the early days. Instead of starting a service at boot and leaving it running forever, you can tell systemd to listen on a port. When the first connection arrives, systemd starts the service on the fly and hands it the connection. The service can shut down when idle and be re-awakened by the next request. For a server running many services that are used infrequently, this saves memory and startup time. It is the pattern that made early Linux phones responsive.

And there is security hardening. Modern systemd lets you add directives to your unit files that restrict what a service is allowed to do. You can prevent a service from accessing the network, restrict it to a read-only filesystem, deny it the ability to create new user accounts, hide entire directory trees from its view, and drop kernel capabilities it does not need. These are not theoretical features. They are things you can add to your PärCel service files, your Årebladet backend, your Popcorn Photobooth workers, and know that even if the application has a vulnerability, the damage is contained.

The Binary Log Debate

One of the longest-running complaints about systemd is that it replaced plain-text log files with a binary journal. The old system, syslog, wrote human-readable text files. You could grep them, tail them, pipe them through awk. The journal, managed by a component called journald, stores logs in a binary format that requires the journalctl tool to read.

Poettering's argument was that structured binary logs allow features that text logs cannot support: efficient indexing by time, by service, by priority level. You can ask journalctl to show you only the logs from a specific service in the last hour at error level or above. With text files, that requires knowing the exact log format and writing a custom filter. With the journal, it is one command.

The counter-argument, and it is a legitimate one, is that text files are universal. They survive disk corruption better. They can be processed by any tool in any language. They do not require a specific program to read. And for system administrators who had spent decades building muscle memory around grep and awk and sed, being told to learn a new tool felt like being told to switch from driving to riding a horse.

In practice, most modern Linux systems run both. journald captures the logs, and you can configure it to forward everything to syslog as well, giving you both the structured queries and the plain text files. It is a compromise that satisfies almost nobody completely and almost everybody partially, which is perhaps the most systemd outcome possible.

The Service That Runs Your Services

Whether you love systemd or merely tolerate it, the reality is that it is the foundation under your entire server infrastructure. Every deploy you do through rsync ends with a systemctl restart. Every boot of your pinkserver goes through systemd before Docker gets a chance to start its containers. Every log message from every service passes through its journal. It is the invisible choreographer that ensures your services start in the right order, restart when they fail, and shut down cleanly when the server needs to reboot.

Lennart Poettering did what the Unix philosophy explicitly warned against. He built a big system that does many things. And then the entire Linux world adopted it because, despite all the arguments and the forum wars and the death threats, it solved problems that the old way could not. The shell scripts were beautiful in their simplicity, but the world they were built for, a world of single-purpose servers with a handful of services, was already gone by the time systemd arrived.

Your VPS runs dozens of interconnected services. They have dependencies. They crash. They need restarting. They need log management and security isolation and scheduled tasks. systemd handles all of that in a consistent, documented, well-tested framework that works the same way on every Linux distribution. The man from Guatemala City built a system that the world depends on, and then he left to build something new, and the system kept running.

In the next episode, we go deeper. Past the services, past the scheduler, past the filesystem, all the way down to the kernel itself. Because there is a technology hidden inside your Linux kernel that lets you run custom programs in the most privileged space your computer has, without rebooting, without risk, and without most people even knowing it exists. It is called eBPF, and Brendan Gregg at Netflix called it superpowers for Linux. He was not exaggerating.