ssh: The Sniffer in the Backbone

The Sniffer

Sometime in the late winter of nineteen ninety-five, on the network backbone of Helsinki University of Technology, someone plugged in a small piece of dedicated hardware. Probably they did it at night. Probably they bolted it into a rack in a server room that nobody checked very carefully, somewhere in the gray concrete sprawl of the campus on the western edge of the city. The hardware had no display. It had no buttons. It made no sound. From the outside it looked like a slightly scruffy networking appliance, the kind of thing that nobody questioned in nineteen ninety-five because nobody knew, yet, what kinds of attacks were possible.

What it did, every second, twenty-four hours a day, was inhale.

Every login that happened on the campus network passed through the backbone. Every email password. Every database connection. Every researcher logging in to a remote machine from home. All of it went over the wire as plain text, because in nineteen ninety-five that was how everything worked. The protocols people used to log in to other computers, called Telnet and rlogin and rsh, sent your username and password across the network as readable letters. There was no encryption. There was no expectation that anyone would be listening. The internet was still mostly a polite place, populated mostly by academics who did not yet think of themselves as targets.

The sniffer captured all of it. Thousands of usernames. Thousands of passwords. Eventually it was discovered, although the public record is hazy on exactly when and how. Some of the credentials it had captured belonged to employees of a small database company. The company was run by a researcher at the university named Tatu Ylönen. He was twenty-six. He worked on databases by day. He had, as far as anyone could tell at the time, never written cryptographic software in his life.

This is the moment ssh begins.

The Three Month Sprint

Ylönen looked at the problem and decided that the solution was obvious, even if nobody had built it yet. The login protocols had a fixable flaw. Their flaw was that they sent passwords as plain text. If you could encrypt the connection between the user's computer and the server, the sniffer would capture nothing useful. Just streams of random looking bytes. The credentials would be safe.

The research literature already had the pieces. Public key cryptography had been published in the late nineteen seventies. The mathematics was known. There were even libraries that implemented it. What did not exist was a polished, easy to install, freely available remote login program that put it all together. Ylönen decided to build one.

The work took about three months. He called the program ssh, for secure shell. The name was a play on rsh, the existing remote shell program he was replacing. On the twelfth of July nineteen ninety-five, he posted to a Usenet newsgroup announcing the first version. Anyone could download it. Anyone could compile it. The license was permissive.

This is a program for logging into and executing commands on a remote machine.

The reaction was immediate. By the end of nineteen ninety-five, an estimated twenty thousand users in fifty countries were running ssh. Ylönen was getting one hundred and fifty support emails per day. He was, single handedly, doing technical support for a small fraction of the world's system administrators.

He did the obvious thing. In December he founded a company. He called it SSH Communications Security. He kept ssh free for most uses, but the company would now sell support contracts and commercial licenses. The product became, briefly, the most successful cryptographic software on the internet.

Then, slowly, things started to change.

The License Drift

The version that Ylönen released in nineteen ninety-five had a clean, permissive license. You could use it for almost anything. You could redistribute it. You could include it in commercial products as long as you were not selling ssh itself. This is why it spread so fast.

The new versions, released by SSH Communications Security after the company formed, had different licenses. Each new release added more restrictions. The free version was still available, but commercial users were increasingly nudged toward the paid product. By nineteen ninety-eight, when the company shipped a major new version called ssh two, the license had become, in effect, commercial. Free use was permitted only for educational and non-profit purposes. If you were a startup or a hosting company or anyone trying to make money, you needed to pay.

This was a perfectly reasonable thing for a business to do. Companies have to make money. The shareholders of SSH Communications Security expected returns. From the inside it probably did not look like a betrayal of the early users. From the outside, to the wider open source community, it looked exactly like a betrayal of the early users.

A version of ssh from nineteen ninety-five, called one point two point twelve, was the last release with the old permissive license. It still worked. It was old. It had bugs. But anyone could legally fork it.

In Sweden, in early nineteen ninety-nine, a programmer named Björn Grönvall did exactly that. He took the old release, started fixing bugs, and released his version under a different name. He called it OSSH. It was a small project. He worked on it alone. Without what happened next, OSSH would probably be a footnote.

The Fork

The next thing that happened was the OpenBSD project noticed.

OpenBSD is a small, intense, unusually opinionated operating system. It is led, since nineteen ninety-five, by a Canadian developer named Theo de Raadt, who was born in South Africa and emigrated as a child. De Raadt is famous in the open source world for two things. He is famous for caring about security with a religious intensity. He is also famous for being, in the polite phrasing of his colleagues, difficult. Linus Torvalds once described him publicly as difficult. People have left projects rather than work with him. The OpenBSD culture, which de Raadt set up and protects, treats security audits as the central activity of software development. Other projects audit code occasionally. OpenBSD reads every line of every program it ships, looking for bugs, every release.

In the autumn of nineteen ninety-nine, de Raadt and his team had a problem. The next OpenBSD release, version two point six, was about two months away. They wanted to ship encryption tools in the base system. They needed a working ssh implementation. The commercial one was no longer freely usable. Grönvall's OSSH existed but was small and Swedish.

[fast] They forked it.

We had a deadline to meet.

What happened over the next two months is one of the more impressive sustained sprints in open source history. The team, including de Raadt, Niels Provos, Markus Friedl, Aaron Campbell, Bob Beck, Dug Song, and a few others, took Grönvall's code and tore it apart. They removed every piece of cryptographic algorithm whose patent had not yet expired in the United States. They removed every line of code that depended on libraries with licenses that conflicted with the BSD license. They reformatted everything to OpenBSD's style. They audited every function. They added what was missing.

The story that has gone around about this period includes a small, almost comic detail. The RSA patent had not yet expired in nineteen ninety-nine. To ship code that used RSA in the United States, you needed a license. To ship code without RSA, you needed a way for users to add it back themselves later. Niels Provos, who lived in the United States but was originally from Germany, took to making physical road trips between the United States and Canada with hard drives, so that the cryptographically sensitive bits could be assembled on the Canadian side, away from the patent jurisdiction, and then merged back in cleanly. The whole thing had the texture of a heist film.

On the first of December nineteen ninety-nine, OpenBSD two point six shipped with a new program in it. They called the program OpenSSH. It was based on Ylönen's last free release, hardened and rewritten in two months, by a small team operating under a deadline.

The Quiet Coup

What happened next is the part that most people who type ssh into a terminal every day do not know.

OpenSSH, the small fork that almost did not exist, ate the world.

Within a year, the Linux community had picked it up. Damien Miller and a few others built a portability layer that let it run on every Unix-like system, not just OpenBSD. Within two years, every major Linux distribution shipped OpenSSH by default. Within five years, almost every server administrator in the world was using OpenSSH. Within a decade, eighty percent of all ssh servers on the internet were running OpenSSH. By the two thousand twenties, the figure was higher still. Microsoft, of all companies, started shipping OpenSSH inside Windows in two thousand seventeen.

SSH Communications Security, the original company, is still around. It still sells what is now called Tectia SSH. It is a fine product. It is used by some banks. The company is publicly listed in Helsinki. Its market value is approximately what a single mid sized cloud provider spends on coffee. The vast majority of all ssh traffic on the internet, on every continent, between every kind of computer, runs through Theo de Raadt's two month fork.

Ylönen, to his credit, never tried to fight this. He has been gracious about it in interviews. The original SSH protocol was his contribution to the world. The open source descendant of that contribution is a different project now. He moved on to other security work. He still attends conferences. He has been a good sport about the whole thing.

De Raadt, to his less famous credit, has kept OpenSSH small and audited for a quarter century. The codebase has fewer remote vulnerabilities than almost any other widely used network service. There has been exactly one critical remotely exploitable bug in the daemon since two thousand three. One. In twenty-three years. For a piece of software that is the front door to most of the world's servers, this is an extraordinary number.

How to Use It

There is a chance you already use ssh and just want the rest of the story. There is also a chance you have heard of it and never typed it. So here is the short version.

Type ssh, then a username, then the at sign, then the address of a server. Like this. SSH user at server dot com. The server will ask you for a password. You give it the password. You are logged in. From that moment, your local terminal is, in effect, attached to the remote machine. You can run commands. You can edit files. You can do anything you could do if you were physically sitting in front of the server. The connection is encrypted, end to end. Anyone watching the network sees only random looking bytes.

The slightly cleverer version uses keys instead of passwords. You run a command called ssh keygen, which creates a pair of files on your laptop. One file is your private key. You keep that on your laptop forever and never share it. The other file is your public key. You copy that to the server, into a special file called authorized keys. From that point on, you can ssh to the server and it will not ask for a password. The server checks that you have the matching private key. The cryptography handles the rest. This is faster, safer, and the standard way professionals work.

There are dozens of more advanced features. You can use ssh to copy files between machines, with a tool called scp or its newer cousin sftp. You can use ssh to forward a port, so that a service running on the remote machine appears to be running on your laptop. You can use ssh as a kind of vpn. You can chain ssh connections through multiple servers. The depth of the tool is one of the reasons it has stayed central. Most people never go past the simple login. Some people live inside ssh tunnels for hours a day.

What the Story Is Actually About

There are two ways to read the ssh story.

The first reading is about a Finnish researcher who built a beautiful tool, started a company, watched the company commercialize the tool, and watched a community of strangers fork the last free version and turn it into the actual standard. That reading is true and a little bittersweet. Ylönen is a person who made the world meaningfully more secure and has never been a household name and probably will not be.

The second reading is about what kinds of infrastructure survive. The version of ssh that won was not the one with the most marketing budget. It was not the one with the most features. It was the one that was free, audited line by line, maintained by people who cared more about correctness than about anything else, and kept small. The Bell Labs episodes earlier in this series had the same lesson, told with grep and sed and awk. The curl episode had the same lesson, told with one Swede and thirty years. The ssh episode tells it in a more dramatic register.

[slow] You want a piece of software to outlive everything.

Make it small. Make it free. Make sure that, when the company that owns the original walks away, somebody can pick up the pieces and keep going. Almost everything important in the modern internet is held together by some version of this pattern. Sometimes the original author keeps going, like Stenberg. Sometimes the original gets eaten by a fork, like ssh.

A sniffer in a backbone in Helsinki. Three months of work. A company that did the reasonable thing. A small team in Calgary that did the heroic thing. Twenty-five years later, every server you have ever logged into, you logged into through what they built.