The Linux Kernel Crisis of Two Thousand Five

Patches by Email

On a Tuesday in late March, two thousand five, a kernel developer named Andrew Morton sat down at his desk and began doing what he did every single day. He opened his email client and started reading patches. Not five patches. Not fifty. Hundreds of them. Plain text diffs, the raw language of code changes, flowing into his inbox from developers on six continents. Each one a small proposal for how the Linux kernel should be different tomorrow than it is today. Morton would read the patch, decide if it made sense, test it against his own copy of the kernel source tree, and either apply it or write back with questions. Then he would move on to the next one. And the next one. And the next one. Every single day.

This was how the most important open source project in the world had been built for over a decade. From nineteen ninety-one, when Linus Torvalds first posted his hobby operating system to a Usenet newsgroup, through two thousand two, the Linux kernel had no version control system at all. Developers would write their changes, format them as patches, and email them to the Linux Kernel Mailing List. Subsystem maintainers, the lieutenants in Linus's informal hierarchy, would collect patches relevant to their area, test them, bundle them together, and forward them up the chain. At the top of that chain sat Linus himself, applying patches to his personal copy of the source tree and cutting releases when he felt the code was ready.

For a while, this worked. When Linux had a hundred contributors, one person could read the patches. When it had five hundred, the lieutenant system distributed the load. But by the late nineteen nineties, the kernel had thousands of active developers, and the system was cracking under the weight. In nineteen ninety-eight, Larry McVoy, a kernel developer and the CEO of a small software company called BitMover, posted a message to the Linux Kernel Mailing List that captured the problem with painful clarity. It was clear, he wrote, that their fearless leader was at the moment a bit overloaded, and patches were getting lost.

Getting lost. Months of work by talented developers, disappearing into an inbox that one person could not possibly keep up with. Not rejected on technical merit. Not discussed and set aside. Just lost. Swallowed by the sheer volume of email that powered the most ambitious collaborative software project in history.

The scale of the problem was staggering. By two thousand five, a new kernel release could involve over ten thousand patches touching millions of lines of code. Each of those patches arrived as an email. Each had to be read, evaluated, tested, and either applied or sent back with feedback. The kernel was the largest, fastest-moving collaborative software project in history, and its development infrastructure was a mailing list and one person's willingness to read email all day.

As we heard in the last two episodes, CVS and Subversion existed. Linus rejected both. CVS could not handle the kernel's workflow. It was centralized, it was slow at scale, and its branching was so painful that teams avoided it. Subversion fixed some of CVS's worst bugs but kept the same fundamental architecture, one server, one truth, one point of failure. For a project with thousands of contributors scattered across the globe, working in different time zones, on different schedules, often without reliable internet connections, the centralized model was a ceiling. And the kernel had already hit it.

Larry McVoy's Generous Offer

Larry McVoy was not an outsider to the kernel world. He was born in nineteen sixty-two in Concord, Massachusetts, earned his computer science degrees at the University of Wisconsin-Madison, and went to work at Sun Microsystems in the late nineteen eighties. At Sun, he did something that would shape the next two decades of his life. He worked on a peer-to-peer source code management system called TeamWare. More specifically, he wrote a tool called NSElite, which evolved into TeamWare. NSElite is the grandparent of BitKeeper. They share a lot of similarities.

McVoy understood something that most version control designers of the nineteen nineties missed. The future was distributed. Instead of one central server holding the truth, every developer should have a complete copy of the entire repository. You could work offline. You could branch without asking permission. You could merge with confidence because the system tracked what had been merged and what had not. TeamWare had shown him this was possible. BitKeeper was his vision for how to do it right.

But McVoy was also a pragmatist with a complicated relationship to the open source world. While at Sun in the early nineteen nineties, he and several other high-profile Unix community members had urged the company to open-source SunOS, their flagship Unix product, in cooperation with Novell. The proposal would have created a copyleft version of SunOS at a time before Linux had even reached its one-point-oh release. McVoy predicted, accurately, that Linux would displace commercial Unix if the companies did not act. Sun did not listen.

He left Sun, founded BitMover, and built BitKeeper as a commercial product. But he also started working with the Linux kernel around version zero-point-ninety-seven in nineteen ninety-two, and he created LMbench, a widely used kernel benchmarking tool. McVoy was a kernel insider. He knew the developers. He knew the workflow. He knew the pain.

In two thousand two, he made Linus an offer. BitMover would give free BitKeeper licenses to open source projects. The Linux kernel, the biggest and most visible of them all, could use the best version control system in the world at no cost. There were conditions, of course. But the tool was extraordinary. Distributed architecture. Fast merging. Branch tracking that actually worked. Everything the kernel needed and everything CVS could not provide.

I'm not a free software zealot. I'll use the best tool for the job.

Linus said yes. And half the open source community lost its mind.

The Uneasy Truce

The philosophical objections were immediate and loud. Richard Stallman, founder of the GNU Project and the Free Software Foundation, the person who had spent two decades arguing that software freedom was a moral imperative, was appalled. The most important free software project in the world was going to depend on a proprietary tool? Stallman wrote to the Linux Kernel Mailing List in October two thousand two, calling out what he saw as the fundamental problem with BitKeeper's license. He described it as the spirit of the whip hand.

Stallman's summary of what the license really said beneath the legal language was blunt: you have no right to use BitKeeper, only temporary privileges that we can revoke. That phrase, the spirit of the whip hand, captured his view that the license was not generosity. It was control. And the license did say something very close to that. The free version of BitKeeper, called the Community Use license, came with strings that would have been unthinkable for any open source tool. You could use BitKeeper for free, but only if you did not participate in the development of a competing version control system. Not just a competing commercial product. Any competing system. If you contributed to CVS, to Subversion, to GNU Arch, to anything that managed source code, you were forbidden from using the free version of BitKeeper.

There was more. The Community Use license required something called Open Logging. Every time you committed a change, metadata about that commit, the changeset, the log message, the repository structure, was automatically transmitted to BitMover's servers at a site called openlogging dot org within seven days. BitMover could see what every free user was working on. Not the source code itself, but the shape of the work. The commit messages. The branch structure. The pace of development. It was as if the library let you borrow books for free but kept a detailed log of every page you turned.

And you had to upgrade when BitMover told you to. And you had to make your repository available on demand. The free license was a gift, but a gift with surveillance built in and an expiration date that the giver controlled.

Think about what that non-compete clause meant in practice. If you were a kernel developer who used the free version of BitKeeper, you were forbidden from contributing to any open source version control project. You could not help improve CVS. You could not submit a patch to Subversion. You could not experiment with GNU Arch or Darcs in your spare time. The very act of using BitKeeper to work on open source code prevented you from working on the open source tools that managed code. The developer community was being divided into two camps: those who accepted McVoy's terms and got the best tool available, and those who refused on principle and kept working with patches and email.

Alan Cox, one of the most prominent kernel developers and a longtime free software advocate, refused to use BitKeeper. He was not alone. A significant faction of kernel developers rejected the tool on principle, continuing to work with patches and email while the rest of the community moved to BitKeeper. The kernel community was split, not over code, but over philosophy. And the split had practical consequences. Developers who refused BitKeeper could not easily pull from or push to the BitKeeper repositories. They were, in effect, second-class citizens in the kernel development process, locked out of the most efficient workflow because they would not agree to a license that restricted their freedom to work on other software.

But here is the thing that made the situation genuinely complicated. BitKeeper was good. It was really, really good. For the developers who used it, the kernel workflow transformed overnight. Whole repositories could be forked and merged easily. Linus could pull changes from subsystem maintainers with a single command instead of wading through hundreds of email patches. The distributed architecture meant developers could work independently, commit offline, experiment on branches, and synchronize when they were ready. BitKeeper did not just make kernel development easier. It showed the entire open source world what version control could be.

And McVoy was not some faceless corporation extracting value from the community. He was a kernel developer himself. He knew these people personally. He had given them a tool worth tens of thousands of dollars per license, for free, because he believed in the kernel project. The generosity was real. The strings were also real. And for three years, from two thousand two to early two thousand five, everyone pretended the strings did not matter.

Tridgell Pulls the Thread

Andrew Tridgell is an Australian computer programmer born in nineteen sixty-seven, and if you have ever shared files between a Windows machine and a Linux machine, you have almost certainly used his work. He holds a PhD from the Australian National University, where his doctoral thesis was on efficient algorithms for sorting and synchronizing files, the theoretical foundation for rsync. He created Samba, the software that lets Linux speak Microsoft's file sharing protocol, by connecting to Windows servers and painstakingly observing how they communicated. He also co-invented the rsync algorithm, the remarkably efficient method for synchronizing files across a network that only transfers the differences. Both achievements required the same skill: looking at a proprietary protocol, figuring out how it worked, and building an open alternative. In the open source world, Tridgell was a folk hero. He had spent his career making proprietary systems interoperable with free ones.

In January two thousand five, Tridgell joined the Open Source Development Labs on a one-year fellowship. OSDL was a nonprofit consortium funded by companies like IBM, Hewlett-Packard, and Intel, created to promote Linux in the enterprise. Linus Torvalds also worked at OSDL. So Tridgell and Linus were, technically, colleagues.

Tridgell and Linus had known each other for years through the Linux and open source community. They were not enemies. They were colleagues with different views on what mattered. And Tridgell had a problem with BitKeeper. Not with the software itself, but with the fact that the kernel's history, the complete record of every change to the most important open source project in the world, was locked inside a proprietary format. If you wanted to examine that history, to search it, to analyze it, to build tools around it, you needed BitKeeper. And if you needed BitKeeper, you needed to agree to BitMover's license. And if you agreed to that license, you could never contribute to an open source version control system again.

So Tridgell did what Tridgell does. He looked at the protocol.

What he did was disarmingly simple. BitKeeper ran a network service on port five thousand. Tridgell opened a terminal, typed telnet to connect to a BitKeeper server on that port, and then typed one word: help. The server responded with a list of available commands. One of those commands was clone. Tridgell ran the clone command. The server returned a large volume of output, and that output was a simple series of SCCS files, the standard source code control format that predates even RCS.

People keep believing I'm some kind of reverse engineering wizard. Getting a handle on the BitKeeper network protocol turned out to be rather easier than that. There was not a whole lot to it.

That was it. No binary analysis. No decompilation. No cryptographic cracking. He connected to a public-facing network service, asked it what it could do, and it told him. The irony was even deeper than that. Users did not even need to know to type help. If you just pressed enter without typing anything, the server itself would suggest it, responding with an error message that said, try help.

Later, at the Linux Conference Australia in two thousand five, Tridgell demonstrated the entire process live during a keynote. He walked through each step. And according to attendees, the audience called out each command in unison. Telnet. Help. Clone. Hundreds of people in the room replicated the feat in about two minutes. Whatever this was, it was not the kind of deep reverse engineering that requires years of expertise. It was a network service telling anyone who asked exactly how it worked.

Tridgell had a specific argument for why this was legitimate. He had never bought BitKeeper. He had never installed BitKeeper. He had never agreed to BitKeeper's license. The license restricted what BitKeeper users could do. Tridgell was not a BitKeeper user. He was someone connecting to a publicly accessible network service and reading its responses. This was exactly the same approach he had used to build Samba, connecting to Windows servers and observing the protocol. Nobody had called that reverse engineering illegal. It was, in fact, the foundational technique of interoperability research.

I have never bought or owned BitKeeper. I have never agreed to its license. I was analyzing the protocol ethically, as I had done with Samba.

Tridgell used what he learned to begin building a tool called SourcePuller, free software that could extract data from BitKeeper repositories without requiring a BitKeeper license. His goal was not to clone BitKeeper. It was to give kernel developers access to the kernel's own history without the strings attached.

But to Larry McVoy, SourcePuller was not a noble interoperability project. It was an attack on his business. BitKeeper's value depended on being the only way to access the data it managed. If anyone could extract that data with a free tool, the incentive to buy commercial licenses disappeared. McVoy had built something real, spent years of his life and his own money on it, and given it away for free to the very community that was now trying to replicate it. From his perspective, Tridgell was not liberating data. He was pulling the rug out from under the person who had laid it down for them.

The Licenses Disappear

McVoy learned about Tridgell's work around February twenty-third, two thousand five. His reaction was immediate and unequivocal. This was a violation of the spirit, if not the letter, of the arrangement between BitMover and the kernel community. McVoy had given free licenses with the explicit understanding that nobody would reverse-engineer the protocol or build competing tools. Tridgell was doing both.

What followed was weeks of tense negotiation behind the scenes. Linus tried to mediate. He and McVoy were friends, or at least friendly colleagues. They had both been part of the kernel community for years. They both worked in the orbit of OSDL. But McVoy was not interested in compromise. He wanted Tridgell to stop. Specifically, according to multiple accounts, McVoy demanded that OSDL fire Tridgell. OSDL refused.

I'm far from blameless, but the majority of this problem is an open source community problem. They simply do not want to play with non-open source.

On April sixth, two thousand five, Linus posted a message to the Linux Kernel Mailing List with the subject line Kernel SCM saga. The tone was resigned, even wry. He acknowledged what everyone already suspected.

Ok, as a number of people are already aware, we've been trying to work out a conflict over BK usage over the last month or two, and it feels like longer. Right now the only real thing that has happened is that I've decided to not use BK mainly because I need to figure out the alternatives, and rather than continuing things as normal, I decided to bite the bullet and just see what life without BK looks like. So far it's a gray and bleak world.

A gray and bleak world. From the person who had built the most successful open source project in history. That sentence tells you everything about how good BitKeeper had been and how devastating its loss was.

The same day, BitMover formally announced that it would stop providing the free version of BitKeeper to the open source community. The deadline for the transition was July first, two thousand five. After that date, if you wanted to use BitKeeper, you had to buy a commercial license. Three years of free access, three years of building workflows and infrastructure and muscle memory around a tool, over. The announcement hit Slashdot within hours. No More BitKeeper Linux, read the headline. The comments section erupted with hundreds of developers arguing about who was to blame, whether Tridgell had been reckless or heroic, whether McVoy had been generous or calculating, and what on earth the kernel was going to do now.

Three days later, a second Slashdot story appeared: Linus Drops BitKeeper. By then, Linus had already started coding. The open source community was still debating. Linus was building.

Linus was furious, but not at McVoy. He was furious at Tridgell. In a post to the Real World Technologies forum titled Hypocrisy the Worst of Human Traits, Linus tore into the man who had pulled the thread.

He screwed people over. He tore down something new and impressive because he could. He didn't write a better source code management system than BitKeeper. He didn't even try. That wasn't his goal.

The accusation was that Tridgell had not been trying to improve the world. He had just been trying to undermine BitKeeper, and in doing so, he had destroyed the tool that thousands of kernel developers depended on. Tridgell, in Linus's telling, was a vandal, not a liberator.

But others pointed out the obvious contradiction. Tridgell's method for analyzing BitKeeper's protocol was identical to his method for building Samba. Connect to a network service. Observe the protocol. Build an open implementation. When Tridgell did this to Microsoft, the open source world called him a hero. When he did it to BitMover, Linus called him a saboteur. The technique had not changed. Only the target had.

And beneath the personal drama, a deeper question loomed. If the kernel's history, the complete record of the most important open source project, could be held hostage by a licensing dispute, then something was fundamentally wrong with the arrangement. Not wrong with McVoy specifically. Wrong with the model. Any proprietary dependency, no matter how generous the terms, came with a kill switch that someone else controlled.

Linus Makes a List

The day after Linus's announcement, the Linux Kernel Mailing List erupted. Two hundred and five emails flew between April sixth and April twelfth, as the community tried to figure out what came next. Developers advocated for their favorite alternatives. Monotone. GNU Arch. Bazaar-ng. Darcs. Subversion. The creators of some of these tools jumped into the thread personally, pitching their projects.

Linus evaluated each one against a mental list of requirements that BitKeeper had taught him to demand. He needed speed above everything else. Not "fast enough." Not "acceptable performance." He needed operations that were essentially instantaneous. He gave a specific example that revealed just how far ahead BitKeeper had been: a source control system that takes thirty seconds to apply a patch and update its metadata is useless when you need to synchronize with a maintainer who has two hundred and fifty patches waiting. At thirty seconds per patch, that is over two hours of waiting just to catch up. BitKeeper handled this in seconds.

He needed a distributed architecture. Every developer must have a complete copy of the repository. No central server requirement. No single point of failure. Offline work had to be a first-class capability, not an afterthought.

He needed data integrity. Strong safeguards against corruption, whether accidental or malicious. If a bit flips on a hard drive, the system must detect it. If someone tampers with the history, the system must refuse to cooperate.

And he had one more requirement, less technical but equally firm. He looked at CVS and said: take it as an example of what not to do. If in doubt, make the exact opposite decision.

Take CVS as an example of what not to do. If in doubt, make the exact opposite decision.

On April eighth, two days into the mailing list discussion, Linus addressed the Monotone suggestion directly. Monotone was arguably the most promising existing tool. It was distributed, it used cryptographic hashes for integrity, and its design philosophy aligned with what the kernel needed. But Linus tested it and found it catastrophically slow.

In the meantime, and because Monotone really is that slow, here's a quick challenge for you, and any crazy hacker out there: if you want to play with something really nasty but also very, very fast...

That ellipsis was not a rhetorical pause. It was the moment Linus went from evaluating other people's tools to building his own. The challenge he issued was the first hint that something new was coming. Something nasty, in his word. Something very, very fast.

The Subversion suggestion was deflected so persistently that someone on the Apache Subversion team eventually published a page titled Please Stop Bugging Linus Torvalds About Subversion. Subversion was still centralized. It had improved on CVS in almost every practical way, as we heard in the last episode, but it had kept the cathedral architecture. One server, one truth. For the kernel, that was a non-starter. The lesson of BitKeeper was not just that distributed version control was nice to have. It was that centralized version control was a dependency on infrastructure you did not control. Whether that infrastructure was a company's server or a company's license, the risk was the same.

GNU Arch had the right philosophy, fully distributed, but its interface was notoriously hostile. Darcs, written in Haskell, was elegant in theory but could grind to a halt on complex merge histories. Bazaar-ng was promising but young. And every one of them was slower than what Linus had grown accustomed to with BitKeeper. BitKeeper processed patches in fractions of a second. The alternatives took seconds or tens of seconds per operation. When you are integrating thousands of patches from hundreds of developers, that difference is the difference between a workflow that works and one that does not.

The irony was not lost on the community. Linus's willingness to use a proprietary tool had set his standards so high that no free alternative could meet them. BitKeeper had shown him what version control should feel like, and nothing else came close.

The Sunday That Changed Everything

April third, two thousand five was a Sunday. Three days before the formal announcement on the mailing list, before the two hundred and five emails, before the arguments about Monotone and Darcs and GNU Arch, Linus had already made his decision. He released Linux kernel version two-point-six-point-twelve-release-candidate-two, the last kernel release that would ever be managed without Git, and he sat down at his computer in Portland, Oregon.

What happened next is the subject of the next episode, a story of two weeks that changed the entire software industry. But let me tell you what Linus was carrying into that Sunday.

He was carrying a decade of watching email patches get lost in his inbox. He was carrying the memory of how BitKeeper had transformed kernel development and the fury that it had been taken away. He was carrying three years of experience with what a distributed version control system should feel like, the speed, the flexibility, the ability to work without asking anyone's permission. He was carrying a list of requirements that no existing tool could satisfy. And he was carrying the conviction, born from watching centralized systems fail and proprietary dependencies get pulled, that the only tool you can trust is one you own.

McVoy had given the kernel community something genuinely valuable. The free BitKeeper licenses were not a trap. They were a gift from someone who cared about the project and believed his tool could help. And it did help, enormously, for three years. But the gift came with conditions, and conditions can change. The generosity of a proprietary license is always conditional. It lasts exactly as long as the licensor wants it to last, and not one day longer.

This is the argument that proprietary versus open source advocates have been having since Richard Stallman wrote the GNU Manifesto in nineteen eighty-five. And the BitKeeper crisis is the most vivid illustration of why it matters. McVoy was not a villain. Tridgell was not a saboteur. Linus was not a hypocrite. They were three smart people caught in a structural tension that no amount of goodwill could resolve. When your critical infrastructure depends on someone else's permission, you are one disagreement away from a crisis. Not because the other person is malicious. Because that is how property works. The owner decides.

The open source community talks about freedom, and critics often dismiss that as ideology. But the BitKeeper crisis was not ideology. It was practical. Thousands of developers, millions of lines of code, an entire workflow, yanked away because of a licensing dispute between two people who were technically colleagues. If that does not make the case for owning your tools, nothing does.

There is one more detail worth knowing. Eleven years after the crisis, in May two thousand sixteen, BitMover quietly released BitKeeper as open source software under the Apache license. By then, Git had conquered the world so completely that the gesture was more epitaph than event. The tool that had been too proprietary for the kernel community to keep was now free, in both senses of the word, and nobody needed it anymore. The Slashdot headline captured the mood: Eleven Years After Git, BitKeeper Is Open-Sourced. The top comment was a single word. Ironic.

On that Sunday in April two thousand five, Linus did not know any of that. He did not know he was about to build the most widely used version control system in the world. He just knew he needed something fast, something distributed, something that nobody could take away from him. He sat down and started typing.

What he built in the next two weeks would change software forever. That story picks up in the next episode.

The Linux kernel was built with two commands long before Git existed: diff and patch. diff old-file new-file, and out comes a precise list of what changed, line by line. Take that output, feed it to patch, and the changes get applied. That is the entire workflow. Linus received thousands of these diffs by email, read them with his own eyes, and applied them one at a time. It sounds primitive because it is primitive. But diff and patch are still here. When you run git diff today, you are looking at the same format, the same plus signs and minus signs, that powered kernel development by email for a decade. Git did not replace them. It wrapped them in something smarter and kept them running underneath.