MQTT: Oil, Satellites, and Your Router

Two Engineers and a Pipeline

In nineteen ninety-nine, somewhere in the American Midwest, two engineers had a problem that had nothing to do with the internet. Andy Stanford-Clark worked for IBM in England, based at the company's old Hursley Park laboratories in Hampshire. Arlen Nipper worked for a company called Arcom, later renamed Cirrus Link. Their client was Phillips sixty-six, the oil company, and the job was straightforward on paper: get sensor data from remote sections of oil pipeline back to headquarters.

The pipelines stretched across hundreds of miles of flat, empty land. There was no fibre optic cable running alongside them. There was no cellular coverage in most of these locations. The only option was satellite, and satellite bandwidth in the late nineties was brutally expensive. Every byte cost real money. The sensors out in the field were small, low-powered devices with almost no processing capacity. They needed to report pressure readings, flow rates, temperature, and they needed to do it reliably, even when the satellite link dropped out for minutes at a time.

Before these two got involved, the pipeline industry had hundreds of proprietary protocols, each one tied to a specific manufacturer. Modbus, Allen-Bradley, Honeywell, each vendor had their own way of moving data, and none of them talked to each other. Andy later described it as a tower of Babel. Every installation was a custom integration job. Every upgrade meant ripping out one vendor's protocol and bolting on another.

Andy and Arlen decided they would build something different. Not another proprietary protocol owned by one company, but an open specification that anyone could implement. Their design philosophy came down to three words: compact, simple, implementable. The protocol had to work on devices with almost no memory. It had to survive unreliable connections. And it had to be so straightforward that a single developer could read the entire specification in an afternoon and build a working implementation before the week was out.

The Publish-Subscribe Trick

What Andy and Arlen designed was built on an idea called publish and subscribe, which sounds academic but is actually beautifully intuitive. Think of it like a bulletin board in a village hall. Anyone can walk in and pin a note to a specific section of the board. Anyone else can walk in and say, I want to see everything posted in the weather section. The person posting the note does not need to know who will read it. The person reading does not need to know who posted it. They never have to meet, or even be in the building at the same time.

In MQTT, the bulletin board is called a broker. The sections are called topics. A sensor out on a pipeline might publish a message to a topic called pipeline slash north slash pressure. Any system that has subscribed to that topic gets the message. The sensor does not need to know the address of the monitoring system. The monitoring system does not need to know the address of the sensor. They just agree on the name of the topic, and the broker handles the rest.

This was a radical departure from how most industrial systems worked at the time. The traditional approach was called poll and response. A central computer would dial up each remote sensor in sequence, ask for its reading, wait for the answer, then move on to the next one. If you had three hundred sensors, you dialled three hundred times. If one sensor was slow to respond, everything behind it in the queue waited. It was like a teacher calling on every student by name to ask if they had a question, instead of just saying, raise your hand when you have something to say.

The publish-subscribe model turned this on its head. Sensors reported when they had something new. The broker sorted the messages and delivered them to whoever was listening. The whole thing was asynchronous, which means the sender and the receiver did not have to be active at the same time. If a subscriber went offline, the broker could hold messages and deliver them when the subscriber reconnected. If a publisher went offline, the broker could send a pre-arranged "last will" message to anyone who was listening, saying, this device has disconnected.

The Name That Stuck

The letters MQTT originally stood for MQ Telemetry Transport. The MQ part came from IBM's existing messaging product, MQ Series, which Andy had worked on before. The telemetry part was obvious, it was about remote measurement. But the name was always slightly misleading, because MQTT is not really a message queue. A queue implies messages lining up in order, waiting to be consumed one by one. MQTT is more like a broadcast, one message going out to everyone who cares about that topic, simultaneously.

Over the years, the community started treating MQTT as just a name, not an acronym. The official standards body, OASIS, eventually dropped the full expansion entirely. Today, MQTT is MQTT, the same way HTML stopped being "hypertext markup language" in most people's minds and just became the thing web pages are made of.

The smallest possible MQTT message is just two bytes. Two bytes. That is sixteen bits. For comparison, a single HTTP request to load a web page might be several hundred bytes before you even get to the content. This is why MQTT survived on satellite links where bandwidth cost real money. Every byte mattered, and Andy and Arlen squeezed the protocol down until there was nothing left to remove.

The Decade of Silence

Here is the part of the story that rarely gets told. After Andy and Arlen finished the specification in nineteen ninety-nine, almost nothing happened. The protocol worked. Phillips sixty-six used it. A few other oil and gas companies adopted it. But outside the industrial world, nobody cared. Andy and Arlen toured the conference circuit, doing what Andy later called "the Arlen and Andy show," travelling to oil and gas industry events trying to drum up interest. They had some modest success, but the snowball effect they hoped for never arrived.

Andy once joked that their modest plan for world domination was that one day, all devices would speak TCP/IP, and all those devices would speak MQTT to each other. In two thousand, that sounded like a fantasy. Broadband was still a luxury. The smartphone did not exist. The idea of connecting your thermostat to the internet would have sounded like a joke.

For almost a full decade, MQTT sat quietly in the industrial sector, doing its job on pipelines and substations and weather monitoring stations. The specification was stable, owned by IBM, and not particularly accessible to the wider developer community. Andy continued his work at IBM, eventually becoming the company's Chief Technology Officer for the UK and Ireland, working from a sixteenth-century cottage on the Isle of Wight. He kept one eye on MQTT, waiting for the world to catch up.

The Mosquitto Moment

The tipping point came in two thousand and eight, when an open-source MQTT broker called Mosquitto was released. Suddenly, anyone with a Linux machine could run their own MQTT broker for free. You did not need IBM's commercial infrastructure. You did not need a licence. You could install Mosquitto on a Raspberry Pi and start experimenting.

Then IBM made a move that changed everything. In two thousand and ten, they released the MQTT specification as royalty-free. Anyone could implement it without paying IBM a cent. The following year, they contributed MQTT client libraries to the Eclipse Foundation under a project called Paho, giving the developer community ready-made tools to build with.

The timing was extraordinary. The smartphone had arrived. Arduino boards were cheap and everywhere. Sensor hardware was getting smaller and cheaper by the month. People were starting to connect things to the internet that had never been connected before. Thermostats, light bulbs, door locks, weather stations, plant moisture sensors. The industry needed a lightweight protocol that could run on tiny devices with limited power and spotty connectivity. They needed exactly what Andy and Arlen had built for oil pipelines eleven years earlier.

By twenty fourteen, MQTT was an OASIS standard. By twenty sixteen, it was an ISO standard. Facebook chose it as the protocol underneath Messenger, because it was light enough to work on unreliable mobile connections in developing countries. Amazon Web Services built their IoT platform around it. Microsoft Azure did the same. The protocol that two engineers designed for measuring pressure in pipes was now carrying billions of chat messages and sensor readings every day.

The Anatomy of a Conversation

Let's slow down and look at what actually happens when a device sends an MQTT message, because this is the part that matters when you start building things yourself.

First, the device connects to the broker. This is a persistent TCP connection, not a one-off request like loading a web page. The device opens the connection once and keeps it alive, sending tiny keep-alive pings at regular intervals so the broker knows it is still there. This is important, because maintaining an open connection is much cheaper than opening and closing new connections for every message.

When the device has data to send, it publishes a message to a topic. Topics are hierarchical, separated by forward slashes. You might have home slash living room slash temperature, or factory slash line three slash motor two slash vibration. This hierarchy is not enforced by the protocol, it is just a convention, but it makes everything vastly easier to organise.

Subscribers can use wildcards to listen to multiple topics at once. A plus sign matches one level, so home slash plus slash temperature would catch temperature readings from every room. A hash sign matches everything below a certain point, so factory slash hash would give you every message from every sensor in the factory. This wildcard system is one of the things that makes MQTT so flexible for monitoring dashboards.

MQTT also has three quality of service levels. Level zero means fire and forget, the message is sent once and nobody checks if it arrived. Level one means the broker acknowledges receipt, and the sender will retry if it does not hear back. Level two is the full handshake, guaranteeing the message is delivered exactly once, no duplicates, no losses. Each level adds overhead, so you choose based on what you can afford to miss. A temperature reading every five seconds? Level zero is fine, you will get the next one. A security alert? Level two, you cannot afford to lose it.

Your Router Speaks This Language

Here is where the story circles back to something sitting on your desk. Your Teltonika RUTM fifty-one, the cellular router, has MQTT built in. It can publish telemetry data, signal strength, connected devices, data usage, temperature, to any MQTT broker you point it at. Out of the box, right now, with no code to write.

If you run Mosquitto on your Raspberry Pi, your router can start sending its status updates there every few seconds. Your pinkserver becomes the broker, the central bulletin board. From there, anything that subscribes to those topics gets the data. A Python script that logs it to PostgreSQL. A dashboard that shows real-time signal quality. An alert system that pings you when your cellular connection drops below a threshold.

This is how the invisible plumbing of a smart home or a smart office actually works. Not through some proprietary app, not through a cloud service that might shut down next year, but through an open protocol designed twenty-five years ago for oil pipelines, running on hardware you own, storing data in a database you control.

I really only foresaw that it would be useful in the SCADA world, but then we saw uses for it in other industries. Vehicle telematics, agriculture, healthcare. A whole lot of different industries.

Andy Stanford-Clark could not have imagined that his protocol would end up running on a cellular router in a house in Jämtland, feeding data into a local PostgreSQL database called Pärkit. But that is the beautiful thing about open protocols. They go where they are needed, and they outlive the intentions of their creators.

What This Opens Up

MQTT is not just a protocol for reading sensor values. It is a pattern for thinking about how devices communicate. Once you have a broker running, everything changes. Your BMW telemetry data, instead of going to a custom endpoint, could publish to MQTT topics. Your weather station, your air quality sensor, your printer, anything that can speak TCP can be taught to speak MQTT.

The real power is in the decoupling. The sensor does not know or care what happens to its data after it publishes. The dashboard does not know or care where the data comes from. You can add a new subscriber without touching the publisher. You can replace a sensor without reconfiguring the dashboard. Everything connects through topics, and topics are just names you choose.

Twenty-five years ago, two engineers built the lightest possible way to move data across the most expensive possible connection. The protocol they created was so well designed that it went a decade without needing a single change. Today it carries chat messages for billions of people and sensor readings for millions of devices. And tomorrow, it might be the backbone of a real-time dashboard for your local newspaper's readers, showing road conditions and weather and emergency alerts across an entire municipality, all flowing through a two-byte protocol invented for oil pipelines in the American Midwest.

That router on your shelf is already speaking the language. The question is what you want it to say. And the next question, the one we will pick up in the next episode, is how you make sure nobody else is listening when it does. Because once you start moving data between devices, you need a tunnel. And the story of how one security researcher built the most elegant tunnel ever made is almost too good to believe.