The Three Heretics: Prefect, Dagster, Temporal, and the Fight Against Airflow

The Mailbox Of Someone Else's Pain

For three years, Jeremiah Lowin sat on the project management committee of Apache Airflow. This is the body that governs an Apache project — the small group of senior contributors who decide what gets merged, what gets released, and what direction the codebase takes. Being on the committee meant Lowin received thousands of emails from people using Airflow in production. People who were stuck. People whose pipelines had failed at three in the morning. People who had tried every retry setting and still could not figure out why their data was wrong.

In isolation, every one of those emails looked unique. Each company had its own infrastructure, its own quirks, its own particular flavor of disaster. But Lowin had spent his earlier career as a risk manager at hedge funds, and risk managers are professionally trained to look for patterns hiding inside noise. In aggregate, the emails started telling a different story than any individual one of them did. The same universal problems were showing up over and over.

He tried, for a long time, to fix these problems inside Airflow. At some point, he concluded that they could not be fixed inside Airflow. That conclusion turned into a company. The company is called Prefect, and it is the first of three projects that, between roughly twenty eighteen and twenty twenty six, have tried to convince the data engineering profession that Airflow's foundational bets were wrong.

Negative Engineering

To understand what Prefect was actually trying to fix, you have to start with a concept Lowin coined in a blog post in twenty eighteen, which became the company's founding idea. He called it negative engineering.

Positive engineering is what we typically think engineers do. They write code to achieve an objective. Negative engineering is when engineers write defensive code to make sure the positive code actually runs.

The negative work is the retry logic, the timeout handlers, the dead-letter queues, the failure alerts, the deduplication checks, the state recovery code. The stuff you write to make sure the actual valuable code, the part that does the actual computation, does not silently break in production. Lowin's claim, based on those thousands of emails, was that engineers spend ninety percent of their time on negative engineering and roughly ten percent on the positive work they were actually hired to do. If you could move that ratio even slightly, by automating away some of the negative engineering, you would double the throughput of an entire profession.

His first attempt was a tool he wrote for his own use at the investment firm where he worked. He named it Tin Man, after the Wizard of Oz character, because its job was to perform repeated mechanical tasks. He had no intention of open-sourcing it. Then, in the fall of twenty seventeen, two engineers at other firms saw what he had built and said, in slightly different words, that they would pay for it if it were a product. So he made it a product. He brought in a colleague named Chris White from Capital One as his first engineer and co-founder. They released Prefect as open source in early twenty eighteen and got over five hundred GitHub stars in the first few days, against an internal expectation of about a hundred.

Dynamic Flows And Hybrid Execution

Where Airflow treated a pipeline as a static graph that you defined in advance, Prefect treated a pipeline as a Python function. You wrote a regular function. You decorated it with the word flow. The functions you called from inside that function were tasks. If your flow needed to branch based on runtime data, you used a regular Python if statement. If it needed to loop a variable number of times, you used a regular Python for loop. The graph was implicit in the code, not declared up front.

This was a real philosophical difference. Airflow's directed acyclic graph had to be known before the pipeline ran. Prefect's graph emerged as the pipeline executed. For static pipelines this made no difference. For dynamic ones — pipelines whose shape depended on the data they were processing — Prefect felt natural and Airflow felt like fighting the framework.

The second difference was hybrid execution. Prefect split the orchestration layer from the execution layer. The scheduling, the state tracking, the observability — all of that ran in Prefect's cloud service. Your actual code ran wherever you wanted it to. On your laptop. On your own Kubernetes cluster. Inside your own data center, behind your own firewall. Prefect Cloud never needed to see your data. It just needed to know whether each task had finished or failed.

For a startup with no platform team, this was enormous. You did not need to operate a scheduler. You did not need to maintain a database. You bought the orchestration as a service, and you only ran the parts that actually had to touch your data.

The GraphQL Co-Creator

While Lowin was building Prefect on the East Coast, a different engineer in San Francisco was reaching the same broad conclusion through a completely different door.

Nick Schrock had spent eight years at Facebook, where he founded a team called Product Infrastructure. That team's mission was to make Facebook's product engineers more productive by building better internal frameworks for them. The team's most famous external output was something called GraphQL — a query language for application programming interfaces that Facebook open-sourced in twenty fifteen and that has since become a major standard. Schrock was one of GraphQL's co-creators. Across the hall from him, a different group of engineers was building something called React, which became the most widely used user interface framework in the world.

When Schrock left Facebook in twenty seventeen, he asked himself a specific question. Frontend engineers had spent the last decade getting better and better tools — React, type systems, automated testing, hot reloading, a whole culture of treating user interface code as serious software. Data engineers had not had any of that. They were still writing pipelines that looked, structurally, like bash scripts from nineteen ninety-eight.

He founded a company called Elementl in twenty eighteen — the same year Lowin founded Prefect, by complete coincidence — and started building a project called Dagster. The company has since renamed itself Dagster Labs. The product is also called Dagster. Schrock himself once described the founding insight this way.

Data pipelines should follow software engineering best practices.

That sentence sounds bland. It is not bland. It is the seed of a fundamentally different way of thinking about pipelines.

Assets, Not Tasks

Where Airflow modeled pipelines as graphs of tasks, and Prefect modeled pipelines as Python functions, Dagster modeled pipelines as graphs of assets. An asset is a piece of data that exists in your system. A table in your warehouse. A model file in your storage bucket. A feature vector in your machine learning store. Each asset has an upstream parent, the thing it was computed from, and downstream children, the things computed from it.

[serious] This sounds like a small reframing. It is not. Once you describe your pipeline as a graph of data assets, the orchestrator can answer questions it could not answer before. If the upstream table did not change, the downstream table does not need to be rebuilt. If the model file is older than the feature data, the model needs retraining. If a user asks where a number on a dashboard came from, the system can walk back through the asset graph and show every transformation that produced it. The asset graph is also a lineage graph. The lineage graph is also a dependency graph. They are the same thing.

For teams that worked with dbt — a tool that builds data warehouse models — Dagster felt obvious. Dbt already thought in terms of models, which is essentially the same thing as assets. Dagster could read a dbt project and automatically build the asset graph from it. The marriage of dbt and Dagster became one of the standard stacks in modern analytics engineering. The asset model also turned out to be exactly the right abstraction for machine learning pipelines, where the inputs and outputs really are versioned datasets and versioned models rather than abstract tasks.

By early twenty twenty six, Dagster had around twelve thousand stars on GitHub, against Airflow's thirty-eight thousand. The community was smaller, the documentation was thinner, but the developer experience was visibly better, and the growth curve was steep.

The Outsider

The third heretic is a different kind of project entirely. It is not really a data pipeline tool at all. It is a general-purpose distributed execution platform that has, almost by accident, become a serious option for orchestration work.

The project is called Temporal. It was founded in twenty nineteen by Maxim Fateev and Samar Abbas. The two of them had been working on the same problem, in different forms, for a literal decade. They first met at Amazon Web Services in two thousand and nine, where they helped build the Simple Workflow Service, an early cloud product for coordinating asynchronous tasks. They parted ways for a few years — Abbas went to Microsoft and built something called Durable Task Framework, which eventually shipped as the engine behind Azure Durable Functions — and then they reunited at Uber, where they created an open source project called Cadence.

Cadence at Uber ran enormous workloads. It coordinated rides, it ran fraud detection chains, it handled refund flows. In twenty nineteen, Fateev and Abbas left Uber to found Temporal, which is essentially Cadence rewritten from scratch with the lessons of the previous decade baked in. Cadence is still maintained at Uber, but the broader community has consolidated around Temporal.

Durable Execution

What Temporal sells is a concept called durable execution. The core idea is that every step of your code is recorded as an event in a persistent event history. If a worker crashes halfway through a ten-step process, Temporal replays the event history on a fresh worker and resumes exactly where the old one left off. The code did not have to be designed to be resumable. The durability comes from the platform, not from the application.

The goal is a fault-oblivious stateful execution environment.

Translated out of the academic register, that means you write code as if failures simply did not exist, and the runtime handles every retry, every restart, every crash recovery behind your back. Your business logic does not have to think about machine failures at all. The platform takes care of it.

This is a categorically different bet than Airflow, Prefect, or Dagster are making. Those three are tools for moving data around on a schedule. Temporal is a tool for making any long-running process resilient — payment flows, user onboarding chains, multi-step business processes that span days or weeks, and increasingly, agent workflows powered by large language models. In a twenty twenty six talk, Fateev described durable execution as the natural fit for artificial intelligence agents, which can take minutes or hours to run, can wait on human approvals, and can fail in unpredictable ways. The same primitives that make payment chains reliable also make agent loops reliable.

Temporal is not the right tool for a nightly extract-transform-load pipeline. It is overkill for that. But for the kinds of long-running, branchy, unpredictable workflows that did not really exist as a category until recently, Temporal has built a comfortable lead. The company reported over three thousand paying customers in May of twenty twenty six and revealed serverless workers at its annual conference.

The Picture In Twenty Twenty Six

So where does all of this leave somebody picking an orchestrator today. The honest answer, which most experienced engineers will give you if you press them, is that the right choice depends almost entirely on what kind of work you are orchestrating.

For traditional scheduled data pipelines, with a wide range of integrations and a deep operational community, Apache Airflow remains the safest default. The three point zero rewrite addressed the worst of the old criticisms. Multi-team support in three point two solved the platform-engineering pain. Anything you might need to integrate with — Snowflake, BigQuery, Databricks, Kubernetes, Kafka, any major cloud — there is already a battle-tested operator for it. The community is the largest by a wide margin.

For analytics teams that have built their stack around dbt, that care about asset lineage, and that want pipelines to behave like well-engineered software, Dagster has become the strongest choice. The asset model is genuinely better for that work. The local development experience is meaningfully ahead.

For small teams that want orchestration with minimal operational overhead, especially teams that need dynamic pipelines or event-driven flows, Prefect remains the cleanest option. The hybrid execution model means you do not run a scheduler. The Python-native authoring style means a product engineer can write a flow without learning a new framework.

For long-running stateful processes that need to survive infrastructure failures — payment systems, fulfillment chains, agent loops, anything that runs for hours or days and absolutely cannot lose state — Temporal is in a category by itself. It is not really competing with the other three. It is solving a different problem that happens to look superficially similar.

What This Argument Was Really About

The interesting thing is that these four projects, taken together, represent four different answers to one underlying question. What is a pipeline, fundamentally. Airflow says a pipeline is a graph of tasks. Prefect says a pipeline is a Python function. Dagster says a pipeline is a graph of data assets. Temporal says a pipeline is a durable program.

Each of those answers is correct for some kinds of work and wrong for others. The argument between these four projects, which has played out in conference talks and blog posts and Twitter threads for the better part of a decade, is not really an argument about which one will win. It is an argument about whether there is one right abstraction for all kinds of orchestration work, or whether the problem space is actually four problem spaces in a trench coat. The answer, increasingly, looks like the second one.

Jeremiah Lowin started Prefect because he thought he could fix Airflow's problems from the outside. Nick Schrock started Dagster because he thought engineering hygiene was missing from the data world. Maxim Fateev started Temporal because he had been chasing the same idea — durable execution — across three different companies over a decade. None of them killed Airflow. None of them are going to. But together, they have permanently changed what the word orchestration means in twenty twenty six.

The directed acyclic graph that Maxime Beauchemin pushed in October twenty fourteen is still standing. It just no longer stands alone.