This week Netflix open sourced Maestro, under the Apache 2.0 license, at github.com/Netflix/maestro.
A correction up front, because the first version of this article got it wrong: Maestro isn't Netflix's recommendation engine or its content management system. It's a workflow orchestrator, the system that schedules and runs Netflix's data pipelines and machine learning workflows. Less glamorous, arguably more useful.
What a workflow orchestrator does
A data platform runs thousands of jobs that depend on each other. Ingest yesterday's viewing logs, then clean them, then compute aggregates, then retrain a model, then publish the results, and only start step four once steps one through three have finished successfully. Some jobs run hourly, some daily, some when a file arrives.
An orchestrator is the system that knows those dependencies, starts each job when its inputs are ready, retries failures, tracks what ran when and lets people see and fix what's stuck. Most data teams use one. Apache Airflow is the best-known open source example, with Dagster, Prefect, Argo Workflows and Temporal covering related ground.
What makes Maestro different
Netflix built Maestro to replace an older internal scheduler called Meson, which was struggling with the company's growth. According to Netflix, Maestro now runs thousands of workflows for hundreds of teams, averaging around half a million jobs a day and peaking near two million. A few things stand out:
- Scale by design. Maestro is horizontally scalable, with its state in a distributed database, so adding capacity means adding nodes.
- Not just DAGs. Most orchestrators only support directed acyclic graphs, pipelines that flow one way. Maestro also supports cyclic workflows and reusable patterns like
foreachloops, subworkflows and conditional branches, which matter for machine learning work like hyperparameter sweeps and backfills. - Several ways in. Workflows can be defined declaratively or programmatically and managed through an API, so teams don't all have to adopt one style.
- Jobs in any container. Steps can run notebooks, Docker images or arbitrary code, rather than being tied to one execution engine.
- Signals and events. Workflows can wait on and emit signals, so one team's pipeline can trigger another's when data is ready.
It's a Java service, and like most things Netflix runs, it's built with its own environment in mind. Running it outside Netflix will take some integration work.
Who should look at it
- Teams outgrowing Airflow, particularly at very high job volumes or with workflows that don't fit neatly into acyclic graphs.
- Platform teams building an internal data or ML platform who want an orchestrator designed for multi-tenant scale from the start.
- Anyone curious about how a large streaming company runs its data platform. The design documents and Netflix's tech blog posts about Maestro are worth reading even if you never deploy it.
For most small and mid-sized teams, Airflow's ecosystem, managed services and hiring pool will still make it the more practical choice.
Why Netflix gives this away
Netflix has a long open source track record: the Chaos Monkey and the Simian Army, Eureka, Hystrix, Zuul, Spinnaker, Titus, Metaflow and more. Its reasons, as it has explained them over the years, are practical. Open sourcing internal tools improves their quality, because engineers write better code for an outside audience. It attracts contributions and feedback. And it helps hiring, because engineers can see the kind of work they'd be doing.
There's also a less obvious benefit for everyone else. Netflix's business is streaming video, not selling workflow software, so it has no reason to relicense Maestro or turn it into a paid product later. Open source from companies whose product is something else tends to stay open. The Apache license, with its patent grant, makes that safer still.
If you run a data platform, Maestro is worth an afternoon of reading. It's rare to see the internals of a system running at this scale, and rarer still to be allowed to use it.