Mesos Graduates from Apache Incubation

Wednesday, 24 July 2013

The Apache Software Foundation (ASF) has announced the graduation of Apache Mesos, the open source cluster manager that is used and heavily supported by Twitter, from its incubator.

 Mesos, which was released last year, acts as a layer of abstraction between applications and pools of servers, helping avoid the necessity of creating separate clusters to run individual frameworks and instead making it possible to optimize how jobs are executed across shared machines.

What’s new

During incubation, Mesos had four releases and has continued to mature within Twitter’s production environment. Some of its improvements include: increased flexibility to support new application frameworks in several programming languages and cgroups isolation, in order to decrease interference across tasks and guarantee cpu/memory availability.

The Mesos community now includes new core committers, an active developer mailing list where future releases and patches are discussed, and increasing interest in running additional frameworks on Mesos. Currently, Mesos can run multiple frameworks, including Apache Hadoop, MPI, Jenkins, Storm, and Spark as well as other applications and custom frameworks.

How Twitter uses Mesos

Twitter actively contributes to and uses Mesos –– it is the cornerstone of our elastic compute infrastructure. A number of key Twitter services run in production on Mesos including: analytics, typeahead, ads and more, and we rely on it to build all our new services. By using Mesos, these services can scale and leverage a shared pool of servers across data centers efficiently.

Furthermore, Mesos has transformed the way developers think about launching new services at Twitter. Instead of thinking about static machines, engineers think about resources like CPU, memory and disk. The result has been a reduction in the time between prototyping and launching new services efficiently running on Mesos.

Future Work

Graduation is a fantastic milestone, but only the beginning for Mesos. We look forward to continuing to help grow the Mesos community and fostering an ecosystem around the project.

In addition to continuing to contribute to the Mesos core, we are preparing to open source Aurora, a service scheduler used to schedule jobs onto Mesos within Twitter. Aurora provides many of the primitives that allow stateless services to quickly deploy and scale, while providing good assurances for fault tolerance. We hope Aurora will make it easier to adopt and use Mesos in the broader ecosystem.

How to get involved

Mesos is still a young project and there’s plenty of areas to contribute. If you’re interested in test driving Mesos, visit the recently-redesigned Mesos website where you’ll find a getting started guide, documentation, and community section. We look towards the community to help improving the documentation as we continue to grow.

You can also create and contribute to Mesos frameworks. For example, Airbnb built their distributed and fault-tolerant scheduler Chronos on top of Mesos. As a replacement for cron, Chronos is a great example of the types of frameworks we hope to see built by the community.

And, this Thursday, Twitter is hosting a meetup event focused on building and running frameworks on Mesos, featuring talks by Twitter’s Vinod Kone on “Building new frameworks: Jenkins as a Case Study” and Brenden Matthews from Airbnb, presenting “Hadoop on Mesos.” Videos from the meetup will be posted online after the event and future meetups are being planned.

Acknowledgements

Mesos was originally created at the University of California at Berkeley’s RAD Lab (the research center also responsible for the original development of Apache Spark). Apache Mesos contributors include Ali Ghodsi, Benjamin Hindman (@benh), Vinod Kone (@vinodkone), Ben Mahler (@bmahler), Andy Konwinski (@andykonwinski), Dave Lester (@davelester), Thomas Marshall, Brenden Matthews, Charles Reiss, Ross Allen, Chris Mattmann, Tom White, Brian McCallister and Matei Zaharia (@matei_zaharia).