How Protocol Buffers are Used in Mesos Framework Development



Apache Mesos

In this blog I will discuss the role of protocol buffers in Mesos framework development. Protocol buffers are used extensively for messaging and serialization inside Mesos and when developing Mesos frameworks. I will show how to use its APIs when developing a framework using examples from the Mesos Elasticsearch project.

Message Serialization

Like any distributed system, the components of a Mesos framework have to periodically send messages to each other. For instance, a scheduler sends a message to the executor telling it to use 2 CPUs and 1 GB of memory. Another example is an executor that sends back a ‘TASK_RUNNING’ status to the scheduler. Because message passing is so essential in building distributed systems, it is important to use a serialization system to program and deliver messages in a flexible manner. Mesos has chosen to use protocol buffers to solve this problem. There are many other serialization formats with trade-offs between certain features and performance characteristics.

Protocol buffers

Protocol buffers were created by Google back in 2008 for developers to define their own protocols and message formats. Protocol buffers generate code in C++, Java and Python. Mesos allows developers to create frameworks in several languages and use the generated protocol buffer APIs for that language. For instance, the Mesos Elasticsearch framework is written in Java and in it we use the generated Protos API classes. Let me show you an example of how to use these classes in the context of developing the Mesos Elasticsearch framework and afterwards I will show how protocol buffers work inside Mesos itself.

Example – Requesting Resources

Checkout out this snippet from the Mesos Elasticsearch scheduler: the registered  method. This method is called whenever a framework successfully registered with the Mesos master. It received several Protos  objects, the frameworkID  and the masterInfo. Inside the method we build up a list of resources and request them via the SchedulerDriver .

When you look at the request object you see a typical use of the protocol buffers APIs. It starts by creating a builder for one of the inner classes under the Protos  class using newBuilder . After the resources have been added the build  method is called to construct the request. Now let’s look at how the resources are created. The buildResources method delegates to a helper class Resources  which builds the resources. See the snippet below.

This class has factory methods for creating CPU, disk, memory and port ranges resources. It uses the same builder pattern as described earlier. Note that many Protos types are complex nested structures. For instance the Scalar type is used inside the CPUs, disk and memory resources.

Mesos & Protocol Buffers

Generating protocol buffer code works by defining a .proto file and running the protoc compiler. Take a look at include/mesos.proto in the Mesos sources for the definition of the Resource type. Below is part of the definition of the Resource message. It contains several fields. A resource has a required type and it can contain different values: a scalar, ranges or a set. These fields are marked optional to support these different types. This is the ‘union trick’ that is mentioned in the comment. For more info see the protocol buffer techniques page from Google.

The Resource type also contains nested messages. For brevity I show one them here: ReservationInfo. This message allows a framework to reserve resources by role and is currently being developed under MESOS-2018 – Dynamic reservation. The code is generated through Mesos’ build process which is based on autotools. contains definitions for protocol files as well as the commands needed to generated the code.

We’d Like to Hear From You!

Hopefully this blog gives you a good overview of using protocol buffers in the context of Mesos frameworks development. We would like to hear from your experiences with using Mesos, Mesos frameworks, Docker or related technologies. Also if you have any questions, a problem you want to solve, leave a comment below, talk to me on Twitter at @Frank_Scholten or contact us.

Leave a Reply

Your email address will not be published. Required fields are marked *