At Container Solutions, we’ve been working hard on minimesos, our testing infrastructure for Mesos frameworks. To make it easier for frameworks to find their components, and for the Mesos master and agents to find Zookeeper, we’re adding service discovery.
Our first experiment uses Hashicorp’s Consul and Gliderlabs’ Registrator.
Consul is a great service discovery catalogue. We especially like the ability to query the catalogue over regular DNS. There’s a key/value store as well, it’s entirely distributed, and data center aware. And it can run in a container! This is great for our purpose.
Consul’s catalogue needs to be filled by some external tool. This is where Registrator comes in. Registrator is a Docker container that plugs into the Docker event stream (run `docker events` in one terminal, then start a container in another and see what happens!) Registrator inspects all started containers and registers any available mapped ports in Consul.
Now we can start any container, point its DNS server to Consul, and query the catalogue.
Here I am using Weave’s Scope to visualise this process.
On the right, there’s the combo of Zookeeper, Mesos master, and one Mesos agent. All the way on the left are Consul and Registrator. The Docker daemon is not pictured.
I’ve created a small Go application that returns the hostname of the container it’s running in on port 8080. Find the code at github.com/ContainerSolutions/hostnamer
This container is called `containersol/hostnamer` and is connected to `remmelt/debian`. This is a Debian Jessie container with some extra tools like dig, watch, and wget.
The hostnamer service is the service that wants to be found. We can start the service using Marathon, or by running the container directly.
The container that wants to find the hostnamer service, that wants to use Consul’s service discovery features, should be started with `--dns 172.17.0.1`. There’s the tricky question of how to discover the service discovery service itself, which we’ve sidestepped by exposing Consul’s DNS service on the `docker0` interface. Benefit is that we now know where to find Consul, downside is that we can only run one Consul at a time, which means we can only run one minimesos cluster at the same time. This will be fixed in a later version.
Now we can dig the hostnamer service from the Debian container:
`dig hostnamer.service.consul SRV` will return
```
;; ANSWER SECTION:
hostnamer.service.consul. 0 IN SRV 1 1 8080 f821e5c35fe1.node.dc1.consul.
;; ADDITIONAL SECTION:
f821e5c35fe1.node.dc1.consul. 0 IN A 172.17.0.2
```
Scaling up the service using Marathon, and re-running dig:
```
;; ANSWER SECTION:
hostnamer.service.consul. 0 IN SRV 1 1 8080 f821e5c35fe1.node.dc1.consul.
hostnamer.service.consul. 0 IN SRV 1 1 8080 f821e5c35fe1.node.dc1.consul.
hostnamer.service.consul. 0 IN SRV 1 1 8080 f821e5c35fe1.node.dc1.consul.
;; ADDITIONAL SECTION:
f821e5c35fe1.node.dc1.consul. 0 IN A 172.17.0.2
f821e5c35fe1.node.dc1.consul. 0 IN A 172.17.0.10
f821e5c35fe1.node.dc1.consul. 0 IN A 172.17.0.11
f821e5c35fe1.node.dc1.consul. 0 IN A 172.17.0.8
f821e5c35fe1.node.dc1.consul. 0 IN A 172.17.0.9
```
Consul returns the nodes in the answer section in random order, providing us with a rudimentary load balancer.
What we can also do from the Debian container is wget:
`wget -qO- hostnamer.service.consul:8080`
This will return a random node each time. We’ve successfully created a highly available service!
In the future, this will enable us to link different Zookeeper or Mesos master instances together, and load balance over them.
Previous article
