Lightning Image Deployment with ImageWolf

 

by Adrian Mouat

Modern web developers can push to staging or production multiple times a day. Each time this happens, there is a delay while images are distributed to the nodes, especially if the images need to be pulled from a remote registry. This can be a significant issue, forcing developers and ops to wait much longer than necessary while a deployment rolls out.

Container Solutions have recently been looking at one way to alleviate this problem, by using bittorrent to rapidly distribute images across the cluster. This work resulted in a proof of concept, known as ImageWolf.

ImageWolf is deployed to a cluster and will listen for pushes to a registry. When an image is pushed, it will pull the image and distribute to all other nodes in the cluster. When the image is subsequently run on the cluster, it should start almost instantaneously regardless of the node it is scheduled on.

The ImageWolf PoC runs on Docker Swarm and can integrate with both the Docker Hub and local registries. The ImageWolf image should run on both ARM and x86 architectures (ImageWolf was tested on a Raspberry Pi cluster). An example of deploying and running ImageWolf can be seen in the following video:

Please let us know if ImageWolf is of interest to you. Assuming there is enough interest we will produce a port for Kubernetes (hopefully straightforward) and start work on a production quality version.

The following two tabs change content below.

Adrian Mouat

Adrian Mouat is Chief Scientist at Container Solutions and the author of the O'Reilly book "Using Docker". He has been a professional software developer for over 10 years, working on a wide range of projects from small webapps to large data mining platforms.

Latest posts by Adrian Mouat (see all)

2 Comments

  1. This would work great for seeding our CI slave cluster since our build images (for ephemeral slave containers) tend to be very large and oftentimes slow to pull and extract (especially from ECR). So, when a new image is created, we want to auto-distribute to each node in the cluster.

    What about a port to Rancher, that’s our current preferred container management solution ?

Leave a Reply

Your email address will not be published. Required fields are marked *