Running Docker Containers with Systemd

You can get by running Docker containers with shell scripts, or with Docker Compose (if you don’t mind ignoring the “don’t use in production” warnings), but for some use cases, it’s preferable to take advantage of the host init system/process manager. It seems that every major distro is moving to systemd these days, so that’s what I’ll look at in this post.

Using systemd or an equivalent is particularly useful if you have another, possibly non-containerized service that is dependent on the container. However, even developers of pure container applications may find advantages in using systemd and it’s worth noting that CoreOS is built around systemd and Docker.

If you follow the official Docker documentation for using systemd, you’ll see that they advise creating the containers manually with docker create and only using docker start and docker stop in the service file. I’m not a huge fan of this advice, as it makes it more difficult to migrate the setup between hosts or to restart the service with a fresh container — it would be better if the service file included all its dependencies. This is the approach taken by CoreOS, and the one I want to show in this blog post.

As an example, we’ll consider systemdizing a dockerized redis. I’ll be using a CentOS 7 distro for this, but it should be very similar on other systemd distros. Pretty much all you need is the following service file:

There’s a few things worth pointing out:

  • The container is clearly dependent on having Docker running, hence the Requires line. The After line is also needed to avoid race conditions.
  • Before we start the container, we first stop and remove any existing container with the same name and then pull the latest version of the image. The “-” at the start means systemd won’t abort if the command fails.
  • This means that our container will be started from scratch each time. If you want to persist data then you’ll need to do something with volumes or volume containers, or change the code to restart the old container if it exists.
  • We’ve used TimeoutStartSec=0 to turn off timeouts, as the docker pull may take a while.

If you save this file to /etc/systemd/system/docker.redis.service and run systemctl start docker.redis, systemd will start up the Redis container (remember that this may take some time if it needs to pull the redis image). We can then access it manually, or set up another service that is dependent on it. For example, if we have an application foo which is running in a container and dependent on the redis service, we can use the following service file:

Now if the redis container fails, systemd will automatically restart both the redis service and the dependent foo service.

This setup works pretty well most of the time. But there is a major problem. systemd isn’t monitoring the container itself, it’s really monitoring the client. If the client detaches from the container for whatever reason (e.g. a network problem), systemd will kill the container, even though it may be functioning fine. Conversely, if the container dies but the client remains running, systemd won’t do anything. What we really want is for systemd to monitor the container instead of the client1. And there is a solution that does just that, systemd-docker.

systemd-docker works by wrapping the docker command and moving the container process into the cgroup of the systemd service unit when it starts. Our redis example would look something like:

This is simpler, but one issue is that the systemd-docker command is hiding the fact that stopped containers with the same name will be removed. I had to add the argument --cgroups name=systemd to work around an issue with CentOS 7 and cgroups (other distros shouldn’t need this argument). If you do want to try systemd-docker, note that you currently have to build from source as the “go get” functionality is broken due to a docker dependency at the time of writing.

So, in short, if you want to use systemd, you need to think about it a bit more than perhaps was first obvious. If you’re just playing with some tooling and don’t need super-reliable up-time, the code in this blog post should work just fine. Otherwise, you should be using systemd-docker or addressing the container monitoring issue in some other way.

 


1.This situation is currently being worked and we are likely to see a solution that doesn’t require hacks or third-party tools soon. See the following issues for more info:

The following two tabs change content below.

Adrian Mouat

Adrian Mouat is Chief Scientist at Container Solutions and the author of the O'Reilly book "Using Docker". He has been a professional software developer for over 10 years, working on a wide range of projects from small webapps to large data mining platforms.

Latest posts by Adrian Mouat (see all)

16 Comments

    • Hi Timur,

      You’re definitely correct that it would be worth revisiting this. I’m not sure I have time at the minute, but it might overlap with some stuff I’m planning to look into soon.

      Thanks for your comment.

      Adrian.

  1. Just going to mention a pitfall related to this topic. You *must* run the container in non-daemonized mode. If you run your docker run command in daemonized mode (docker run -d ), systemd will keep on restarting the container concluding that the service crashed.

    • Yes, and this leads back to my points in the article about systemd monitoring the client rather than the server. Nowadays, it would be worth looking at runc or rkt, both of which I believe can solve this problem.

  2. There is a bug in your post. The following lines…

    > ExecStartPre=-/usr/bin/docker stop redis
    > ExecStartPre=-/usr/bin/docker rm redis
    > ExecStartPre=/usr/bin/docker pull redis
    > ExecStart=/usr/bin/docker run –rm –name %n redis

    …should read as:

    ExecStartPre=-/usr/bin/docker stop %n
    ExecStartPre=-/usr/bin/docker rm %n
    ExecStartPre=/usr/bin/docker pull %n
    ExecStart=/usr/bin/docker run –rm –name %n redis

    • C’n’P glitch on my side as well 😉

      …should read as:

      ExecStartPre=-/usr/bin/docker stop %n
      ExecStartPre=-/usr/bin/docker rm %n
      ExecStartPre=/usr/bin/docker pull redis
      ExecStart=/usr/bin/docker run –rm –name %n redis

    • I’ve updated the post to use %n, but I think it worked with the previous code as long as you called the file redis.service.

  3. Hi,

    I need your help in running the containers with systemd service files.

    I have created a simple systemd service file lxcsystem.service where I have the following content

    [Unit]
    Description=service for container process
    After=network.target

    [Service]
    Type=oneshot
    ExecStart=/bin/sh /lib/systemd/system/lxccontainers.sh
    ExecStop=/usr/bin/lxc-stop -n Cont1

    [Install]
    WantedBy=multi-user.target

    and my bash script file lxccontainers has the following content.

    #! /bin/sh
    echo “Creating an LXC Container”
    sleep 1
    lxc-create -n Cont1 -t busybox
    echo “Starting an LXC Container”
    lxc-start -n Cont1
    sleep 5

    Its a very simple service and script file, where in the LXC container gets created and started through the .service file. When I run this, I can see the container gets created and starts. But with in few seconds, when the .service file gets deactivated, the lxc container also gets into stop mode which is not my requirement.

    My main aim is create an lxc container, start and attach to a sample process on a platform. As the container goes into stop state as soon as the .service file gets deactivated, I am not able to attach a process to container correctly. I tried with the Type of service file with forking, simple and I see different error at different times. I am not sure whether my approach itself is correct or not.

    Do you have any idea why container gets into stopped state in this scenario?

    Thanks
    Sreeni.

  4. I found that by adding

    RemainAfterExit=true

    Fixed my issue of systemd stopping the container.

    ExecStart=/usr/bin/docker run -it –name mysql-server -v /u01/data/mysql:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=teampw -d mysql:latest
    ExecStop=/usr/bin/docker stop -t 2 mysql-server
    ExecStopPost=/usr/bin/docker rm -f mysql-server

Leave a Reply

Your email address will not be published. Required fields are marked *