read

In this blog post I’ll walk you through a Docker container setup orchestrated with Salt. This post is inspired by a post from Erik Kristensen and Dave Boucha’s video on Managing Docker Containers at Scale. I use Salt heavily both at work and on other projects for remote execution (patching critical bugs like shellshock) as well as for configuration management. It’s a great tool with so many usage scenarios, you can fix almost any issue with a salt state.

Salt is an open source configuration management and remote execution application. Salt is written with the intent of making central system management and configuration as simple, yet as flexible as possible.” Wikipedia

Docker and it’s now famous containers has had an uprise like no other in DevOps community. I started using Docker at version 0.5 in August of 2013, and I instantly fell in love. Easy to use, reproducibility, speed of development and the Dockerfiles and build process

Docker is an open-source project that automates the deployment of applications inside software containers, by providing an additional layer of abstraction and automation of operating system–level virtualization on Linux. Wikipedia

Both Docker and Salt are extremely interesting to use when running bare-metal servers as well as cloud services. Salt is provisioning and providing base systems and Docker is containing each needed application to provide the best possible services.

Let’s get down to business!

  • Goal: To deploy containers on as many hosts as needed, and as many containers on each host as needed.
  • Not covered: Building docker containers. I usually do it with Jenkins and pushing to either a private repository or using Docker Hub.

Salt has a system of state modules and execution module. States are a representation of how minions should look. Execution module is commands sent to minions for execution but can’t be called from within a state. Find out more in the salt docs.

Jekyll rendering is playing tricks on me, so please check out state files at Github.

1. Setting variables

Start with setting a number of variables which will be used later. As you can see I’m using both pillars, grains and strings. This will make the state file easier to reuse. noofcontainers is used to set how many containers will be started.

{% set name           = 'node-demo'   %}
{% set registryname   = 'jacksoncage' %}
{% set tag            = salt['pillar.get']('imagetag', "latest") %}
{% set containerid    = salt['grains.get']('id') %}
{% set hostport       = '808' %}
{% set hostip         = grains['ip_interfaces']['eth0'][0] %}
{% set noofcontainers = range(10) ‰}

2. Pulling docker image (docker pull)

Using a nodejs demo application as image, and we need to pull it down from Docker Hub.

{{ name }}-image:
  docker.pulled:
    - name: {{ registryname }}/{{ name }}
    - tag: {{ tag }}
    - force: True

3. Stop container if newer image exists (docker stop)

We already pulled down the latest image, we are now looking into the already running container to see if there is a newer image downloaded. If there is a newer we stop the old container.

{{ name }}-stop-if-old-{{ no }}:
  cmd.run:
    - name: docker stop {{ containerid }}-{{ name }}-{{ no }}
    - unless: docker inspect --format '{{ .Image }}' {{ containerid }}-{{ name }}-{{ nr }} | grep $(docker images --no-trunc | grep "{{ registryname }}/{{ name }}" | awk '{ print $3 }')
    - require:
      - docker: {{ name }}-image

4. Removing container if newer image exists (docker rm)

Same as before, remove after stopped.

{{ name }}-remove-if-old-{{ no }}:
  cmd.run:
    - name: docker rm {{ containerid }}-{{ name }}-{{ no }}
    - unless: docker inspect --format '{{ .Image }}' {{ containerid }}-{{ name }}-{{ nr }} | grep $(docker images --no-trunc | grep "{{ registryname }}/{{ name }}" | awk '{ print $3 }')
    - require:
      - cmd: {{ name }}-stop-if-old-{{ no }}

5. Starting container (docker run/docker.installed)

Installing container by invoking docker.installed. Also setting hostname, port and environments variables.

{{ name }}-container-{{ no }}:
  docker.installed:
    - name: {{ containerid }}-{{ name }}-{{ no }}
    - hostname: {{ containerid }}-{{ name }}-{{ no }}
    - image: {{ registryname }}/{{ name }}:{{ tag }}
    - ports:
        - "8080/tcp"
    - environment:
        - EXECUTER: "forever -f start"
        - APP: "index.js"
    - require_in: {{ name }}-{{ no }}
    - require:
      - docker: {{ name }}-image

6. Keeping container running (docker run/docker.running)

Starting container by invoking docker.running. This also sets port binding on host.

{{ name }}-{{ no }}:
  docker.running:
    - container: {{ containerid }}-{{ name }}-{{ no }}
    - port_bindings:
        "8080/tcp":
            HostIp: "{{ hostip }}"
            HostPort: "{{ hostport }}{{ no }}"

Complete state file and top file

/srv/salt/containers/applications/node-demo.sls

{% set name           = 'node-demo'   %}
{% set registryname   = 'jacksoncage' %}
{% set tag            = salt['pillar.get']('imagetag', "latest") %}
{% set containerid    = salt['grains.get']('id') %}
{% set hostport       = '808' %}
{% set hostip         = grains['ip_interfaces']['eth0'][0] %}
{% set noofcontainers = range(5) ‰}

{{ name }}-image:
  docker.pulled:
    - name: {{ registryname }}/{{ name }}
    - tag: {{ tag }}
    - force: True

{% for no in noofcontainers %}
{{ name }}-stop-if-old-{{ no }}:
  cmd.run:
    - name: docker stop {{ containerid }}-{{ name }}-{{ no }}
    - unless: docker inspect --format '{{ .Image }}' {{ containerid }}-{{ name }}-{{ nr }} | grep $(docker images --no-trunc | grep "{{ registryname }}/{{ name }}" | awk '{ print $3 }')
    - require:
      - docker: {{ name }}-image

fetch_out_of_band:
module.run:
  - name: docker.stop
  - opts: 'timeout=20'

{{ name }}-remove-if-old-{{ no }}:
  cmd.run:
    - name: docker rm {{ containerid }}-{{ name }}-{{ no }}
    - unless: docker inspect --format '{{ .Image }}' {{ containerid }}-{{ name }}-{{ nr }} | grep $(docker images --no-trunc | grep "{{ registryname }}/{{ name }}" | awk '{ print $3 }')
    - require:
      - cmd: {{ name }}-stop-if-old-{{ no }}

{{ name }}-container-{{ no }}:
  docker.installed:
    - name: {{ containerid }}-{{ name }}-{{ no }}
    - hostname: {{ containerid }}-{{ name }}-{{ no }}
    - image: {{ registryname }}/{{ name }}:{{ tag }}
    - ports:
        - "8080/tcp"
    - environment:
        - EXECUTER: "node"
        - APP: "index.js"
    - require_in: {{ name }}-{{ no }}
    - require:
      - docker: {{ name }}-image

{{ name }}-{{ no }}:
  docker.running:
    - container: {{ containerid }}-{{ name }}-{{ no }}
    - port_bindings:
        "8080/tcp":
            HostIp: "{{ hostip }}"
            HostPort: "{{ hostport }}{{ no }}"
{%- endfor %}

/srv/salt/top.sls

base:
  'minion1':
    - containers.applications.node-demo

Deploying

Complete state can now be executed on needed minions.

salt minion1 state.sls containers.applications.node-demo

Salt minion will now run node-demo state file create all containers. I would then use something like haproxy or nginx to add all running containers to a hostname. Using the proxy approach there will be no downtime as each container will be stopped=>removed=>created=>started in sequence.

Conclusion

There are a couple of ugly workarounds here, some which could be removed with the next release of Salt codename Helium. As of Helium unless can be used in any state so cmd.run could be replaced with a docker module instead.

{{ name }}-stop-if-old-{{ nr }}:
  module.run:
    - name: docker.stop
    - opts: '{{ containerid }}-{{ name }}-{{ nr }} [timeout=30]'
    - unless: docker inspect --format '{{ .Image }}' {{ containerid }}-{{ name }}-{{ nr }} | grep $(docker images --no-trunc | grep "registry.bloglovin.com:5000/{{ name }}" | grep "{{ tag }}" | awk '{ print $3 }')
    - require:
      - docker: {{ name }}-image

{{ name }}-remove-if-old-{{ nr }}:
  module.run:
    - name: docker.kill
    - opts: '{{ containerid }}-{{ name }}-{{ nr }}'
    - unless: docker inspect --format '{{ .Image }}' {{ containerid }}-{{ name }}-{{ nr }} | grep $(docker images --no-trunc | grep "registry.bloglovin.com:5000/{{ name }}" | grep "{{ tag }}" | awk '{ print $3 }')
    - require:
      - module: {{ name }}-stop-if-old-{{ nr }}

Next blog post will go into Salt Reactor system based on the state file above.

References

Blog Logo

Love Billingskog Nyberg


Published

Image

jacksoncage

A blog about sysadmin, devops, automation, containers and awesomeness!

Back to Overview