Deploy Docker Containers on Mesos 0.20

With the release of Mesos 0.20.0 five days ago Docker became a first class citizen. Although not all features are supported yet Docker is now integrated into Mesos without having to use an external containerizer with Deimos.

For those that are interested in how the Docker Containerizer was integrated in Mesos 0.20.0 and some of the background on challenges encountered and future plans please read the Docker on Mesos 0.20 post on Timothy Chen’s site. There are also a couple documents posted to the Apache Mesos and Mesosphere Marathon github repositories that are enligthening: Docker Containerizer and Upgrading Marathon 0.6.x to 0.7.x.

Building on the Setup Standalone Mesos on Ubuntu post I will walk through installing Docker and running the Mesos Slave with the “docker” containerizer option. I assume that you already have a node with Mesos Master & Slave 0.20.0, Marathon 0.6.1, and Zookeeper 3.4.5 running installed using packages from the Mesosphere repository.

Install Docker

Each slave that has the Docker containerizer should have Docker installed (version >= 1.0.0). The latest version of Docker available in the Ubuntu repositories at the moment is 0.9.1, so we’ll want to grab a newer version from Docker’s repositories.

  1. Import the Docker Release Tool Key:
    $ sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9
  2. Add the Docker repository:
    $ echo "deb https://get.docker.io/ubuntu docker main" | \ sudo tee /etc/apt/sources.list.d/docker.list
  3. Download package lists and information of latest versions:
    $ sudo apt-get -y update
  4. Install the lxc-docker package:
    $ sudo apt-get install lxc-docker
  5. Verify Docker is working as expected by running the command below to download the libmesos/ubuntu image and start bash in a container:
    $ sudo docker run -i -t libmesos/ubuntu /bin/bash

Build Marathon from Source

The current packaged version of the Marathon framework, 0.6.1, needs to be updated to take advantage of the new integrated Docker containerizer. Until Marathon 0.7.0 is released the best way to experiment is build from the latest master on the Marathon Github repository.

  1. First we need to uninstall the current Marathon 0.6.1 package and all of its configuration files:
    $ sudo apt-get -y --purge remove marathon
  2. Install Open JDK 7:
    $ sudo apt-get install -y openjdk-7-jdk
  3. Install Git:
    $ sudo apt-get install -y git
  4. You’ll also need SBT (Simple Build Tool for Scala software) to build Marathon so install it from the Scala SBT site:
    $ cd /tmp $ wget http://dl.bintray.com/sbt/debian/sbt-0.13.5.deb $ sudo dpkg -i sbt-0.13.5.deb
  5. Clone the Marathon repository:
    $ cd /tmp $ git clone https://github.com/mesosphere/marathon.git
  6. Build Marathon:
    $ cd /tmp/marathon $ sbt assembly
  7. Install Marathon:
    $ sudo mv /tmp/marathon/ /opt/
  8. Use the start script to bring up Marathon in the background. The options –master and –zk identify the local Zookeeper that the Mesos Master and Slave are registered with:
    $ cd /opt/marathon $ sudo nohup ./bin/start --master zk://localhost:2181/mesos --zk zk://localhost:2181/marathon > /opt/marathon/nohup.log 2> /opt/marathon/nohup.log < /dev/null &

Enable Docker Containerizer on Mesos Slave

Each Mesos Slave must be launched with “docker” as one of the containerizers.

  1. Set the --containerizers=docker,mesos command line parameter:
    $ echo "docker,mesos" | sudo tee /etc/mesos-slave/containerizers
  2. Increase the executor timeout to 5 minutes 1:
    $ echo "5mins" | sudo tee /etc/mesos-slave/executor_registration_timeout
  3. Restart the Mesos Slave:
    $ sudo service mesos-slave restart

Verify Marathon is Functioning

You should be able to access the Marathon web console via http://nodeipaddress:8080/. The Marathon git repository we cloned contains some examples which we can use to verify Marathon is functioning properly and that applications can be created and run on a Mesos Slave. To keep things simple we’ll start a Python HTTP server.

  1. Start the Python Simple HTTP Server using the Marathon REST API:
    $ cd /opt/marathon/examples/ $ curl -i -H 'Content-Type: application/json' -d @PythonSimpleHTTPServer.json localhost:8080/v2/apps
  2. Use the REST API GET /v2/tasks call to list all running tasks and identify which port the Python Simple HTTP Server is running on. From the sample command output below you can see the port is 31907.
    $ curl -X GET -H "Content-Type: application/json" localhost:8080/v2/tasks | python -m json.tool { "tasks": [ { "appId": "/http", "host": "vagrant-ubuntu-trusty-64", "id": "http.0d974417-2d9d-11e4-a509-56847afe9799", "ports": [ 31907 ], "stagedAt": "2014-08-27T03:48:48.910Z", "startedAt": "2014-08-27T03:53:25.192Z", "version": "2014-08-27T03:48:46.778Z" } ] }
  3. If you use your browser to access http://nodeipaddress:31907 (substitute your node’s IP address and port identified in the previous REST call response) you should see output similar to the screen below.

http://frankhinek.com/wp-content/uploads/2014/08/2014-08-26_marathon-python-http.png

Deploy Docker Container with Marathon

To deploy this test Docker container we will POST a JSON task description to the Marathon instance running on our node.

  1. Create the Docker task description file under /tmp/Docker.json with the following content:
    { "container": { "type": "DOCKER", "docker": { "image": "libmesos/ubuntu" } }, "id": "ubuntu", "instances": "1", "cpus": "0.5", "mem": "128", "uris": [], "cmd": "while sleep 10; do date -u +%T; done" }
  2. Post the Docker.json file to Marathon using the following command:
    $ curl -X POST -H "Content-Type: application/json" http://localhost:8080/v2/apps -d@Docker.json
  3. Open the Marathon web console (http://nodeipaddress:8080/) you will see the Docker application running with an instance count of 0/1. Once the Docker image is pulled by the Mesos Slave the instance count should change to 1/1.
  4. We can also verify the container is running using the docker ps command on the Mesos Slave.

http://frankhinek.com/wp-content/uploads/2014/08/2014-08-26_docker-ps.png

Listed below is another JSON task description you can use for test purposes. You can access the Python3 web server (http://nodeipaddress:port) running in the Docker container using the port dynamically assigned when each instance starts.

  1. As explained by Connor Doyle, “The executorregistrationtimeout starts counting down when a containerizer starts launching a container. In the Docker containerizer, launch does 1) fetch files, 2) docker pull, 3) docker run, and 4) run the executor with docker wait. So, the reasoning behind increasing the registration timeout is mainly to allow time for pulling large docker images.” My thanks to Connor and Tim Chen for providing the background on this tweak.