A Week with Docker

This is a bit of a brain dump rather than a fully polished and edited post, in the unlikely event it gets any significant viewership I’ll try to clean it up.

It is some initial thoughts and experiences with using Docker for about a week. Specifically I was getting JIRA up and running in Docker on Rackspace for my own use (and it directly receives crash reports from some of my iOS apps). I also used it on my home server to set up a build environment for a JIRA plugin.

Update - Rackspace Cloud Servers Advice

Rackspace Cloud Servers are currently configured with no swapfile. I strongly recommend following the instructions to create a swapfile otherwise once you hit the limit of your server memory it will kill a process (probably your biggest which could well be Java).

General Docker Thoughts

Docker is nice, new containers spin up quickly, the documentation is OK. It is still pre-1.0 so moving fast and much of the online articles about it are out of date already. This isn’t really the first place you should look for information so I’m generally not going to explain syntax or concepts except to give my understanding of the difference between images and containers.

Containers

A container is a runnable environment that you can think of as a lightweight virtual machine. You will normally not run any services or daemons in the container but instead invoke a single command within it (which may be to run a server that would often be run as a daemon). Containers may be running or stopped.

Images

These are frozen checkpoints of containers from which NEW containers may be created. Images may be checked into repositories. For me some of the documentation and examples has a little too much focus on images rather than the containers themselves.

Commands

docker images

This lists the images (optionally in a tree structure).

docker ps

This lists the RUNNING containers unless you add the -a argument to see them all. For starting out it would be nice if docker containers was an alias for docker ps -a as it wasn’t initially obvious how to see stopped containers..

Name your containers

For your sanity I suggest naming your containers from the very beginning. This makes, starting, stopping, deleting and attaching to them much easier. Names are automatically generated but they are a bit long for convenience. When you start combining containers names are essential.

You can’t rename your containers

At least I don’t think so yet. I believe at the moment to rename a container you have to make an image of it (docker commit), then run the image with the name you want it to have (after deleting any existing containers with that name).

Docker on Debian

I think to do it simply you need to be running Jessie (current Unstable) as the Sid kernel is too old according to the Docker website. Jessie seems to run fine. This was an issue on my home server as I’d just migrated to Debian from Ubuntu.

Disk space

You can quite quickly fill up a number of GB of space. The default location is in /var (/var/lib/docker). My initial partitioning didn’t have a very large /var. So I created a new partition for docker using lvm, moved the contents of the docker folder over and set up a symlink. A settings change would have been an alternative approach. Again this was an issue on my home

Docker files

I haven’t yet created Docker files and have done experimentation from the commandline inside the containers to get it working. Now I have things working I’ll try to get it set up for easy rebuilding.

The JIRA server setup

Three containers:

  1. Postgresql database exposing port 5432 only internally within the docker host.
  2. JIRA host itself exposing 8080.
  3. Apache2 server to run in front as the SSL endpoint rather than have JIRA itself handling the SSL.

Postgresql

This host is fairly vanilla with not much custom setup using Postgres 9.3 from the Postgres apt packages. Created a jira database and user. This can definitely be shared by other containers that required databases. I still need to set up DB backups and will probably create a separate container on which can be started by the host’s cron to perform DB backups.

JIRA

I’ve currently got the container running with a hardcoded database IP address rather than using the environment variables created by Docker as the dbsetup.xml file for Docker doesn’t seem to be process environment variables. My next step will be create a script to start the JIRA process that will initially write a new dbsetup.xml file with the correct current IP address and port for the database. I also discovered that if you use the JIRA installer then the Tomcat server is included so you don’t need to install the Tomcat packages from Ubuntu. The setup I have is customised slightly so that it works behind the Apache SSL termination.

The default configuration for JIRA uses too much memory for my small virtual host and a single user (plus crashreporting from my iOS apps - when things are going badly I can get several crashes a day so the load is very small). Reducing the memory available dramatically improved the performance and there is more tweaking still to be done as I think I can still squeeze it significantly smaller.

Apache2

Obviously needs the appropriate SSL certificates and also needs a number of mods enabling which isn’t too difficult. The config files can pick up the environment variables to forward the requests to the JIRA server. The environment variables should be referenced as ${VARIABLE_NAME}

I also need to serve some static files (non-SSL and a different host) and I will probably add this to the same container as I am running on a fairly small virtual host and two Apache instances seems wasteful.

Update - Investigating why the JIRA container was being killed

After running for some hours the JIRA container was dying. docker logs containerName had a last entry of "Killed". Fortunately docker events -since 'YYYY-MM-DD` lists the times of container starts stops and deaths. From this I could see the time Jira was killed. Searching for that time in the host /var/log/kern.log revealed that the kernel had killed the Java process running Jira due to running out of memory. This is what led me to realise the need to add swap on my cloud server. It would have been harder to find without quick access to the time of the process stop event.

I did try to use the docker option to limit the memory available to the Jira container but I think that caused Java to be killed when that limit had been reached so possibly useful for protecting the host but it doesn't help keep the process running within the limits.

JIRA Build Environment

I rarely use Java so don’t have it installed on most of my machines but I wanted to build this version of the JIRA Connect Mobile plugin as it seemed to have some improvements over Atlassian’s own version in terms of grouping crash reports. This would obviously require Maven and Java and the JIRA plugin development library. As I don’t regularly use Java I preferred it not to insert itself irremovably into my system so I decided to install and build it within a container.

This worked really well and I will use docker for other occasional use build environments. I didn't set up the test server but made the resulting binaries available on bitbucket which I then installed on my containerised Jira server and tested there.