Monday, September 5, 2016

Using Docker and a Private Registry with VPN On Windows

Wasn’t that a very specific title? Docker has a very good documentation and reading that alone is enough for most of the straightforward tasks we might want to do. But as always some practical tasks are not straightforward, hence this blog. What we are going to see here today is how to setup docker toolbox on a Windows machine, make it work even when VPN is connected, make it talk to a private, insecure docker registry (that is why VPN) and configure it so it can run docker compose and see how we can set this config as a one-time activity. That’s quite a mouthful, but yes this is what we are going to do. All ready? Let us begin then.

Install Docker Toolbox

Go and download the docker toolbox and install it. That should create a shortcut called “Docker Quickstart Terminal”. Run it. That should show you an error about virtualization.

Enable Virtualization

Restart your machine, enter the BIOS settings and enable virtualization. It may be under advanced settings. On this Laptop, it is under the advanced settings -> device configurations and is named as: “Virtualization Technology (VTx)”. Whatever be the name, enable it.
Docker requires a Linux kernel, and since Windows machines lack it (of course!), docker toolbox runs a lightweight Linux distro called boot2docker in a virtualbox, hence the virtualization setting.

A Handy Tip

This tutorial will require you to copy and paste quite some shell commands, it is better we make that easy. Exit the quickstart terminal. Right click the shortcut, click properties -> options and enable ‘Quick Edit’ mode and save. It might ask for permission. This should now enable paste just by right clicking the mouse, to copy just select the text with mouse. While we are at that, also consider increasing the buffer and window size to suite your taste.

Start Up the VM

Make sure you are not connected to VPN and use the Quickstart Terminal shortcut again, this time it should proceed to validate if the boot2docker image is latest, or it shall pull the latest image, then it shall create a VM, get an IP, setup some ssh keys and finally the whale should appear with a terminal. Run the following commands to get a hang of docker running on windows:

docker -v
docker version
then docker run hello-world
docker images
docker ps -a

(And do read the output of hello-world, it describes how docker works).

The Disappointment

Feeling happy? Now for a little disappointment, connect VPN and try again. Errors errors everywhere. Disconnect VPN. What happened: Docker is running in a virtualbox on your machine, which gets an IP in local range (normally:, and you are talking to it over ssh. Once VPN is up, it sets the new routes and sends the 192.168.* range traffic out over VPN and your commands never reach your VM running docker. The most popular solution to this is setting a port forwarding and is documented on many blogs/github issues. Let’s just do that.

A new Beginning

Ensure you are not on VPN and remove the default VM, not necessary, but reduces the confusion. So in the quickstart terminal:

docker-machine rm default

And confirm. We are now going to create a new VM, let us call it ‘custom’. So type in:

docker-machine create -d virtualbox custom
eval "$(docker-machine env custom)"

It might take a couple of minutes, it is almost the same process as the first time. What we did is created a VM named custom and setup the environment to talk to this VM instead of the default. Mark this step, cause if anything goes wrong in the following steps, this is the one you should get back to to start over. Just be sure to use a new name, docker currently does not allow reusing names for VMs, so next time you may not be able to create a VM called custom. A new name should work just fine.

Battling With VPN

Now we shall create the a port forwarding on the virtual machine, binding the default docker port (2376) on localhost/ to forward to this VM, whatever the ip of it.

docker-machine stop custom
/c/Program\ Files/Oracle/VirtualBox/VBoxManage.exe modifyvm "custom" --natpf1 "docker,tcp,,2376,,2376"
docker-machine start custom
docker ps -a

If you changed the location of virtualbox installation, please use appropriate path to vboxmanage. Assuming it was successful, last command should show you a table with all containers. You can use UI to do that as well: Open VirtualBox, stop the VM, open settings -> network -> NAT adapter -> advanced -> Port forwarding. Click add rule and use the same values as above (comma separates columns). If the command was successful, you should see the rule listed at the same location. Also, this is the place to add an entry if you need any port exposed from a docker container and use it with VPN enabled; for example your application’s tomcat port.

We are not done yet, a few more commands:

export DOCKER_HOST="tcp://localhost:2376"
alias docker="docker --tlsverify=false"

Kudos to this smart guy for that alias. In other posts, you might find IP of the VM (which does not work), public IP of your machine, or even loopback IP ( being used, which might work but I would advise against that. Use ‘localhost’ instead; this and the TLS setting has to do with running docker-compose.

Now enable VPN and enjoy docker. This is where your journey ends if you are not using a private registry; but if you are, then continue.

Configuring Private Insecure Registry

Ensure that VPN is down, and ssh into the docker-machine. We want to enable it to talk to an insecure registry. A private docker registry does not need a name, but docker images in a non-docker-hub registry require that they be tagged with the URL of the registry prefixed to the usual repository name. They say it is for transparency, helps in identifying where the image originates from. Hence, it would be advisable to have a host-name even if your registry is private and has a static IP. That way even if you change the IP of the registry for whatever reason, you do not have to update all images/tags/compose ymls, shell scripts and whatever else is using them. Let us say our registry is hosted at:, on port 5000 and this being insecure, of course, is accessible only over VPN.
This step is intentionally manual, to avoid risks of breaking something else:

docker-machine ssh custom
sudo vi /var/lib/boot2docker/profile

In the EXTRA_ARGS, before the closing quote, add this line:

(I would ensure a blank line before the quote, as there already was) Save the file and exit vi (:wq). We now need to restart the docker daemon for changes to take effect:

sudo /etc/init.d/docker stop
Ensure service is down:  sudo /etc/init.d/docker status
sudo /etc/init.d/docker start
Ensure service is up: sudo /etc/init.d/docker status

Exit the VM by typing exit in terminal. (BTW, there is restart command too)

Using the registry

Now let us try pushing and pulling from this registry. In the quickstart terminal:

docker tag hello-world
docker push
docker rmi
docker run

What we did is tagged an image with the registry, pushed it to the private registry, removed the local copy and run the image by pulling from this registry.

Docker Compose

Next step is to get docker compose up and running with this setup. Actually, we are already ready, everything that we need to run docker-compose is taken care of in the previous steps. Most importantly docker-host configuration. You see, the TLS certs allow only for docker-machine IP and localhost to be used even when we disable verification, but we have already taken that into account and we have already configured our private registry. All set. Just connect VPN, navigate to your directory with docker-compose.yml file and hit: docker-compose up. You should see the images in compose file getting pulled and executed.

Starting the quickstart terminal second time

When you restart the quickstart terminal you might find that it recreates the ‘default’ VM and configures the environment to use it. That is okay, it does not bother us. But what does bother us is that none of the docker commands are working with VPN again. Please keep reading..

Consecutive starts of quickstart terminal

Well, we have to reconfigure the terminal every time to use our VM of choice. Here is how to do it:
Always make sure that you start the terminal when VPN is down. Starting with VPN up has never worked for me; and then run these commands:

eval "$(docker-machine env custom)"
export DOCKER_HOST="tcp://localhost:2376"
alias docker="docker --tlsverify=false"

Yes, every time you start the terminal. There is a way to avoid this, read on.

One Time Setup: For The Brave Among Us

From this point on, you are entering undocumented territory and are on your own. If something breaks, do not come looking for me. :) And before making any modifications, take a backup.
If you notice, the shortcut points to a shell script called ‘’. We are going to modify this script to auto-configure our environment every time it is called. Navigate to docker installation directory (directory that quickstart shortcut is pointing to) and open the (After creating a backup) file in a text editor.

Change 1: On line number 10 which looks like: VM=${DOCKER_MACHINE_NAME-default}
change that line to: VM=custom. Custom here is the name of our VM. This saves you from typing the eval line every time.

Change 2: On line 66/67, in “Setting Env” step, after the existing eval command add the following lines:

eval "DOCKER_HOST=\"tcp://localhost:2376\""
eval "DOCKER_TLS_VERIFY=\"0\""
eval "alias docker=docker --tlsverify=false"

These handle rest of the config. That is all, save and exit the file and we are ready to roll. This may break when an update the docker toolbox is installed which overwrites the file, may not work if the script changes in future, may break things I am not aware of, hence only for the brave. Besides, I do not use a Windows machine daily, so you guys would be first to know if it starts breaking ;). Let me know and we will figure it out.

Saturday, September 3, 2016

Redis Cluster: Fact Sheet (Not Just Issues)

Redis and the Redis clustering works very differently from the other data stores and data store clusters. The differences are not always as obvious and may come up as realizations down the line while using Redis, like what happened in our case. We are using a Redis cluster, with which, fortunately, we have not faced many issues so far. But that does not mean we will not and we shall need to be prepared.

Recently we were working on getting a Redis cluster up and working with docker compose and was enlightened to some of the differences which later led to disillusionment for me. Thought that there should be a ‘document of facts‘ on Redis and Redis cluster which people/myself can refer to. So I decided to create one, enjoy:

  1. Redis is great as a single server.
  2. In a Redis cluster, all your masters behave as if they are simultaneously active (not sure if they all are masters at the same time technically, but they behave as such).
  3. Every master in a cluster knows every other master/node in the cluster.
  4. There is no single master looking over the orchestration job.
  5. The masters, during clustering (sharding) agree upon the division of load: who shall have which hash slots.
  6. Each master speaks only for itself. If you ask for a key, and if the hash-slot for the same happens to be on the master you asked, it will return a value. Otherwise it returns a ‘redirection’ to the master that has the slot for this key.
  7. It is then the client’s job to resend the request to this new master based on the redirection.
  8. Clients try to sync up with master for which hash-slots lie with which master in order to speed up the retrieval.
  9. Every master knows other master by IP and IP only. It is not possible to use a hostname.
  10. The knowledge about the other nodes in the cluster is stored in a file called: nodes.conf. Although the extension is gives an impression of user modifiable configuration file, it is not a file for humans to modify.
  11. Every master must know other master by actual public ip, it is not possible to use a loopback (like If you do that it ends up in a max-redirection error. How it works is, when a client asks for a key and server responds with a redirection, the ‘smart’ client is expected to follow this redirection and get the value from this other node. Now the ‘dumb’ server responds with only ip it knows other node by, that is your loopback on the Redis server. But this ‘smart’ client (Jedis) is not smart enough to understand that the loopback is actually of the node and apparently starts looking for a Redis node on its own host! Whatever.. Just avoid doing that.
  12. When two nodes meet to form a cluster, one of them has to forgo its data. Either one must be empty.
  13. Replicas are not within master or any other nodes for that matter. Unlike what we know about clustering in Elastic Search or Kafka like services, replicas in Redis are independent nodes. So if you want a replication factor of 2 and have 3 masters, you effectively need 3 * 2 + 3 = 9 nodes in the cluster.
  14. If a master drops off, it is not possible to bring it back into cluster with data. Implication of point 12.
  15. If you need to perform any updates to any of the nodes/servers, take point 12 and 14 into consideration. Take out the master, upgrade, flush and reconnect as a slave, that is how it works.
  16. Converting a single server to cluster is not supported officially. There is one blog of a smart person showing a workaround for such a migration. Inverse of this, cluster to single server shall be equally painful.
  17. Redis / Redis Clustering is not officially supported on a Windows machine. There are unofficial ways to achieve something of the sort, the MSOpenTech’s Redis implementation, which now also supports clusters.
  18. The Java client, Jedis, has two different classes, one for connecting to a single standalone (JedisClient) and other for connecting to a cluster (JedisClusterClient). So if you decide to use the cluster in production, you cannot choose to use a single server during development. Implication is un-necessary load on your laptops. It can be managed by using environment aware wiring. We worked around by creating a jar, with a class that on post-construct just replaces the cluster-client reference of our internal cache utility class with a single server jedis-client. Just placing this jar on classpath during development solves it for us.
  19. Running Redis cluster in docker has its own pain points, on that later. (A different fact sheet for docker soon.)
  20. Extending point 11, if you have two network interfaces on the nodes, and have two isolated networks for two services that use this Redis cluster, how will that work out? Such is a setup is expected in a docker compose, where we isolate the service into different networks. Will need to see how Redis behaves in such a setup.

Although it was not the intention, while reading what I wrote I realized that the points above do look like a rant. In spite of these Redis is a solid, fast cache store and I love it for that. These are merely a few nuisances and related implications which we learnt about and experienced in our use of Redis cluster. Please use them only as points to ponder on when designing your application. Also, these nuisances are based on the state of Redis and Redis cluster at the time of writing which will change in time to come.