Monday, September 5, 2016

Using Docker and a Private Registry with VPN On Windows

Wasn’t that a very specific title? Docker has a very good documentation and reading that alone is enough for most of the straightforward tasks we might want to do. But as always some practical tasks are not straightforward, hence this blog. What we are going to see here today is how to setup docker toolbox on a Windows machine, make it work even when VPN is connected, make it talk to a private, insecure docker registry (that is why VPN) and configure it so it can run docker compose and see how we can set this config as a one-time activity. That’s quite a mouthful, but yes this is what we are going to do. All ready? Let us begin then.


Install Docker Toolbox

Go and download the docker toolbox and install it. That should create a shortcut called “Docker Quickstart Terminal”. Run it. That should show you an error about virtualization.


Enable Virtualization

Restart your machine, enter the BIOS settings and enable virtualization. It may be under advanced settings. On this Laptop, it is under the advanced settings -> device configurations and is named as: “Virtualization Technology (VTx)”. Whatever be the name, enable it.
Docker requires a Linux kernel, and since Windows machines lack it (of course!), docker toolbox runs a lightweight Linux distro called boot2docker in a virtualbox, hence the virtualization setting.


A Handy Tip

This tutorial will require you to copy and paste quite some shell commands, it is better we make that easy. Exit the quickstart terminal. Right click the shortcut, click properties -> options and enable ‘Quick Edit’ mode and save. It might ask for permission. This should now enable paste just by right clicking the mouse, to copy just select the text with mouse. While we are at that, also consider increasing the buffer and window size to suite your taste.


Start Up the VM

Make sure you are not connected to VPN and use the Quickstart Terminal shortcut again, this time it should proceed to validate if the boot2docker image is latest, or it shall pull the latest image, then it shall create a VM, get an IP, setup some ssh keys and finally the whale should appear with a terminal. Run the following commands to get a hang of docker running on windows:


docker -v
docker version
then docker run hello-world
docker images
docker ps -a


(And do read the output of hello-world, it describes how docker works).


The Disappointment

Feeling happy? Now for a little disappointment, connect VPN and try again. Errors errors everywhere. Disconnect VPN. What happened: Docker is running in a virtualbox on your machine, which gets an IP in local range (normally: 192.168.99.100), and you are talking to it over ssh. Once VPN is up, it sets the new routes and sends the 192.168.* range traffic out over VPN and your commands never reach your VM running docker. The most popular solution to this is setting a port forwarding and is documented on many blogs/github issues. Let’s just do that.


A new Beginning

Ensure you are not on VPN and remove the default VM, not necessary, but reduces the confusion. So in the quickstart terminal:


docker-machine rm default


And confirm. We are now going to create a new VM, let us call it ‘custom’. So type in:


docker-machine create -d virtualbox custom
eval "$(docker-machine env custom)"

It might take a couple of minutes, it is almost the same process as the first time. What we did is created a VM named custom and setup the environment to talk to this VM instead of the default. Mark this step, cause if anything goes wrong in the following steps, this is the one you should get back to to start over. Just be sure to use a new name, docker currently does not allow reusing names for VMs, so next time you may not be able to create a VM called custom. A new name should work just fine.


Battling With VPN

Now we shall create the a port forwarding on the virtual machine, binding the default docker port (2376) on localhost/127.0.0.1 to forward to this VM, whatever the ip of it.


docker-machine stop custom
/c/Program\ Files/Oracle/VirtualBox/VBoxManage.exe modifyvm "custom" --natpf1 "docker,tcp,,2376,,2376"
docker-machine start custom
docker ps -a


If you changed the location of virtualbox installation, please use appropriate path to vboxmanage. Assuming it was successful, last command should show you a table with all containers. You can use UI to do that as well: Open VirtualBox, stop the VM, open settings -> network -> NAT adapter -> advanced -> Port forwarding. Click add rule and use the same values as above (comma separates columns). If the command was successful, you should see the rule listed at the same location. Also, this is the place to add an entry if you need any port exposed from a docker container and use it with VPN enabled; for example your application’s tomcat port.


We are not done yet, a few more commands:


export DOCKER_HOST="tcp://localhost:2376"
export DOCKER_TLS_VERIFY="0"
alias docker="docker --tlsverify=false"


Kudos to this smart guy for that alias. In other posts, you might find IP of the VM (which does not work), public IP of your machine, or even loopback IP (127.0.0.1) being used, which might work but I would advise against that. Use ‘localhost’ instead; this and the TLS setting has to do with running docker-compose.


Now enable VPN and enjoy docker. This is where your journey ends if you are not using a private registry; but if you are, then continue.


Configuring Private Insecure Registry

Ensure that VPN is down, and ssh into the docker-machine. We want to enable it to talk to an insecure registry. A private docker registry does not need a name, but docker images in a non-docker-hub registry require that they be tagged with the URL of the registry prefixed to the usual repository name. They say it is for transparency, helps in identifying where the image originates from. Hence, it would be advisable to have a host-name even if your registry is private and has a static IP. That way even if you change the IP of the registry for whatever reason, you do not have to update all images/tags/compose ymls, shell scripts and whatever else is using them. Let us say our registry is hosted at: dockerregistry.example.com, on port 5000 and this being insecure, of course, is accessible only over VPN.
This step is intentionally manual, to avoid risks of breaking something else:


docker-machine ssh custom
sudo vi /var/lib/boot2docker/profile


In the EXTRA_ARGS, before the closing quote, add this line: --insecure-registry=dockerregistry.example.com:5000


(I would ensure a blank line before the quote, as there already was) Save the file and exit vi (:wq). We now need to restart the docker daemon for changes to take effect:


sudo /etc/init.d/docker stop
Ensure service is down:  sudo /etc/init.d/docker status
sudo /etc/init.d/docker start
Ensure service is up: sudo /etc/init.d/docker status


Exit the VM by typing exit in terminal. (BTW, there is restart command too)


Using the registry

Now let us try pushing and pulling from this registry. In the quickstart terminal:


docker tag hello-world dockerregistry.example.com:5000/hello-world
docker push dockerregistry.example.com:5000/hello-world
docker rmi dockerregistry.example.com:5000/hello-world
docker run dockerregistry.example.com:5000/hello-world


What we did is tagged an image with the registry, pushed it to the private registry, removed the local copy and run the image by pulling from this registry.


Docker Compose

Next step is to get docker compose up and running with this setup. Actually, we are already ready, everything that we need to run docker-compose is taken care of in the previous steps. Most importantly docker-host configuration. You see, the TLS certs allow only for docker-machine IP and localhost to be used even when we disable verification, but we have already taken that into account and we have already configured our private registry. All set. Just connect VPN, navigate to your directory with docker-compose.yml file and hit: docker-compose up. You should see the images in compose file getting pulled and executed.


Starting the quickstart terminal second time

When you restart the quickstart terminal you might find that it recreates the ‘default’ VM and configures the environment to use it. That is okay, it does not bother us. But what does bother us is that none of the docker commands are working with VPN again. Please keep reading..


Consecutive starts of quickstart terminal

Well, we have to reconfigure the terminal every time to use our VM of choice. Here is how to do it:
Always make sure that you start the terminal when VPN is down. Starting with VPN up has never worked for me; and then run these commands:


eval "$(docker-machine env custom)"
export DOCKER_HOST="tcp://localhost:2376"
export DOCKER_TLS_VERIFY="0"
alias docker="docker --tlsverify=false"


Yes, every time you start the terminal. There is a way to avoid this, read on.


One Time Setup: For The Brave Among Us

From this point on, you are entering undocumented territory and are on your own. If something breaks, do not come looking for me. :) And before making any modifications, take a backup.
If you notice, the shortcut points to a shell script called ‘start.sh’. We are going to modify this script to auto-configure our environment every time it is called. Navigate to docker installation directory (directory that quickstart shortcut is pointing to) and open the start.sh (After creating a backup) file in a text editor.


Change 1: On line number 10 which looks like: VM=${DOCKER_MACHINE_NAME-default}
change that line to: VM=custom. Custom here is the name of our VM. This saves you from typing the eval line every time.


Change 2: On line 66/67, in “Setting Env” step, after the existing eval command add the following lines:

eval "DOCKER_HOST=\"tcp://localhost:2376\""
eval "DOCKER_TLS_VERIFY=\"0\""
eval "alias docker=docker --tlsverify=false"

These handle rest of the config. That is all, save and exit the file and we are ready to roll. This may break when an update the docker toolbox is installed which overwrites the file, may not work if the script changes in future, may break things I am not aware of, hence only for the brave. Besides, I do not use a Windows machine daily, so you guys would be first to know if it starts breaking ;). Let me know and we will figure it out.

No comments:

Post a Comment