Setting up a Continuous Integration Environment with Jenkins and Docker PART 4 Private Docker Registry

In Part 4 of this series I will be walking through the steps that were needed to set up my own Private Docker Registry on a CentOS 6 Linux server.

Step 1) Install Prerequisites  
The Docker registry is written in python, thus we need to install python development utilities and some libraries.

> yum install python-pip phython-devel libevent libevent-devel pyliblzma jjpython-gunicorn ...lots of output ... Is this ok [y/N]: y

=== may not be needed
yum install mod_wsgi.x86_64
yum install  python-wsgiproxy.noarch
yum install python-moksha-wsgi.noarch python-wsgi-jsonrpc.noarch

====

Step 2) Install and Configure Docker Registry

>pip install docker-registry ...lots of output ... Cleaning up...

By default Docker saves its data under the /tmp directory.  I will create a permanent location, then will need to configure Docker to to use the new location.

> mkdir /var/docker-registry

Locate the file config_sample.yml. On my system it is located in /usr/lib/python2.6/site-packages/config. Now copy the file to config.yml. 

> cd /usr/lib/python2.6/site-packages/config > cp config_sample.yml config.yml

Edit config.yml, then replace any reference to /tmp with /var/docker-registry

Change this:

 sqlalchemy_index_database: _env:SQLALCHEMY_INDEX_DATABASE:sqlite:////tmp/docker-registry.db

To this:

 sqlalchemy_index_database: _env:SQLALCHEMY_INDEX_DATABASE:sqlite:////var/docker-registry/docker-registry.db

Change this:
local: &local
    <<: *common
    storage: local

    storage_path: _env:STORAGE_PATH:/tmp/registry

To this:

local: &local
    <<: *common
    storage: local

    storage_path: _env:STORAGE_PATH:/var/docker-registry/registry

Change this:

glance: &glance
    <<: *common
    storage: glance
    storage_alternate: _env:GLANCE_STORAGE_ALTERNATE:file

    storage_path: _env:STORAGE_PATH:/tmp/registry

To this:

glance: &glance
    <<: *common
    storage: glance
    storage_alternate: _env:GLANCE_STORAGE_ALTERNATE:file

    storage_path: _env:STORAGE_PATH:/var/docker-registry/registry

If you want to do something more complex like using AWS S3 buckets, Google Cloud Storage, or Openstack you can configure it in this file.

Now that the config is in place let's try to start the server.

> gunicorn --access-logfile - --debug -k gevent -b 0.0.0.0:5000 -w 1 docker_registry.wsgi:application 07/Jan/2015:16:52:32 +0000 WARNING: Cache storage disabled! 07/Jan/2015:16:52:32 +0000 WARNING: LRU cache disabled! 07/Jan/2015:16:52:32 +0000 DEBUG: Will return docker-registry.drivers.file.Storage

Your output should be similar.
Go ahead and hit CTRL-C to kill the process.

STEP 3) Docker Registry as a Service on start up

First check to see if the Docker Registry is configured as a service

> chkconfig --list | grep docker docker-registry 0:off   1:off   2:on    3:on    4:on    5:on    6:off

Next start the Docker Registry.  I use restart here just so you can see the output.

> service docker-registry restart Stopping docker-registry: [FAILED] Starting docker-registry: OK

Awesome! it looks like it is up and running, lets verify that docker-registry is up.

> ps -ef | grep docker root 5445 1 0 Jan07 pts/0 00:00:00 /usr/bin/python /usr/bin/gunicorn --access-logfile - --debug --max-requests 100 --graceful-timeout 3600 -t 3600 -k gevent -b 0.0.0.0:5000 -w 4 docker_registry.wsgi:application root 5451 5445 0 Jan07 pts/0 00:00:23 /usr/bin/python /usr/bin/gunicorn --access-logfile - --debug --max-requests 100 --graceful-timeout 3600 -t 3600 -k gevent -b 0.0.0.0:5000 -w 4 docker_registry.wsgi:application root 5452 5445 0 Jan07 pts/0 00:00:25 /usr/bin/python /usr/bin/gunicorn --access-logfile - --debug --max-requests 100 --graceful-timeout 3600 -t 3600 -k gevent -b 0.0.0.0:5000 -w 4 docker_registry.wsgi:application root 5455 5445 0 Jan07 pts/0 00:00:25 /usr/bin/python /usr/bin/gunicorn --access-logfile - --debug --max-requests 100 --graceful-timeout 3600 -t 3600 -k gevent -b 0.0.0.0:5000 -w 4 docker_registry.wsgi:application root 5458 5445 0 Jan07 pts/0 00:00:25 /usr/bin/python /usr/bin/gunicorn --access-logfile - --debug --max-requests 100 --graceful-timeout 3600 -t 3600 -k gevent -b 0.0.0.0:5000 -w 4 docker_registry.wsgi:application root 6211 20583 0 13:46 pts/0 00:00:00 grep docker

STEP 4) Adding Authentication

Borrowing heavily from the References linked to at the bottom of this post I will be implementing Nginx (pronounced Engine X) as a front end for access to the Docker Registry. By default the Docker Registry listens on port 5000. In many labs and environments port 5000 is not available through internal firewalls and certainly not on the open internet. Thus we need a method for forwarding the requests to Docker.  Nginx is a fairly new opensource web server for http, https and other protocols, it can be used as a load balancer and reverse proxy server.  It has features that allow it to handle a higher volume of requests than Apache, by handling requests differently. There are many discussions and blog posts on the web about Nginx vs Apache. I recommend reading a couple. 

Lets verify that htpasswd is installed, this is an Apache tool that can be used to generate encrypted passwords.

> yum provides \*bin/htpasswd Loaded plugins: fastestmirror ... httpd-tools-2.2.15-39.el6.centos.x86_64 : Tools for use with the Apache HTTP Server Repo : base Matched from: Filename : /usr/bin/htpasswd ...

Now we know what httpd package has htpasswd lets try to install it. 

> yum install httpd-tools Loaded plugins: fastestmirror Setting up Install Process Loading mirror speeds from cached hostfile * base: mirror.zetup.net * epel: mirror.proserve.nl * extras: mirror.zetup.net * updates: ftp.plusline.de Package httpd-tools-2.2.15-39.el6.centos.x86_64 already installed and latest version Nothing to do

It has already been installed, if it wasn't installed this would have installed it.
Install Nginx

yum install nginx ...lots of output... Is this ok [y/N]:y

Create a Docker user, create a password for the user when prompted.
After this step we will have a password file with our users, and a Docker registry available.

> htpasswd -c /etc/nginx/docker-registry.htpasswd pete New password: Re-type new password: Adding password for user pete

To add additional users, rerun the above command without the -c flag. The -c flag creates the password file. If you reuse -c you will overwrite the file.


Next I need to tell Nginx to use the created authentication file, and forward requests to the Docker registry. 
Create the file: /etc/nginx/conf.d/docker-registry.conf
then add the following content, and edit as appropriate for your environment.

# For versions of Nginx > 1.3.9 that include chunked transfer encoding support # Replace with appropriate values where necessary upstream docker-registry { server localhost:5000; } server { listen 8080; server_name my.docker.registry.com; # ssl on; # ssl_certificate /etc/nginx/ssl/<servername>.crt; # ssl_certificate_key /etc/nginx/ssl/<servername>.key; proxy_set_header Host $http_host; # required for Docker client sake proxy_set_header X-Real-IP $remote_addr; # pass on real client IP client_max_body_size 0; # disable any limits to avoid HTTP 413 for large image uploads # required to avoid HTTP 411: see Issue #1486 (https://github.com/dotcloud/docker/issues/1486) chunked_transfer_encoding on; location / { # let Nginx know about our auth file auth_basic "Restricted"; auth_basic_user_file docker-registry.htpasswd; proxy_pass http://docker-registry; } location /_ping { auth_basic off; proxy_pass http://docker-registry; } location /v1/_ping { auth_basic off; proxy_pass http://docker-registry; } }

Restart Nginx to activate the virtual host, and test the connections to Docker and Nginx

> service nginx restart Stopping nginx: [ OK ] Starting nginx: [ OK ] >curl localhost:5000 true >curl localhost:8080 <html> <head><title>401 Authorization Required</title></head> <body bgcolor="white"> <center><h1>401 Authorization Required</h1></center> <hr><center>nginx/1.0.15</center> </body> </html>

Great, Docker is running, and Nginx is forwarding the request to Docker.
Now lets try connecting to Docker with the username created earlier.

>curl pete:mypassword@localhost:8080 true

STEP 5) Adding Secure Authentication (SSL)
In the previous step we added basic http authentication, however this is not very secure since connections to the registry are unencrypted.  In this step I'll show you how to enable SSL (https), and setup a self signed certificate.

Lets begin by editing the file: /etc/nginx/conf.d/docker-registry.conf 
Remove the # symbol in front of the SSL lines.  The result should look like this;

ssl on; ssl_certificate /etc/nginx/ssl/<servername>.crt; ssl_certificate_key /etc/nginx/ssl/<servername>.key;

Save the file. Nginx is now configured to use SSL and will look for the certificate and key at the name and location listed in the file. Please note: As far I know there are no standards for the location of your certificate I chose the above paths for convenience .

Make sure you are logged into the server for which you want to create the SSL Certificate then enter the following, making sure to replace <servername> with the fully qualified domain name of your system.
The first thing I need to do is create the key and certificate request.

> cd /etc/httpd/conf > openssl req -new -newkey rsa:2048 -nodes -keyout <servername>.key -out <servername>.csr

Next answer the questions The purpose of the questions is to add randomization in the certificate. Answer the questions with values suitable to your environment. Press Enter to leave the field blank, or to select the default. In my experience the last three questions can be left blank. Make sure that "Common Name", (the Fully Qualified Domain Name) matches the hostname you will use to connect to Docker.

Generating a 2048 bit RSA private key .........+++ ......................................................................................................................................+++ writing new private key to 'myservername.key' ----- You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) [GB]:US State or Province Name (full name) [Berkshire]:MASSACHUSETTS Locality Name (eg, city) [Newbury]:BOSTON Organization Name (eg, company) [My Company Ltd]: Organizational Unit Name (eg, section) []:IT Common Name (eg, your name or your server's hostname) []: <servername> Email Address []: Please enter the following 'extra' attributes to be sent with your certificate request A challenge password []: An optional company name []:

Next Create the following directory, then copy the private key and the Certificate Request to it.

> mkdir /etc/nginx/ssl > cp <servername>.key /etc/nginx/ssl > cp <servername>.csr /etc/nginx/ssl

Creating a self-signed certificate
If you do not plan to have the certificate signed by a Certificate Authority (CA) or if you plan to test the new SSL implementation while the CA is signing your certificate, you can generate a self-signed certificate. This temporary certificate generates errors in the client browser to the effect that the signing certificate authority is unknown and not trusted.
To generate a temporary certificate that is good for 365 days  Issue the following commands:

> cd /etc/nginx/ssl > openssl x509 -req -days 365 -in <servername>.csr -signkey <servername>.key -out <servername>.crt

SSL Test

Restart Nginx to reload the configuration and SSL Keys, if all goes well there should not be any errors.

> service nginx restart Stopping nginx: OK Starting nginx: OK

Lets try a couple of different curl commands.

> cd /etc/nginx/ssl > curl pete:mypassword@localhost:8080 ... 400 Bad Request ... # You can see that Nginx is forwarding us to https, but I'm getting a 400 Bad Request. # Lets try specifying https >curl https://pete:mypassword@localhost:8080 curl: (60) Peer certificate cannot be authenticated with known CA certificates More details here: http://curl.haxx.se/docs/sslcerts.html curl performs SSL certificate verification by default, using a "bundle" of Certificate Authority (CA) public keys (CA certs). If the default bundle file isn't adequate, you can specify an alternate file using the --cacert option. If this HTTPS server uses a certificate signed by a CA represented in the bundle, the certificate verification probably failed due to a problem with the certificate (it might be expired, or the name might not match the domain name in the URL). If you'd like to turn off curl's verification of the certificate, use the -k (or --insecure) option. #Curl is having trouble now because I have created a self signed certificate #and the curl command doesn't trust the the cert, due to the certificate not # being in curl's default trust store. Lets point directly to new certificate. > curl --cacert myserver.crt https://pete:mypassword@localhost:8080 curl: (51) SSL: certificate subject name 'myserver' does not match target host name 'localhost' #Curl still is having trouble because we tried to connect to localhost, but #created the certificate with the fully qualified domain name of the server. #Lets try one more time. > curl --cacert myserver.crt https://pete:mypassword@myserver:8080 true #SUCCESS!!!

STEP 6) Accessing the Docker Registry from another server.

Specific to CentOS6 and possibly Redhat there is an application called docker that is a "docking application for KDE3, and GNOME2". This app will cause conflicts with docker-io. We have several tasks we must do on the client prior being able to login. 

1. Remove the docker app if it is installed.
2. Setup a server as a Docker client.
3. Import the Docker Registry's self-signed certificate into the systems trust store.

Install Docker Client

> rpm -e docker > yum install docker-io ...lots of output... Is this ok [y/N]: y Complete!

Once Docker is installed, you will need to start the docker daemon, then verify that docker is configured to start at boot.

> service docker start Starting cgconfig service: [ OK ] Starting docker: [ OK ] >chkconfig --list docker docker 0:off 1:off 2:on 3:on 4:on 5:on 6:off

If we were to attempt to login now using the command below we would encounter several issues.  I will work through each issue I encountered hopefully this will save you time.

> docker login https://myserver username: password: email:

If your docker registry server is not yet part of your local DNS, add the servers ip address and fully qualified domain name (FQDN) into your client systems /etc/hosts file. On start up the Docker daemon creates a file /var/run/docker.sock. This file has the following ownership (root,docker) and permissions (644).
srw-rw---- 1 root docker 0 Jan 13 17:24 /var/run/docker.sock
When attempting to run the docker login command as a client user, an error similar to the following is returned.

dial unix /var/run/docker.sock: permission denied

A temporary fix is to reset the permissions to by executing the chmod command, however once you restart the docker daemon the permissions return to 644.

> chmod 666 /var/run/docker.sock srw-rw-rw- 1 root docker 0 Jan 13 17:24 /var/run/docker.sock

A more permanent solution is to add your clients username to the docker group. First verify that you have a docker group by looking at the /etc/group file. To add the the docker group and add your client to the docker group, execute the following commands;

> groupadd docker > useradd -aG docker username

After adding the client user to the docker group the user must log out then log back into their account, and the error should go away.

The next thing we need to do is add our self signed certificate to the client systems list of trusted certificates. The following solution is not ideal but it worked for me, I will update this post if I come across a better solution. You will need to go back to your Docker Registry server and copy the registry servers certificate to the client server.

On the registry server,

> cd /etc/nginx/ssl > cat myserver.crt -----BEGIN CERTIFICATE----- MIIDfjCCAmYCCQCS024a3xtBXzANBgkqhkiG9w0BAQUFADCBgDELMAkGA1UEBhMC VVMxFjAUBgNVBAgMDU1BU1NBQ0hVU0VUVFMxDzANBgNVBAcMBkJvc3RvbjESMBAG A1UECgwJTWljcm9zb2Z0MQswCQYDVQQLDAJJVDEnMCUGA1UEAwweZG9ja2VyaHVi LnVzLm1zdWRldi5ub2tsYWIubmV0MB4XDTE1MDExMjIwNDQyNloXDTE2MDExMjIw NDQyNlowgYAxCzAJBgNMBAYTAlVTMRYwFAYDVQQIDA1NQVNTQUNIVVNFVFRTMQ8w DQYDVQQHDAZCb3N0b24xEjAQBgNVBAoMCU1pY3Jvc29mdDELMAkGA1UECwwCSVQx JzAlBgNVBAMMHmRvY2tlcmh1Yi51cy5tc3VkZXYubm9rbGFiLm5ldDCCASIwDQYJ KoZIhvcNAQEBBQADggEPADCCAQoCggEBALHAu+zYTe9dMJL4sSz8ihTADKgwOUPh Szj4JeDYKVMv/N3ihNdwoSVDoy1qldR+Zl86BRD2YHj4i2FUOhlBxyFDLEB+6bMi lKHeh7V2dBpTraALJF4faKyVVRtwvhtvfxvRdP4sS3a2H44oYkWQsnV026TRRhnn bI7AqkYuna8EcjFt1UrRBM3lzDxGwyX1iCyydU9xKS0mRsgtpZXbHS8NBD5mKDD2 breYiaSsdzhLdCxqGuzoULhWP9KJq++gAlahdo1OJjCdrbLYvkNAeVZsayEA8Yf7 4MEej5Ab5SNd3rkOlZDY9pi/W72EDqJpE1vEEiHcmuQshnZFxTADtdcCAwEAATAN BgkqhkiG9w0BAQUFAAOCAQEABs+53+GMpOLIMVaKVxwHUIy2MIzIKE3j3x0W2oXt N3kHi9gYvZoiClw/E+1VKj6ra59vnrptSumFy3gqBPPFa4r9hglb25ITDiIiXy9t UAZWBq8YDdHkOCPfKFtc3P0b+eZ/HiQITfdle9SnNXwAVV9DmC1YTFlMUA3XvlRO DPERTa4RWW1WXA/zkyGbMXRvdqRppOvQQ1uewl3HFk8ZCbbR8BqLbmfcoepn/KAs MWaik/04ARab2sa/xC27ZswyG1VlLD/SjMK6Tu6b8bv72pYbEgDj/ekRWN15BdYA -----END CERTIFICATE-----

Copy the output and paste it to a file on your client server such as /tmp/docker-registry.crt then add it to your systems certificate authority bundle, after making a backup copy of course.

> cd /etc/pki/tls/certs > cp ca-bundle.crt ca-bundle.crt.bak > cat /tmp/docker-registry.crt >> ca-bundle.crt > service docker restart

Finally we are ready to attempt our login, if successful a file (.dockercfg) will be created in the end users home directory.

> docker login https://myserver.com:8080 Username: pete Password: Email: pete@mycompany.com Login Succeeded

Lets see if we can run a simple docker command.

> docker info Containers: 0 Images: 0 Storage Driver: devicemapper Pool Name: docker-253:0-918242-pool Pool Blocksize: 65.54 kB Data file: /var/lib/docker/devicemapper/devicemapper/data Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata Data Space Used: 305.7 MB Data Space Total: 107.4 GB Metadata Space Used: 729.1 kB Metadata Space Total: 2.147 GB Library Version: 1.02.89-RHEL6 (2014-09-01) Execution Driver: native-0.2 Kernel Version: 2.6.32-504.el6.x86_64 Operating System:

At this point I have successfully installed python pre-requisites, docker-io, and Nginx. I have configured docker and Nginx as a reverse proxy to forward requests to docker, setup SSL and several users. Finally I was able to log in to the docker registry and run a simple docker command.  In Part 5 of this tutorial, I will begin using this docker registry. 


References:

How to set up a private docker registry on ubuntu