Setup the secure repository proxy
We will start by creating a folder for the reverse-proxy. This folder will hold the information needed to build a docker image specific for our need. It will hold the configuration for the proxy, which will be Nginx, and it will hold the certificates. This is the quickest and easiest way to build an image, but lacks some re-use potential. For now we will proceed with this simple setup, and we will use self-signed certificates.
In the demo folder, run the following commands.
mkdir reverse
cd reverse
mkdir certs
openssl req \
-newkey rsa:4096 -nodes -sha256 -keyout certs/domain.key \
-x509 -days 365 -out certs/domain.crt
You will be asked to fill in some details like your organisation name etc. These can be entered as you like. The only important question is the FQDN. This is the name by which the user will access the docker repository. This can be an official domain name you own, like docker.mycompany.com, a domain name setup on your local netwerk, a well known ip number (not user friendly), or (like I am using for local development) you can choose a name like mydocker, and add a mapping from mydocker to the correct ip number in your host file on every computer that is using the repository. (Requires root permissions on the clients).
You will see something like this:
Generating a 4096 bit RSA private key
............................................................................................................++
......................................................................................................++
writing new private key to 'certs/domain.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:NL
State or Province Name (full name) [Some-State]:Noord Brabant
Locality Name (eg, city) []:Helmond
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Rubix
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:mydocker
Email Address []:barry@b*********cker.nl
You will now see two files in the certs folder: a domain.crt file containing your public certificate, and a domain.key file containing the private key. Make sure to keep the last one secret, and only use it on the reverse proxy.
Bonus: Generate the certificate using docker
If you are on windows, or just don’t wish to install openssl in order to generate one certificate, try using a docker image to create the certificate:
docker run -it centurylink/openssl sh
mkdir certs
openssl req -newkey rsa:4096 -nodes -sha256 -keyout certs/domain.key -x509 -days 365 -out certs/domain.crt
While the container is running, open a new commandline. You can find the id with docker ps and copy the certificate out of it using docker cp <containerid>:/certs .
Configure Nginx
We now have finished the preparations and are ready to start configuring Nginx.
Create the file https.conf in the folder reverse, and start adding the following upstreams:
upstream docker-releases {
server nexus:8082;
}
upstream docker-snapshots {
server nexus:8083;
}
upstream docker-public {
server nexus:8084;
}
Each upstream refers to a docker repository we configured in Nexus in the previous posts. An upstream is a destination where Nginx can forward it’s requests to. The reference is by hostname and portnumber. The hostname matches the name of the Nexus container in the docker-compose.yml, while the port number matches the http port we defined for each repository individually during the configuration of Nexus.
Next, we add a header field mapping that is required for the docker repository system.
map $upstream_http_docker_distribution_api_version $docker_distribution_api_version {
'' 'registry/2.0';
}
Finally, we start adding the listeners for the inbound requests. The first listener will be on port 443, which is the default https port as well as the default docker registry port. This will allow us to use just mydocker as a destination, without specifying a port number.
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name mydocker mydocker.local;
set $fqdn mydocker;
ssl_certificate /etc/ssl/domain.crt;
ssl_certificate_key /etc/ssl/domain.key;
add_header Strict-Transport-Security "max-age=15768000; includeSubdomains; preload" always;
ssl_protocols TLSv1.1 TLSv1.2;
ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
client_max_body_size 0;
chunked_transfer_encoding on;
location /v2/ {
# Do not allow connections from docker 1.5 and earlier
# docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
return 404;
}
## If $docker_distribution_api_version is empty, the header will not be added.
## See the map directive above where this variable is defined.
add_header 'Docker-Distribution-Api-Version' $docker_distribution_api_version always;
proxy_pass http://docker-public;
proxy_set_header Host $http_host; # required for docker client's sake
proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 900;
}
}
Let’s analyse the above configuration:
- A server definition creates a listener for inbound requests on a given port
- We tell Nginx to listen on port 443for ssl-encrypted http2 requests. The interfaces we use are the ip4 and ip6 interfaces of this host.
- Nginx should expect mydocker or mydocker.local as hostname. This is the hostname you’d type in a browser, before it is resolved to the ip-number. It must match the FQDN of the certificate we created earlier.
- The public certificate and private key are provided. We will have to add the files to the docker image later on.
- A required docker header is added.
- Not all ssl protocols are secure. Some are outdated. Some are not supported by docker. We list the protocols and ciphers we wish to use.
- Docker transfers can be huge. You might want to transfer a 16G image. We remove the max-size limit on the request, so that the client is allowed to send this much information. This also means that we can’t encode the entire request in memory, but we have to use a chunked approach.
- The docker repository we use is version 2, so we expect the path to start with /v2/ which allows us to add a v1 or v3 with different settings if we ever need to.
- Exclude old docker versions that don’t play nice.
- We add the header mapping we defined at the very beginning of the http2.conf file, right after the upstreams. The mapping is required because add_header by itself only allows fixed data.
- Finally, we tell Nginx what to do with the incoming request. The request should be forwared (proxied) to the upstream docker-public, which we defined at the very start. Nexus will need some extra headers again, this time they are related to the way a proxy server talks to the proxied server. It is used to forward information, such as the protocol used between the client and the proxy, the ip number of the client etc. We also set a large timeout, because storing large binaries might take some time.
What did we do?
- We have forwarded the default docker port towards the Nexus docker-public repository. This is the group repository for our docker images, which means that when we pull an image from this default endpoint, the image will be retrieved from one of the following locations: docker-releases, docker-snapshots or docker-hub.
This endpoint allows us to find any docker image we created ourselves, or from the public docker-hub repository on the internet. We don’t need to know in which repository it is stored, all magic is handled by Nexus. Great.
The next step is storing docker images. We don’t want to use the generic port for this, but rather, we would like to specify what kind of image we are storing: is it a snapshot build created during development, or is it a candidate release build that might end up on production?
For this, we introduce two endpoints in Nginx, in a way similar to the default endpoint.
server {
listen 8082 ssl http2;
listen [::]:8082 ssl http2;
server_name mydocker mydocker.local;
set $fqdn mydocker;
ssl_certificate /etc/ssl/domain.crt;
ssl_certificate_key /etc/ssl/domain.key;
add_header Strict-Transport-Security "max-age=15768000; includeSubdomains; preload" always;
ssl_protocols TLSv1.1 TLSv1.2;
ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
client_max_body_size 0;
chunked_transfer_encoding on;
location /v2/ {
# Do not allow connections from docker 1.5 and earlier
# docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
return 404;
}
## If $docker_distribution_api_version is empty, the header will not be added.
## See the map directive above where this variable is defined.
add_header 'Docker-Distribution-Api-Version' $docker_distribution_api_version always;
proxy_pass http://docker-releases;
proxy_set_header Host $http_host; # required for docker client's sake
proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 900;
}
}
and
server {
listen 8083 ssl http2;
listen [::]:8083 ssl http2;
server_name mydocker mydocker.local;
set $fqdn mydocker;
ssl_certificate /etc/ssl/domain.crt;
ssl_certificate_key /etc/ssl/domain.key;
add_header Strict-Transport-Security "max-age=15768000; includeSubdomains; preload" always;
ssl_protocols TLSv1.1 TLSv1.2;
ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
client_max_body_size 0;
chunked_transfer_encoding on;
location /v2/ {
# Do not allow connections from docker 1.5 and earlier
# docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
return 404;
}
## If $docker_distribution_api_version is empty, the header will not be added.
## See the map directive above where this variable is defined.
add_header 'Docker-Distribution-Api-Version' $docker_distribution_api_version always;
proxy_pass http://docker-snapshots;
proxy_set_header Host $http_host; # required for docker client's sake
proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 900;
}
}
Note that the only differences are:
- The listen port has changed for both ip4 and ip6
- The proxy_pass destination has changed to the corresponding upstream.
This concludes our Nginx configuration. You can close the editor on https.conf. All we need to do now is to bundle the software, configuration and certificates in a docker image.
Bundling the package
In the reverse folder, we create a file Dockerfile and add the following content:
FROM nginx:1.15-alpine
COPY certs/* /etc/ssl/
COPY https.conf /etc/nginx/conf.d/
This specifies that we use the official Nginx distribution from docker-hub. We select a specific version to avoid update problems in the future. Our certificates are copied to the location we specified in the configuration file. Finally we copy the configuration file itself to the default location where Nginx expects it to be.
We have ended up with the following file-structure for the reverse proxy image:
/reverse
/certs
/domain.crt
/domain.key
/https.conf
/Dockerfile
Validate your results by typing docker build . on the command-line inside the reverse folder. It should download the base image and add the required files.
Bring it together
We now have a reverse proxy configured to forward all traffic towards the Nexus repository. All we need is to put them together in a single docker-compose environment, so that they can communicate. Go to the root folder of your project and edit the docker-compose.yml file. We will add some lines, so that the result will be:
version: '2'
services:
nexus:
image: sonatype/nexus3:3.12.1
volumes:
- "nexus-data:/nexus-data"
ports:
- "8081:8081"
expose:
- "8082"
- "8083"
- "8084"
- "8085"
- "8086"
- "8087"
- "8088"
- "8089"
reverse-proxy:
build: reverse
ports:
- "443:443"
- "8082:8082"
- "8083:8083"
- "8084:8084"
- "8085:8085"
- "8086:8086"
- "8087:8087"
- "8088:8088"
- "8089:8089"
volumes:
nexus-data:
Everything we added is in the reverse-proxy service:
- The docker image will be identified by the name reverse-proxy
- It is not a downloaded image like nexus, but instead it’s a locally build image that can be found in the folder reverse
- It exposes a number of ports to the outside world, most specifically port 443, 8082 and 8083. The others are there for future use.
Running and testing
Now that we have both Nexus and Nginx in the docker-compose, it is time to start using it. Make sure your previous compose is stopped by typing docker-compose stop
Go to the main directory and build the composition by running docker-compose build (without the . that docker build . uses). You should see output like this:
nexus uses an image, skipping
Building reverse-proxy
Step 1/3 : FROM nginx:alpine
---> ba60b24dbad5
Step 2/3 : COPY https.conf /etc/nginx/conf.d/
---> 49d1e664e3f5
Step 3/3 : COPY certs/* /etc/ssl/
---> 1cc416dc1cd2
Successfully built 1cc416dc1cd2
Successfully tagged demo_reverse-proxy:latest
Now we are ready to run the composition for the first time. Run it with docker-compose up so that it creates missing volumes if needed. To run it afterwards, use docker-compose start instead.
Before we can login, we need to make sure we can find the host mydocker. As discussed before, it needs to be registered. The simplest way is to register the name on the local machine:
On linux, edit the file /etc/hosts
On windows, edit the file C:\Windows\System32\drivers\etc\hosts
Add the following line:
127.0.0.1 mydocker
This tells your computer that any traffic for mydocker will be routed towards the loopback ip number.
Now we can start testing. Try to log in to your repository by entering the following command
docker login mydocker
You will be asked for credentials. Provide the username and password for the user you created.
The command should end with the message “Login succeeded”
You are now logged on to the group repository that also contains the reference to docker-hub. Confirm this by doing a docker pull nginx
It should show:
Using default tag: latest
latest: Pulling from library/nginx
Digest: sha256:9fca103a62af6db7f188ac3376c60927db41f88b8d2354bf02d2290a672dc425
Status: Image is up to date for nginx:latest
Now try a docker push nginx. This should give you a denied message: you have no permissions to push to the nginx image on docker-hub.
Lets store this image in our docker repository. Begin by logging in to our snapshot repo: enter docker login mydocker:8083 and provide the usercredentials.
Tag and push the image:
docker tag nginx mydocker:8083/nginx:latest
docker push mydocker:8083/nginx:latest
You should see the layers being uploaded.
Verify the data in Nexus. 
It should show the nginx image you just uploaded:
You can find more details if you drill-down deeper.
Securing the admin interface
Now that we have secured the Docker interface, we can add the admin interface as well. Edit the https.conf file and add the admin port as an upstream at the start of the file.
upstream admin-page {
server nexus:8081;
}
Scroll down to the server component for port 443. This server contains one location for /v2/. What we want is to lroute trafic on /v2/ towards the Docker repository, and to route other data towards the admin pages. This works because no admin pages exists that use /v2/ as prefix.
Below the location /v2/ we add a new location. Make sure it is still inside the server section for port 443.
location / {
proxy_pass http://admin-page;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 90;
}
Test the new endpoint by rebuilding and starting your docker-compose. Make sure you can log on to the admin page. You’ll most likely need to accept the untrusted certificate before you can continue.
Finally, go to the docker-compose.yml and remove the following lines from the nexus configuration. The port will no longer be public.
ports:
- "8081:8081"
Instead we will open a private port. Inside the expose section of nexus add:
- "8081"
Now our http admin port is no longer visible from the outside world, and we can only access it using the public https endpoint of the proxy.
Recap
We have introduced a reverse-proxy in order to create a secure repository. The proxy provides a https-secured Docker endpoint, which ensures that the transferred data is not intercepted. The data remains private and unmodified during transport.
In our case, we used a self-generated certificate for encryption. One important thing to note is that we didn’t have to install the certificate in docker. Docker accepts our certificate, even if it isn’t signed by a trusted certificate authority. Should future versions of Docker enforce the trust of te certificate, you’ll need to add the public certificate in the certificate folder on every client, which can be found at
c:\Users\<username>\.docker\machine\certs
or
/home/<username>/.docker/machine/certs
Upcoming post
In the next post, we will deploy the Jenkins master in the docker-compose network and set it up as a work orchestration server.
Recent Comments