Posts Tagged "nexus"

This post is part of a series about creating a continues integration platform for home use.

 

  Create an artifact repository

  Configure the artifact repository

  Secure the artifact repository

 Create the Jenkins master

 Add a Jenkins slave

 Creating a sample project.

Setup the secure repository proxy

We will start by creating a folder for the reverse-proxy. This folder will hold the information needed to build a docker image specific for our need. It will hold the configuration for the proxy, which will be Nginx, and it will hold the certificates. This is the quickest and easiest way to build an image, but lacks some re-use potential. For now we will proceed with this simple setup, and we will use self-signed certificates.

In the demo folder, run the following commands.

mkdir reverse
cd reverse
mkdir certs

openssl req \
  -newkey rsa:4096 -nodes -sha256 -keyout certs/domain.key \
  -x509 -days 365 -out certs/domain.crt

You will be asked to fill in some details like your organisation name etc. These can be entered as you like. The only important question is the FQDN. This is the name by which the user will access the docker repository. This can be an official domain name you own, like docker.mycompany.com, a domain name setup on your local netwerk, a well known ip number (not user friendly), or (like I am using for local development) you can choose a name like mydocker, and add a mapping from mydocker to the correct ip number in your host file on every computer that is using the repository. (Requires root permissions on the clients).

You will see something like this:

Generating a 4096 bit RSA private key
............................................................................................................++
......................................................................................................++
writing new private key to 'certs/domain.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:NL
State or Province Name (full name) [Some-State]:Noord Brabant
Locality Name (eg, city) []:Helmond
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Rubix
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:mydocker
Email Address []:barry@b*********cker.nl

You will now see two files in the certs folder: a domain.crt file containing your public certificate, and a domain.key file containing the private key. Make sure to keep the last one secret, and only use it on the reverse proxy.

 

Bonus: Generate the certificate using docker

If you are on windows, or just don’t wish to install openssl in order to generate one certificate, try using a docker image to create the certificate:

docker run -it centurylink/openssl sh
mkdir certs
openssl req -newkey rsa:4096 -nodes -sha256 -keyout certs/domain.key -x509 -days 365 -out certs/domain.crt

While the container is running, open a new commandline. You can find the id with docker ps and copy the certificate out of it using docker cp <containerid>:/certs .

 

Configure Nginx

We now have finished the preparations and are ready to start configuring Nginx.

Create the file https.conf in the folder reverse, and start adding the following upstreams:

upstream docker-releases {
   server nexus:8082;
}
upstream docker-snapshots {
   server nexus:8083;
}
upstream docker-public {
   server nexus:8084;
}

Each upstream refers to a docker repository we configured in Nexus in the previous posts. An upstream is a destination where Nginx can forward it’s requests to. The reference is by hostname and portnumber. The hostname matches the name of the Nexus container in the docker-compose.yml, while the port number matches the http port we defined for each repository individually during the configuration of Nexus.

Next, we add a header field mapping that is required for the docker repository system.

map $upstream_http_docker_distribution_api_version $docker_distribution_api_version {
   '' 'registry/2.0';
}

Finally, we start adding the listeners for the inbound requests. The first listener will be on port 443, which is the default https port as well as the default docker registry port. This will allow us to use just mydocker as a destination, without specifying a port number.

server {
   listen 443 ssl http2;
   listen [::]:443 ssl http2;

   server_name mydocker mydocker.local;
   set $fqdn mydocker;

   ssl_certificate /etc/ssl/domain.crt;
   ssl_certificate_key /etc/ssl/domain.key;

   add_header Strict-Transport-Security "max-age=15768000; includeSubdomains; preload" always;

   ssl_protocols TLSv1.1 TLSv1.2;
   ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
   ssl_prefer_server_ciphers on;
   ssl_session_cache shared:SSL:10m;

   client_max_body_size 0;
   chunked_transfer_encoding on;

   location /v2/ {
      # Do not allow connections from docker 1.5 and earlier
      # docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
      if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
        return 404;
      }

      ## If $docker_distribution_api_version is empty, the header will not be added.
      ## See the map directive above where this variable is defined.
      add_header 'Docker-Distribution-Api-Version' $docker_distribution_api_version always;

      proxy_pass                          http://docker-public;
      proxy_set_header  Host              $http_host;   # required for docker client's sake
      proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
      proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
      proxy_set_header  X-Forwarded-Proto $scheme;
      proxy_read_timeout                  900;
   }
}

Let’s analyse the above configuration:

• A server definition creates a listener for inbound requests on a given port
• We tell Nginx to listen on port 443for ssl-encrypted http2 requests. The interfaces we use are the ip4 and ip6 interfaces of this host.
• Nginx should expect mydocker or mydocker.local as hostname. This is the hostname you’d type in a browser, before it is resolved to the ip-number. It must match the FQDN of the certificate we created earlier.
• The public certificate and private key are provided. We will have to add the files to the docker image later on.
• A required docker header is added.
• Not all ssl protocols are secure. Some are outdated. Some are not supported by docker. We list the protocols and ciphers we wish to use.
• Docker transfers can be huge. You might want to transfer a 16G image. We remove the max-size limit on the request, so that the client is allowed to send this much information. This also means that we can’t encode the entire request in memory, but we have to use a chunked approach.
• The docker repository we use is version 2, so we expect the path to start with /v2/ which allows us to add a v1 or v3 with different settings if we ever need to.
• Exclude old docker versions that don’t play nice.
• We add the header mapping we defined at the very beginning of the http2.conf file, right after the upstreams. The mapping is required because add_header by itself only allows fixed data.
• Finally, we tell Nginx what to do with the incoming request. The request should be forwared (proxied) to the upstream docker-public, which we defined at the very start. Nexus will need some extra headers again, this time they are related to the way a proxy server talks to the proxied server. It is used to forward information, such as the protocol used between the client and the proxy, the ip number of the client etc. We also set a large timeout, because storing large binaries might take some time.

What did we do?

• We have forwarded the default docker port towards the Nexus docker-public repository. This is the group repository for our docker images, which means that when we pull an image from this default endpoint, the image will be retrieved from one of the following locations: docker-releases, docker-snapshots or docker-hub.

This endpoint allows us to find any docker image we created ourselves, or from the public docker-hub repository on the internet. We don’t need to know in which repository it is stored, all magic is handled by Nexus. Great.

The next step is storing docker images. We don’t want to use the generic port for this, but rather, we would like to specify what kind of image we are storing: is it a snapshot build created during development, or is it a candidate release build that might end up on production?

For this, we introduce two endpoints in Nginx, in a way similar to the default endpoint.

server {
   listen 8082 ssl http2;
   listen [::]:8082 ssl http2;

   server_name mydocker mydocker.local;
   set $fqdn mydocker;

   ssl_certificate /etc/ssl/domain.crt;
   ssl_certificate_key /etc/ssl/domain.key;

   add_header Strict-Transport-Security "max-age=15768000; includeSubdomains; preload" always;

   ssl_protocols TLSv1.1 TLSv1.2;
   ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
   ssl_prefer_server_ciphers on;
   ssl_session_cache shared:SSL:10m;

   client_max_body_size 0;
   chunked_transfer_encoding on;

   location /v2/ {
      # Do not allow connections from docker 1.5 and earlier
      # docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
      if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
        return 404;
      }

      ## If $docker_distribution_api_version is empty, the header will not be added.
      ## See the map directive above where this variable is defined.
      add_header 'Docker-Distribution-Api-Version' $docker_distribution_api_version always;

      proxy_pass                          http://docker-releases;
      proxy_set_header  Host              $http_host;   # required for docker client's sake
      proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
      proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
      proxy_set_header  X-Forwarded-Proto $scheme;
      proxy_read_timeout                  900;
   }
}

and

server {
   listen 8083 ssl http2;
   listen [::]:8083 ssl http2;

   server_name mydocker mydocker.local;
   set $fqdn mydocker;

   ssl_certificate /etc/ssl/domain.crt;
   ssl_certificate_key /etc/ssl/domain.key;

   add_header Strict-Transport-Security "max-age=15768000; includeSubdomains; preload" always;

   ssl_protocols TLSv1.1 TLSv1.2;
   ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
   ssl_prefer_server_ciphers on;
   ssl_session_cache shared:SSL:10m;

   client_max_body_size 0;
   chunked_transfer_encoding on;

   location /v2/ {
      # Do not allow connections from docker 1.5 and earlier
      # docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
      if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
        return 404;
      }

      ## If $docker_distribution_api_version is empty, the header will not be added.
      ## See the map directive above where this variable is defined.
      add_header 'Docker-Distribution-Api-Version' $docker_distribution_api_version always;

      proxy_pass                          http://docker-snapshots;
      proxy_set_header  Host              $http_host;   # required for docker client's sake
      proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
      proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
      proxy_set_header  X-Forwarded-Proto $scheme;
      proxy_read_timeout                  900;
   }
}

Note that the only differences are:

• The listen port has changed for both ip4 and ip6
• The proxy_pass destination has changed to the corresponding upstream.

This concludes our Nginx configuration. You can close the editor on https.conf. All we need to do now is to bundle the software, configuration and certificates in a docker image.

Bundling the package

In the reverse folder, we create a file Dockerfile and add the following content:

FROM nginx:1.15-alpine

COPY certs/* /etc/ssl/
COPY https.conf /etc/nginx/conf.d/

This specifies that we use the official Nginx distribution from docker-hub. We select a specific version to avoid update problems in the future. Our certificates are copied to the location we specified in the configuration file. Finally we copy the configuration file itself to the default location where Nginx expects it to be.

We have ended up with the following file-structure for the reverse proxy image:

/reverse
        /certs
              /domain.crt
              /domain.key
        /https.conf
        /Dockerfile

Validate your results by typing docker build . on the command-line inside the reverse folder. It should download the base image and add the required files.

Bring it together

We now have a reverse proxy configured to forward all traffic towards the Nexus repository. All we need is to put them together in a single docker-compose environment, so that they can communicate. Go to the root folder of your project and edit the docker-compose.yml file. We will add some lines, so that the result will be:

version: '2'

services:
  nexus:
    image: sonatype/nexus3:3.12.1
    volumes:
      - "nexus-data:/nexus-data"
    ports:
      - "8081:8081"
    expose:
      - "8082"
      - "8083"
      - "8084"
      - "8085"
      - "8086"
      - "8087"
      - "8088"
      - "8089"
  reverse-proxy:
    build: reverse
    ports:
      - "443:443"
      - "8082:8082"
      - "8083:8083"
      - "8084:8084"
      - "8085:8085"
      - "8086:8086"
      - "8087:8087"
      - "8088:8088"
      - "8089:8089"
      
volumes:
  nexus-data:

Everything we added is in the reverse-proxy service:

• The docker image will be identified by the name reverse-proxy
• It is not a downloaded image like nexus, but instead it’s a locally build image that can be found in the folder reverse
• It exposes a number of ports to the outside world, most specifically port 443, 8082 and 8083. The others are there for future use.

 

Running and testing

Now that we have both Nexus and Nginx in the docker-compose, it is time to start using it. Make sure your previous compose is stopped by typing docker-compose stop

Go to the main directory and build the composition by running docker-compose build (without the . that docker build . uses). You should see output like this:

nexus uses an image, skipping
Building reverse-proxy
Step 1/3 : FROM nginx:alpine
 ---> ba60b24dbad5
Step 2/3 : COPY https.conf /etc/nginx/conf.d/
 ---> 49d1e664e3f5
Step 3/3 : COPY certs/* /etc/ssl/
 ---> 1cc416dc1cd2
Successfully built 1cc416dc1cd2
Successfully tagged demo_reverse-proxy:latest

Now we are ready to run the composition for the first time. Run it with docker-compose up so that it creates missing volumes if needed. To run it afterwards, use docker-compose start instead.

Before we can login, we need to make sure we can find the host mydocker. As discussed before, it needs to be registered. The simplest way is to register the name on the local machine:

On linux, edit the file /etc/hosts

On windows, edit the file C:\Windows\System32\drivers\etc\hosts

Add the following line:

127.0.0.1           mydocker

This tells your computer that any traffic for mydocker will be routed towards the loopback ip number.

 

Now we can start testing. Try to log in to your repository by entering the following command

docker login mydocker

You will be asked for credentials. Provide the username and password for the user you created.

The command should end with the message “Login succeeded”

You are now logged on to the group repository that also contains the reference to docker-hub. Confirm this by doing a docker pull nginx

It should show:

Using default tag: latest
latest: Pulling from library/nginx
Digest: sha256:9fca103a62af6db7f188ac3376c60927db41f88b8d2354bf02d2290a672dc425
Status: Image is up to date for nginx:latest

Now try a docker push nginx. This should give you a denied message: you have no permissions to push to the nginx image on docker-hub.

Lets store this image in our docker repository. Begin by logging in to our snapshot repo: enter docker login mydocker:8083 and provide the usercredentials.

Tag and push the image:

docker tag nginx mydocker:8083/nginx:latest
docker push mydocker:8083/nginx:latest

You should see the layers being uploaded.

Verify the data in Nexus. 

It should show the nginx image you just uploaded:

You can find more details if you drill-down deeper.

 

Securing the admin interface

Now that we have secured the Docker interface, we can add the admin interface as well. Edit the https.conf file and add the admin port as an upstream at the start of the file.

upstream admin-page {
   server nexus:8081;
}

Scroll down to the server component for port 443. This server contains one location for /v2/. What we want is to lroute trafic on /v2/ towards the Docker repository, and to route other data towards the admin pages. This works because no admin pages exists that use /v2/ as prefix.

Below the location /v2/ we add a new location. Make sure it is still inside the server section for port 443.

location / {
   proxy_pass                          http://admin-page;
   proxy_set_header  Host              $http_host;
   proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
   proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
   proxy_set_header  X-Forwarded-Proto $scheme;
   proxy_read_timeout                  90;
}

Test the new endpoint by rebuilding and starting your docker-compose. Make sure you can log on to the admin page. You’ll most likely need to accept the untrusted certificate before you can continue.

Finally, go to the docker-compose.yml and remove the following lines from the nexus configuration. The port will no longer be public.

    ports:
      - "8081:8081"

Instead we will open a private port. Inside the expose section of nexus add:

      - "8081"

Now our http admin port is no longer visible from the outside world, and we can only access it using the public https endpoint of the proxy.

 

Recap

We have introduced a reverse-proxy in order to create a secure repository. The proxy provides a https-secured Docker endpoint, which ensures that the transferred data is not intercepted. The data remains private and unmodified during transport.

In our case, we used a self-generated certificate for encryption. One important thing to note is that we didn’t have to install the certificate in docker. Docker accepts our certificate, even if it isn’t signed by a trusted certificate authority. Should future versions of Docker enforce the trust of te certificate, you’ll need to add the public certificate in the certificate folder on every client, which can be found at

c:\Users\<username>\.docker\machine\certs
or
/home/<username>/.docker/machine/certs

Upcoming post

In the next post, we will deploy the Jenkins master in the docker-compose network and set it up as a work orchestration server.

This post is part of a series about creating a continues integration platform for home use.

 

  Create an artifact repository

  Configure the artifact repository

  Secure the artifact repository

 Create the Jenkins master

 Add a Jenkins slave

 Creating a sample project.

Take your browser to the Nexus login page at http://localhost:8081. Log in with the admin user: At the top-right side of the screen you find the Sign in button. Click it and enter the default credentials

name: admin
password: admin123

Now is a good time to change your credentials to something more secure: click on the admin button in the top bar, and select change password.

After changing your password, proceed towards the Repository administration page by clicking on the clog icon in the top-bar. You should see a navigation menu on the left with different areas for configuration.

Use the left side navigation to go to the Repositories, it’s the second item from the top. It will show you some default repositories that are configured and ready to use. We will go through the details.

 

The repositories in Nexus

Name	Type	Format	Status
maven-central	proxy	maven2	Online – Ready to Connect
maven-public	group	maven2	Online
maven-releases	hosted	maven2	Online
maven-snapshots	hosted	maven2	Online
nuget-group	group	nuget	Online
nuget-hosted	hosted	nuget	Online
nuget.org-proxy	proxy	nuget	Online – Ready to Connect

• The maven-central repository is of type proxy, which means that it doesn’t store data locally, but instead it forwards all requests to the maven-central repository on the internet.
• The maven repositories are using the format maven2, which means that artifacts are stored using the identifier artifactid,groupid,version from the maven build system.
• The maven-releases and maven-snapshots repositories are hosted, which means that the files are stored and managed in this Nexus instance.
• The maven-public repository is of type group. The group consists of the other three maven repositories we just discussed above. (the group members are not visible on this screen). When a group repository receives a request, it tries all the member repositories to find a match, so it aggregates multiple other repositories into a single location.
• The nugget repositories follow the same pattern with a proxy to the internet, a hosted repo for the local data and a group to aggregate both locations into one. It uses the nugget format to identify the artifacts.
• Apart from the identification of the artifacts, each format also has it’s own api used to store and retrieve binaries. By selecting the correct format, you enable the api.

Since we are storing docker images in our build, we will create four extra repositories:

1. A proxy repository to access the master docker repository on the internet
2. A releases repository where we store our final builds, this repo is write-once, read many, so that we can’t accidentally overwrite a published artifact
3. A snapshots repository where we store our work in progress. This repo allows overwriting exiting binaries, so that we can rebuild fast and often.
4. A group repository to aggregate the three previous repositories in one place.

 

The proxy repository

Click on the “Create repository” button

Select docker (proxy)

Name your repo, for example “docker-hub”. It should be marked as Online.

Scroll down and enter the Proxy – Remote storage field. It should read https://hub.docker.com

Mark the checkbox “Use certificates stored in the Nexus truststore to connect to extrnal systems” and click view certificate. It will show some certificate information like the screenshot shown here. Ensure that it is the certificate you expect to see, and press the Add button.

 

Leave the other options as-is, scroll down to the bottom of the page and press “Create repository” to finish.

You now have your first repository, which is a virtual read-only copy of docker hub.

 

The releases repository

Create a new repository. This time, we select type docker (hosted) and name it “docker-releases“.

In the section Repository connectors, we mark the http checkbox, and enter the number 8082 in the data field behind the checkbox. This will make the repository available on the port number 8082 inside the docker container. This will allow us to connect to the repository later on.

Finally we scroll down to the section Hosted. The deployment policy is by default “Allow redeploy”. Since this is a releases repository, and we don’t want to overwrite existing artifacts, we have to select “Disable redeploy” here, so all artifacts become write-once.

Press “Create repository” to finish.

 

The snapshot repository

Create a new repository. Select type docker (hosted) and name it “docker-snapshots“.

In the section Repository connectors, we mark the http checkbox, and enter the number 8083 in the data field behind the checkbox.

Leave all other settings at default.

Press “Create repository” to finish.

 

Aggregating into a single repository

Create a new repository. Select type docker (group) and name it “docker-public“.

In the section Repository connectors, we mark the http checkbox, and enter the number 8084 in the data field behind the checkbox.

Scroll down to the bottom and the other three docker repositories to the group. The order is important here. Nexus will try to find artifacts by trying the repositories from top to bottom. At the top should be docker-releases, then docker-snapshots and finally docker-hub. Add all three and make sure the order is correct.

Press “Create repository” to finish.

 

Summary

We have now added four repositories, as shown in the table below

Name	Type	Format	Status	Purpose
docker-central	proxy	docker	Online – Ready to Connect	Proxy towards docker.io so that we can use public docker images as if they were part of our repository
docker-public	group	docker	Online	One central access point for pulling docker images, regardless of the physical repository where they are stored
docker-releases	hosted	docker	Online	A repository for our final builds. These docker images are protected from accidental overwriting
docker-snapshots	hosted	docker	Online	A repository for our development builds. These docker images can be pushed repeatedly, providing easy of use during development

 

Security in Nexus

Before we can access the repositories, we will have to set up some permissions. We will start by creating a role for docker.

 

Creating the docker role

Enter the role id “nx-docker” and the role name “Docker user“.

Add four priviliges:

• nx-repository-admin-docker-docker-hub.*
• nx-repository-admin-docker-docker-public.*
• nx-repository-admin-docker-docker-releases.*
• nx-repository-admin-docker-docker-snapshots.*
• nx-repository-view-docker-*-*

This will grant rights for all four docker repositories to all users that have this role. Press “Create role” to finalize the role.

Next we can add a user.

 

Create a local user

Navigate to the Users section via the menu on the left side and select create local user. Provide the information for the user you wish to use. Make sure that:

1. Status is Active
2. Roles Granted contains “Docker user“

Press “Create local user” to finish.

 

Activate the docker realm

As a final step, we want to use the docker login system. Therefore we need to activate the docker security realm. In the left side menu, navigate to Realms. It will show the security realms. Add Docker Bearer Token Realm to the active realms and press Save.

 

Conclusion

This concludes the setup of Nexus itself, however we still can’t access Nexus with our tooling. Only the admin interface is exposed. The next blog will guide us through the setup of the reverse proxy, so that we can have a secure connection into the repositories.