Introduction

Let's Encrypt is a "free, automated, and open certificate authority" that provides the browser-trusted certificates required to establish https connections. Whilst the success of Let's Encrypt came as wonderful news for developers everywhere, I still wondered whether it would be possible to create, install and renew certificates on my NGINX server without incurring downtime.

Maybe I haven't been looking hard enough, but all of the articles I read on the subject of installing and renewing Let's Encrypt certificates seemed to involve something like:

• run the certificate management agent on a live web server
• restart the web server / container

This process would work just fine if you only ever had to do it once, but things aren't quite that simple in the real world, and as soon as renewal time rolls around then a reboot will be required. A reboot will mean that visitors are very likely to experience an outage, and everything will suck on a grand scale until the application spins back up again.

The bad news is that working around this problem will require a small amount of extra effort, but the good news is that it's totally possible to use free certificates without sacrificing valuable uptime.

The method that I'll show you today isn't necessarily the only way to achieve a seamless installation / renewal, but it works for my chosen technology stack and method of deployment. I'll be performing SSH operations against a live (virtual) server, which is a little bit dirty in terms of devops, but it's a tradeoff that's probably unavoidable if Let's Encrypt is being used.

There are many different options available with Let's Encrypt, such as "auto renew" and "force renew", which I have neither the time nor the experience to cover to any useful extent, so I'm going to keep things as straightforward as I possibly can, and hopefully you'll be able to adapt the steps to suit your own requirements.

This article has been updated (22nd May 2021): the latest version of Let's Encrypt requires a different installation method, so this article has been updated to reflect that

Goals

By the end of this article, you should be able to:

• generate a Let's Encrypt certificate using an NGINX Docker container
• retrieve the generated keys so that they can be built into a new version of the NGINX Docker image

Prerequisites

• Linux based virtual machine (other platforms should involve similar processes, but I personally use Linux)
• understanding of Docker and NGINX
• understanding of SSH and SSL usage
• understanding of the Linux command line, and ability to install packages on your particular distro

Creating a basic NGINX config

If you don't already have a Let's Encrypt certificate configured in NGINX, you'll need to generate a new one.

Below I've provided a basic configuration that you can modify for your own purposes, but be sure to at least change yourdomain.com to your actual domain.

We'll use the commented section later in the tutorial, so just leave that alone for now

user              nginx;
worker_processes  1;
error_log         /var/log/nginx/error.log warn;
pid               /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    server {
        listen 80 default_server;

        server_name    _;

        location /.well-known/acme-challenge {
            root /var/www/letsencrypt;
        }
    }

    #server {
    #    # SSL configuration
    #    listen 443 ssl default_server;
    #
    #    ssl_certificate /etc/letsencrypt/live/yourdomain.com/fullchain.pem;
    #    ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem;
    #    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    #    ssl_prefer_server_ciphers on;
    #    ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
    #
    #
    #    server_name _;
    #    return 403 "FORBIDDEN";
    #}

    server {
        # SSL configuration
        listen 443 ssl;
        root        /usr/share/nginx/html;
        server_name yourdomain.com www.yourdomain.com;

        location / {
            index  index.html index.htm;
        }
    }
}

The main point of interest here is the first server section:

server {
    listen 80 default_server;

    server_name    _;

    location /.well-known/acme-challenge {
        root /var/www/letsencrypt;
    }
}

Let's Encrypt actually requires an http (as opposed to https) connection to connect to the webserver and confirm that it is the origin of the certificate generation request.

Make a note of that root directory, as we'll need to ensure that it exists later on

The second server block simply serves up the default NGINX greeting file to any requests that it receives over https, which of course can't occur until we have a certificate configured.

server {
    # SSL configuration
    listen 443 ssl;
    root        /usr/share/nginx/html;
    server_name yourdomain.com www.yourdomain.com;

    location / {
        index  index.html index.htm;
    }
}

Modify the above code for your purposes, and place it in your NGINX configuration so that we can build it into a Docker container later on

For the purposes of this tutorial, I'll be referring to this file as nginx.conf

Creating a Let's Encrypt config

It's probably possible to specify all of the necessary options directly on the command line for the Let's Encrypt agent, but personally I prefer to keep them in a file. Be sure to investigate the options that suit you best though.

# the domain that we're creating the certificate for
# if you need multiple domains then separate them with commas (e.g. domaina.com,domainb.com,domainc.com)
domains = yourdomain.com
 
# increase key size
rsa-key-size = 2048 # Or you could use a value of 4096
 
# this version of the api works at the time of writing, but keep checking the Let's Encrypt documentation for updates
server = https://acme-v01.api.letsencrypt.org/directory
 
# an address to use for renewal reminders (this seems to be a required value)
email = you@youremail.com
 
# Let's Encrypt provides a colourful UI, but we can turn this off if we're running things in the terminal
text = True
 
# tells Let's Encrypt to authenticate the origin by creating a random file
authenticator = webroot

# the random file (mentioned above) is created in in the .well-known/acme-challenge/ directory by default, but we specify a different directory here to make things easier when using a Docker container
webroot-path = /tmp

Modify the above code for your purposes, and place it into a file that we can build into the Docker container. For the purposes of this tutorial, I'll refer to this file as le.conf

Local directory structure

Before we go any further, let's take a look at what we have so far.

We should have two files:

• nginx.conf: contains our main NGINX configuration
• le.conf: contains our Let's Encrypt configuration

We'll eventually be transferring both of these files into a Docker container, so I like to keep the structure of my local project as similar as possible to the structure that will exist inside the container. This isn't essential, but I find that it leads to fewer path related errors.

The relevant part of my project structure currently looks like the following, so be sure to take account of any differences in your structure as we move forward:

nginx
└───etc
.   └───letsencrypt
.   .   └───configs
.   .       |   le.conf
.   └───nginx
.       |  nginx.conf

Creating a Docker file

The next step is to build an NGINX image containing our configuration. There are quite a few different ways to set up NGINX, but here's a nice simple version that works fine as a demonstration:

FROM nginx:1.13.8-alpine
LABEL maintainer="Fizzymatt"

# Set up directory structure
RUN mkdir /etc/letsencrypt \
    && mkdir -p /var/www/letsencrypt \
    && rm /etc/nginx/conf.d/default.conf

# Copy files from the host
ADD ./nginx/etc/nginx/nginx.conf /etc/nginx/nginx.conf
ADD ./nginx/etc/letsencrypt /etc/letsencrypt

Let's have a quick look at what's going in that Dockerfile.

The RUN commands first create a couple of directories that we'll need, and then remove that default NGINX configuration, as we'll be using the new one that we created previously instead.

RUN mkdir /etc/letsencrypt \
    && mkdir -p /var/www/letsencrypt \
    && rm /etc/nginx/conf.d/default.conf

The first ADD command then copies the nginx.conf file from our local project (assuming that your directory structure is the same as mine, of course) and into the specified location in the NGINX image.

ADD ./nginx/etc/nginx/nginx.conf /etc/nginx/nginx.conf

The second ADD command does a similar thing, but with the entire contents of the letsencrypt directory.

ADD ./nginx/etc/letsencrypt /etc/letsencrypt

So just to clarify things: when your NGINX image is built (based on the little demo project that I have here), it should contain - amongst other things - the following files and directories:

• /etc/letsencrypt/configs/le.conf
• /var/www/letsencrypt
• /etc/nginx/nginx.conf

The method used to actually build your NGINX image will differ depending on your needs. Personally I use a cloud based CI solution, but your build process might be entirely different.

However you decide to run the NGINX container, you must mount the /tmp directory (which will eventually be created on the live server when we run the Let's Encrypt client) into the /var/www/letsencrypt directory inside the container. As I'll describe below, this is to allow the remote Let's Encrypt authentication service to verify that a file exists by making a request to the NGINX container, even though that file was created by an instance of the Let's Encrypt client that is running directly on the live server (so outside the NGINX container)

However you choose to do it, build your NGINX container and start it running on your server. At this stage you'll need to make sure that traffic from at least port 80 can reach your container, regardless of whether that's arriving directly or through a proxy

Generating a certificate

With your container up and running on your server, it's finally time to do what we came here for. Let's generate a certificate.

In a real life scenario the following process should all be automated, perhaps using a tool such as Ansible, but for this demonstration I'll be running everything manually. I'll also be describing everything as simply as I possibly can, so be sure to do your own research around the associated techniques and technologies (particularly security)

Connect to your live server (which should now have a running NGINX container, as described in the previous section) using SSH

If you don't already have Git installed, then install it now. The following command should work for a Centos based distro, but modify according to your setup:

yum install git -y

Use Git to clone the Let's Encrypt repo:

git clone https://github.com/letsencrypt/letsencrypt /opt/letsencrypt

If the /opt/letsencrypt directory already exists (which would indicate that you might have an existing Let's Encrypt installation) then you'll get an error

Transfer the le.conf file (that we created previously) across to /etc/letsencrypt/configs/le.conf on your live server, creating the directory structure if necessary. You might want to use the scp command if you're doing this manually over SSH

The latest version of Let's Encrypt requires a package called snapd, so let's get that installed

Run the following command to install epel-release:

yum install epel-release -y

Install the snapd package itself:

yum install snapd -y

Configure snapd:

systemctl enable --now snapd.socket

Enable classic snap support:

ln -s /var/lib/snapd/snap /snap

Install classic certbot using snap:

snap install --classic certbot

The command that we'll run to generate the certificate is as follows:

certbot certonly --config /etc/letsencrypt/configs/le.conf

The --config flag tells Let's Encrypt to create a certificate using the custom config (le.conf) that we we provided.

The certonly subcommand tells Let's Encrypt that we want to generate the certificate without installing it. Remember that we're currently connected to our live server, but not to our NGINX container (which is where the certificate should be installed), so we don't want to perform an installation at our current location.

Run the command described previously in order to generate the certificate

The Let's Encrypt client will now create a randomly named file in the /tmp directory that we specified in our le.conf config file.

We ran our NGINX container so that the /tmp directory (on the server) was mounted into volume /var/www/letsencrypt (in the NGINX container). This means that the randomly named file should immediately be available to NGINX at /var/www/letsencrypt, as soon as it is created in /tmp (if you check the /tmp directory, you should see the temporary file that was created).

The Let's Encrypt client also instructs a remote authentication service to send a request for the same random file to the live server, which should serve it up using the route that we created in our NGINX config:

server {
    listen 80 default_server;

    server_name    _;

    location /.well-known/acme-challenge {
        root /var/www/letsencrypt;
    }
}

If the remote service can verify the existence of the file that we just created with the client tool, then the authentication process has been successful and you should see console output similar to:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for yourdomain.com
Using the webroot path /tmp for all unmatched domains.
Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/yourdomain.com/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/yourdomain.com/privkey.pem

Retrieving the .pem files

So far, we have a certificate file and a private key file on our live server. They're no good to us there, so we'll need to retrieve them so that we can build them into a new Docker image that will eventually be used to replace the NGINX container that's currently running.

Ideally this would be an automated process, but for now you can get hold of the files manually, possibly with the scp command if you're using SSH.

You should find the files in whichever directory you created at /etc/letsencrypt/live/.

Whatever your chosen method, download the keys and store them in a location that will be accessible when we build the new NGINX image. Personally I like to store the contents of the .pem files inside secret variables in my CI solution, but use whatever method suits you best

Think very carefully about where you store the contents of privkey.pem. You don't want this to fall into the wrong hands

Extended NGINX Configuration

Now we need to modify our NGINX config so that it'll accept https connections and use our freshly generated certificate to verify the identity of our server.

We saw this code earlier on, but notice that I've uncommented the server block that deals with the SSL files.

user              nginx;
worker_processes  1;
error_log         /var/log/nginx/error.log warn;
pid               /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    server {
        listen 80 default_server;

        server_name    _;

        location /.well-known/acme-challenge {
            root /var/www/letsencrypt;
        }
    }

    server {    
        # SSL configuration
        listen 443 ssl default_server;
    
        ssl_certificate /etc/letsencrypt/live/yourdomain.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem;
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_prefer_server_ciphers on;
        ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
    
    
        server_name _;
        return 403 "FORBIDDEN";
    }

    server {
        # SSL configuration
        listen 443 ssl;
        root        /usr/share/nginx/html;
        server_name yourdomain.com www.yourdomain.com;

        location / {
            index  index.html index.htm;
        }
    }
}

The finer points of the SSL configuration are beyond the scope of this article, and will be something that you'll likely tailor to your own requirements in any case.

For now, just focus on the ssl_certificate and ssl_certificate_key paths. They should point to the certificate and key files respectively, using paths within the NGINX container, the location of which will eventually be created once we've rebuilt the NGINX image and started a new instance of the container.

Modify the above code for your purposes, place it in your NGINX configuration, and rebuild the NGINX image so that it also contains the fullchain.pem and privkey.pem files at the correct locations

Deployment

How you manage your deployment will depend upon your own circumstances, but I like to use a blue / green style setup that involves having a floating ip pointed at the current live server (virtual in my case).

Once I've generated the necessary certificate files and built them into a new NGINX image, I spin the container up on a brand new server and then alter the floating ip so that it points at the new server rather than the original one.

My new server is then considered to be "live", and the old one can be destroyed as soon as I've run all my post deployment tests and I'm sure that everything is working ok.

You'll only be able to achieve a truly zero downtime deployment if you're using a load balancer to manage traffic, but the steps I've described here should remain the same. Using a single node, downtime will only occur while the floating ip is altered, which should be a matter of milliseconds

Deploy your new NGINX image, start a container, and ensure that requests are being served over https using a valid Let's Encrypt certificate

Renewing the certificate

Once you everything up and running with a Let's Encrypt certificate, the process of renewal should be more straight forward.

I like to spin up a new virtual machine (as described above), and then perform all the steps from the Generating a certificate section onwards, with the exception of the NGINX config changes (described in the Extended NGINX Configuration section), which should already be in place in your build environment.

The commands should be exactly the same, but be aware that it might be necessary to pass the --force-renewal flag to the Let's Encrypt client if your certificates aren't close to their renewal date. For example:

/opt/letsencrypt/certbot-auto --agree-tos --force-renewal --config /etc/letsencrypt/configs/le.conf certonly

Don't forget to retrieve the new .pem files and make them available to the build process when recreating the new NGINX image, and be aware that Let's Encrypt currently allows a maximum of around five certificate renewal requests in a seven day period

Summary

So that's a very stripped down example of how I generate a Let's Encrypt certificate for a containerised project.

Retrieving the .pem files and recreating the NGINX image should allow for zero downtime deployments where a load balancer is involved, and close to zero downtime where a single node is used.

Realistically, the steps I've described should be automated and performed in conjunction with a CI tool if they're to work well in a production environment.

As always, you should do your own research into all of the technologies used here, and use the information that I've provided as a springboard towards your own solution. As I mentioned at the beginning of the article, this is just a technique that happens to work well for me, and I make no claims to its suitability for other people.

If you find my article useful, then please don't forget to share a link to it :)