Installing and renewing a Let's Encrypt certificate without downtime
Let's Encrypt is a "free, automated, and open certificate authority" that provides the browser-trusted certificates required to establish https connections. Whilst the success of Let's Encrypt came as wonderful news for developers everywhere, I still wondered whether it would be possible to create, install and renew certificates on my NGINX server without incurring downtime.
Maybe I haven't been looking hard enough, but all of the articles I read on the subject of installing and renewing Let's Encrypt certificates seemed to involve something like:
- run the certificate management agent on a live web server
- restart the web server / container
This process would work just fine if you only ever had to do it once, but things aren't quite that simple in the real world, and as soon as renewal time rolls around then a reboot will be required. A reboot will mean that visitors are very likely to experience an outage, and everything will suck on a grand scale until the application spins back up again.
The bad news is that working around this problem will require a small amount of extra effort, but the good news is that it's totally possible to use free certificates without sacrificing valuable uptime.
The method that I'll show you today isn't necessarily the only way to achieve a seamless installation / renewal, but it works for my chosen technology stack and method of deployment. I'll be performing SSH operations against a live (virtual) server, which is a little bit dirty in terms of devops, but it's a tradeoff that's probably unavoidable if Let's Encrypt is being used.
There are many different options available with Let's Encrypt, such as "auto renew" and "force renew", which I have neither the time nor the experience to cover to any useful extent, so I'm going to keep things as straightforward as I possibly can, and hopefully you'll be able to adapt the steps to suit your own requirements.
By the end of this article, you should be able to:
- generate a Let's Encrypt certificate using an NGINX Docker container
- retrieve the generated keys so that they can be built into a new version of the NGINX Docker image
- Linux based virtual machine (other platforms should involve similar processes, but I personally use Linux)
- understanding of Docker and NGINX
- understanding of SSH and SSL usage
- understanding of the Linux command line, and ability to install packages on your particular distro
Creating a basic NGINX config
If you don't already have a Let's Encrypt certificate configured in NGINX, you'll need to generate a new one.
Below I've provided a basic configuration that you can modify for your own purposes, but be sure to at least change yourdomain.com to your actual domain.
The main point of interest here is the first server section:
Let's Encrypt actually requires an http (as opposed to https) connection to connect to the webserver and confirm that it is the origin of the certificate generation request.
The second server block simply serves up the default NGINX greeting file to any requests that it receives over https, which of course can't occur until we have a certificate configured.
Creating a Let's Encrypt config
It's probably possible to specify all of the necessary options directly on the command line for the Let's Encrypt agent, but personally I prefer to keep them in a file. Be sure to investigate the options that suit you best though.
Local directory structure
Before we go any further, let's take a look at what we have so far.
We should have two files:
- nginx.conf: contains our main NGINX configuration
- le.conf: contains our Let's Encrypt configuration
We'll eventually be transferring both of these files into a Docker container, so I like to keep the structure of my local project as similar as possible to the structure that will exist inside the container. This isn't essential, but I find that it leads to fewer path related errors.
The relevant part of my project structure currently looks like the following, so be sure to take account of any differences in your structure as we move forward:
Creating a Docker file
The next step is to build an NGINX image containing our configuration. There are quite a few different ways to set up NGINX, but here's a nice simple version that works fine as a demonstration:
Let's have a quick look at what's going in that Dockerfile.
The RUN commands first create a couple of directories that we'll need, and then remove that default NGINX configuration, as we'll be using the new one that we created previously instead.
The first ADD command then copies the nginx.conf file from our local project (assuming that your directory structure is the same as mine, of course) and into the specified location in the NGINX image.
The second ADD command does a similar thing, but with the entire contents of the letsencrypt directory.
So just to clarify things: when your NGINX image is built (based on the little demo project that I have here), it should contain - amongst other things - the following files and directories:
The method used to actually build your NGINX image will differ depending on your needs. Personally I use a cloud based CI solution, but your build process might be entirely different.
Generating a certificate
With your container up and running on your server, it's finally time to do what we came here for. Let's generate a certificate.
The command that we'll run to generate the certificate is as follows:
The --config flag tells Let's Encrypt to create a certificate using the custom config (le.conf) that we we provided.
The --agree-tos flag automatically provides a positive answer to any questions at installation, which isn't necessary while we're running the command manually, but will come in very handy if you wish to automate things later.
The certonly subcommand tells Let's Encrypt that we want to generate the certificate without installing it. Remember that we're currently connected to our live server, but not to our NGINX container (which is where the certificate should be installed), so we don't want to perform an installation at our current location.
The Let's Encrypt client will now create a randomly named file in the /tmp directory that we specified in our le.conf config file.
We ran our NGINX container so that the /tmp directory (on the server) was mounted into volume /var/www/letsencrypt (in the NGINX container). This means that the randomly named file should immediately be available to NGINX at /var/www/letsencrypt, as soon as it is created in /tmp (if you check the /tmp directory, you should see the temporary file that was created).
The Let's Encrypt client also instructs a remote authentication service to send a request for the same random file to the live server, which should serve it up using the route that we created in our NGINX config:
If the remote service can verify the existence of the file that we just created with the client tool, then the authentication process has been successful and you should see console output similar to:
Retrieving the .pem files
So far, we have a certificate file and a private key file on our live server. They're no good to us there, so we'll need to retrieve them so that we can build them into a new Docker image that will eventually be used to replace the NGINX container that's currently running.
Ideally this would be an automated process, but for now you can get hold of the files manually, possibly with the scp command if you're using SSH.
You should find the files in whichever directory you created at /etc/letsencrypt/live/.
Extended NGINX Configuration
Now we need to modify our NGINX config so that it'll accept https connections and use our freshly generated certificate to verify the identity of our server.
We saw this code earlier on, but notice that I've uncommented the server block that deals with the SSL files.
The finer points of the SSL configuration are beyond the scope of this article, and will be something that you'll likely tailor to your own requirements in any case.
For now, just focus on the ssl_certificate and ssl_certificate_key paths. They should point to the certificate and key files respectively, using paths within the NGINX container, the location of which will eventually be created once we've rebuilt the NGINX image and started a new instance of the container.
How you manage your deployment will depend upon your own circumstances, but I like to use a blue / green style setup that involves having a floating ip pointed at the current live server (virtual in my case).
Once I've generated the necessary certificate files and built them into a new NGINX image, I spin the container up on a brand new server and then alter the floating ip so that it points at the new server rather than the original one.
My new server is then considered to be "live", and the old one can be destroyed as soon as I've run all my post deployment tests and I'm sure that everything is working ok.
Renewing the certificate
Once you everything up and running with a Let's Encrypt certificate, the process of renewal should be more straight forward.
I like to spin up a new virtual machine (as described above), and then perform all the steps from the Generating a certificate section onwards, with the exception of the NGINX config changes (described in the Extended NGINX Configuration section), which should already be in place in your build environment.
The commands should be exactly the same, but be aware that it might be necessary to pass the --force-renewal flag to the Let's Encrypt client if your certificates aren't close to their renewal date. For example:
So that's a very stripped down example of how I generate a Let's Encrypt certificate for a containerised project.
Retrieving the .pem files and recreating the NGINX image should allow for zero downtime deployments where a load balancer is involved, and close to zero downtime where a single node is used.
Realistically, the steps I've described should be automated and performed in conjunction with a CI tool if they're to work well in a production environment.
As always, you should do your own research into all of the technologies used here, and use the information that I've provided as a springboard towards your own solution. As I mentioned at the beginning of the article, this is just a technique that happens to work well for me, and I make no claims to its suitability for other people.
If you find my article useful, then please don't forget to share a link to it :)