Skip to main content

Automation of the certificate renewal

Current state​

As of beginning of 2022 we use certbot manually every 3 months to generate a new multi-domain wildcard certificate for all our microservices. The whole process takes like half an hour, but still would be nice to have it fully automated.

Copr​

Copr uses certbot as well, but with automated renewals, so can we do it in a similar way?

Well, I'm not sure. They run certbot and the renewal service & timer (systemd equivalent of cron job) on the same machine(s) as web server(s). But we run our httpd/nginx in containers and probably don't want to have more processes (httpd/nginx + crond) running in each of the containers. For nginx, it'd also mean a separate image with crond & certbot because we use official nginx image. But some people do that.

Or maybe running two containers in one pod? Does anyone has experience with that?

With RWO volumes it's also not possible to run certbot+crond in another pod with the certificate on a shared volume (because the RWO volume can't be mounted to more containers).

Other options​

Both, copr and us use certbot certonly, they just use --standalone while we use --manual. But there are more plugins, so can some of them be or any help?

Certbot plugins​

There are special plugins for apache and nginx which might help a lot with automation, but the problem with running certbot/crond+httpd/nginx in the same container still persists.

There are also DNS plugins which would probably allow us to create/renew the certificate(s) in a different container, but there's no plugin for Google Domains, just Google Cloud DNS which is a different service.

OCP way​

Wait, there must be some Openshift/k8s native way to this.

Openshift-acme​

is what we used to utilize as a first version of our cert management, but then we switched to DNS challenge, removed it and probably don't want to return.

Cert-manager​

is a Kubernetes add-on, which can be installed in OCP as operator but requires cluster-admin privileges to do so. After installation, one also needs to configure a specific ACME issuer with solver being either:

mod_md​

mod_md is Apache module for automated TLS cert provisioning/renewal using ACME (Let's Encrypt). One just has to tweak the httpd configuration and everything else should be automated. A few howtos:

The thing is that we haven't configured httpd directly since we started using mod_wsgi-express which configures httpd for us. So we'd need to figure out how to tweak the mod_wsgi-express generated httpd.conf before the server starts Or do we need to request mod_wsgi to support mod_md in a similar way it supports mod_ssl?

We'd been actually installing mod_md prior to moving to mod_wsgi-express but AFAIK never used it (directly).

Use a web server with automatic HTTPS, like Caddy​

How difficult the switch from httpd (API, dashboard) and nginx (metrics) would be, has not been researched.

Output​

There's no straightforward way to automate the certificate generation/renewal completely. All options look like they'd still need a lot of experimentation and/or some compromises.