Centralising Certificate Management - Part 1

With encrypted protocols now the only acceptable way to present services to the Internet, most organisations will find that all their edge hosts must have certificates. Unless you have shares in a certificate authority or you are a natural contrarian, the most reliable way to deliver and renew certificates to your edge hosts will be via the ACME protocol. That generally means using Let's Encrypt (I've heard that some commercial suppliers support it as well).

But do we really want all our edge hosts managing their own certificates? Remembering that to request, prove control of the domain and fetch a certificate, requires far more access and permissions than should make you comfortable.

Edge or Central?

A web server or mail server which uses an ACME certificate, should not also contain credentials for updating DNS or have privileges to edit web content. Yet this is exactly the most common method used to request and fetch an ACME certificate.

If you follow a 'least privilege' principle for securing your systems, you only want content and credentials that are absolutely required to deliver your service and no more.

What we really want is a trusted system, one that isnt on the network edge, from where we can push the needed material to our public facing systems.

Luckily this sort of design pattern is not new or unusual. In fact if you use any kind of configuration management or orchestration, you're probably very close to this design already.

Central Controller

It's a common pattern to use a central host or hosts which run Ansible to push configuration to all other systems. This is true whether you use Ansible Tower/AWX or stand alone Ansible playbooks.

DNS-01

The ACME protocol supports 3 challenge types, but dns-01 makes the most sense for a host that we are trying not to expose to external systems. The dns-01 challenge process requires that a DNS record be created to prove domain control. So the controlling host will need to send the update to the DNS hosting service.

ACME Controller

Our central ansible controller will become an ACME Controller. Here are the functions we'll need to automate:

1. Create an ACME account, or use a pre-existing one. This account will contain contact details for any certificate related alerts we want to receive.
2. Generate certificate signing requests (CSRs) for all the domain labels for which we want certificates. A private key will need to be generated for each.
3. Connect to the ACME service and fetch the challenge value. This is the random string we'll need to add into DNS.
4. Update DNS with the challenge string. Each domain label will require its own unique record.
5. Fetch the certificate, once DNS has been updated.
6. Restart or reload any service using an old certificate.
7. Publish new TLSA records for each service and remove any old records.

You can do all of the above using a mix of acme client software, bash or python and cron. But I prefer to keep my configuration logic in Ansible, so that's what I'll be writing about in [Part 2]({% post_url 2020-03-15-centralising-certificate-management-part-2 %}).

This series

• Part 1
• Part 2
• Part 3