Kubernetes Self Signing and Trusting your CA

What's in This Post

Here are some quick steps to assist you in creating a private CA within Kubernetes that you can then trust on the Linux hosts within your perimeter.

Install cert-manager

The instructions for installing cert-manager are quite straight forward, so I wont include any further details here.

Create the CA

cert-manager provides some very good instructions on creating a self-signed CA. However I prefer to use a single Cluster Issuer. Using different issuers might allow an organisation to tailor certificate policies per namespace, which inturn might be used by separate organisations. In my case, my lab replicates a single organisation operational model, so a common set of policies covering all certificates in the k8s cluster makes a lot more sense.

 2apiVersion: cert-manager.io/v1
 3kind: ClusterIssuer
 5  name: selfsigned-cluster-issuer
 7  selfSigned: {}
 9apiVersion: cert-manager.io/v1
10kind: Certificate
12  name: my-lab-root
13  namespace: cert-manager # Default cert-manager namespace for cluster wide resources
15  isCA: true
16  commonName: my-lab-selfsigned-ca
17  secretName: my-root-secret
18  privateKey:
19    algorithm: ECDSA
20    size: 256
21  issuerRef:
22    name: selfsigned-cluster-issuer
23    kind: ClusterIssuer
24    group: cert-manager.io
26apiVersion: cert-manager.io/v1
27kind: ClusterIssuer
29  name: my-lab-root-issuer # using the self signed issuer caused problems with importing the crt into linux
31  ca:
32    secretName: my-root-secret

The example above only differs from the cert-manager documented example by creating a ClusterIssuer.

Updating Trust Anchors

Now that we have a CA that will be issuing certificates to endpoints in Kubernetes, we'll need to add the root certificate into the trust store of any clients. The certificate is stored in base64 form within the my-root-secret we created above. We can see the certificates and private key by looking at the contents of the secret:

 1user@k8s-master:~$ kubectl get secret my-root-secret -n cert-manager -o yaml
 3apiVersion: v1
 5  ca.crt: 1234567890ABCDEFG=
 6  tls.crt: 1234567890ABCDEFG=
 7  tls.key: abcdefg1234567890=
 8kind: Secret
10  annotations:
11    cert-manager.io/alt-names: ""
12    cert-manager.io/certificate-name: my-lab-root
13    cert-manager.io/common-name: my-lab-selfsigned-ca
14    cert-manager.io/ip-sans: ""
15    cert-manager.io/issuer-group: cert-manager.io
16    cert-manager.io/issuer-kind: ClusterIssuer
17    cert-manager.io/issuer-name: selfsigned-cluster-issuer
18    cert-manager.io/uri-sans: ""
19  creationTimestamp: "2021-10-26T23:13:17Z"
20  name: my-root-secret
21  namespace: cert-manager
22  resourceVersion: "12975422"
23  uid: abcd2039-5866-42e0-b078-63457c6e6c22
24type: kubernetes.io/tls

We only need the ca.crt value in order to update our clients' trust stores. We'll also need to decode the output from base64. We can do all that with a single line:

1user@k8s-master:~$ kubectl get secret my-root-secret -n cert-manager -o yaml | grep ca.crt | cut -d ':' -f 2 | sed -e 's/^[ \t]*//' | base64 -d > my-ca.pem
2user@k8s-master:~$ cat my-ca.pem

Trust the CA on your clients

To update your Centos hosts to trust the newly created CA you'll need to update the ca-bundle on each host. Copy the CA certificate pem file to each client's trust anchor source directory. Then use the update-ca-trust extract command to force a rebuild of the bundled certificates to now include your CA.

1user@my-host01:~$ cp my-ca.pem /etc/pki/ca-trust/source/anchors/
2user@my-host01:~$ update-ca-trust extract

We can check to see if the new CA has been added by looking at the updated bundle file:

1user@my-host01:~$ head /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
2# my-lab-selfsigned-ca