A clean start

I currently use 3 different cloud providers and some local hardware for my test lab. I've been happy with all of them for most of my usage in the last 10 years. Recently one of the providers broke their backup job (which I pay $$$ for) and was unable to fix it for several weeks. I'd wanted to consolidate my usage anyway, so this was the excuse I needed to start moving my hosts.

Some of my hosts are more than 14 years old. They are still up to date and using the latest kernels, thank you Gentoo. But I'd like to start again and use a more templated configuration. I also want to template the networks and access control lists for my lab. My home lab is small enough that I could just set everything up manually, but where's the fun in that?

Where to begin?

I'm moving most of my workloads to AWS and only keeping my privileged Ansible controller in my home lab. So I have two initial tasks.

1. Set up my ansible controller to access aws (this post)
2. Deploy some basic scaffolding into AWS for my workloads (my next post)
3. Set up ansible's inventory (my third post)

Setting up Ansible authentication to AWS

Ansible comes with AWS support builtin and there's also some community collections offering additional functionality. Initially I'll only need the builtin support. Let's install the AWS collection first:

1[runner@controller]$ ansible-galaxy collection install amazon.aws

AWS Credentials

Ansible can use several methods to authenticate with AWS.

1. You can put your credentials into ~/.aws/credentials and Ansible will use those.
2. You can use environment variables:

1[runner@controller]$ export awx_access_key="AAABBBCCC"`

3. You can keep your credentials in an Ansible vault file:

1ansible-vault encrypt my-access-keys.yaml

4. If you're using Tower/AWX there's the credentials feature which will store these for you.
5. You can decide that Access Keys are so 2021 and use IAM Roles Anywhere.

I strongly believe in following best practices at home in order to reinforce good habits for my professional life. Managing secrets is one of those areas where we can get lazy at home, then we find excuses to be equally lazy in our work place dev environments and before we know it, our production secrets haven't been stored safely or haven't been rotated in years. The first 4 options above can be secured and rotated, but they rely on operational discipline not present in many workplaces and certainly not in my lab. IAM Roles Anywhere allows us to use client certificates to authenticate with AWS. So I can use my pre-existing certificate life cycle tools and the certificate expiry means I can't use the same cert forever. It's also new and shiny.

Deploying the CA

Check out my post about deploying a CA using CFSSL and Ansible. I end up with a root CA and an intermediate with which I'll be signing my hosts' CSRs for the purposes of authenticating to AWS.

Create a role

My privileged Ansible controller will need to manage EC2 hosts. So I'll need to create a role for that purpose.

1. Select IAM from the AWS console home

2. Select Roles from the IAM page

3. Choose the Create Role button

4. Find and select the Permissions Policies you want to use.

   

IAM Roles Anywhere set up

With the role created, go to the bottom of the Roles page and you'll see a Roles Anywhere section. Hit the Manage button. We need to first set up a trust anchor, which is high falutin' security person speak for the Certificate Authority which will sign all the host certificates. Whether you followed my post or you use a different PKI solution, you will have a public certificate for your issuing CA.

1[signer@my-pki-host]$ cat issuer-key.pem
2-----BEGIN EC PRIVATE KEY-----
3MMMMMMMMMMAAAAAAADDDDDDDDDDDDDDDeeeeeeee
4...

1[signer@my-pki-host]$ cat issuer.pem
2-----BEGIN CERTIFICATE-----
3MMMMMMAAAAAAaabbbcccdddeeefffggghhh
4...

That's the certificate we want! Take a copy of the contents of that file. We'll use that to set up the trust anchor.

Trust Anchor

1. Back in the AWS console, select Create a trust anchor.

2. Give your trust anchor a name: Trusty-mctrust-face.

3. If you're using the AWS Certificate Manager Private CA you can use it's ID here. Otherwise if you're using your own CA select the External certificate bundle option. Paste the content of the public key file we found earlier, in to the text box.

Profile

Next we create a profile. This can act as the outer boundary for any permissions you assign to hosts authenticated via IAM Roles Anywhere. Since the profile could contain many roles, some of which may contain access you want to restrict, you can use policies within the profile to limit access in addition to what is already defined within the role.

1. Hit Create profile.

2. Give the profile a name. I recommend describing the hosts or access type.

3. Select at least one role. In this case, the role we created earlier.

Hit Create a profile and you're done!

Deploying IAM Roles Anywhere on your hosts

So we have a trust anchor with at least one role within one profile. I didn't set up IAM Roles Anywhere just for a single host and neither should you. Since this is a post about Ansible, let's try to deploy the necessary files to any host that needs it, using a playbook. The key difference between using IAM Roles Anywhere and plain old access_key and secret_key values is that you'll need to fetch credentials on-demand. So instead of having a static entry in our ~/.aws/credentials file, we'll include a program call within the ~/.aws/config file.

Fetching credentials for IAM Roles Anywhere can be done using a helper script. You can read that page or you can be lazy and use the template here.

We'll need to deploy a ~/.aws/config file, so I'll be using a Jinja2 template. If you're brave you could use Ansible's builtin.lineinfile module to achieve something similar. My aws_config.j2 template is very simple:

1# templates/aws_config.j2
2# Note the name of the certificate and private key. Check the value of {{ansible_hostname}} if these files are not found.
3[default]
4region=ap-southeast-2
5credential_process = {{localbin}}/aws_signing_helper credential-process --certificate {{aws_dir}}/{{ansible_hostname}}.pem --private-key {{aws_dir}}/{{ansible_hostname}}-key.pem --trust-anchor-arn {{ta_arn}} --profile-arn {{profile_arn}} --role-arn {{role_arn}}

The config file requires the ARN (found in the AWS console) for the Trust Anchor (ta_arn), the profile to use for the host (profile_arn) and the role (role_arn). Since my lab wont be that complex, I'll use a single vault file for each Trust Anchor<->Profile<->Role combination I plan to use.

1# vault/default_host_role.yaml
2  ta_arn: "111122223333444"
3  profile_arn: "555666777888"
4  role_arn: "999000111222"

Encrypt the file with ansible-vault.

1[runner@controller]$ ansible-vault encrypt vault/default_host_role --vault-password-file ~/.ansible/vault

I won't be copying the certificates or generating CSRs with this playbook. I like to keep my playbooks as simple as possible.

The playbook below will copy the IAM Roles Anywhere helper file to the host and validate the checksum. Unfortunately AWS has not signed the checksum, so there's no signature to validate. The playbook will set up the required directories, restricting access to a named user and group. Since copying the aws_config.j2 template to ~/.aws/config is potentially destructive, the file will be backed up on change.

 1---
 2  - name: Deploy IAM roles anywhere compatible config file.
 3    hosts: "{{ target| default('localhost')}}"
 4    gather_facts: yes
 5    become: yes
 6    vars:
 7      signing_helper_url: "https://s3.amazonaws.com/roles-anywhere-credential-helper/CredentialHelper/latest/linux_amd64/aws_signing_helper"
 8      signing_helper_checksum: "308424e4ecd002ae3eb155c443b941e08ed66032868567f519eccf8d2c8c5387"
 9      user: "aws_runner" # this is the aws running user
10      group: "aws_runner"
11      home_dir: "/home"
12      localbin: "{{home_dir}}/{{user}}/.local/bin"
13      aws_dir: "{{home_dir}}/{{user}}/.aws"
14      # ta_arn: Found in vault
15      # profile_arn: Found in vault
16      # role_arn: Found in vault
17    vars_files: "{{ vault_file| default ('vault/default_host_role.yaml') }}"
18    tasks:
19      - name: Create local bin directory if not present
20        file:
21          path: "{{localbin}}"
22          mode: '0770'
23          owner: "{{user}}"
24          group: "{{group}}"
25          state: directory
26      - name: Create .aws directory if not present
27        file:
28          path: "{{aws_dir}}"
29          mode: '0770'
30          owner: "{{user}}"
31          group: "{{group}}"
32          state: directory
33      - name: Ensure credential helper is present
34        ansible.builtin.get_url:
35          checksum: "sha256:{{signing_helper_checksum}}"
36          url: "{{signing_helper_url}}"
37          dest: "{{home_dir}}/{{user}}/.local/bin/aws_signing_helper"
38          mode: '0550'
39          owner: "{{user}}"
40          group: "{{group}}"
41      - name: Copy config file
42        ansible.builtin.template:
43          src: aws_config.j2
44          dest: "{{aws_dir}}/config"
45          backup: true
46          owner: "{{user}}"
47          group: "{{group}}"
48          mode: '0640'

You can now use the AWS CLI or SDK as you would normally and they will fetch credentials via IAM Roles Anywhere instead of relying on statically configured secrets.

Summary

I've set up IAM Roles Anywhere for those hosts which will not be moving to AWS, but which may need to consume an AWS service. In particular, my privileged Ansible controller will need access for the next phase.

Get the code

Checkout the files on github

Resources

• The official IAM Roles Anywhere docs
• The official IAM Roles Anywhere blog