One of the biggest challenges operations teams face is SSL certificate issuing and renewal. Often this is because different applications, like vendor appliances, have a complicated renewal process. Others can be because corporations simply miss the renewal date. If large enterprises like Microsoft sometimes fail at this, what hope do the rest of us have?Other posts in this series:

1. Configure Vault
2. Configure Ansible

I recently decided to improve this process in HobbitCloud by ensuring that certificate operations on all non-vendor appliances would now be automatic. It’s an added hassle I just don’t need, and any process that can be automated, should be.

In HobbitCloud we have a two-tier PKI, consisting of an offline OpenSSL root CA and an online issuing CA running HashiCorp Vault. The latter is provisioned as a three-node cluster using Raft storage, and all internal hosts and services secured by SSL are issued certificates from this CA.

In the past the request, issuing and renewal process has mostly been manual, using the Vault CLI tool on my workstation. However, as all compute workloads are now provisioned using vRealize Automation and Ansible, this process also needs to be automated.

Happily HashiCorp has just the tool for the job with the Vault Agent. Simply put, this is a binary executable, together with a config file and two templates which define where the issued certificates will be stored.

Whilst installing Vault Agent is trivially simple, like any manual process, it does not scale well. With this in mind, I will be using Ansible Tower to deploy Vault Agent.

Configure Vault

For this exercise I will assume you already have a Vault CA up and running and are able to successfully request and issue certificates from it.

To deploy the Vault Agent using Ansible we will be employing two service accounts which will use AppRole authentication. These will be:

• sa_ansible
• sa_vault-agent

This will enable Ansible to connect to Vault (using sa_ansible) to read in the role ID and secret ID of sa_vault-agent. These will outputted to files, which the Vault Agent, once installed, will use to authenticate to the Vault cluster.

First, enable AppRole authentication:

vault auth enable approle

Create and apply a policy for the sa_vault-agent service account. This just needs the ability to create and update certificates from the PKI engine (substitute the name of your CA accordingly):

cat acl_sa_vault-agent.hcl | vault policy write acl_sa_vault-agent -

Create the sa_vault-agent role:

vault write auth/approle/role/sa_vault-agent \
  token_ttl=0m \
  token_num_uses=0 \
  secret_id_num_uses=0 \
  token_no_default_policy=false \
  token_policies="acl_sa_vault-agent"

Create and apply a policy for the sa_ansible role. This just needs the ability to retrieve the role ID and secret ID for the Vault Agent service account:

cat acl_sa_ansible.hcl | vault policy write acl_sa_ansible -

Now create the sa_ansible AppRole:

vault write auth/approle/role/sa_ansible \
  token_ttl=0m \
  token_num_uses=0 \
  secret_id_num_uses=0 \
  token_no_default_policy=false \
  token_policies="acl_sa_ansible"

Finally, get the role ID for the sa_ansible service account from Vault and generate a secret ID:

vault read auth/approle/role/sa_ansible/role-id

vault write -f auth/approle/role/my-role/secret-id

Record both of these for later.

Test Vault using Postman

Now we have a role ID and secret ID for sa_ansible, we need to confirm it can also retrieve the same for the sa_vault-agent role. While we can use cURL to do this quite easily, we can automate a part of this process using Postman.

Create a new environment in Postman for Vault. I suggest a variable for the Vault address and one for the token. Supply a value for the former and leave the latter blank:

Create a new tab using the following details:

• Method: POST
• URL: https://{{vault-fqdn}}/v1/auth/approle/login
• Header:
  • Content-Type: application/json
• Body: Json containing the role ID and secret ID (generated above) for the sa_ansible role:

• Tests: Paste in the following:

Once authenticated using the AppRole role ID and secret ID, this will enable us to store the generated token for further use.

Click Send and verify you get a 200 response code, a client_token in the payload, and this same value is reflected in the “token” environment variable:

Test: Retrieve the sa_vault-agent role ID

Create a new tab using the following:

• Method: GET
• URL: https://{{vault-fqdn}}/v1/auth/approle/role/sa_vault-agent/role-id
• Header:
  • Content-Type: application/json
  • X-Vault-Token: {{token}}

The result should be another 200, with a role ID in the data field.

Test: Generate a secret ID for sa_vault-agent role

Create one final tab, this time using:

• Method: POST
• URL: https://{{vault-fqdn}}/v1/auth/approle/role/sa_vault-agent/secret-id
• Header:
  • Content-Type: application/json
  • X-Vault-Token: {{token}}

Again, you should receive a 200, and a secret ID field. This proves that the sa_ansible role has the permissions to read the role ID for sa_vault-agent and generate a new secret ID.

Coming Up

In part 2 I will document the various Ansible playbooks and configuration files we shall need, along with how to create the job in Tower.