I’m a big fan of Kubernetes. It has taken a while to get my head around it, and deploying/operating it every day has helped me on my path to mastering it. As part of that journey, I decided that the HobbitCloud infrastructure would become “Kubernetes first”, and as much of it would be migrated as possible. The first item was Vault.

In this post, I will show how to deploy an initial Vault cluster, before moving on to how to migrate an existing installation.

In the following example, Vault will be installed in the “vault” namespace.

Certificates

Before we begin any installation we need to mint our SSL cert. Our Kubernetes deployment will expose Vault to clients outside the cluster using a LoadBalancer service and will terminate SSL connections on each Vault pod.

To enable each pod to trust each other when starting up, it is important to ensure the SSL certificate contains the SANs for not only each node but also the load-balanced services.

In my environment, Vault will be deployed to the “vault” namespace and will be externally accessible at https://vault.hobbitcloud.com. Therefore my SANs would be:

• vault.hobbitcloud.com
• vault.svc.cluster.local
• vault.vault.svc.cluster.local
• vault-internal
• vault-0.vault-internal
• vault-1.vault-internal
• vault-2.vault.internal
• 127.0.0.1

Whilst it is a long list, it will make sure all endpoints are covered.

When you mint the certs, be sure to name them tls.crt, tls.key and the CA chain vault.ca.

Installation

The first step is to create an initial manifest which creates and correctly labels the namespace:

Apply this using:

kubectl apply -f vault.yaml

Once we have minted our Vault SSL certificate, we need to create a secret for it. To do this we use:

kubectl create secret generic -n vault tls-server --from-file=tls.crt=tls.crt --from-file=tls.key=tls.key --from-file=vault.ca=vault.ca

Clone the HashiCorp Helm repo:

helm repo add hashicorp https://helm.releases.hashicorp.com

Next, we need to extract the values so we can modify them to suit our needs:

helm show values hashicorp/vault > values.yaml

Edit the values file as needed. Things to look for would be:

• Define the storage class for dataStorage and auditStorage
• Mounting the tls-server secret as an extraVolume
• Setting the image tag (I have set mine to “1.14” as I will be performing an upgrade at a later date)

Once you are happy with the values file, deploy the Helm chart:

helm install vault hashicorp/vault -n vault -f values.yaml

Once deployed, check the pods come up successfully:

kubectl get po -n vault --watch

Post-installation

Now that Vault is successfully installed and the pods are running, we need to initialize it. To do that, use:

kubectl exec -n vault --stdin=true --tty=true vault-0 -- vault operator init

Again, this assumes you have installed Vault into the “vault” namespace.

That will generate a list of tokens. Use these tokens to unseal Vault on the vault-0 pod:

kubectl exec -n vault --stdin=true --tty=true vault-0 -- vault operator unseal 73Jlw7BHHFczFIRmLo3J4YdePOf4xGIxXcjuj1LDq7e4
kubectl exec -n vault --stdin=true --tty=true vault-0 -- vault operator unseal HVy7fm+ehO5Um36GR7736Rn+IsPYc3zgUGs7BrW1WP+/
kubectl exec -n vault --stdin=true --tty=true vault-0 -- vault operator unseal XhFinzJUhnTLBV0HtNuvmtl8UNQ6JY0yzErmeSlbRmbX

(The above tokens were taken from a test environment which has long since been torn down).

Repeat the above steps on vault-1 and vault-2.

Check the unseal was successful (on any node):

kubectl exec -n vault --stdin=true --tty=true vault-0 -- vault status

Migration

We now have a new Vault installation, which unfortunately contains no data. Let’s migrate from our existing platform to the new one.

In your terminal, define the following variable and point it at your existing installation:

export VAULT_ADDR=https://<insert existing Vault IP/DNS here>:8200

Login into Vault using your preferred method. If for some reason you don’t trust the SSL certificate, append “—tls-skip-verify”.

Next, take a snapshot of your existing installation:

vault operator raft snapshot save mysnap.snap

Next, we need to get the IP of the Kubernetes service we just deployed:

kubectl get svc -n vault

There will be a number of services shown – we’re after the one labelled “vault-ui”.

Redo the $VAULT_ADDR variable defined above, this time point to the new IP address.

Login to Vault, supplying the token generated when we initialized Vault:

vault login

Restore the snapshot taken previously:

vault operator raft snapshot restore -force ~/mysnap.snap

Voila! Vault has now been restored with your data.

In my next post I will show how to upgrade Vault and restore any needed plugins.