Recently I demonstrated how to install HashiCorp Vault on Kubernetes using Helm. Whilst it’s great to get up and running, upgrades are also an important part of application lifecycle management. In this post I’ll demonstrate how we can use rolling upgrades to bring Vault to the latest version, along with any plugins you may be using in your infrastructure.

Deploying HashiCorp Vault in Kubernetes with Helm is relatively straightforward. However upgrades are just as important, and there are a couple of things that need to be done to achieve a successful upgrade.

First off, check your pods are all running fine. In the following examples, the Vault namespace is simply called “vault”:

kubectl get po -n vault

Hopefully, you will have three pods, all running fine.

Before we make any changes it is a good idea to perform a backup, just in case.
From the command line, set your VAULT_ADDR variable and login using your preferred method. HobbitCloud Vault is set to use Active Directory:

vault login -method=ldap username=<insert username here>

Save a snapshot:

vault operator raft snapshot save mybackup.snap

Next, we need to update Helm. Assuming you already have the repo as that is what you deployed from:

helm repo update

Depending on how your initial deployment went, you will probably have a supplementary overrides or values file. However this may have changed with the new version, so it’s best to check if there are any changes:

helm show values hashicorp/vault > newvalues.yaml

In my case, the values file was pretty much the same except for the version number. At the time of writing, I just increased the deployment version tag from “1.14.0” to “1.15.2”.

Upgrade the deployment using Helm, again assuming your namespace is “vault”:

helm upgrade vault hashicorp/vault -n vault -f newvalues.yaml

That will upgrade the deployment, but nothing will change until the existing pods are deleted.

The way to achieve this is to delete the standby pods first. Find out which ones these are using:

kubectl get pods -n vault -l vault-active=false

Delete these pods in turn:

kubectl delete po -n vault vault-<<insert first standby pod>>
kubectl delete po -n vault vault-<<insert second standby pod>>

Repeat the command one last time to remove the final (active) pod.

Check all pods are now recreated and running:

kubectl get po -n vault --watch

If all the pods are indeed running then congratulations, you have successfully upgraded HashiCorp Vault!

Update Any Plugins

The last remaining task is to replace any plugins you may use, as these are removed when the pods are destroyed if you haven’t set any persistence for them.

In the following example, I will use the vault-secrets-gen plugin as this is an important plugin in my infrastructure automation.

First, umount the existing plugin (in my case “gen”):

vault secrets disable gen

Copy the plugin binary to each pod:

kubectl cp -n vault vault-secrets-gen vault-0:/vault/plugins
kubectl cp -n vault vault-secrets-gen vault-1:/vault/plugins
kubectl cp -n vault vault-secrets-gen vault-2:/vault/plugins

Export the checksum of the plugin:

export SHA256=$(sha256sum ./vault-secrets-gen | cut -d' ' -f1)

Register the plugin:

vault plugin register -sha256="${SHA256}" -command="vault-secrets-gen" secret secrets-gen

Enable the plugin:

vault secrets enable -path="gen" -plugin-name="secrets-gen" plugin

The plugin should now be registered and enabled (and visible in the UI).

Test the plugin, which for the vault-secrets-gen is as simple as running:

vault write gen/passphrase words=4

Or:

vault write gen/password length=36 symbols=0