Integrating HashiCorp Vault into your cluster

Vault abstracts away authentication via auth methods and authorization via policies.

One of my teammates took an initiative to improve the way secrets were managed in the source code. He brought up HashiCorp Vault, which is a secret management solution that enables developers to handle the storage and retrieval of sensitive information with ease.

My responsibility was to set up Vault in one of the existing Amazon EKS clusters. While integrating Vault into the system, I encountered some configuration issues, which forced me to learn deeper about many different components of Vault. The more I dived into it, the more I realized how useful and powerful it was.

So, let's start with a high-level overview of vault architecture. It is roughly composed of four components:

  • Vault server: a vault server can be deployed with a helm chart, which would generate a set of required resources easily and quickly.
  • Storage backend (e.g. Consul, MySQL, etc): MySQL (v8.0.30) docker image is used. A script containing sql statements in the docker-entrypoint-initdb.d file is mounted, such that initial set up is executed when a MySQL container starts up.
  • Authentication backend (third-party services such as AWS, GitHub, LDAP, Okta or built-in AppRole): AppRole auth method is used.
  • Client: users, machines, applications that consume secrets

In order to retrieve a secret from vault, a token is required. This token has a policy attached to it, and the policy specifies granular access (which action on which path is allowed for this particular client). For instance, one policy would state that only "read" action is allowed at /secret/data/webapp path.

To acquire a token, a machine or an application needs to authenticate with vault against a given auth method. One example of an auth method is AppRole. To authenticate with AppRole, the following credentials should be provided:

  • ROLE_ID: the role id is like a static username that identifies an application. it can be shared across multiple instances of the same app, and can be embedded into a machine image.

  • SECRET_ID: the secret id is like a dynamically generated password that is associated with a particular role id.

The acquired token can then be provided to make API requests to vault server. Best practice regarding the AppRole authentication is to use two different channels to provide role id and secret id, but never both. For instance, we can obscure the role id from jenkins (by embedding into a machine image), but allow jenkins to deliver a wrapped secret id to the application.

References

  • https://www.vaultproject.io/docs/what-is-vault