Integrations with external secret management solutions

Introduction

In the realm of Kubernetes, secrets are regular objects protected by Kubernetes RBAC rules. That’s why there is no encryption applied to the secret data, only base 64 encoding.

Encrypting the native secrets inside Kubernetes requires etcd database encryption.

Integration with an external secrets management solution or provider can be done in several ways, depending how the secrets are delivered to pods.

Integration types

Native Kubernetes secrets synchronization

This type of integration uses special custom objects defined via CRDs and an operator which retrieves the secrets from external vault and creates a k8s secret containing the data from vault secret.

Examples:

Hashicorp Vault secrets operator: https://github.com/hashicorp/vault-secrets-operator - integration with Hashicorp Vault
External secrets operator: https://github.com/external-secrets/external-secrets - integration with AWS Secrets Manager, Hashicorp Vault, Google Secrets Manager, Azure KV and others; this one can also sync k8s secrets into external vault secrets
Bitnami sealed secrets https://github.com/bitnami-labs/sealed-secrets - a slightly different solution, the secrets are kept encrypted in git and the encryption keys are stored in the k8s cluster

Kubernetes manifests modification at rendering time

This type of integration alters the secrets by manipulating the k8s manifests before sending them to the k8s API.

Examples:

Helm Secrets plugin https://github.com/jkroepke/Helm-secrets, using vals (https://github.com/Helmfile/vals), can integrate with lot of secret backends:
Vault
AWS SSM Parameter Store
AWS Secrets Manager
AWS S3
GCP Secrets Manager
Google Sheets
SOPS-encrypted files
Terraform State
Argocd vault plugin https://argocd-vault-plugin.readthedocs.io/en/stable/ - integration with Hashicorp Vault; it might use annotations or specific inline path in the secret; annotations usage is limited to one Vault path

Kubernetes manifests modification or injection using k8s mutation webhooks

These types of integrations are using a webhook to change the definition of a pod (or deployment/statefulset/daemonset/job) by adding a sidecar container and in-memory volume

Examples:

Hashicorp Vault agent injector https://developer.hashicorp.com/vault/docs/deploy/kubernetes/injector - it is automatically injecting the vault-agent into the pod and uses it to populate an in-memory volume with secrets retrieved from Hashicorp Vault. It is using pod annotations to specify the vault secrets path

Kubernetes csi secrets storage driver solutions

These solutions use a special type of CSI driver, meant to be used with secrets mounted in in-memory volumes.

Examples:

Hashicorp CSI Secrets provider https://developer.hashicorp.com/vault/docs/deploy/kubernetes/csi - integration with Hashicorp vault
Azure Keyvault CSI Secrets provider https://github.com/Azure/secrets-store-csi-driver-provider-azure - integration with AKS KV
AWS Secrets manager CSI Secrets provider https://github.com/aws/secrets-store-csi-driver-provider-aws - integration with AWS Secrets manager
and so on: https://github.com/Kubernetes-sigs/secrets-store-csi-driver

In this case, the pod definition has to be changed to mount a CSI storage volume instead of a native k8s secret.

Limitations and compatibility issues

The first two integration types (NativeK8sSecretsSync/AlterationAtRenderingTime) do not require any change from the application perspective since they are still using native Kubernetes secrets. However, the templating has to push custom objects instead of native k8s secrets in case of the first type NativeK8sSecretsSync.
The last two integration methods (K8sManifestInjection/K8sCSISecrets) require the application to read the secrets from files instead of environment variables.
AlterationAtRenderingTime method type can’t deal out of box with secrets rotation.

External management solutions compatible with SOT Helm deployment

Currently, SOT Helm Deployment is not compatible with any of the methods described in the K8sManifestInjection or K8sCSISecrets.

From the category of NativeK8sSecretsSync methods it supports HashiCorp Vault integration via Vault Secrets Operator.

The following methods are certified to work with it:

Helm secrets plugin

Helm Secrets plugin can be used directly with Helm override files since it is agnostic of templated manifests (recommended choice).

ArgoCD vault plugin

SOT Helm deployment supports OOB ArgoCD Vault plugin.

In order to use this special integration one has to explicitly pass the secrets as Helm override values. This has to be done for each individual secret that should be handled via the ArgoCD vault plugin using ArgoCD Vault inline placeholders. The automatic secret generation offered by the SOT operators will be disabled in this case. The following example shows how a secret can be configured via an override:

global:
  nexeedHost: myhost.mydomain.com
  modules:
    portal:
      enabled: true
      macmaPortalAdminUser: <path:path/to/the/secret#userkey>
      macmaPortalAdminPassword: <path:path/to/the/secret#passwordkey>

where path/to/the/secret is the path to the secret in HashiCorp Vault and userkey is the key corresponding to the user stored in vault while passwordkey is the key corresponding to the password.

If, for example, one uses the kv secret engine and creates a folder called global inside of it, the path will be kv/data/global.

Vault static operator integration

Another secret provider that can be used is Vault Static Secret Operator for HashiCorp Vault.

Before starting to use this integration one needs to:

Save the secrets in Vault. An important aspect is to avoid saving secrets values that should end in the same Kubernetes secret at different vault paths. If during chart rendering, a secret is composed from vault references pointing to different path, the process will fail.
Configure the Vault authorization. This includes choosing an authentication method to the vault and setting up an authentication policy. The utility-toolkit works with two authentication methods, appRole and Kubernetes.
Install and configure the Vault Secret Operator. The most important resources to configure after VSO installation is a VaultAuth object. This specifies where the address of vault is, the auth mechanism, and depending on the authentication type, other required details.
A good example for these steps is provided here: Configuring Vault Secret Operator with AppRole ID – HashiCorp Help Center. In this example, the VaultAuth is configured for the appRole authentication.

VSO usage in the SOT Helm chart

The SOT Helm chart can generate VaultStaticSecrets when a flag and a set of parameters are passed at the global level. In this case the secret values are expected to be a specific vault reference. These two important component will be detailed next.

The vault flag and vault options.
- The first parameter is secretsProvider. When this has value "vso" it changes completely how the secrets are processed.
- The second parameter that it’s now required is vsoOptions. This is an object that contains all the settings needed for VSO mode.
  - vaultAuthRef - this field should contain VaultAuth name and if it’s not in the same namespace also the name;
  - vaultMount - this field is the name of the mount in Vault. This value is common for all secretes. Default: kv;
  - useRolloutRestartTargets - VaultStaticSecret object can restart the objects using/mounting a secret that it creates. This field is a boolean. When it’s true the VaultStaticSecret object will contain in rolloutRestartTargets fields all the objects using the generated secret;
  - vaultAuthMethod - this field accepts one of these two values, either appRole or kubernetes;
  - vaultAuthConfigAppRole - this is required only when vaultAuthMethod is set with appRole. When using appRole a secret containing the secret ID of the appRole is required in each namespace where VaultStaticSecrets are deployed. This value it’s an object with 2 members, detailed next;
    
    secretName: this specify the name of the secret to be created. This should match the name from the VaultAuth object at appRole.secretRef;
    
    secretValue: this should be secret key for the role configured in VaultAuth; This is the value that it will end up in the secret.
The vault secret reference is similar to this string: ref+vso://path/toSecret[secretkeyInVault]?2.
- ref+vso:// - this part it’s just a convention aligned with other secret reference models (i.e. in Argo);
- path/toSecret - this it’s the path to the secret in vault separated with "/";
- [secretkeyInVault] - this specifies that at the path mentioned previously, the secret named secretkeyInVault should be consumed;
- ?2 - this is the version of the secret and can be omitted. If this is omitted the last version will be used.