Sync Secret from Vault to K8S using External Secrets Operator

When doing work on many micro services, i have to find a way to manage environment variable for them, previously i used AWS Secret Manager but when working on multiple environment with many account, the effort for manage IAM Account, Role, Permission is very large, so i switched to Vault, this help me reduce a lot of working on secret management. Now i can centralize all secret in just one Vault server. When i work on many environment i just need to isolate the secret engine path for them. One token with read permission is enough to retrieve the secret on that environment(or account).

In this post i will enable key-vault secret engine for development environment and sync to k8s using https://github.com/external-secrets/external-secrets/

1. Setup Vault server

For demo purpose, i will setup a single node using docker-compose .

❯ tree vault-server 
vault-server
├── docker-compose.yml
└── vault.json
0 directories, 2 files

The content of compose file and vault.json is same as below.

version: "3.9"
networks:
default:
name: learn-vault
driver: bridge
volumes:
vaultlog:
vaultfile:
services:
vault:
image: vault:1.12.0
ports:
- 8200:8200
restart: always
volumes:
- vaultlog:/vault/logs
- vaultfile:/vault/file
- ./vault.json:/vault/config/vault.json
entrypoint: vault server -config=/vault/config/vault.json
{
"storage": {
"file": {
"path": "/vault/file"
}
},
"listener": {
"tcp": {
"address": "0.0.0.0:8200",
"tls_disable": "true"
}
},
"default_lease_ttl": "8640h",
"max_lease_ttl": "86400h",
"ui": true,
"log_level": "Info",
"disable_mlock": true
}

To start up vault run the up command.

gitops-k8s/vault-server on  master on ☁️  (ap-southeast-1) 
❯ docker compose up -d
[+] Running 4/4
⠿ Network learn-vault Created 0.0s
⠿ Volume "vault-server_vaultlog" Created 0.0s
⠿ Volume "vault-server_vaultfile" Created 0.0s
⠿ Container vault-server-vault-1 Started 0.4s

gitops-k8s/vault-server on  master on ☁️ (ap-southeast-1)

Now, you can login to http://localhost:8200 to start using Vault. For me i already deploy it here https://vault.dongnguyen.link

2. Create Vault Secret Key Vault Engine.

2.1 First you need to login to Vault server.

gitops-k8s/vault-server on  master on ☁️  (ap-southeast-1) 
❯ export VAULT_ADDR=https://vault.dongnguyen.link

gitops-k8s/vault-server on  master on ☁️ (ap-southeast-1)
❯ vault login hvs.XXXXXXXXXXXXX
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key Value
--- -----
token hvs.XXXXXXXXXXXXX
token_accessor YYYYYYYYYYYYYYYYY
token_duration ∞
token_renewable false
token_policies ["root"]
identity_policies []
policies ["root"]

gitops-k8s/vault-server on  master on ☁️ (ap-southeast-1)

2.2 Enable new secret engine for development environment.

gitops-k8s/vault-server on  master on ☁️  (ap-southeast-1) 
❯ vault secrets enable -version=2 --path=development kv
Success! Enabled the kv secrets engine at: development/

gitops-k8s/vault-server on  master on ☁️ (ap-southeast-1)

2.3 Now on Vault UI, you will see this.

Vault Secret Engine List

2.4 Create a api secret.

On Terminal run vault cli to create a secret with some value.

❯ vault kv put -mount=development api PORT=30006              
==== Secret Path ====
development/data/api

======= Metadata =======
Key Value
--- -----
created_time 2023-01-10T04:28:01.729528959Z
custom_metadata <nil>
deletion_time n/a
destroyed false
version 1

gitops-k8s/vault-server on  master [?] on ☁️ (ap-southeast-1)

Now, open Vault UI you will see this.

Secret List

Click to view development/api secret.

development/api secret

2.5. Create Policy and Token to use with K8s.

Run Vault CLI to create it. The content of this HCL file is

path "sys/mounts"
{
capabilities = ["read"]
}
path "development/*"
{
capabilities = ["list"]
}
path "development/data/*"
{
capabilities = ["read"]
}
gitops-k8s/vault-server on  master [?] on ☁️  (ap-southeast-1) 
❯ vault policy write development-reader ../vault-policy/development-reader-policy.hcl
Success! Uploaded policy: development-reader

gitops-k8s/vault-server on  master [?] on ☁️ (ap-southeast-1)

Now on Vault UI, go to the policy tab, you will see this.

Vault policy list

Click to development-reader to see it.

2.6 Create a token with the development-reader policy.

gitops-k8s/vault-server on  master [?] on ☁️  (ap-southeast-1) 
❯ vault token create -policy=development-reader -period=8640h
Key Value
--- -----
token hvs.CAESIMO8teS1Z8Hl-oUjlvumwwqYnVWt2kQebR2g-ZZPWnehGh4KHGh2cy5XaXZocHp4bk9UYzdnVlpzdk8yWW9xdEg
token_accessor kM5etWc7DbFwPtg0Ih42N4uO
token_duration 8640h
token_renewable true
token_policies ["default" "development-reader"]
identity_policies []
policies ["default" "development-reader"]

gitops-k8s/vault-server on  master [?] on ☁️ (ap-southeast-1)

Now you can open another terminal, login using this token and try to read the development/api secret.

gitops-k8s/vault-server on  master [?] on ☁️  (ap-southeast-1) 
❯ export VAULT_ADDR=https://vault.dongnguyen.link

gitops-k8s/vault-server on  master [?] on ☁️ (ap-southeast-1)
❯ vault login hvs.CAESIMO8teS1Z8Hl-oUjlvumwwqYnVWt2kQebR2g-ZZPWnehGh4KHGh2cy5XaXZocHp4bk9UYzdnVlpzdk8yWW9xdEg
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key Value
--- -----
token hvs.CAESIMO8teS1Z8Hl-oUjlvumwwqYnVWt2kQebR2g-ZZPWnehGh4KHGh2cy5XaXZocHp4bk9UYzdnVlpzdk8yWW9xdEg
token_accessor kM5etWc7DbFwPtg0Ih42N4uO
token_duration 8639h58m47s
token_renewable true
token_policies ["default" "development-reader"]
identity_policies []
policies ["default" "development-reader"]

gitops-k8s/vault-server on  master [?] on ☁️ (ap-southeast-1)
❯ vault kv get -format=json development/api | jq ".data.data"
{
"PORT": "30006"
}

gitops-k8s/vault-server on  master [?] on ☁️ (ap-southeast-1)


As you can see, we can able to read api secret. Now it time to sync it to k8s using the token above.

3. Install External Secret Operator on K8s.

3.1 Install using Helm

helm repo add external-secrets https://charts.external-secrets.io

helm install external-secrets \
external-secrets/external-secrets \
-n external-secrets \
--create-namespace \
--set installCRDs=true
Install ESO using Helm

4. Sync secret

4.1 Create a secret on k8s to store vault token for development environment. First we need to get the base64 encoded vault for vault token.

get base64 encoded vault for vault token

Then create a secret to store vault token in external-secrets namespace.

apiVersion: v1
kind: Secret
metadata:
name: development-vault-token
namespace: external-secrets
type: Opaque
data:
token: aHZzLkNBRVNJTU84dGVTMVo4SGwtb1VqbHZ1bXd3cVluVld0MmtRZWJSMmctWlpQV25laEdoNEtIR2gyY3k1WGFYWm9jSHA0Yms5VVl6ZG5WbHB6ZGs4eVdXOXhkRWcK
kubectl create vault token secret

Now open K8S Dashboard, you will see it.

4.2 Create a ClusterSecretStore.

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
name: vault-backend
spec:
provider:
vault:
server: "https://vault.dongnguyen.link"
path: "development"
version: "v2"
auth:
tokenSecretRef:
name: "development-vault-token"
key: "token"
namespace: external-secrets

Run the command below

kubectl create -f vault-cluster-secret-store.yml

If you open k8s dashboard, in the CRD session, you will see it.

4.3 Sync development/api secret to development namespace.

Use kubectl to create a secret same as below. Before you you need to create namespace development .

kubectl create namespace development

The development-api.yaml file content

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: api
namespace: development
spec:
refreshInterval: "15s"
secretStoreRef:
name: vault-backend
kind: ClusterSecretStore
target:
name: api
dataFrom:
- extract:
key: development/api

Now create a external secret , this secret is refer to development/api secret on Vault server and map it to api secret in development namespace on K8s cluster.

Wait some second, now you can check it in development namespace.

That is all, thank for reading !

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store