Sync Secret from Vault to K8S using External Secrets Operator
When doing work on many micro services, i have to find a way to manage environment variable for them, previously i used AWS Secret Manager but when working on multiple environment with many account, the effort for manage IAM Account, Role, Permission is very large, so i switched to Vault, this help me reduce a lot of working on secret management. Now i can centralize all secret in just one Vault server. When i work on many environment i just need to isolate the secret engine path for them. One token with read permission is enough to retrieve the secret on that environment(or account).
In this post i will enable key-vault secret engine for development environment and sync to k8s using https://github.com/external-secrets/external-secrets/
1. Setup Vault server
For demo purpose, i will setup a single node using docker-compose .
❯ tree vault-server
vault-server
├── docker-compose.yml
└── vault.json
0 directories, 2 filesThe content of compose file and vault.json is same as below.
version: "3.9"
networks:
default:
name: learn-vault
driver: bridge
volumes:
vaultlog:
vaultfile:
services:
vault:
image: vault:1.12.0
ports:
- 8200:8200
restart: always
volumes:
- vaultlog:/vault/logs
- vaultfile:/vault/file
- ./vault.json:/vault/config/vault.json
entrypoint: vault server -config=/vault/config/vault.json{
"storage": {
"file": {
"path": "/vault/file"
}
},
"listener": {
"tcp": {
"address": "0.0.0.0:8200",
"tls_disable": "true"
}
},
"default_lease_ttl": "8640h",
"max_lease_ttl": "86400h",
"ui": true,
"log_level": "Info",
"disable_mlock": true
}To start up vault run the up command.
gitops-k8s/vault-server on master on ☁️ (ap-southeast-1)
❯ docker compose up -d
[+] Running 4/4
⠿ Network learn-vault Created 0.0s
⠿ Volume "vault-server_vaultlog" Created 0.0s
⠿ Volume "vault-server_vaultfile" Created 0.0s
⠿ Container vault-server-vault-1 Started 0.4s
gitops-k8s/vault-server on master on ☁️ (ap-southeast-1)
❯Now, you can login to http://localhost:8200 to start using Vault. For me i already deploy it here https://vault.dongnguyen.link
2. Create Vault Secret Key Vault Engine.
2.1 First you need to login to Vault server.
gitops-k8s/vault-server on master on ☁️ (ap-southeast-1)
❯ export VAULT_ADDR=https://vault.dongnguyen.link
gitops-k8s/vault-server on master on ☁️ (ap-southeast-1)
❯ vault login hvs.XXXXXXXXXXXXX
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.
Key Value
--- -----
token hvs.XXXXXXXXXXXXX
token_accessor YYYYYYYYYYYYYYYYY
token_duration ∞
token_renewable false
token_policies ["root"]
identity_policies []
policies ["root"]
gitops-k8s/vault-server on master on ☁️ (ap-southeast-1)
❯2.2 Enable new secret engine for development environment.
gitops-k8s/vault-server on master on ☁️ (ap-southeast-1)
❯ vault secrets enable -version=2 --path=development kv
Success! Enabled the kv secrets engine at: development/
gitops-k8s/vault-server on master on ☁️ (ap-southeast-1)
❯2.3 Now on Vault UI, you will see this.
2.4 Create a api secret.
On Terminal run vault cli to create a secret with some value.
❯ vault kv put -mount=development api PORT=30006
==== Secret Path ====
development/data/api
======= Metadata =======
Key Value
--- -----
created_time 2023-01-10T04:28:01.729528959Z
custom_metadata <nil>
deletion_time n/a
destroyed false
version 1
gitops-k8s/vault-server on master [?] on ☁️ (ap-southeast-1)
❯ Now, open Vault UI you will see this.
Click to view development/api secret.
2.5. Create Policy and Token to use with K8s.
Run Vault CLI to create it. The content of this HCL file is
path "sys/mounts"
{
capabilities = ["read"]
}
path "development/*"
{
capabilities = ["list"]
}
path "development/data/*"
{
capabilities = ["read"]
}gitops-k8s/vault-server on master [?] on ☁️ (ap-southeast-1)
❯ vault policy write development-reader ../vault-policy/development-reader-policy.hcl
Success! Uploaded policy: development-reader
gitops-k8s/vault-server on master [?] on ☁️ (ap-southeast-1)
❯
Now on Vault UI, go to the policy tab, you will see this.
Click to development-reader to see it.
2.6 Create a token with the development-reader policy.
gitops-k8s/vault-server on master [?] on ☁️ (ap-southeast-1)
❯ vault token create -policy=development-reader -period=8640h
Key Value
--- -----
token hvs.CAESIMO8teS1Z8Hl-oUjlvumwwqYnVWt2kQebR2g-ZZPWnehGh4KHGh2cy5XaXZocHp4bk9UYzdnVlpzdk8yWW9xdEg
token_accessor kM5etWc7DbFwPtg0Ih42N4uO
token_duration 8640h
token_renewable true
token_policies ["default" "development-reader"]
identity_policies []
policies ["default" "development-reader"]
gitops-k8s/vault-server on master [?] on ☁️ (ap-southeast-1)
❯Now you can open another terminal, login using this token and try to read the development/api secret.
gitops-k8s/vault-server on master [?] on ☁️ (ap-southeast-1)
❯ export VAULT_ADDR=https://vault.dongnguyen.link
gitops-k8s/vault-server on master [?] on ☁️ (ap-southeast-1)
❯ vault login hvs.CAESIMO8teS1Z8Hl-oUjlvumwwqYnVWt2kQebR2g-ZZPWnehGh4KHGh2cy5XaXZocHp4bk9UYzdnVlpzdk8yWW9xdEg
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.
Key Value
--- -----
token hvs.CAESIMO8teS1Z8Hl-oUjlvumwwqYnVWt2kQebR2g-ZZPWnehGh4KHGh2cy5XaXZocHp4bk9UYzdnVlpzdk8yWW9xdEg
token_accessor kM5etWc7DbFwPtg0Ih42N4uO
token_duration 8639h58m47s
token_renewable true
token_policies ["default" "development-reader"]
identity_policies []
policies ["default" "development-reader"]
gitops-k8s/vault-server on master [?] on ☁️ (ap-southeast-1)
❯ vault kv get -format=json development/api | jq ".data.data"
{
"PORT": "30006"
}
gitops-k8s/vault-server on master [?] on ☁️ (ap-southeast-1)
❯
As you can see, we can able to read api secret. Now it time to sync it to k8s using the token above.
3. Install External Secret Operator on K8s.
3.1 Install using Helm
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets \
external-secrets/external-secrets \
-n external-secrets \
--create-namespace \
--set installCRDs=true
4. Sync secret
4.1 Create a secret on k8s to store vault token for development environment. First we need to get the base64 encoded vault for vault token.
Then create a secret to store vault token in external-secrets namespace.
apiVersion: v1
kind: Secret
metadata:
name: development-vault-token
namespace: external-secrets
type: Opaque
data:
token: aHZzLkNBRVNJTU84dGVTMVo4SGwtb1VqbHZ1bXd3cVluVld0MmtRZWJSMmctWlpQV25laEdoNEtIR2gyY3k1WGFYWm9jSHA0Yms5VVl6ZG5WbHB6ZGs4eVdXOXhkRWcK
Now open K8S Dashboard, you will see it.
4.2 Create a ClusterSecretStore.
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
name: vault-backend
spec:
provider:
vault:
server: "https://vault.dongnguyen.link"
path: "development"
version: "v2"
auth:
tokenSecretRef:
name: "development-vault-token"
key: "token"
namespace: external-secretsRun the command below
kubectl create -f vault-cluster-secret-store.yml
If you open k8s dashboard, in the CRD session, you will see it.
4.3 Sync development/api secret to development namespace.
Use kubectl to create a secret same as below. Before you you need to create namespace development .
The development-api.yaml file content
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: api
namespace: development
spec:
refreshInterval: "15s"
secretStoreRef:
name: vault-backend
kind: ClusterSecretStore
target:
name: api
dataFrom:
- extract:
key: development/apiNow create a external secret , this secret is refer to development/api secret on Vault server and map it to api secret in development namespace on K8s cluster.
Wait some second, now you can check it in development namespace.
That is all, thank for reading !
