Secret Management
Secrets never live in this repository. They are stored in a Vault on each cluster, and the External Secrets operator copies them into Kubernetes Secrets.
Vault
Every cluster (mocha, turing, doppio) runs its own Vault — OpenBao, deployed by
the bank-vaults operator — in the bank-vaults
namespace. The Vault custom resource is in <cluster>/system/bank-vaults/vault/vault.yaml.
The operator handles what used to be manual:
- Init and unseal. It initialises Vault with 5 key shares (threshold 3) and
stores them, along with the root token, in the
vault-unseal-keysSecret of thebank-vaultsnamespace. A restarted pod is unsealed automatically. - Configuration. The
externalConfigblock of the CR creates thekvKV v2 engine, a read-onlyreaderpolicy onkv/*, and a Kubernetes auth roledefaultbound to that policy. - TLS. Vault listens on HTTPS with a self-signed CA, published in the
vault-tlsSecret.
Key in vault-unseal-keys |
Content |
|---|---|
vault-root |
Root token |
vault-unseal-0 … vault-unseal-4 |
Unseal key shares |
mise tasks
The tasks in mise.toml read the token and keys from that Secret and run
vault inside the Vault pod. They target $VAULT_CONTEXT, or the current
kubectl context.
export VAULT_CONTEXT=mocha-corium
mise run vault:status # seal status of every Vault pod
mise run vault:unseal # unseal pods the operator left sealed
mise run vault:kv list kv/
mise run vault:kv get kv/rmfakecloud
mise run vault:kv get -field=jwt_secret kv/rmfakecloud
mise run vault:kv put kv/foo user=admin password="$(openssl rand -hex 24)"
mise run vault:kv patch kv/foo password=- # value read from stdin
vault:kv passes its arguments straight to vault kv, so any subcommand works
(metadata delete, -format=json, …). Only Vault's output goes to stdout,
which makes it safe to pipe into jq.
External Secrets
The External Secrets operator reads Vault through a single ClusterSecretStore,
vault-backend (<cluster>/system/external-secret/cluster-store.yaml). It
authenticates with the Kubernetes auth method — no token is stored anywhere —
so it can only read, through the reader policy.
An app declares an ExternalSecret that points at a path under kv/:
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: rmfakecloud
spec:
refreshInterval: "1h"
secretStoreRef:
name: vault-backend
kind: ClusterSecretStore
target:
name: rmfakecloud
creationPolicy: Owner
data:
- secretKey: jwt_secret
remoteRef:
key: rmfakecloud # kv/rmfakecloud
property: jwt_secret
The workflow for a new secret:
- Write the value:
mise run vault:kv put kv/<app> key="$(openssl rand -hex 24)" - Commit only the
ExternalSecret, never the value. - Force a resync if needed:
kubectl annotate externalsecret <name> force-sync=$(date +%s) --overwrite