Encrypt Kubernetes secrets at rest

The fix/etc/kubernetes/enc/encryption-config.yaml

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources: ["secrets"]
    providers:
      - kms:                    # prefer a KMS provider (external key)
          apiVersion: v2
          name: mykms
          endpoint: unix:///var/run/kmsplugin/socket.sock
      - aescbc:                 # fallback: a local key, better than nothing
          keys:
            - name: key1
              secret: <BASE64_32_BYTE_KEY>
      - identity: {}            # allows reading old, unencrypted secrets

Then reference it: --encryption-provider-config=/etc/kubernetes/enc/encryption-config.yaml on the API server. On managed clusters this is often on by default — verify rather than configure.

The misconception this page exists to correct

Kubernetes Secrets are not encrypted. They are base64-encoded, and base64 is not encryption — it’s a reversible transform with no key, decodable by anyone, instantly:

kubectl get secret db -o jsonpath='{.data.password}' | base64 -d

That’s the whole “decryption.” A great many people believe that because a Secret looks like gibberish in kubectl get -o yaml, it’s protected. It isn’t. The base64 is there so binary values survive YAML, not for confidentiality.

Stored in etcd, a Secret is plaintext behind base64. Anyone who can read etcd — a backup, a snapshot, a disk, a compromised etcd node — reads every secret in the cluster with a base64 -d. Database passwords, TLS keys, API tokens, cloud credentials: all of it.

Encryption at rest, and why KMS is better

Turning on encryption at rest means the API server encrypts Secret values before writing them to etcd, so an etcd read yields ciphertext.

There are two provider styles, and the difference matters:

• aescbc / aesgcm (local keys) — the encryption key lives in the EncryptionConfiguration file on the control plane node. Better than nothing: an etcd backup alone no longer leaks secrets. But an attacker who compromises the control plane node has both the ciphertext and the key.
• kms (a KMS provider) — the key lives in an external system (cloud KMS, Vault), and the API server calls out to it. Now the encryption key is not on the node, so an etcd snapshot and a node compromise still don’t yield the plaintext without also compromising the KMS. This is the meaningfully stronger option and what you want in production.

Order matters in the provider list: the first provider encrypts; all of them can decrypt. Keep identity last during migration so existing unencrypted secrets remain readable, then remove it once everything is re-encrypted.

Existing secrets aren’t encrypted retroactively

Enabling encryption only affects secrets written after it’s on. Everything already in etcd stays as it was. You have to rewrite them:

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

That reads and re-writes every secret, which re-encrypts them with the new provider. Until you do this, your old secrets are still plaintext in etcd no matter what the config says.

Verify it worked

Check whether the API server has an encryption config at all:

kubectl -n kube-system get pod -l component=kube-apiserver \
  -o jsonpath='{.items[0].spec.containers[0].command}' | tr ',' '\n' | grep encryption-provider

No output means secrets are stored in plaintext — the finding on a self-managed cluster.

Then prove it — read a secret straight from etcd and see if it’s ciphertext:

sudo ETCDCTL_API=3 etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/default/db | hexdump -C | head

If you can see the password in that output, it’s not encrypted. Encrypted secrets show a provider prefix (k8s:enc:aescbc: or k8s:enc:kms:) followed by ciphertext.

Confirm the base64 point to anyone who doubts it:

kubectl get secret db -o jsonpath='{.data.password}' | base64 -d; echo

That prints the plaintext password from any secret you can get — which is why RBAC on secrets matters as much as encryption.

Before you apply this

A wrong encryption config can make secrets unreadable — including to the API server. If the key is lost or the provider list is misordered, the API server can’t decrypt existing secrets, and workloads that mount them fail. This is one of the more dangerous changes on the list.

Non-negotiable precautions:

• Back up etcd before enabling encryption, and back up the encryption key somewhere the cluster’s loss doesn’t also lose it. A lost aescbc key with no backup means the encrypted secrets are gone permanently.
• Keep identity in the provider list during migration, last, so pre-existing unencrypted secrets stay readable while you re-encrypt.
• Re-encrypt with kubectl replace and verify before removing identity. Remove it too early and old secrets become unreadable.

KMS adds a dependency the cluster now needs to function. If the KMS endpoint is unreachable, the API server can’t decrypt secrets, and secret-consuming operations stall. The KMS must be highly available and the plugin healthy — this trades “key on the node” for “hard dependency on the KMS,” which is the right trade in production but is a real operational commitment.

On multi-control-plane clusters, every API server needs the same config and keys, rolled out together. A cluster where one API server can decrypt and another can’t produces intermittent, maddening failures.

Related

• Secure etcd — where secrets live; encryption protects the data, etcd access protects the store.
• RBAC least privilege — encryption doesn’t help against a subject that can get secrets.