Secure etcd

The fix/etc/kubernetes/manifests/etcd.yaml

spec:
  containers:
    - command:
        - etcd
        - --client-cert-auth=true
        - --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
        - --cert-file=/etc/kubernetes/pki/etcd/server.crt
        - --key-file=/etc/kubernetes/pki/etcd/server.key
        - --peer-client-cert-auth=true
        - --listen-client-urls=https://127.0.0.1:2379,https://10.0.1.5:2379

On managed clusters, etcd is the provider’s — you don’t run it and can’t reach it. This page is for self-managed control planes.

Why it matters

etcd is the cluster’s only database. Every object — deployments, RBAC, config, and every Secret — is stored in etcd. And unless you’ve turned on encryption at rest, those secrets are in etcd as base64, which is to say in the clear.

So etcd access is total cluster compromise, and it’s quieter than compromising the API server — there’s no RBAC in the way, no audit log of object access, just raw key-value reads. An attacker who reaches etcd reads every secret, and can write directly to cluster state (create an admin binding, plant a workload) bypassing the API server entirely.

This is why etcd is the crown jewels, and why “who can reach port 2379” is one of the most important questions in the cluster.

Client-certificate authentication is the gate

etcd’s protection is mutual TLS with client-certificate authentication. --client-cert-auth=true means etcd rejects any client that doesn’t present a certificate signed by its CA — and the API server holds such a certificate, so it gets in; nothing else does.

Without --client-cert-auth, etcd may accept any TLS connection, or worse, plain connections — and then anyone who reaches 2379 reads everything. The peer flag (--peer-client-cert-auth) does the same for etcd-to-etcd traffic in a multi-node etcd cluster, so a rogue host can’t join the etcd cluster the way a rogue Erlang node joins RabbitMQ.

kubeadm sets all of this up correctly by default — so on a kubeadm cluster the job is verifying it, and confirming nobody loosened it. Hand-rolled etcd is where it’s often wrong.

Never expose 2379/2380, and lock the certs and snapshots

• Port 2379 (client) and 2380 (peer) must be reachable only by the control plane. On a stacked control plane (etcd on the same nodes as the API server), loopback plus the peer network. Never the internet, never the pod network.
• The etcd certificate files and data directory (/var/lib/etcd, /etc/kubernetes/pki/etcd) must be root-only. The client cert is a cluster-admin credential in disguise — anyone who can read it can talk to etcd as the API server.
• Snapshots are the whole cluster in a file. etcdctl snapshot save produces a copy of every secret. Encrypt those backups and control who can read them as tightly as etcd itself — a snapshot in an open bucket is the breach without touching the cluster.

Verify it worked

Check the auth flags on the running etcd:

sudo grep -E 'client-cert-auth|peer-client-cert-auth|listen-client-urls' \
  /etc/kubernetes/manifests/etcd.yaml

Want --client-cert-auth=true. Its absence, or listen-client-urls on 0.0.0.0, is the finding.

Prove etcd rejects an unauthenticated client — this should fail:

sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  endpoint health 2>&1 | head -1

That fails without certs (good). With the certs, it works:

sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key endpoint health

Confirm 2379 isn’t reachable from outside — from another host:

nc -zv -w5 your.control-plane 2379 2>&1 | tail -1

A timeout is the goal. Check the file permissions:

sudo ls -l /etc/kubernetes/pki/etcd/server.key /var/lib/etcd

Expect root-only (600/700). Anything group- or world-readable on the key is a cluster-admin credential left lying around.

Before you apply this

etcd is the one component where a mistake loses the whole cluster — not “an outage,” the cluster’s entire state. Every precaution here is about not doing that.

Back up before you touch it:

sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /root/etcd-backup.db

Store that snapshot securely — it is the secrets — and off the node.

Editing etcd.yaml restarts etcd, which pauses the control plane. The API server can’t function without etcd, so this is a control-plane outage while it restarts — running workloads keep serving, but no API operations. A syntax error leaves etcd down and the cluster unmanageable; keep a backup of the manifest and watch the kubelet logs.

On a multi-node etcd cluster, never restart a majority at once. etcd needs a quorum; take out more than one of three (or two of five) simultaneously and the cluster loses quorum and stops accepting writes — which stops the whole Kubernetes control plane. Change one etcd member at a time, confirm the cluster is healthy (endpoint health on all members) before the next.

Tightening --client-cert-auth can lock out the API server if its client certificate isn’t trusted by etcd’s CA — which on a hand-rolled cluster is easy to get wrong. Confirm the API server’s etcd client cert is signed by the CA in --trusted-ca-file before enabling client-cert-auth, or the API server can’t reach its own datastore and the cluster is down.

Related

• Encrypt secrets at rest — so an etcd read yields ciphertext, not plaintext.
• Secure the API server — the only thing that should talk to etcd.