Enable Kubernetes audit logging

The fix/etc/kubernetes/audit-policy.yaml

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  # Don't log secret/configmap bodies — just that they were accessed
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets", "configmaps"]
  # Log changes at RequestResponse, reads at Metadata
  - level: RequestResponse
    verbs: ["create", "update", "patch", "delete"]
  - level: Metadata

Then wire it up on the API server: --audit-policy-file=/etc/kubernetes/audit-policy.yaml and --audit-log-path=/var/log/kubernetes/audit.log. On managed clusters, audit logging is usually a provider setting (CloudWatch, Cloud Logging) rather than these flags.

Why it matters

Unlike audit logging in MongoDB or Elasticsearch, Kubernetes audit logging is free — it’s a core API server feature, no license required. But it’s off by default: a stock API server records nothing about who did what.

That gap is the difference between investigating an incident and guessing at one. Every meaningful action goes through the API server — who read which secret, who created that pod, who changed that RBAC binding, which service account authenticated from where. Without an audit policy, none of it is recorded, and after a compromise you have no way to answer the questions that matter: what did they touch, and what do you need to rotate.

This is medium because it closes no exposure — it changes what you can reconstruct. But it’s free and the reconstruction is genuinely valuable, which is why it ranks above the paid-only audit pages on the datastores.

The four levels

An audit policy assigns each request one of four levels, which controls how much is recorded:

Level	Records
None	Nothing — use to drop noise
Metadata	Who, what, when, verb, resource — not the request/response body
Request	Metadata plus the request body
RequestResponse	Metadata plus request and response bodies

The skill is matching level to resource. RequestResponse on everything produces an enormous, expensive log — and, critically, logs secret contents. Metadata on secrets and configmaps records that they were accessed without recording their values; RequestResponse on mutations captures what changed for the objects where that matters. Rules are evaluated top to bottom, first match wins, so put the specific low-level rules (secrets → Metadata) first.

Don’t log your secrets into the audit log

This is the trap worth stating plainly, because the naive “log everything at RequestResponse” advice creates it: at Request or RequestResponse level, the audit log captures the body of the request — and for a secret, that’s the secret.

You’d have taken every secret in the cluster and copied it into a log file that goes to your log aggregator, with different access controls and retention than etcd. That’s the same mistake as the Postgres log_statement trap, at cluster scale.

Always log secrets and configmaps at Metadata — the first rule in the policy above does exactly this, before any broader rule can catch them.

Ship it off the node

An audit log on the control plane node is readable by anyone who compromises the control plane — the same people you’re trying to have a record of. And it competes for disk with the thing whose health you depend on. Ship it to an external system (a log pipeline, SIEM, or the cloud provider’s logging), and set rotation on the local file so it can’t fill the disk:

- --audit-log-maxsize=100      # MB per file
- --audit-log-maxbackup=10
- --audit-log-maxage=30

Verify it worked

Check whether audit logging is configured at all:

kubectl -n kube-system get pod -l component=kube-apiserver \
  -o jsonpath='{.items[0].spec.containers[0].command}' | tr ',' '\n' | grep audit

No --audit-policy-file means the API server is recording nothing — the finding on a self-managed cluster.

Confirm it’s actually writing, and that secrets aren’t in it:

sudo tail -1 /var/log/kubernetes/audit.log | jq '{user:.user.username, verb, resource:.objectRef.resource, level}'

Then the important negative check — secrets must not appear at RequestResponse:

sudo grep '"resource":"secrets"' /var/log/kubernetes/audit.log | \
  jq -r 'select(.level=="RequestResponse") | .objectRef.name' | head

Any output here means secret contents are in your audit log — fix the policy so secrets are Metadata only.

Generate an event and confirm it’s captured:

kubectl get secrets -n kube-system >/dev/null
sudo tail -5 /var/log/kubernetes/audit.log | jq -r 'select(.objectRef.resource=="secrets") | .user.username' | tail -1

Before you apply this

Audit logging at too high a level fills the disk, and a full disk on the control plane node degrades or stops the API server — which would make this the page that causes the outage it was meant to help investigate. Set the --audit-log-max* rotation flags, ship logs off the node, and start with a conservative policy (Metadata broadly, RequestResponse only for mutations) rather than logging everything at the highest level.

Editing the API server manifest to add these flags restarts the API server — a brief control-plane outage on a single-control-plane cluster, and a syntax error leaves it down. Back up the manifest, and note the audit policy file must exist and be valid before you reference it, or the API server won’t start.

The audit log is sensitive. It records who accessed what, which is reconnaissance gold — and, if the policy is wrong, may contain secret values. Protect it like the data it describes: off the node, access-controlled, encrypted in transit to wherever it lands. An attacker who reads your audit log learns your cluster’s structure and your admins’ patterns.

Tune the noise or you’ll ignore it. A policy that logs every get/list/watch from every controller buries the signal — Kubernetes controllers are extremely chatty. Drop the high-volume read traffic from system components to None, keep mutations and secret access, so the log is something a human might actually read during an incident.

Related

• Secure the API server — where the audit policy is wired in.
• RBAC least privilege — the audit log shows whether your RBAC is being tested.