It’s 3 AM. Your phone buzzes. The Slack channel is on fire. “App’s down. Database credentials invalid.” Your heart races as you SSH into the cluster, only to find a leaked API key has let a hacker spin up rogue pods. Sound like a nightmare? Welcome to the high-stakes world of Kubernetes secrets management — where one misstep can burn your cluster to the ground. But fear not, because you’re about to become the hero of this tech thriller. Let’s dive into how to securely inject secrets into Kubernetes, the right way. Buckle up! 🚀

Creating Secrets Without Losing Your Mind

Imagine secrets as the keys to your digital kingdom — API tokens, database passwords, the works. Kubernetes makes it easy to store them, but easy doesn’t mean secure. Let’s start with the basics.

You can create a secret imperatively with a one-liner:

kubectl create secret generic db-credentials \
  --from-literal=username=admin \
  --from-literal=password=SuperSecret123

Boom! You’ve got a Secret object inside Kubernetes, containing the username and password.. But if you prefer managing infrastructure as code (and you should), use a YAML manifest:

It’s better to manage secrets as code using YAML files:

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
data:
  username: YWRtaW4=   # "admin" in base64
  password: U3VwZXJTZWNyZXQxMjM=   # "SuperSecret123" in base64

Notice the base64?

⚠️ It’s not encryption — just encoding to make data YAML-friendly.

What does that even mean, YAML-friendly?.. 🤦🏽‍♂️, uhhmm, I think I might even be confusing myself too with that word. Let’s explain what that meant in the Kubernetes scene and why we are doing that.

Kubernetes Secrets must be valid and safe to transport over the network and store in JSON or YAML. Base64 ensures the data doesn’t break YAML syntax, introduce invalid characters, or become unreadable across systems. That sounds better and about right..

So anyone with a base64 decoder (aka anyone) can read it. That’s your first plot twist: secrets aren’t secure by default. 😱

WAIT WHAT?!!!

Yeah, you read right again…

The Etcd Trap and Encryption at Rest

When you create a secret, Kubernetes sends it to the API Server, which stores it in etcd, the cluster’s database. Here’s the kicker: by default, etcd stores secrets as base64-encoded plaintext. If a hacker gets into etcd, it’s game over.

To fix this, we can enable encryption at rest. Basically, we can tell the API-Server — "hey man!!, before you store our secrets in the etcd, do encrypt them with any of the keys (or methods) in this file". We can achieve this by creating EncryptionConfiguration Resource:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - kms:   # First try an external KMS provider (like AWS KMS)
          name: myKmsProvider
          endpoint: unix:///tmp/kms.socket
      - aesgcm:   # Fallback to a local symmetric key (AES-GCM)
          keys:
            - name: key1
              secret: pFq3kFJodLRQk3JKe/b5kj2kNjY4e09qRbRQzk7uO9I=  # Base64-encoded AES key
      - identity: {}   # Final fallback: no encryption (only for dev/test!)

So what does this do?

This tells Kubernetes to encrypt secrets using an external KMS (like AWS KMS), falling back to a local AES-GCM key if needed. The AES key (e.g., pFq3kFJodLRQk3JKe/b5kj2kNjY4e09qRbRQzk7uO9I=) is sensitive—store it in a vault like HashiCorp Vault or AWS Secrets Manager, not Git.

Pro tip: Use tools like Ansible to fetch it securely during bootstrap

This is beyond the scope of this article, I would have shown how this can be achieved, nevertheless, perhaps that can be the topic of my next article, so keep an eye out

Here’s how it works: The API Server base64-decodes your secret, encrypts it with the configured key, and stores the ciphertext in etcd. When a pod needs the secret, the API Server decrypts it and delivers base64-encoded plaintext. Pods don’t need the encryption key — Kubernetes handles the heavy lifting.

Key rotation best practice:

For key rotation (because hackers don’t sleep), add a new provider key or AES-GEM key to the config, make it the first key, and phase out the old one after re-encrypting secrets. It’s like changing your locks every few months. 🔐

To explain further how this works in detials, when the api server wants to encrypt a key, it uses the first provider if available or the first key in the AES-GCM key config, but when it wants to decrypts it tries each keys in the config till it hits a key that can decrypt the encrypted data. Normally, when you rotate keys, say every 4 months, usually by moving the original first key or provider to the second key, then adding a new key, new data will be encrypted with the new key, but old data will still be encrypted with the old key.

Locking Down Access with RBAC

Secrets are powerful, so limit who can touch them. Use Role-Based Access Control (RBAC) to ensure only trusted pods or users can access them. Here’s an example:

# Role allowing reading only secrets
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: prod
  name: secret-reader
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "list"]

# Bind a service account to the role
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: prod
  name: read-secrets
subjects:
- kind: ServiceAccount
  name: app-service-account
roleRef:
  kind: Role
  name: secret-reader
  apiGroup: rbac.authorization.k8s.io

Now, only app-service-account in the prod namespace can read secrets. No one else gets a peek.

Best Practices to Sleep Soundly

You’re almost the hero. To seal the deal, follow these best practices:

• Use a secrets manager: Tools like Vault or AWS KMS keep encryption keys safe.
• Encrypt at rest: Always enable it for etcd.
• Least privilege: Restrict RBAC and network policies to minimize access.
• Audit access: Log who’s touching secrets and why.
• Rotate secrets: Change them regularly, like your toothbrush.
• Never log secrets: Mask them in logs to avoid leaks.

My DevOps instructor did tell me a story about a team losing over $50k because a leaked secret let hackers mine crypto on their cluster. I can only imagine what a nightmare it was for the team, the company, investors, and possibly customers at large.. Don’t be that team. Build a fortress around your secrets from day one, and you’ll sleep like a baby.

Your Next Step

Kubernetes secrets management isn’t a checkbox — it’s a mindset. Treat every secret like your bank PIN. Encrypt, restrict, monitor, repeat . What’s your secret strategy? Drop a comment below or send a DM let’s keep the DevOps vibes secure. 💪