Secrets Encryption at Rest: Protecting the Database

In Module 3, we discussed a hard truth: Kubernetes Secrets are NOT encrypted by default. They are merely Base64 encoded. If a hacker gets access to your master node and peeks into the etcd database, they can see every database password, every OpenAI API key, and every private certificate in plain text.

To be truly secure—and to meet compliance standards like SOC2 or HIPAA—you must enable Encryption at Rest.

Encryption at Rest ensures that the moment a secret is written to the physical disk (within etcd), it is scrambled using a high-strength encryption key. If that disk is stolen or leaked, the data inside is useless without the key. In this lesson, we will master the EncryptionConfiguration, learn how to integrate with a Cloud KMS (Key Management Service), and understand the workflow of Rotating Encryption Keys.

1. The Default State: Encoding is not Encryption

Base64 is an encoding, intended for safely transmitting binary data. It is not a security measure.

echo "S3J5cHRvbg==" | base64 --decode -> Result: Krypton

Anyone with a terminal can "Decrypt" a standard Kubernetes secret. This is why we need actual cryptographic encryption at the storage layer.

2. The EncryptionConfiguration File

To enable encryption, you must provide a configuration file to the Kubernetes API Server. This file defines which resources should be encrypted and which provider (algorithm) should be used.

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets # We want to encrypt all secrets
      - configmaps # Optional: encrypt sensitive configs
    providers:
      - kms: # THE BEST PRACTICE: Use Cloud KMS
          name: aws-kms
          endpoint: unix:///var/run/kms-plugin/socket.sock
          cachesize: 100
          timeout: 3s
      - aescbc: # Fallback: Local AES key (Less secure)
          keys:
            - name: key1
              secret: < 32-byte-base64-encoded-key >

3. Envelope Encryption: The Gold Standard

When using a Cloud KMS (like AWS KMS or GCP KMS), Kubernetes uses a pattern called Envelope Encryption.

Data Encryption Key (DEK): Kubernetes generates a unique, random key for every single secret.
Encryption: The secret is encrypted using the DEK.
Key Encryption Key (KEK): The DEK is then sent to AWS/GCP KMS, where it is encrypted by a "Master Key."
Storage: Both the Encrypted Secret and the Encrypted DEK are stored in etcd.

Why this matters: Your "Master Key" never leaves the high-security cloud vault. Even if someone steals your entire cluster, they cannot decrypt the secrets without the permission of the cloud provider.

4. Visualizing the Encryption Path

graph TD
    User["kubectl create secret"] -- "Plaintext" --> API["API Server"]
    
    subgraph "Encryption Logic"
        API -- "Request Encryption" --> KMS_Plug["KMS Plugin / Socket"]
        KMS_Plug -- "Forward to" --> Cloud["AWS / GCP KMS Vault"]
        Cloud -- "Return Encrypted Key" --> KMS_Plug
    end
    
    API -- "Encrypted Bytes" --> ETCD["etcd Database (Safe)"]
    
    style ETCD fill:#9f9,stroke:#333
    style Cloud fill:#f96,stroke:#333

5. Converting Existing Secrets

Enabling encryption only affects New secrets. Your existing secrets in etcd are still in plain text!

To encrypt your entire database after enabling the configuration, you must "Touch" every secret to force a rewrite:

# This command forces the API server to read and immediately write back every secret
kubectl get secrets --all-namespaces -o json | kubectl replace -f -

6. Practical Example: Auditing Encryption

On a master node, you can use etcdctl (the etcd CLI) to verify if your secrets are actually scrambled.

# Query the physical etcd data directly
ETCDCTL_API=3 etcdctl get /registry/secrets/default/my-db-pass

If you see v1:kms:... followed by a wall of random characters, your encryption is working. If you see plain text or recognizable Base64, your configuration is failing.

7. AI Implementation: Multi-Cloud Secrets Management

If you are running your AI research on GCP but your production agent on AWS, you have two different KMS providers.

The Multi-Cluster Security Strategy:

Identity Federation: Use OIDC to allow your clusters to "Talk" to each other's KMS.
External Secrets (Module 7.3): Instead of managing etcd encryption yourself, use the External Secrets Operator to pull data from a single, central AWS Secrets Manager. This way, even if your GCP cluster's etcd is compromised, it contains no secrets—only the "Pointer" of where to fetch them from the secure AWS vault.

8. Summary and Key Takeaways

Base64 is NOT Encryption: It is only encoding.
etcd: The heart of the cluster where plain text secrets live by default.
EncryptionConfiguration: The manifest that tells K8s to start scrambling data.
KMS Provider: Use managed cloud services (AWS/GCP/Azure) for master key management.
Envelope Encryption: Protects your data by encrypting the keys that encrypt the data.
Rotation: Always have a plan for rotating your master keys annually.

Congratulations!

You have completed Module 10: Security in Kubernetes. You have transformed your cluster from an open playground into a hardened fortress. You are now prepared to handle the most sensitive data and the most high-stakes AI deployments.

Next Stop: In Module 11: CI/CD with Kubernetes, we will move from security to speed: Helm, GitHub Actions, and GitOps workflows.

9. SEO Metadata & Keywords

Focus Keywords: Kubernetes secrets encryption at rest tutorial, K8s EncryptionConfiguration example, AWS KMS Kubernetes secrets integration, etcd encryption best practices, encrypting existing secrets K8s, envelope encryption explained.

Meta Description: Don't leave your passwords in plain text. Master the ultimate security layer of Kubernetes: Encryption at Rest. Learn how to configure cloud-managed KMS keys to scramble your etcd database and protect your AI and web services from data breaches.

Secrets encryption at rest