Hanso Group

Effectively managing secrets—such as API keys, database passwords, and encryption keys—is a critical aspect of running secure applications in Kubernetes. While Kubernetes provides native secrets functionality, building a robust secrets management strategy requires understanding both the capabilities and limitations of these native features, as well as the ecosystem of tools that extend them. In this article, we’ll explore best practices for secrets management in Kubernetes and evaluate some of the leading tools that can help enhance your security posture.

Understanding Kubernetes Secrets

Kubernetes Secrets are objects that store sensitive information such as passwords, tokens, and keys. They’re designed to reduce the risk of exposing sensitive data compared to hard-coding it in pod specifications or container images.

How Kubernetes Secrets Work

At its core, a Kubernetes Secret is a resource that contains a map of strings. Here’s an example of creating a Secret using YAML:

apiVersion: v1
kind: Secret
metadata:
  name: database-credentials
type: Opaque
data:
  username: YWRtaW4=      # Base64 encoded "admin"
  password: UGEkJHcwcmQ=  # Base64 encoded "Pa$$w0rd"

Once created, Secrets can be mounted as files in pods or exposed as environment variables:

apiVersion: v1
kind: Pod
metadata:
  name: database-client
spec:
  containers:
  - name: database-client
    image: myapp:1.0
    env:
    - name: DB_USERNAME
      valueFrom:
        secretKeyRef:
          name: database-credentials
          key: username
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: database-credentials
          key: password

Limitations of Kubernetes Secrets

While convenient, native Kubernetes Secrets have several important limitations:

1. Base64 Encoding, Not Encryption: By default, Secret data is only base64 encoded, not encrypted. Anyone with API access can retrieve and decode secrets.

2. Stored in etcd: Secrets are stored in etcd, which means anyone with access to etcd can access all secrets.

3. Limited Access Controls: Kubernetes RBAC can restrict Secret access, but lacks fine-grained controls.

4. No Versioning or Rotation: Native Secrets have no built-in versioning or rotation capabilities.

5. No Audit Trail: There’s no built-in way to track who accessed a secret and when.

6. No Integration with External Systems: Native Secrets don’t integrate with external key management systems.

These limitations highlight why organizations often need additional tools for production-grade secrets management.

Best Practices for Kubernetes Secrets

Before exploring third-party solutions, let’s establish some fundamental best practices for managing secrets in Kubernetes.

1. Enable Encryption at Rest

By default, Secrets are stored unencrypted in etcd. Enable encryption at rest by configuring the API server with an encryption configuration:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-key>
      - identity: {}

Save this to a file (e.g., encryption-config.yaml) and update your API server configuration to use it:

--encryption-provider-config=/etc/kubernetes/encryption-config.yaml

2. Implement Least Privilege with RBAC

Restrict access to Secrets using Role-Based Access Control (RBAC):

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: secret-reader
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "list"]
  resourceNames: ["database-credentials"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-secrets
  namespace: default
subjects:
- kind: ServiceAccount
  name: myapp
  namespace: default
roleRef:
  kind: Role
  name: secret-reader
  apiGroup: rbac.authorization.k8s.io

3. Use Namespaces for Isolation

Place applications and their secrets in dedicated namespaces to provide logical isolation:

kubectl create namespace myapp
kubectl -n myapp create secret generic database-credentials \
  --from-literal=username=admin \
  --from-literal=password=Pa$$w0rd

4. Avoid Using Secrets in Pod Definitions

Instead of directly referencing secrets in pod definitions (which might be stored in version control), use Helm, Kustomize, or other templating tools to inject secrets during deployment.

5. Implement Network Security Policies

Restrict which pods can communicate with the Kubernetes API server to limit secret access:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-server-restriction
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: restricted-app
  policyTypes:
  - Egress
  egress:
  - to:
    - ipBlock:
        cidr: 10.0.0.0/16
        except:
        - 10.0.0.1/32  # API server IP

6. Regular Audit and Rotation

Establish procedures for regular auditing and rotation of secrets. While Kubernetes doesn’t provide this natively, your organization should implement processes for periodic review and updates.

External Secrets Management Tools

To address the limitations of native Kubernetes Secrets, several external tools have emerged. Let’s examine the most popular options.

HashiCorp Vault

Vault is a comprehensive secrets management solution that can integrate with Kubernetes. It provides encryption, fine-grained access control, dynamic secrets, secret rotation, and auditing.

Setting Up Vault with Kubernetes

1. Install Vault using Helm:

helm repo add hashicorp https://helm.releases.hashicorp.com
helm install vault hashicorp/vault

2. Configure Vault’s Kubernetes authentication:

## Configure auth method
vault auth enable kubernetes

## Configure Kubernetes authentication
vault write auth/kubernetes/config \
    kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
    token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
    kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
    issuer="https://kubernetes.default.svc.cluster.local"

3. Create a policy for your application:

vault policy write myapp - <<EOF
path "secret/data/myapp/*" {
  capabilities = ["read"]
}
EOF

4. Create a Kubernetes role that maps to this policy:

vault write auth/kubernetes/role/myapp \
    bound_service_account_names=myapp \
    bound_service_account_namespaces=default \
    policies=myapp \
    ttl=1h

5. Store a secret in Vault:

vault kv put secret/myapp/database username="admin" password="Pa$$w0rd"

Using Vault with the Vault Agent Injector

The Vault Agent Injector is a mutation webhook that injects a Vault Agent container into pods, which then retrieves secrets from Vault and writes them to a shared volume:

apiVersion: v1
kind: Pod
metadata:
  name: database-client
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/agent-inject-secret-database-creds: "secret/data/myapp/database"
    vault.hashicorp.com/agent-inject-template-database-creds: |
      {{- with secret "secret/data/myapp/database" -}}
      export DB_USERNAME="{{ .Data.data.username }}"
      export DB_PASSWORD="{{ .Data.data.password }}"
      {{- end -}}
    vault.hashicorp.com/role: "myapp"
spec:
  serviceAccountName: myapp
  containers:
  - name: app
    image: myapp:1.0
    command: ["sh", "-c", "source /vault/secrets/database-creds && ./my-app"]

External Secrets Operator (ESO)

External Secrets Operator is a Kubernetes operator that integrates external secret management systems with Kubernetes. It fetches secrets from external APIs and creates Kubernetes Secret objects.

Setting Up External Secrets Operator

1. Install ESO using Helm:

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
   --namespace external-secrets \
   --create-namespace

2. Configure a SecretStore (in this example, for AWS Secrets Manager):

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secretstore
  namespace: default
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-west-1
      auth:
        jwt:
          serviceAccountRef:
            name: aws-sa

3. Define an ExternalSecret:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
  namespace: default
spec:
  refreshInterval: "15m"
  secretStoreRef:
    name: aws-secretstore
    kind: SecretStore
  target:
    name: database-credentials
    creationPolicy: Owner
  data:
  - secretKey: username
    remoteRef:
      key: myapp/database
      property: username
  - secretKey: password
    remoteRef:
      key: myapp/database
      property: password

ESO supports many providers including AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, HashiCorp Vault, and others.

Sealed Secrets

Sealed Secrets, developed by Bitnami, addresses the specific challenge of managing Kubernetes Secrets in Git repositories. It uses asymmetric cryptography to encrypt secrets that can only be decrypted by the controller running in the cluster.

Setting Up Sealed Secrets

1. Install Sealed Secrets using Helm:

helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets sealed-secrets/sealed-secrets

2. Install the kubeseal CLI tool:

## Linux
wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.23.0/kubeseal-0.23.0-linux-amd64.tar.gz
tar -xvf kubeseal-0.23.0-linux-amd64.tar.gz
sudo install -m 755 kubeseal /usr/local/bin/kubeseal

## macOS
brew install kubeseal

3. Seal a Secret:

## Create a regular Secret definition
kubectl create secret generic database-credentials \
  --from-literal=username=admin \
  --from-literal=password=Pa$$w0rd \
  --dry-run=client -o yaml > database-credentials.yaml

## Seal the Secret
kubeseal --format yaml < database-credentials.yaml > sealed-database-credentials.yaml

4. Apply the SealedSecret:

kubectl apply -f sealed-database-credentials.yaml

The controller will create a regular Secret from the SealedSecret, which your applications can consume normally.

SOPS (Secrets OPerationS)

SOPS, developed by Mozilla, is a tool that supports encrypting YAML, JSON, ENV, and INI files with various encryption providers. When used with Kubernetes, it often pairs with tools like Flux or Helm for GitOps-based deployments.

Using SOPS with Kubernetes

1. Install SOPS:

## Linux
wget https://github.com/mozilla/sops/releases/download/v3.7.3/sops-v3.7.3.linux.amd64
chmod +x sops-v3.7.3.linux.amd64
sudo mv sops-v3.7.3.linux.amd64 /usr/local/bin/sops

## macOS
brew install sops

2. Create a GPG key for encryption:

gpg --gen-key

3. Encrypt a Secret:

## Create a regular Secret definition
cat > database-credentials.yaml << EOF
apiVersion: v1
kind: Secret
metadata:
  name: database-credentials
type: Opaque
data:
  username: YWRtaW4=
  password: UGEkJHcwcmQ=
EOF

## Encrypt with SOPS
sops --encrypt --pgp <fingerprint> database-credentials.yaml > database-credentials.enc.yaml

4. To use with Flux, install the SOPS controller:

flux create source git sops-example \
  --url=https://github.com/user/repo \
  --branch=main

flux create kustomization sops-example \
  --source=sops-example \
  --path=./path/to/resources \
  --decryption-provider=sops \
  --decryption-secret=sops-gpg

AWS Secrets Manager and AWS KMS

For AWS environments, the combination of AWS Secrets Manager and AWS KMS provides robust secrets management. There are multiple approaches to integration:

Option 1: Using External Secrets Operator

As shown above, ESO can be configured to work with AWS Secrets Manager.

Option 2: Direct Integration via IAM Roles for Service Accounts (IRSA)

1. Enable IRSA on your EKS cluster.

2. Create an IAM policy for Secrets Manager access:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret"
      ],
      "Resource": "arn:aws:secretsmanager:region:account-id:secret:myapp/*"
    }
  ]
}

3. Create a service account with the IAM role:

eksctl create iamserviceaccount \
  --name myapp-sa \
  --namespace default \
  --cluster my-cluster \
  --attach-policy-arn arn:aws:iam::ACCOUNT_ID:policy/MySecretManagerPolicy \
  --approve

4. Use AWS SDK in your application to retrieve secrets directly.

Comparing Solutions

To help you choose the right approach for your environment, here’s a comparison of the solutions discussed:

Feature	Native K8s Secrets	Vault	External Secrets Operator	Sealed Secrets	SOPS
Encryption at Rest	With configuration	Yes	Depends on provider	Yes	Yes
Fine-grained Access Control	Limited	Yes	Depends on provider	No	No
Versioning	No	Yes	Depends on provider	No	With Git
Secret Rotation	No	Yes	Depends on provider	No	No
Audit Trail	No	Yes	Depends on provider	No	With Git
Cloud Provider Integration	No	Yes	Yes	No	Yes
Complexity	Low	High	Medium	Low	Medium
GitOps Friendly	No	With add-ons	Yes	Yes	Yes

Real-World Implementation Patterns

Based on our experience working with various organizations, here are some common implementation patterns:

Pattern 1: Vault + Kubernetes for Enterprise Environments

For large enterprises with complex security requirements:

1. Use Vault as the central secrets store
2. Use Vault’s Kubernetes authentication
3. Deploy the Vault Agent Injector for secret injection
4. Implement strict network policies
5. Use audit logging and monitoring

Pattern 2: Cloud Provider Solution for Cloud-Native Applications

For applications running exclusively in one cloud provider:

1. Use the cloud provider’s secrets management service (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault)
2. Use External Secrets Operator to sync secrets to Kubernetes
3. Implement IAM/Role-based access controls
4. Set up automated secret rotation

Pattern 3: Sealed Secrets for GitOps Workflows

For teams using GitOps with tools like Flux or ArgoCD:

1. Use Sealed Secrets or SOPS to encrypt secrets in Git
2. Implement proper Git access controls
3. Use automated pipelines for secret deployment
4. Implement monitoring and alerting for secret usage

Pattern 4: Hybrid Approach for Multi-Cloud

For organizations spanning multiple clouds:

1. Use Vault as the central secrets management platform
2. Implement External Secrets Operator for Kubernetes integration
3. Configure authentication for each cloud provider
4. Implement consistent policies across environments

Advanced Topics

As your secrets management strategy matures, consider these advanced topics:

Secret Rotation

Implement automatic secret rotation to limit the impact of potential breaches:

Using Vault’s Dynamic Secrets:

apiVersion: v1
kind: Pod
metadata:
  name: database-client
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/agent-inject-secret-database-creds: "database/creds/readonly"
    vault.hashicorp.com/role: "myapp"
spec:
  serviceAccountName: myapp
  containers:
  - name: app
    image: myapp:1.0

Using AWS Secrets Manager with Rotation:

aws secretsmanager rotate-secret \
  --secret-id myapp/database \
  --rotation-lambda-arn arn:aws:lambda:region:account:function:rotation-function

Secrets Auditing and Monitoring

Implement comprehensive auditing to track secret usage:

1. Enable Kubernetes audit logging for API server
2. Configure Vault audit logging:

vault audit enable file file_path=/vault/logs/audit.log

3. Set up alerts for suspicious access patterns:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: secret-alerts
spec:
  groups:
  - name: secrets
    rules:
    - alert: ExcessiveSecretAccess
      expr: rate(apiserver_request_total{resource="secrets",verb="get"}[5m]) > 100
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Excessive secret access detected"

Integrating with CI/CD Pipelines

Securely manage secrets in CI/CD pipelines:

1. For GitHub Actions, use GitHub Secrets:

name: Deploy
on: [push]
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Deploy
      env:
        API_KEY: ${{ secrets.API_KEY }}
      run: ./deploy.sh

2. For Jenkins, use Jenkins Credentials:

pipeline {
  agent any
  environment {
    API_KEY = credentials('api-key')
  }
  stages {
    stage('Deploy') {
      steps {
        sh './deploy.sh'
      }
    }
  }
}

Conclusion

Effective secrets management in Kubernetes requires a thoughtful approach that extends beyond native capabilities. By combining appropriate tools, implementation patterns, and operational practices, organizations can build a robust secrets management strategy that balances security and usability.

Remember that no single approach is perfect for all scenarios. The right solution depends on your specific requirements, including:

• Regulatory and compliance requirements
• Organizational size and complexity
• Existing security infrastructure
• Cloud provider environments
• Development workflows

Start with the fundamentals: enable encryption at rest, implement least privilege, and establish basic operational practices. As your needs evolve, gradually incorporate more advanced solutions and practices.

Finally, keep in mind that secrets management is just one component of a comprehensive security strategy. It should be complemented by network security, image scanning, runtime protection, and other security controls to provide defense in depth for your Kubernetes environments.

References

1. Kubernetes Documentation: Secrets. https://kubernetes.io/docs/concepts/configuration/secret/

2. HashiCorp Vault Documentation. https://www.vaultproject.io/docs

3. External Secrets Operator Documentation. https://external-secrets.io/latest/

4. Sealed Secrets Documentation. https://github.com/bitnami-labs/sealed-secrets

5. Mozilla SOPS Documentation. https://github.com/mozilla/sops