External Secrets Operator with AWS Secrets Manager on Kubernetes

Posted on May 16, 2022

Updated

External Secrets Operator (ESO) synchronizes values from AWS Secrets Manager into Kubernetes Secrets so applications can use native Kubernetes secret references. On Amazon EKS, workload identity can supply temporary AWS credentials to the controller. The copied values still require Kubernetes access controls, and updating them does not automatically reload every application.

This guide uses an existing Amazon EKS cluster and a dedicated demo secret. You will install ESO, connect its controller to AWS through EKS Pod Identity, create a namespaced SecretStore and ExternalSecret, and check consumption without printing credentials.

Choose your authentication and consumption approach

ESO creates Kubernetes Secrets. Use it when your application or deployment tooling expects those objects. If you prefer mounted secret files, evaluate Secrets Store CSI Driver with AWS Secrets and Configuration Provider (ASCP); Kubernetes Secret synchronization is optional in that design. An application can also retrieve values directly through an AWS SDK, taking responsibility for caching, retries and refresh behavior.

SituationAuthentication routeWhat to configure
EKS environment supporting Pod IdentityEKS Pod IdentityAgent availability, IAM role and association for the ESO controller service account. The store uses the default AWS credential chain.
EKS environment using IRSAIAM roles for service accountsOIDC provider and trust policy, service account role annotation, and the documented ESO IRSA configuration.
Other Kubernetes distributionsEnvironment-specific credentialsChoose a supported credential mechanism and restrict its scope. The EKS association commands below do not apply.

Pod Identity and IRSA are different configurations. Do not add auth.jwt.serviceAccountRef to the Pod Identity store below. See the ESO AWS authentication guide for both routes.

Tested environment

Technical validation completed on 30 October 2026 using a disposable cluster in eu-central-1. The exact tests compared dummy secret values in memory without logging them, proved denial of an unrelated secret, checked periodic updates and pod restart behavior, and verified ownership and deletion behavior.

ComponentTested version
Amazon EKS and worker1.37.1, build eks-577a51e
Worker OSAmazon Linux 2023.12.20261026
ESO Helm chart and application2.14.0
Pod Identity agentv1.4.1-eksbuild.3
Local toolsAWS CLI 2.38.4, kubectl 1.37.1, Helm 3.19.0
Demo imageOfficial BusyBox 1.37.0, pinned by digest in the commands below

The execution test covers the EKS Pod Identity path below. IRSA, other Kubernetes distributions, customer-managed KMS keys, CSI mounts and real database credential rotation are discussed alternatives or extensions, and were not executed in this test.

Check prerequisites

  • A disposable EKS environment with eligible nodes and Pod Identity support. Confirm the Pod Identity agent is available; EKS Auto Mode handles this differently from clusters where you install the agent yourself. Check the AWS agent setup guide.
  • Configured AWS CLI, kubectl and Helm, with the intended AWS account and cluster context selected.
  • Provisioning permissions to create a demo secret, IAM role and inline policy, pass that role, and create a Pod Identity association. These are separate from ESO’s runtime permissions.
  • Kubernetes permissions to install ESO and CRDs, create namespaces and workloads, and inspect synchronization status.
  • Network access for image pulls and the AWS endpoints required by your authentication and secret retrieval path.
  • No conflicting ESO installation, app namespace or demo resources. This walkthrough creates a controller identity with access to one demo secret.

Before running the examples, set these variables in your shell. The commands pin ESO chart 2.14.0 and the official BusyBox 1.37.0 image by digest. Set the cluster name to your disposable EKS cluster and confirm that the region matches it.

export AWS_REGION=eu-central-1
export CLUSTER_NAME='your-disposable-cluster'
export ESO_CHART_VERSION='2.14.0'
export DEMO_IMAGE='docker.io/library/busybox@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec'
export SECRET_NAME='cloudhero-eso-demo'
export ROLE_NAME='cloudhero-eso-demo'
aws sts get-caller-identity
kubectl config current-context
kubectl version --client
helm version

Keep the tested versions above with this guide, and rerun the checks when upgrading the cluster or chart. The manifests below use external-secrets.io/v1. ESO stopped serving v1beta1 in v0.17.0.

Configure AWS access and install ESO

Create a demo secret and a scoped runtime policy

Create a local JSON file containing dummy values only. Use a protected file and your approved secret entry process for actual credentials; never commit real values or paste them into a shared terminal.

umask 077
cat > demo-secret.json <<'EOF'
{"username":"demo-user","password":"demo-only-value"}
EOF
SECRET_ARN=$(aws secretsmanager create-secret \
  --region "$AWS_REGION" --name "$SECRET_NAME" \
  --secret-string file://demo-secret.json \
  --query ARN --output text)
export SECRET_ARN
cat > read-secret-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": [
      "secretsmanager:GetSecretValue",
      "secretsmanager:DescribeSecret",
      "secretsmanager:GetResourcePolicy",
      "secretsmanager:ListSecretVersionIds"
    ],
    "Resource": "$SECRET_ARN"
  }]
}
EOF

This policy permits reads on the one demo secret. It grants no secret creation, update or deletion actions to ESO. Additional operations such as discovery or batch retrieval need their own permissions; they are not part of this example. If you use a customer-managed KMS key, configure the required decrypt permission and key policy for the consuming role. See the ESO Secrets Manager provider documentation.

Associate an IAM role with the ESO controller service account

The role belongs to the ESO controller’s service account in this example. The demo application reads a Kubernetes Secret and does not need this AWS role.

cat > pod-identity-trust.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"Service": "pods.eks.amazonaws.com"},
    "Action": ["sts:AssumeRole", "sts:TagSession"]
  }]
}
EOF
ROLE_ARN=$(aws iam create-role --role-name "$ROLE_NAME" \
  --assume-role-policy-document file://pod-identity-trust.json \
  --query Role.Arn --output text)
aws iam put-role-policy --role-name "$ROLE_NAME" \
  --policy-name ReadDemoSecret --policy-document file://read-secret-policy.json
kubectl create namespace external-secrets
kubectl -n external-secrets create serviceaccount external-secrets
ASSOCIATION_ID=$(aws eks create-pod-identity-association \
  --region "$AWS_REGION" --cluster-name "$CLUSTER_NAME" \
  --namespace external-secrets --service-account external-secrets \
  --role-arn "$ROLE_ARN" --query association.associationId --output text)

Use your organization’s role trust restrictions for production. Allow time for IAM and association propagation. See how EKS Pod Identity works.

Install the pinned ESO chart and verify its resources

helm repo add external-secrets https://charts.external-secrets.io
helm repo update
helm upgrade --install external-secrets external-secrets/external-secrets \
  --namespace external-secrets --version "$ESO_CHART_VERSION" \
  --set installCRDs=true \
  --set serviceAccount.create=false \
  --set serviceAccount.name=external-secrets \
  --wait --timeout 5m
kubectl -n external-secrets rollout status deployment/external-secrets --timeout=120s
kubectl wait --for=condition=Established \
  crd/secretstores.external-secrets.io \
  crd/externalsecrets.external-secrets.io --timeout=120s

Expected result: Helm finishes successfully, the controller rollout completes and both CRDs are established. Confirm the deployment name against the selected chart if you customize release settings. A list of pods alone does not establish that the CRDs are ready. The ESO installation guide explains chart and CRD management.

Create and verify synchronization

A SecretStore describes the provider and authentication context. An ExternalSecret describes the source values and the Kubernetes Secret ESO should maintain. Use a namespaced store here because the example serves one application namespace.

kubectl create namespace app
cat > app-secrets.yaml <<EOF
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: aws-secrets-manager
  namespace: app
spec:
  provider:
    aws:
      service: SecretsManager
      region: $AWS_REGION
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: app-secret
  namespace: app
spec:
  refreshPolicy: Periodic
  refreshInterval: 1m
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: app-secret
    creationPolicy: Owner
    deletionPolicy: Retain
  dataFrom:
    - extract:
        key: $SECRET_NAME
EOF
kubectl apply -f app-secrets.yaml
kubectl -n app wait --for=condition=Ready secretstore/aws-secrets-manager --timeout=120s
kubectl -n app wait --for=condition=Ready externalsecret/app-secret --timeout=120s
kubectl -n app get externalsecret app-secret
kubectl -n app get secret app-secret

Expected result: the store and ExternalSecret report Ready, and a Kubernetes Secret named app-secret exists with the two JSON properties. These commands show status and metadata rather than secret values. dataFrom.extract expands properties from this one JSON secret; it does not import all secrets in the AWS account.

The absent auth block is deliberate: the provider uses the controller’s AWS credential chain. A namespaced SecretStore limits where the object can be referenced, but it does not restrict the underlying shared controller IAM identity. For multiple tenants, design per-store permissions, trust and Kubernetes RBAC together; see ESO security best practices.

creationPolicy: Owner makes the ExternalSecret own its target Secret. Deleting that owner can garbage-collect the target. deletionPolicy: Retain addresses provider-side disappearance and does not override owner deletion. Review the lifecycle rules before adopting a policy.

Consume the Kubernetes Secret without logging its values

Map only the keys the application needs. This demonstration checks that the environment contains values, then emits a value-free marker. The DEMO_IMAGE variable above identifies the official BusyBox image by its immutable digest.

cat > app-deployment.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-demo
  namespace: app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-demo
  template:
    metadata:
      labels:
        app: app-demo
    spec:
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: app
          image: $DEMO_IMAGE
          command: ["sh", "-c"]
          args:
            - 'test -n "\$APP_USERNAME" && test -n "\$APP_PASSWORD" && echo "Secret keys are available" && sleep 3600'
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          env:
            - name: APP_USERNAME
              valueFrom:
                secretKeyRef:
                  name: app-secret
                  key: username
            - name: APP_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: app-secret
                  key: password
          resources:
            requests:
              cpu: 10m
              memory: 16Mi
            limits:
              cpu: 100m
              memory: 32Mi
EOF
kubectl apply -f app-deployment.yaml
kubectl -n app rollout status deployment/app-demo --timeout=120s
kubectl -n app logs deployment/app-demo

Expected log output: Secret keys are available. Do not substitute env, echo a credential, or decode the Secret into logs. This test checks key availability, not whether a database accepts a credential.

Test source updates and application reload

ESO’s periodic refresh can update the Kubernetes Secret after the source changes. It does not replace environment variables inside an already running container. Roll out new pods or implement an appropriate application reload strategy. See the Kubernetes consumption documentation.

kubectl -n app get secret app-secret -o jsonpath='{.metadata.resourceVersion}{"\n"}'
cat > demo-secret-v2.json <<'EOF'
{"username":"demo-user","password":"another-demo-only-value"}
EOF
aws secretsmanager put-secret-value --region "$AWS_REGION" \
  --secret-id "$SECRET_ARN" --secret-string file://demo-secret-v2.json \
  --query VersionId --output text
kubectl -n app get externalsecret app-secret \
  -o jsonpath='{.status.refreshTime}{"\n"}'
kubectl -n app get secret app-secret --watch \
  -o custom-columns=NAME:.metadata.name,VERSION:.metadata.resourceVersion

Wait for a new refresh and a changed target resourceVersion, then stop the watch with Ctrl+C. Keep the ExternalSecret Ready and check diagnostics if the update fails. A one-minute interval is a schedule, not a guaranteed delivery deadline. A metadata change is evidence of reconciliation, not proof that the application is using the new value.

kubectl -n app rollout restart deployment/app-demo
kubectl -n app rollout status deployment/app-demo --timeout=120s
kubectl -n app logs deployment/app-demo

With an actual application, verify successful authentication against the downstream service after rotation. File-mounted consumers need to re-read or reload changed files; subPath mounts do not receive the same automatic updates. ESO does not change the upstream database credential by itself. Review refresh policies and Kubernetes Secret update behavior.

Troubleshoot synchronization failures

SymptomFirst checks
No match for the requested kind or versionVerify the CRDs exist and are established; use v1 with a compatible ESO release.
Store is not ReadyInspect store status, controller events and logs. Confirm region, network access, controller service account, agent availability and Pod Identity association.
AWS AccessDeniedCheck the role’s permission policy and secret ARN. For customer-managed keys, check KMS permissions and key policy. Do not solve this by attaching SecretsManagerReadWrite.
Secret or JSON property not foundCheck the exact name, region, account, expected version and JSON structure. Keep sensitive values out of diagnostic output.
ExternalSecret is Ready but the app has old credentialsConfirm source and target updates, then check whether the application needs a rollout or file reload.
kubectl -n app describe secretstore aws-secrets-manager
kubectl -n app describe externalsecret app-secret
kubectl -n external-secrets logs deployment/external-secrets --tail=100

With ESO 2.14.0, a failed ExternalSecret may show only a generic provider error in its condition and Kubernetes event. Inspect the controller log for the detailed AWS error and assumed role before changing permissions.

Inspect output locally and redact it before sharing. These diagnostics should not include secret values, but application errors and configuration can still contain sensitive details.

Plan production access and operations

Restrict who can read Kubernetes Secrets, create pods that consume them, and create or modify stores and ExternalSecrets. Protect secret storage, controller permissions, audit logs and backups. A shared ClusterSecretStore is useful when centrally managed access is intentional; constrain its eligible namespaces and IAM role scope.

Monitor controller health and failed synchronization, choose refresh frequency based on credential requirements and API usage, and decide how to handle source outages or deletion. Source synchronization, credential rotation and application reload are three separate responsibilities. Test the full sequence before depending on it.

Clean up the isolated demo

Remove only resources created for this exercise. Save the association identifier from setup, and make sure the named secret and role are demo resources. AWS secret deletion below uses a recovery window.

kubectl -n app delete deployment app-demo
kubectl -n app delete externalsecret app-secret
kubectl -n app delete secretstore aws-secrets-manager
aws eks delete-pod-identity-association --region "$AWS_REGION" \
  --cluster-name "$CLUSTER_NAME" --association-id "$ASSOCIATION_ID"
aws secretsmanager delete-secret --region "$AWS_REGION" \
  --secret-id "$SECRET_ARN" --recovery-window-in-days 7
aws iam delete-role-policy --role-name "$ROLE_NAME" --policy-name ReadDemoSecret
aws iam delete-role --role-name "$ROLE_NAME"
rm -f demo-secret.json demo-secret-v2.json

Delete the ExternalSecrets and wait for their cleanup to finish while the ESO controller is still running. ESO 2.14.0 uses cleanup finalizers; uninstalling the controller first can leave those objects or their namespace stuck in Terminating. Restore the controller to let it finish rather than removing finalizers as a routine cleanup step.

Deleting the ExternalSecret should remove its owned target Secret; verify the result. If the ESO release was installed solely for this isolated exercise, uninstall it with Helm and remove the manually created controller service account when no longer needed. Treat CRD removal separately: deleting CRDs can affect every dependent object in the cluster. Keep shared namespaces and installations.

Common questions

Does ESO keep secrets out of Kubernetes?

No. ESO synchronizes external values into Kubernetes Secrets. The external service remains the source, but the copy in the cluster still needs access controls and appropriate storage protection. Evaluate CSI mounts without optional Secret synchronization or direct SDK retrieval if avoiding that copy is a requirement.

Does updating AWS Secrets Manager restart a pod?

No. ESO can refresh the Kubernetes Secret, but a running container’s environment variables remain unchanged. Arrange a rollout or a consumption method with the required reload behavior.

Should every application use ClusterSecretStore?

No. A namespaced SecretStore is a clear starting point for one application. A ClusterSecretStore supports deliberate central management, with explicit namespace eligibility and access boundaries. Neither object scope substitutes for scoped AWS permissions.

Can I use IRSA instead of Pod Identity?

Yes. Configure the documented IRSA trust and service account route rather than combining it with the Pod Identity example. Choose the route supported by your cluster and operating model.

Get help reviewing your Kubernetes secret delivery

CloudHero can help assess IAM scope, Kubernetes access boundaries, synchronization and application reload behavior. Explore our Kubernetes engineering services or talk to our team about your environment.