External Secrets Operator (ESO) synchronizes values from AWS Secrets Manager into Kubernetes Secrets so applications can use native Kubernetes secret references. On Amazon EKS, workload identity can supply temporary AWS credentials to the controller. The copied values still require Kubernetes access controls, and updating them does not automatically reload every application.
This guide uses an existing Amazon EKS cluster and a dedicated demo secret. You will install ESO, connect its controller to AWS through EKS Pod Identity, create a namespaced SecretStore and ExternalSecret, and check consumption without printing credentials.
Choose your authentication and consumption approach
ESO creates Kubernetes Secrets. Use it when your application or deployment tooling expects those objects. If you prefer mounted secret files, evaluate Secrets Store CSI Driver with AWS Secrets and Configuration Provider (ASCP); Kubernetes Secret synchronization is optional in that design. An application can also retrieve values directly through an AWS SDK, taking responsibility for caching, retries and refresh behavior.
| Situation | Authentication route | What to configure |
|---|---|---|
| EKS environment supporting Pod Identity | EKS Pod Identity | Agent availability, IAM role and association for the ESO controller service account. The store uses the default AWS credential chain. |
| EKS environment using IRSA | IAM roles for service accounts | OIDC provider and trust policy, service account role annotation, and the documented ESO IRSA configuration. |
| Other Kubernetes distributions | Environment-specific credentials | Choose a supported credential mechanism and restrict its scope. The EKS association commands below do not apply. |
Pod Identity and IRSA are different configurations. Do not add auth.jwt.serviceAccountRef to the Pod Identity store below. See the ESO AWS authentication guide for both routes.
Tested environment
Technical validation completed on 30 October 2026 using a disposable cluster in eu-central-1. The exact tests compared dummy secret values in memory without logging them, proved denial of an unrelated secret, checked periodic updates and pod restart behavior, and verified ownership and deletion behavior.
| Component | Tested version |
|---|---|
| Amazon EKS and worker | 1.37.1, build eks-577a51e |
| Worker OS | Amazon Linux 2023.12.20261026 |
| ESO Helm chart and application | 2.14.0 |
| Pod Identity agent | v1.4.1-eksbuild.3 |
| Local tools | AWS CLI 2.38.4, kubectl 1.37.1, Helm 3.19.0 |
| Demo image | Official BusyBox 1.37.0, pinned by digest in the commands below |
The execution test covers the EKS Pod Identity path below. IRSA, other Kubernetes distributions, customer-managed KMS keys, CSI mounts and real database credential rotation are discussed alternatives or extensions, and were not executed in this test.
Check prerequisites
- A disposable EKS environment with eligible nodes and Pod Identity support. Confirm the Pod Identity agent is available; EKS Auto Mode handles this differently from clusters where you install the agent yourself. Check the AWS agent setup guide.
- Configured AWS CLI, kubectl and Helm, with the intended AWS account and cluster context selected.
- Provisioning permissions to create a demo secret, IAM role and inline policy, pass that role, and create a Pod Identity association. These are separate from ESO’s runtime permissions.
- Kubernetes permissions to install ESO and CRDs, create namespaces and workloads, and inspect synchronization status.
- Network access for image pulls and the AWS endpoints required by your authentication and secret retrieval path.
- No conflicting ESO installation, app namespace or demo resources. This walkthrough creates a controller identity with access to one demo secret.
Before running the examples, set these variables in your shell. The commands pin ESO chart 2.14.0 and the official BusyBox 1.37.0 image by digest. Set the cluster name to your disposable EKS cluster and confirm that the region matches it.
export AWS_REGION=eu-central-1
export CLUSTER_NAME='your-disposable-cluster'
export ESO_CHART_VERSION='2.14.0'
export DEMO_IMAGE='docker.io/library/busybox@sha256:a7dbc319b1438115a207e11930dc4bb3e643dcb3ec0bdd694bb36cd8fad289ec'
export SECRET_NAME='cloudhero-eso-demo'
export ROLE_NAME='cloudhero-eso-demo'
aws sts get-caller-identity
kubectl config current-context
kubectl version --client
helm version
Keep the tested versions above with this guide, and rerun the checks when upgrading the cluster or chart. The manifests below use external-secrets.io/v1. ESO stopped serving v1beta1 in v0.17.0.
Configure AWS access and install ESO
Create a demo secret and a scoped runtime policy
Create a local JSON file containing dummy values only. Use a protected file and your approved secret entry process for actual credentials; never commit real values or paste them into a shared terminal.
umask 077
cat > demo-secret.json <<'EOF'
{"username":"demo-user","password":"demo-only-value"}
EOF
SECRET_ARN=$(aws secretsmanager create-secret \
--region "$AWS_REGION" --name "$SECRET_NAME" \
--secret-string file://demo-secret.json \
--query ARN --output text)
export SECRET_ARN
cat > read-secret-policy.json <<EOF
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": [
"secretsmanager:GetSecretValue",
"secretsmanager:DescribeSecret",
"secretsmanager:GetResourcePolicy",
"secretsmanager:ListSecretVersionIds"
],
"Resource": "$SECRET_ARN"
}]
}
EOF
This policy permits reads on the one demo secret. It grants no secret creation, update or deletion actions to ESO. Additional operations such as discovery or batch retrieval need their own permissions; they are not part of this example. If you use a customer-managed KMS key, configure the required decrypt permission and key policy for the consuming role. See the ESO Secrets Manager provider documentation.
Associate an IAM role with the ESO controller service account
The role belongs to the ESO controller’s service account in this example. The demo application reads a Kubernetes Secret and does not need this AWS role.
cat > pod-identity-trust.json <<'EOF'
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"Service": "pods.eks.amazonaws.com"},
"Action": ["sts:AssumeRole", "sts:TagSession"]
}]
}
EOF
ROLE_ARN=$(aws iam create-role --role-name "$ROLE_NAME" \
--assume-role-policy-document file://pod-identity-trust.json \
--query Role.Arn --output text)
aws iam put-role-policy --role-name "$ROLE_NAME" \
--policy-name ReadDemoSecret --policy-document file://read-secret-policy.json
kubectl create namespace external-secrets
kubectl -n external-secrets create serviceaccount external-secrets
ASSOCIATION_ID=$(aws eks create-pod-identity-association \
--region "$AWS_REGION" --cluster-name "$CLUSTER_NAME" \
--namespace external-secrets --service-account external-secrets \
--role-arn "$ROLE_ARN" --query association.associationId --output text)
Use your organization’s role trust restrictions for production. Allow time for IAM and association propagation. See how EKS Pod Identity works.
Install the pinned ESO chart and verify its resources
helm repo add external-secrets https://charts.external-secrets.io
helm repo update
helm upgrade --install external-secrets external-secrets/external-secrets \
--namespace external-secrets --version "$ESO_CHART_VERSION" \
--set installCRDs=true \
--set serviceAccount.create=false \
--set serviceAccount.name=external-secrets \
--wait --timeout 5m
kubectl -n external-secrets rollout status deployment/external-secrets --timeout=120s
kubectl wait --for=condition=Established \
crd/secretstores.external-secrets.io \
crd/externalsecrets.external-secrets.io --timeout=120s
Expected result: Helm finishes successfully, the controller rollout completes and both CRDs are established. Confirm the deployment name against the selected chart if you customize release settings. A list of pods alone does not establish that the CRDs are ready. The ESO installation guide explains chart and CRD management.
Create and verify synchronization
A SecretStore describes the provider and authentication context. An ExternalSecret describes the source values and the Kubernetes Secret ESO should maintain. Use a namespaced store here because the example serves one application namespace.
kubectl create namespace app
cat > app-secrets.yaml <<EOF
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
name: aws-secrets-manager
namespace: app
spec:
provider:
aws:
service: SecretsManager
region: $AWS_REGION
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: app-secret
namespace: app
spec:
refreshPolicy: Periodic
refreshInterval: 1m
secretStoreRef:
name: aws-secrets-manager
kind: SecretStore
target:
name: app-secret
creationPolicy: Owner
deletionPolicy: Retain
dataFrom:
- extract:
key: $SECRET_NAME
EOF
kubectl apply -f app-secrets.yaml
kubectl -n app wait --for=condition=Ready secretstore/aws-secrets-manager --timeout=120s
kubectl -n app wait --for=condition=Ready externalsecret/app-secret --timeout=120s
kubectl -n app get externalsecret app-secret
kubectl -n app get secret app-secret
Expected result: the store and ExternalSecret report Ready, and a Kubernetes Secret named app-secret exists with the two JSON properties. These commands show status and metadata rather than secret values. dataFrom.extract expands properties from this one JSON secret; it does not import all secrets in the AWS account.
The absent auth block is deliberate: the provider uses the controller’s AWS credential chain. A namespaced SecretStore limits where the object can be referenced, but it does not restrict the underlying shared controller IAM identity. For multiple tenants, design per-store permissions, trust and Kubernetes RBAC together; see ESO security best practices.
creationPolicy: Owner makes the ExternalSecret own its target Secret. Deleting that owner can garbage-collect the target. deletionPolicy: Retain addresses provider-side disappearance and does not override owner deletion. Review the lifecycle rules before adopting a policy.
Consume the Kubernetes Secret without logging its values
Map only the keys the application needs. This demonstration checks that the environment contains values, then emits a value-free marker. The DEMO_IMAGE variable above identifies the official BusyBox image by its immutable digest.
cat > app-deployment.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: app-demo
namespace: app
spec:
replicas: 1
selector:
matchLabels:
app: app-demo
template:
metadata:
labels:
app: app-demo
spec:
automountServiceAccountToken: false
securityContext:
runAsNonRoot: true
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
containers:
- name: app
image: $DEMO_IMAGE
command: ["sh", "-c"]
args:
- 'test -n "\$APP_USERNAME" && test -n "\$APP_PASSWORD" && echo "Secret keys are available" && sleep 3600'
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
env:
- name: APP_USERNAME
valueFrom:
secretKeyRef:
name: app-secret
key: username
- name: APP_PASSWORD
valueFrom:
secretKeyRef:
name: app-secret
key: password
resources:
requests:
cpu: 10m
memory: 16Mi
limits:
cpu: 100m
memory: 32Mi
EOF
kubectl apply -f app-deployment.yaml
kubectl -n app rollout status deployment/app-demo --timeout=120s
kubectl -n app logs deployment/app-demo
Expected log output: Secret keys are available. Do not substitute env, echo a credential, or decode the Secret into logs. This test checks key availability, not whether a database accepts a credential.
Test source updates and application reload
ESO’s periodic refresh can update the Kubernetes Secret after the source changes. It does not replace environment variables inside an already running container. Roll out new pods or implement an appropriate application reload strategy. See the Kubernetes consumption documentation.
kubectl -n app get secret app-secret -o jsonpath='{.metadata.resourceVersion}{"\n"}'
cat > demo-secret-v2.json <<'EOF'
{"username":"demo-user","password":"another-demo-only-value"}
EOF
aws secretsmanager put-secret-value --region "$AWS_REGION" \
--secret-id "$SECRET_ARN" --secret-string file://demo-secret-v2.json \
--query VersionId --output text
kubectl -n app get externalsecret app-secret \
-o jsonpath='{.status.refreshTime}{"\n"}'
kubectl -n app get secret app-secret --watch \
-o custom-columns=NAME:.metadata.name,VERSION:.metadata.resourceVersion
Wait for a new refresh and a changed target resourceVersion, then stop the watch with Ctrl+C. Keep the ExternalSecret Ready and check diagnostics if the update fails. A one-minute interval is a schedule, not a guaranteed delivery deadline. A metadata change is evidence of reconciliation, not proof that the application is using the new value.
kubectl -n app rollout restart deployment/app-demo
kubectl -n app rollout status deployment/app-demo --timeout=120s
kubectl -n app logs deployment/app-demo
With an actual application, verify successful authentication against the downstream service after rotation. File-mounted consumers need to re-read or reload changed files; subPath mounts do not receive the same automatic updates. ESO does not change the upstream database credential by itself. Review refresh policies and Kubernetes Secret update behavior.
Troubleshoot synchronization failures
| Symptom | First checks |
|---|---|
| No match for the requested kind or version | Verify the CRDs exist and are established; use v1 with a compatible ESO release. |
| Store is not Ready | Inspect store status, controller events and logs. Confirm region, network access, controller service account, agent availability and Pod Identity association. |
| AWS AccessDenied | Check the role’s permission policy and secret ARN. For customer-managed keys, check KMS permissions and key policy. Do not solve this by attaching SecretsManagerReadWrite. |
| Secret or JSON property not found | Check the exact name, region, account, expected version and JSON structure. Keep sensitive values out of diagnostic output. |
| ExternalSecret is Ready but the app has old credentials | Confirm source and target updates, then check whether the application needs a rollout or file reload. |
kubectl -n app describe secretstore aws-secrets-manager
kubectl -n app describe externalsecret app-secret
kubectl -n external-secrets logs deployment/external-secrets --tail=100
With ESO 2.14.0, a failed ExternalSecret may show only a generic provider error in its condition and Kubernetes event. Inspect the controller log for the detailed AWS error and assumed role before changing permissions.
Inspect output locally and redact it before sharing. These diagnostics should not include secret values, but application errors and configuration can still contain sensitive details.
Plan production access and operations
Restrict who can read Kubernetes Secrets, create pods that consume them, and create or modify stores and ExternalSecrets. Protect secret storage, controller permissions, audit logs and backups. A shared ClusterSecretStore is useful when centrally managed access is intentional; constrain its eligible namespaces and IAM role scope.
Monitor controller health and failed synchronization, choose refresh frequency based on credential requirements and API usage, and decide how to handle source outages or deletion. Source synchronization, credential rotation and application reload are three separate responsibilities. Test the full sequence before depending on it.
Clean up the isolated demo
Remove only resources created for this exercise. Save the association identifier from setup, and make sure the named secret and role are demo resources. AWS secret deletion below uses a recovery window.
kubectl -n app delete deployment app-demo
kubectl -n app delete externalsecret app-secret
kubectl -n app delete secretstore aws-secrets-manager
aws eks delete-pod-identity-association --region "$AWS_REGION" \
--cluster-name "$CLUSTER_NAME" --association-id "$ASSOCIATION_ID"
aws secretsmanager delete-secret --region "$AWS_REGION" \
--secret-id "$SECRET_ARN" --recovery-window-in-days 7
aws iam delete-role-policy --role-name "$ROLE_NAME" --policy-name ReadDemoSecret
aws iam delete-role --role-name "$ROLE_NAME"
rm -f demo-secret.json demo-secret-v2.json
Delete the ExternalSecrets and wait for their cleanup to finish while the ESO controller is still running. ESO 2.14.0 uses cleanup finalizers; uninstalling the controller first can leave those objects or their namespace stuck in Terminating. Restore the controller to let it finish rather than removing finalizers as a routine cleanup step.
Deleting the ExternalSecret should remove its owned target Secret; verify the result. If the ESO release was installed solely for this isolated exercise, uninstall it with Helm and remove the manually created controller service account when no longer needed. Treat CRD removal separately: deleting CRDs can affect every dependent object in the cluster. Keep shared namespaces and installations.
Common questions
Does ESO keep secrets out of Kubernetes?
No. ESO synchronizes external values into Kubernetes Secrets. The external service remains the source, but the copy in the cluster still needs access controls and appropriate storage protection. Evaluate CSI mounts without optional Secret synchronization or direct SDK retrieval if avoiding that copy is a requirement.
Does updating AWS Secrets Manager restart a pod?
No. ESO can refresh the Kubernetes Secret, but a running container’s environment variables remain unchanged. Arrange a rollout or a consumption method with the required reload behavior.
Should every application use ClusterSecretStore?
No. A namespaced SecretStore is a clear starting point for one application. A ClusterSecretStore supports deliberate central management, with explicit namespace eligibility and access boundaries. Neither object scope substitutes for scoped AWS permissions.
Can I use IRSA instead of Pod Identity?
Yes. Configure the documented IRSA trust and service account route rather than combining it with the Pod Identity example. Choose the route supported by your cluster and operating model.
Get help reviewing your Kubernetes secret delivery
CloudHero can help assess IAM scope, Kubernetes access boundaries, synchronization and application reload behavior. Explore our Kubernetes engineering services or talk to our team about your environment.