External Secrets Operator

Now we can explore integrating with Secrets Managed using the External Secrets operator. This has already been installed in our EKS cluster:

~$kubectl -n external-secrets get pods

NAME                                                READY   STATUS    RESTARTS   AGE

external-secrets-6d95d66dc8-5trlv                   1/1     Running   0          7m

external-secrets-cert-controller-774dff987b-krnp7   1/1     Running   0          7m

external-secrets-webhook-6565844f8f-jxst8           1/1     Running   0          7m

~$kubectl -n external-secrets get sa

NAME                  SECRETS   AGE

default               0         7m

external-secrets-sa   0         7m

As we can see theres a ServiceAccount named external-secrets-sa which is tied to an IAM role via IRSA, with access to AWS Secrets Manager for retrieving secrets information.

~$kubectl -n external-secrets describe sa external-secrets-sa | grep Annotations

Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::1234567890:role/eks-workshop-external-secrets-sa-irsa

In addition to that, we'll need to create a new cluster resource called ClusterSecretStore which is a cluster-wide SecretStore that can be referenced by all ExternalSecrets from all namespaces.

~/environment/eks-workshop/modules/security/secrets-manager/cluster-secret-store.yaml

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: "cluster-secret-store"
spec:
  provider:
    aws:
      service: SecretsManager
      region: $AWS_REGION
      auth:
        jwt:
          serviceAccountRef:
            name: "external-secrets-sa"
            namespace: "external-secrets"

~$cat ~/environment/eks-workshop/modules/security/secrets-manager/cluster-secret-store.yaml \

| envsubst | kubectl apply -f -

Take a deeper look at this newly created resources specifications.

~$kubectl get clustersecretstores.external-secrets.io

NAME                   AGE   STATUS   CAPABILITIES   READY

cluster-secret-store   81s   Valid    ReadWrite      True

~$kubectl get clustersecretstores.external-secrets.io cluster-secret-store -o yaml | yq '.spec'

provider:

  aws:

    auth:

      jwt:

        serviceAccountRef:

          name: external-secrets-sa

          namespace: external-secrets

    region: us-west-2

    service: SecretsManager

You can see here, that it's using a JSON Web Token (jwt), referenced to the ServiceAccount we just checked, to sync with AWS Secrets Manager.

Let's move forward and create an ExternalSecret that describes what data should be fetched from AWS Secrets Manager, how the data should be transformed and saved as a Kubernetes Secret. Then we can patch our catalog Deployment to use the External Secret as source for the credentials.

Kustomize Patch

~/environment/eks-workshop/modules/security/secrets-manager/external-secrets/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../../../base-application/catalog
  - external-secret.yaml
patches:
  - path: deployment.yaml

Deployment/catalog

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/created-by: eks-workshop
    app.kubernetes.io/type: app
  name: catalog
  namespace: catalog
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/component: service
      app.kubernetes.io/instance: catalog
      app.kubernetes.io/name: catalog
  template:
    metadata:
      annotations:
        prometheus.io/path: /metrics
        prometheus.io/port: "8080"
        prometheus.io/scrape: "true"
      labels:
        app.kubernetes.io/component: service
        app.kubernetes.io/created-by: eks-workshop
        app.kubernetes.io/instance: catalog
        app.kubernetes.io/name: catalog
    spec:
      containers:
        - env:
            - name: DB_USER
              valueFrom:
                secretKeyRef:
                  key: username
                  name: catalog-external-secret
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: password
                  name: catalog-external-secret
          envFrom:
            - configMapRef:
                name: catalog
          image: public.ecr.aws/aws-containers/retail-store-sample-catalog:0.4.0
          imagePullPolicy: IfNotPresent
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 3
          name: catalog
          ports:
            - containerPort: 8080
              name: http
              protocol: TCP
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            periodSeconds: 5
            successThreshold: 3
          resources:
            limits:
              memory: 512Mi
            requests:
              cpu: 250m
              memory: 512Mi
          securityContext:
            capabilities:
              drop:
                - ALL
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            runAsUser: 1000
          volumeMounts:
            - mountPath: /tmp
              name: tmp-volume
      securityContext:
        fsGroup: 1000
      serviceAccountName: catalog
      volumes:
        - emptyDir:
            medium: Memory
          name: tmp-volume

Diff

             - name: DB_USER
               valueFrom:
                 secretKeyRef:
                   key: username
-                  name: catalog-db
+                  name: catalog-external-secret
             - name: DB_PASSWORD
               valueFrom:
                 secretKeyRef:
                   key: password
-                  name: catalog-db
+                  name: catalog-external-secret
           envFrom:
             - configMapRef:
                 name: catalog
           image: public.ecr.aws/aws-containers/retail-store-sample-catalog:0.4.0

~$kubectl kustomize ~/environment/eks-workshop/modules/security/secrets-manager/external-secrets/ \

| envsubst | kubectl apply -f-

~$kubectl rollout status -n catalog deployment/catalog --timeout=120s

Check the newly created ExternalSecret resource.

~$kubectl -n catalog get externalsecrets.external-secrets.io

NAME                      STORE                  REFRESH INTERVAL   STATUS         READY

catalog-external-secret   cluster-secret-store   1h                 SecretSynced   True

Verify that the resource has a SecretSynced status, which means that it was successfully synchronized from AWS Secrets Manager. Let's take a closer look to this resources specifications.

~$kubectl -n catalog get externalsecrets.external-secrets.io catalog-external-secret -o yaml | yq '.spec'

dataFrom:

  - extract:

      conversionStrategy: Default

      decodingStrategy: None

      key: eks-workshop/catalog-secret

refreshInterval: 1h

secretStoreRef:

  kind: ClusterSecretStore

  name: cluster-secret-store

target:

  creationPolicy: Owner

  deletionPolicy: Retain

Notice the key and the secretStoreRef parameters pointing to the secret we stored on AWS Secrets Manager, and the ClusterSecretStore previously created. Also the refreshInterval is set to 1 hours which means that the value from this secret will be checked and refreshed every hour.

But how do we use this ExternalSecret in our Pods? After we create this resource it automatically created a Kubernetes secret with the same name in the namespace.

~$kubectl -n catalog get secrets

NAME                      TYPE     DATA   AGE

catalog-db                Opaque   2      21h

catalog-external-secret   Opaque   2      1m

catalog-secret            Opaque   2      5h40m

Take a deeper look in this secret.

~$kubectl -n catalog get secret catalog-external-secret -o yaml | yq '.metadata.ownerReferences'

- apiVersion: external-secrets.io/v1beta1

  blockOwnerDeletion: true

  controller: true

  kind: ExternalSecret

  name: catalog-external-secret

  uid: b8710001-366c-44c2-8e8d-462d85b1b8d7

See that it has an ownerReference that points to External Secrets Operator.

Now check that the catalog pod is already updated with the values from this new secret, and it's up and running!

~$kubectl -n catalog get pods

NAME                       READY   STATUS    RESTARTS   AGE

catalog-777c4d5dc8-lmf6v   1/1     Running   0          1m

catalog-mysql-0            1/1     Running   0          24h

~$kubectl -n catalog get deployment catalog -o yaml | yq '.spec.template.spec.containers[] | .env'

- name: DB_USER

  valueFrom:

    secretKeyRef:

      key: username

      name: catalog-external-secret

- name: DB_PASSWORD

  valueFrom:

    secretKeyRef:

      key: password

      name: catalog-external-secret

Conclusion

In conclusion there is no best option on choosing between AWS Secrets and Configuration Provider (ASCP) vs. External Secrets Operator (ESO) in order to manage your secrets stored on AWS Secrets Manager.

Both tools have their specific advantages, for example, ASCP can help you avoid exposing secrets as environment variables, mounting them as volumes directly from AWS Secrets Manager into a Pod, the drawback is the need to manage those volumes. In the other hand ESO makes easier the Kubernetes Secrets lifecycle management, having also a cluster wide SecretStore, however it doesn't allow you to use Secrets as volumes. It all depends on your use case, and having both can bring you a lot more flexibility and security with Secrets Management.