Managing Kubernetes secrets with AWS Parameter Store

Managing secrets in a Kubernetes cluster can be overwhelming at a large scale having more configuration for microservices. Ever wonder how you store those there? As the majority of the running workloads inside the Kubernetes cluster need some kind of configuration or secrets. Kubernetes secrets help to decouple those secrets from the application codebase and reuse them inside the container pod runtime when required. You won’t be using every time imperative way to create those Kubernetes secrets objects by using the kubectl client to hit on those Kube API requests. What if you need to change or edit secrets often which are stored in the etcd database? It’s good to store on it rather than the pod definition or container image itself either private or public itself.

Secrets are stored by default in the insecure base64 format.

What if those Kubernetes resources required external secrets, especially from third parties like hashicorp vault, aws secret manager, or any other platform? It doesn’t come up natively support on Kubernetes. Hence External Secret Operator came into the big picture of the bond between those bridges. It integrates external secrets platform to make API calls to retrieve secret data and injects it as a native Kubernetes secret object. External Secret Operator uses the operator/controller patterns that extend the Kubernetes running application using Custom Resources Definition.

The External Secret Operator manages secrets via Custom Resource Definitions. ExternalSecret, SecretStore, and ClusterSecretStore are user-friendly wrappers around the external API that store and manage secrets on your behalf.

• SecretStore: Custom Resource Definition used for the authentication and access management of the external secret store to fetch the secret from the external API.
• ExternalSecret: Custom Resource Definition that specifies what secret data to fetch. It references SecretStore which knows how to access that data. The controller uses the external secret as a blueprint to create secrets and sync them at a time interval you choose.
• ClusterSecretStore — A global, cluster-wide SecretStore that can be referenced from all namespaces. You can use it to provide a central gateway to your secret provider.

If secrets from external API got changed, the controller reconciles the state in the cluster and updates the secret. In this blog, we will be managing the secrets with the aws parameter store via External Secret Operator on Elastic Kubernetes Service. We will create the deployment where we will mount secrets retrieving from the parameter store and expose it into the pod via environment variables.

Ensure you have the following components and tools

• Kubectl
• Helm
• AWS Account and Access
• EKS Cluster

To achieve our requirements, we will create an IAM role with the attached policy to read secrets which will associate it with Kubernetes Service Account. The following commands will deploy an External Secrets Operator to my EKS Cluster.

helm repo add external-secrets https://charts.external-secrets.io
helm repo update
helm install external-secrets external-secrets/external-secrets

As pod’s container can able to call API requests to AWS Service via cli or SDK with AWS Identity and Access management permission. Instead of creating aws credentials, you can associate the IAM role with a Kubernetes service account and configure your pods to use the service account. We will create an IAM role, and bind the policy to be able to read secrets from the parameter store for the Kubernetes service account. To use AWS Identity and Access Management (IAM) roles for service accounts, an IAM OIDC provider must exist for your cluster. Here I will be using eksctl, you can provision via declarative way using terraform too.

eksctl utils associate-iam-oidc-provider --cluster your-cluster --approve --region your-region

We use the principle of least privilege on every environment to limit the permissions for the external-secrets IAM user to only access the path of the needed secret.

cat >policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ssm:GetParameter"
        "ssm:GetParametersByPath"
      ],
      "Resource": "arn:aws:ssm:region:id:parameter/*"
    }
  ]
}
EOF

Create an IAM policy

aws iam create-policy --policy-name my-policy --policy-document file://policy.json

Create an IAM role and associate it with a Kubernetes service account. The following command ‘eksctl create iamserviceaccount’ takes an IAM policy arn as an argument, creates an IAM role associated with the given policy, and maps a service account to that role.

eksctl create iamserviceaccount --name my-service-account --namespace your-namespace --cluster your-cluster --role-name "my-role" \
    --attach-policy-arn arn:aws:iam::id:policy/get-parameter-store-policy --approve

Confirm that the Kubernetes service account is annotated with the role.

kubectl describe serviceaccount my-service-account -n your-namespace

Create and deploy SecretProviderClass custom resource and by using provider: aws

Demo here we make a simple deployment in the EKS cluster running out the single replica of Nginx mounting the secrets retrieved via ExternalSecret object. Let’s create some env on the AWS parameter store.

cat <<EOF | kubectl apply -f -
apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: secret-env
spec:
  backendType: systemManager
  dataFrom:
    - /development/nginx-secret/env
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: secret-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      nginx: secret
  template:
    metadata:
      annotations:
      labels:
        nginx: secret
    spec:
      containers:
        - name: secret-nginx
          image: nginx:latest
          imagePullPolicy: Always
          envFrom:
            - secretRef:
                name: secret-env
EOF

We can evaluate the secret value by typing:

kubectl get secrets secret-env -o json

Output

{
    "apiVersion": "v1",
    "data": {
        "cred": "c29tZXRoaW5n",
        "env": "ZGV2ZWxvcG1lbnQ="
    },
    "kind": "Secret",
    "metadata": {
        "creationTimestamp": "2022-11-29T07:54:29Z",
        "name": "secret-env",
        "namespace": "default",
        "ownerReferences": [
            {
                "apiVersion": "kubernetes-client.io/v1",
                "controller": true,
                "kind": "ExternalSecret",
                "name": "secret-env",
                "uid": "6987878d-9711-4147-b4ac-788c861ad823"
            }
        ],
        "resourceVersion": "278798378",
        "uid": "12efc64a-d373-4884-99f5-5d236abefc12"
    },
    "type": "Opaque"
}

Else you can also print the env inside the pod using the exec command.

Output:

kubectl exec -it pod-name -- env

Conclusion

Kudos to you, you made it to the end. You can finally able to setup up the whole process for managing secrets using External Secret Manager with AWS Parameter Store. Kicked off by GoDaddy to solve the issue of reading the information from a third-party service like AWS Parameter Store and injecting the values as Kubernetes Secrets.

Feel free to provide feedback or reach out to me via Twitter. I look forward to hearing from you!

Are you interested in blog updates? You can also subscribe to my blog newsletter.