The Basics of GitOps and Secure Deployment in Kubernetes

Today, I would like to discuss secrets and GitOps in the K8S world.

Some of my friends who are not DevOps/SRE engineers and most developers asked me too many times how to store Secrets/API keys/environment variables within the deployment process.

I hope this article will help to understand with a trivial explanation of the basics of how-to store secrets in the deployment process in the GitOps Way.

First things first

Plan of our journey:

a small introduction about GitOps
deploy k8s cluster ( EKS )
encrypt sensitive data ( SOPS )
deploy ArgoCD ( GitOps tool )
deploy our application in the GitOps way
inference

GitOps and why it is so important

Let's imagine you have some Git repo and deploy your application into Kubernetes via Jenkins, AKA IaC style (Infrastructure as a code). At first look, this architecture is magnificent. We have stored our code in repo, and we deploy it via any CI\CD tool.

But why is GitOps so major? The answer to that question lies in how Kubernetes API works.

When you apply something into Kubernetes, the API will make you aware that the syntax is acceptable or not. If it's OK, it will give you the lowdown that your resources are declaratively written. Kubernetes will deploy everything you declared.

But the first problem is that k8s does not guarantee that and the second one is that if somebody deletes/change/update information about any component or resource, you will never know until it is late.

The GitOps way implies that what is in the repo will be in Kubernetes. It literally syncs the states between your repo and Kubernetes. If an unspecified person changes something in Kubernetes ( delete or update config map, for example ), the GitOps tool will sync state and overwrite changes. It works in both ways.

It's easy with EKS

I will cover the deployment process with eksctl tool. I assume that you are already familiar with AWS and with K8S, so I will not explain much on it.
First of all, you need to install eksctl.

brew install eksctl

For security reasons, it is not a best practice to use AWS access keys. That's why we are going to use IAM Roles to have the ability to use the AWS KMS service within the Kubernetes pods. Hence, we need to create an AWS IAM Policy for KMS keys and assign this Policy to our EKS Worker group, which we will discuss later in this article. I have prepared two git repo for this reason. You are free to use them.

So what we are going to do next is:

Create IAM role and put ARN reference into the EKS config
Create KMS key
Deploy EKS cluster with KMS Policy

git clone https://github.com/n-g-s/argocd.git
cd argocd/eks
aws iam create-policy --policy-name AmazonKMSFullAccess --policy-document file://kms_policy.json
aws kms create-key --tags TagKey=ENV,TagValue=DEV --description "GitOps KMS key"
aws kms create-alias --alias-name alias/gitops --target-key-id  48161235-0e48-4aac-9a2c-9b2f99d9e167 # Optional, but if you would like to have clear visibility, you should do that
vim eks.yaml # Change line 27 to your AWS account.
eksctl create cluster -f eks.yaml

You should see something like this in the KMS section of the AWS console.

How to encrypt the data with SOPS

In this section, we will discuss how to encrypt the data with SOPS. I've drawn a small explanation of how SOPS works with encryption and the KMS keys. In a few words, SOPS is a tool written by Mozilla to encrypt YAML\JSON and other formats by encryption only values and not keys. More information about this tool you can find here.

To install SOPS, you need to follow this command. We will encrypt with SOPS the official Jenkins helm chart values.yaml and push it to the repo.

brew install sops
git clone https://github.com/n-g-s/argocd-example.git
cd argocd-example/charts
export SOPS_KMS_ARN="arn:aws:kms:us-east-1:591415181778:key/48161235-0e48-4aac-9a2c-9b2f99d9e167"
vim values.yaml # Change the password on line 106
sops -e values.yaml > values_encrypted.yaml
git add . 
git commit -S -m "Added Jenkins Helm chart with encrypted values file"
git push

ArgoCD - Declarative GitOps CD for Kubernetes

ArgoCD is a tool which we will deploy into Kubernetes and which will do our deployment process. I've prepared a simple schema of how ArgoCD will do the job.

Argo CD is un-opinionated about how secrets are managed and let's you decide how to handle this problem independently by allowing you to customize the deployment workflow with any tool. You can read more about it here, but in a few words, you need to do two things:

1. It would be best if you injected the tool via custom tooling. You will have two options:

Adding Tools Via Volume Mounts
Build Your Own Image ( which I choose )


ARG ARGOCD_VERSION=v1.7.8

FROM argoproj/argocd:${ARGOCD_VERSION}

ARG SOPS_VERSION="v3.6.1"

# Switch to root for the ability to perform install
USER root

# Install tools needed for your repo-server to retrieve & decrypt secrets, render manifests 
# (e.g. curl, awscli, gpg, sops)
RUN apt-get update && \
    apt-get install -y \
        curl \
        awscli \
        gpg && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && \
    curl -o /usr/local/bin/sops -L https://github.com/mozilla/sops/releases/download/${SOPS_VERSION}/sops-${SOPS_VERSION}.linux && \
    chmod +x /usr/local/bin/sops

# Switch back to non-root user
USER argocd

2. Register a new plugin in argocd-cm ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
  name: argocd-cm
data:
  configManagementPlugins: |
    - name: sops
      init:
        command: ["sh", "-xc"]
        args: ["sops -d values_encrypted.yaml > values_decrypted.yaml"]
      generate:
        command: ["sh", "-xc"]
        args: ["helm template -n ${ARGOCD_APP_NAMESPACE} ${ARGOCD_APP_NAME} -f values_decrypted.yaml ."]

Let's deploy the ArgoCD with our changes.

kubectl create ns argocd
kubens argocd # change namespace
kubectl apply -f https://raw.githubusercontent.com/n-g-s/argocd/main/manifests/install.yaml
kubectl port-forward svc/argocd-server -n argocd 8080:443
brew install argocd # cli tool
kubectl get pods -n argocd -l app.kubernetes.io/name=argocd-server -o name | cut -d'/' -f 2 # Initial password
argocd login 127.0.0.1:8080 # user admin and password 
argocd account update-password # change password to a new one

Now you can login with your new password into UI via your Web browser

Last but not least - Application deployment!

Full description of declarative you can read from official documentation, but the only thing I would like to mention.

plugin - sops
inject environment variables passed to the plugin

So let's deploy our application and check if the password is correct with our encryption.

kubectl apply -f https://raw.githubusercontent.com/n-g-s/argocd-example/main/application.yaml

kubectl port-forward service/jenkins 8090:8080 -n jenkins

Inference

As you can see from above, the secured deployment process is not rocket science. There a variety of options on how to do that and how to manage it. The conclusion from my side is - don't be afraid of trying something new.
I hope this article explained the basics of GitOps and secure deployment.
Few links to read:

FluxCD - Alternative CD tool
List of Encryption tools - there are a lot of them.