Delivering Cloud Infrastructure the GitOps Way with Crossplane and ArgoCD

In this blog, we will delve into the exciting world of modern infrastructure management using Crossplane and ArgoCD. , a methodology that brings the principles of and collaboration to . In this blog, we'll guide you through the steps of leveraging Crossplane and ArgoCD to deliver infrastructure the GitOps way. GitOps version control infrastructure as code (IaC) Please be aware that this blog marks the beginning of the Crossplane + ArgoCD series. In this installment, we'll delve into the fundamental configuration required to deploy infrastructure using the GitOps approach. Subsequent blog posts will provide more comprehensive insights into various concepts. Prerequisites Docker kind, there are many ways to install kind, please refer to the link to install as you see fit. kubectl, we will use this to interact with the cluster. Github account Helm to install crossplane charts AWS Account as in this blog we will create an SQS resource in AWS. What are we going to create/observe? At a high level we will do the following - Private Github Repository setup with ArgoCD Kubernetes cluster with Crossplane and ArgoCD Create an AWS SQS queue the GitOps way Import an existing AWS SQS queue the GitOps way Delete the resource from cluster and observe that ArgoCD brings it back kubernetes Delete the resource from AWS Console and observe that Crossplane brings it back Create a Private GitHub Repository Assuming you already have a GitHub account. The following steps will create the GitHub repository and generate a personal access token. ArgoCD will watch this repository to sync the resources in the Kubernetes cluster. Create a new private GitHub repository as shown in the screenshot below, we will name it as , if it's not available you can use any name - xargocd-gitops After the successful creation of the repository it will look as below - Create Personal access tokens (classic) with scope as below and click on button, make a note of the token, we will need this while configuring ArgoCD to watch our private git repo - repo Generate Token Create Kubernetes cluster using KIND I have explained in detail in an earlier blog how to setup a functional Kubernetes cluster using KIND, hence we will not go into detail again. Please read the for details on KIND. blog Create a KIND cluster as below - kind create cluster --config - <<EOF kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 name: "platformwale" # configure cluster with containerd registry config dir enabled containerdConfigPatches: - |- [plugins."io.containerd.grpc.v1.cri".registry] config_path = "/etc/containerd/certs.d" nodes: - role: control-plane image: "kindest/node:v1.27.3" EOF Make sure the kubectl context is pointing to the kind cluster created above - # set the context as below kubectl config use-context kind-platformwale # validate as below kubectl config current-context Install and Configure ArgoCD Let's install the stable release of the ArgoCD in the KIND cluster we created above. At the time of writing this blog, the ArgoCD stable release was on version . v2.8.0 kubectl create namespace argocd kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml Make sure all the argocd pods are running as below - $ kubectl get po -n argocd NAME READY STATUS RESTARTS AGE argocd-application-controller-0 1/1 Running 0 49s argocd-applicationset-controller-74bd4b8497-5bfcd 1/1 Running 0 49s argocd-dex-server-6c6dfb6597-twwnw 1/1 Running 0 49s argocd-notifications-controller-6786586cb-87nxc 1/1 Running 0 49s argocd-redis-b5d6bf5f5-xbqc5 1/1 Running 0 49s argocd-repo-server-6658f8b96d-6jmwt 1/1 Running 0 49s argocd-server-5fff657769-nnqwm 1/1 Running 0 49s Install as suggested in the link, in my case, I was working on Mac and I installed CLI as below - ArgoCD CLI brew install argocd Port forward argocd service so that you can access the argocd UI at http://localhost:8080 as below - kubectl port-forward svc/argocd-server -n argocd 8080:443 Modify the initial secret for argocd UI by running following commands in another terminal - # open another terminal # make sure your kubecontext is pointing to the cluster you created above kubectl config use-context kind-platformwale # this will stdout the initial password, copy that, you will need it for the command below argocd admin initial-password -n argocd # login using the password from above command, the Username will be `admin` and Password will be the one you copied above argocd login localhost:8080 # update password; it will ask for current password, which is the one from command above and provide a new password argocd account update-password # delete the secret holding initial argocd password kubectl delete secret -n argocd argocd-initial-admin-secret Make sure you are able to use the new password and login to argocd UI using . On successful login you will see something as below - http://localhost:8080 Username: admin Configure ArgoCD to watch the private GitHub repository Now we will configure ArgoCD to watch the Private Github Repository we created in the section above. Assuming you have copied the private access token, you will need to substitute below and apply the manifest - GITHUB_PRIVATE_ACCESS_TOKEN= YOUR_PRIVATE_GITHUB_REPO_URL= YOUR_GITHUB_USERNAME= kubectl apply -f - <<EOF --- # setup argocd secret for private gitops repository apiVersion: v1 kind: Secret metadata: name: xargocd-gitops-credentials namespace: argocd labels: argocd.argoproj.io/secret-type: repository type: Opaque stringData: name: xargocd-gitops url: ${YOUR_PRIVATE_GITHUB_REPO_URL} username: ${YOUR_GITHUB_USERNAME} password: ${GITHUB_PRIVATE_ACCESS_TOKEN} EOF Make sure the secret is created successfully - $ kubectl get secrets -n argocd xargocd-gitops-credentials NAME TYPE DATA AGE xargocd-gitops-credentials Opaque 4 4m1s Configure ArgoCD AppProject CRs The following AppProject CR allows to deploy ArgoCD Application CRs to only namespace within the KIND cluster - argocd SOURCE_REPO= kubectl apply -f - <<EOF # The 'applications-project' AppProject allows a non-admin user to deploy # only Application resources in the 'argocd' namespace. --- apiVersion: argoproj.io/v1alpha1 kind: AppProject metadata: name: applications-project namespace: argocd finalizers: - resources-finalizer.argocd.argoproj.io spec: description: Project for argocd applicatons sourceRepos: - ${SOURCE_REPO} # # Allow this project to deploy only to 'argocd' namespace # destinations: - namespace: argocd server: https://kubernetes.default.svc # # Deny all namespace-scoped resources from being created, except for Application # namespaceResourceWhitelist: - group: 'argoproj.io' kind: Application EOF The following AppProject CR allows to deploy any AWS resources from API to any namespace within the KIND cluster - SQS sqs.aws.upbound.io SOURCE_REPO= kubectl apply -f - <<EOF # 'sqs-project' is used for actual SQS crossplane resources apiVersion: argoproj.io/v1alpha1 kind: AppProject metadata: name: sqs-project namespace: argocd finalizers: - resources-finalizer.argocd.argoproj.io spec: description: Project for deploying crossplane resources to the cluster sourceRepos: - ${SOURCE_REPO} # can deploy to any namespace but within the kind cluster only destinations: - namespace: '*' server: https://kubernetes.default.svc # can deploy any SQS resources clusterResourceWhitelist: - group: 'sqs.aws.upbound.io' kind: '*' EOF Make sure the CRs are installed successfully as below - AppProject $ kubectl get appprojects.argoproj.io -A NAMESPACE NAME AGE argocd applications-project 24s argocd default 60m argocd sqs-project 12s Setup ArgoCD Application Let's configure an argocd Application CR to install any SQS resources under directory under private repository, we will create the directory after the Crossplane is installed. sqs-queues xargocd-gitops REPO_URL= kubectl apply -f - <<EOF --- apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: sqs-queues namespace: argocd finalizers: - resources-finalizer.argocd.argoproj.io spec: project: sqs-project source: repoURL: ${REPO_URL} targetRevision: HEAD path: ./sqs-queues destination: server: https://kubernetes.default.svc namespace: crossplane-system syncPolicy: automated: prune: true selfHeal: true syncOptions: - SyncWaveOrder=true retry: limit: 1 backoff: duration: 5s factor: 2 maxDuration: 1m EOF Make sure the argocd application is created successfully as below - $ kubectl get applications -A NAMESPACE NAME SYNC STATUS HEALTH STATUS argocd sqs-queues Unknown Healthy $ argocd app list NAME CLUSTER NAMESPACE PROJECT STATUS HEALTH SYNCPOLICY CONDITIONS REPO PATH TARGET argocd/sqs-queues https://kubernetes.default.svc crossplane-system sqs-project Unknown Healthy Auto-Prune ComparisonError https://github.com/piyushjajoo/xargocd-gitops ./sqs-queues HEAD Let's fix the for now, this is because directory is not present in the repository just yet, the error will disappear once we setup the directory in next section. ComparisonError sqs-queues Create a directory named and add any dummy text file for e.g. to the directory and commit to git repo. This will fix the argocd app. sqs-queues dummy $ argocd app list NAME CLUSTER NAMESPACE PROJECT STATUS HEALTH SYNCPOLICY CONDITIONS REPO PATH TARGET argocd/sqs-queues https://kubernetes.default.svc crossplane-system sqs-project Synced Healthy Auto-Prune https://github.com/piyushjajoo/xargocd-gitops ./sqs-queues HEAD Install Crossplane and Configure Provider AWS In this section, we will install the Crossplane core and Crossplane AWS Provider controller for SQS resources. Install Crossplane Core, at the time of writing this blog, we Crossplane version - v1.13.2 # add crossplane helm repo helm repo add \ crossplane-stable https://charts.crossplane.io/stable # update the added helm repo helm repo update # to check the current version of the crossplane core helm search repo crossplane-stable # install crossplane core helm chart helm install crossplane \ crossplane-stable/crossplane \ --namespace crossplane-system \ --create-namespace Make sure all the crossplane pods are installed successfully - $ kubectl get po -n crossplane-system NAME READY STATUS RESTARTS AGE crossplane-8697f8cff4-mw5j6 1/1 Running 0 35s crossplane-rbac-manager-6f8dbd9ffd-tkcb9 1/1 Running 0 35s Install Crossplane AWS Provider as below, at time of writing this blog, was at version - provider-aws-sqs v0.38.0 cat <<EOF | kubectl apply -f - apiVersion: pkg.crossplane.io/v1 kind: Provider metadata: name: provider-aws-sqs spec: package: xpkg.upbound.io/upbound/provider-aws-sqs:v0.38.0 EOF Make sure the AWS Provider is installed successfully as below - $ kubectl get providers NAME INSTALLED HEALTHY PACKAGE AGE provider-aws-sqs True True xpkg.upbound.io/upbound/provider-aws-sqs:v0.38.0 29s upbound-provider-family-aws True True xpkg.upbound.io/upbound/provider-family-aws:v0.38.0 25s Also make sure the for SQS are available - api-resources $ kubectl api-resources | grep sqs.aws.upbound.io queuepolicies sqs.aws.upbound.io/v1beta1 false QueuePolicy queueredriveallowpolicies sqs.aws.upbound.io/v1beta1 false QueueRedriveAllowPolicy queueredrivepolicies sqs.aws.upbound.io/v1beta1 false QueueRedrivePolicy queues sqs.aws.upbound.io/v1beta1 false Queue Create IAM User with SQS permissions Crossplane will need AWS IAM Permissions to be able to create SQS queues, in our use case we will simply create an IAM Role with permissions as below from AWS Console. SQSFullAccess Go to in AWS UI IAM > Users console Create an IAM User, you can choose any name you want, on next screen select and search for and create the IAM User. Attach policies directly AmazonSQSFullAccess Once the user is created, click on and go to tab and scroll down to section . View User Security Credentials Access Keys Click on and make sure to copy the values before you hit or at least download the file, as we will need these credentials to setup the for Provider AWS as shown below. Create access key Done .csv ProviderConfig NOTE: there are different ways you can configure credentials for AWS Provider, but for this blog, we will use IAM User credentials, in upcoming blogs we will go over other ways, for e.g. . The various ways of authentication for Provider AWS are mentioned in this . InjectedIdentity document Setup ProviderConfig In this section, we will setup a for Provider AWS, this is a way to make provider AWS controller create resources in your desired AWS Account. ProviderConfig Create the file with the IAM User credentials you copied in the section above, make sure to substitute the values - aws-credentials.txt [default] aws_access_key_id = REPLACE_ME_AWS_ACCESS_KEY_ID aws_secret_access_key = REPLACE_ME_AWS_SECRET_ACCESS_KEY Create the kubernetes secret as below - kubectl create secret \ generic aws-secret \ -n crossplane-system \ --from-file=creds=./aws-credentials.txt Create as below - ProviderConfig cat <<EOF | kubectl apply -f - apiVersion: aws.upbound.io/v1beta1 kind: ProviderConfig metadata: name: default spec: credentials: source: Secret secretRef: namespace: crossplane-system name: aws-secret key: creds EOF This is a way to configure ProviderConfig for provider-aws, i.e. if is not mentioned in the crossplane resource being created, it will by default pick the ProviderConfig. default providerConfigRef default Validate the crossplane and provider aws setup That's it, we are now ready to make a commit to git and see the resource getting created. Or maybe one last validation before we deliver infrastructure through GitOps. Submit the following resource using kubectl and validate if the queue is created in your desired AWS Account in the desired region, notice in the example below we are explicitly specifying , but this will work even if we don't mention the ProviderConfig - Queue providerConfigRef -> default default kubectl apply -f - <<EOF apiVersion: sqs.aws.upbound.io/v1beta1 kind: Queue metadata: name: demo-queue spec: forProvider: name: demo-queue region: us-west-2 providerConfigRef: name: default EOF Give it a few seconds the sqs-queue resource will be created, you can also validate in the AWS Console - $ kubectl get queues.sqs.aws.upbound.io NAME READY SYNCED EXTERNAL-NAME AGE demo-queue True True https://sqs.us-west-2.amazonaws.com/xxxxxxxxx/demo-queue 37s Delete the queue as follows and make sure that the queue is deleted - $ kubectl delete queues.sqs.aws.upbound.io demo-queue queue.sqs.aws.upbound.io "demo-queue" deleted $ kubectl get queues.sqs.aws.upbound.io No resources found See the GitOps + Crossplane + ArgoCD magic We are ready to see Crossplane and ArgoCD in action. Here is the architecture - Create a new SQS Queue Create a commit with the following YAML manifest in directory in the github repository you created earlier, this is the directory you had configured earlier with ArgoCD. In this scenario, we are creating a new resource via GitOps. sqs-queues apiVersion: sqs.aws.upbound.io/v1beta1 kind: Queue metadata: name: platformwale-queue spec: forProvider: name: platformwale-queue region: us-west-2 Your GitHub repo will look as below - Go to argocd UI to your sqs-queues application and hit button if argocd has not yet auto synced the repo, wait for a few seconds and you will see that repo is synced as below, also validate that the is created in the specified AWS Region in configured AWS Account. Refresh platformwale-queue $ kubectl get queues.sqs.aws.upbound.io NAME READY SYNCED EXTERNAL-NAME AGE platformwale-queue True True https://sqs.us-west-2.amazonaws.com/xxxxxxx/platformwale-queue 6m48s Import an existing SQS Queue Let's do the other way around, let's create a Queue from AWS Console, let's call it in region and let's try to import it. platformwale-queue2 us-west-2 Create the queue in with all the default values and copy the value of field . For example - platformwale-queue2 us-west-2 URL https://sqs.us-west-2.amazonaws.com/xxxxx/platformwale-queue2 Submit the manifest below to git repo under directory as earlier, this time you will notice that, it will successfully import the existing resource and will not try to recreate the resource. sqs-queues apiVersion: sqs.aws.upbound.io/v1beta1 kind: Queue metadata: name: platformwale-queue2 annotations: crossplane.io/external-name: https://sqs.us-west-2.amazonaws.com/xxxxx/platformwale-queue2 spec: forProvider: name: platformwale-queue2 region: us-west-2 The magic happens through annotation we specified in our manifest. This is the annotation modified by crossplane after the resource is created and crossplane validates if the resource exists with the value specified in the annotation, if yes, it simply tries to import the resource. crossplane.io/external-name $ kubectl get queues.sqs.aws.upbound.io NAME READY SYNCED EXTERNAL-NAME AGE platformwale-queue True True https://sqs.us-west-2.amazonaws.com/xxxxx/platformwale-queue 16m platformwale-queue2 True True https://sqs.us-west-2.amazonaws.com/xxxxx/platformwale-queue2 15s We will cover importing resource in detail in our upcoming blogs. That's it!! There you have a functional GitOps way of delivering infrastructure using Crossplane + ArgoCD. Automatic Drift Detection Let's be ambitious and , you will see that in sometime argocd will resync and the resource will be back - delete the resource from the cluster # delete the queue $ kubectl delete queues platformwale-queue queue.sqs.aws.upbound.io "platformwale-queue" deleted # queue is gone and you see it being recreated, you can also valdiate in AWS Console $ kubectl get queues NAME READY SYNCED EXTERNAL-NAME AGE platformwale-queue False True 2s Let's be super ambitious and , you will see that in sometime (for me it was ~10 mins, I am sure this is configurable) the resource will be back - delete the resource from AWS UI Console $ kubectl describe queues platformwale-queue .... .... .... Normal CreatedExternalResource 16s (x3 over 22m) managed/sqs.aws.upbound.io/v1beta1, kind=queue Successfully requested creation of external resource Cleanup This is extremely important, you don't want to see unexpected costs in your AWS bill. GitOps makes it even smooth to cleanup. Simply delete the YAML files created under directory and then run , this will delete the resources whose manifests you just deleted. sqs-queues argocd app sync sqs-qeueues --prune $ kubectl get queues No resources found Make sure to you created earlier from AWS Console. delete the IAM User Also if you no longer need a private GitHub repository, you can delete the repository. Also make sure to delete the personal access token you generated if you don't want to continue using it. Please leverage GitHub UI to clean them up. Finally, if you are done using cluster, delete it as below, please don't delete the cluster before deleting the cloud resources, otherwise you will need to clean the cloud resources manually - kind kind delete cluster --name platformwale ## validation $ kind delete cluster --name platformwale Deleting cluster "platformwale" ... Deleted nodes: ["platformwale-control-plane"] Conclusion Therefore, we've observed how Crossplane and ArgoCD can be harnessed to implement the GitOps methodology for delivering infrastructure. This capability holds immense potential for both internal developer teams and platform teams. The built-in abilities for detecting drift and conducting reconciliations offer remarkable advantages that any platform engineer would be enthusiastic about offering to their internal developer groups. Resources Crossplane documentation ArgoCD Documentation Setting Up a Kubernetes Cluster with a Local Registry and Ingress in Docker using KIND Crossplane Providers Author Notes Feel free to reach out with any concerns or questions you have. I will make every effort to address your inquiries and provide resolutions. Stay tuned for the upcoming blog in this series dedicated to Platformwale (Engineers who work on Infrastructure Platform teams). Originally published on Aug 19, 2023. here