Using Spinnaker with Kubernetes for CI/CD

Kubernetes is now the de-facto standard for container orchestration. With more and more organizations adopting Kubernetes, it is essential that we get our fundamental ops-infra in place before any migration. This post will focus on pushing out new releases of the application to our Kubernetes cluster i.e. Continuous Delivery. Pre-requisites Running Kubernetes Cluster (GKE is used for the purpose of this blog). A Spinnaker set-up with . Jenkins CI enabled Github webhook enabled for Jenkins jobs. Strategy Overview : CI System to build the docker image and push to registry. Github + Jenkins : Registry to store docker images. Docker hub : CD System to enable environment and . Spinnaker automatic deployments to Staging supervised deployment to Production Continuous Delivery Pipeline for Kubernetes Continuous Integration System Although this post is about the CD system using Spinnaker. I want to briefly go over the CI pipeline so that the bigger picture is clear. Whenever the master branch of a Git repo gets merged, a Jenkins job is triggered via . Github webhook The commit message for the master merge should include the updated version of the app, and whether it is Kubernetes Deploy action or Patch action. The Jenkins job checks out the repo, builds code and the docker image according to the Dockerfile, and pushes it to Docker hub. It then triggers a and sends a file as a . This properties file contains very crucial info that is consumed by Spinnaker and will be explained later in this post. Spinnaker pipeline trigger.properties build artifact Properties File TAG=1.7-SNAPSHOT ACTION=DEPLOY Continuous Delivery System This is the crucial part. Spinnaker offers a ton of options for Kubernetes deployments. You can either consume manifests from GCS or S3 bucket or you can provide manifest as text within the pipeline. Consuming manifests from GCS or S3 buckets includes more moving parts, and since this is an introductory blog, it is beyond this article's scope right now. However, with that being said, I extensively use that approach as it is best in scenarios where you need to deploy a large number of micro-services running in Kubernetes as such pipelines are highly templatized and re-usable. Today, we will deploy a sample Nginx Service which reads the app version from a file and renders it on the browser. Application code and Dockerfile can be found . The part where is updated can be seen below (This is what the Jenkins job does basically). pom.xml here index.html if_master=` | jq | grep master` [ $? -eq 1 ]; 2 TAG=`grep SNAPSHOT pom.xml | sed ` ACTION=$( | jq -r | cut -d -f2) sed -i -- app/index.html docker build -t vaibhavthakur/nginx-demo: . docker push vaibhavthakur/nginx-demo: TAG= > trigger.properties ACTION= >> trigger.properties #!/bin/bash #Author: Vaibhav Thakur #Checking whether commit was in master of not. echo $payload '.ref' if then echo "Pipeline should not be triggered" exit fi #Getting tag from pom.xml 's|[ ,/,version ]||g' echo $TAG #Getting action from commit message echo $payload '.commits[0].message' ',' #Updating index.html "s/VER/ /g" ${TAG} #Pushing to dockerhub $TAG $TAG echo ${TAG} echo ${ACTION} The manifest for the Nginx deployment and service is below: apiVersion: v1 kind: Namespace metadata: name: nginx --- apiVersion: extensions/v1beta1 kind: Deployment metadata: namespace: nginx name: nginx-deployment spec: replicas: 1 template: metadata: annotations: prometheus.io/path: prometheus.io/scrape: prometheus.io/port: labels: app: nginx-server spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - nginx-server topologyKey: kubernetes.io/hostname containers: - name: nginx-demo image: vaibhavthakur/nginx-demo:1.0-SNAPSHOT imagePullPolicy: Always resources: limits: cpu: 2500m requests: cpu: 2000m ports: - containerPort: 80 name: http --- apiVersion: v1 kind: Service metadata: namespace: nginx name: nginx-service annotations: cloud.google.com/load-balancer-type: Internal spec: ports: - port: 80 targetPort: 80 name: http selector: app: nginx-server : LoadBalancer "/status/format/prometheus" "true" "80" type Steps to Set-up the Pipeline: Create a new application under the tab and add and you to it. Remaining fields can be left blank. Aplications Project Name Email Create a new project under Spinnaker and add your application under it. Also, you can add your staging and production Kubernetes cluster under it. 3. Under the section add the Pipeline. Make sure that in the stage of the pipeline, is enabled and the artifacts are consumed appropriately. Demo-Project configuration Application Trigger Jenkins (Don’t forget to modify it according to your credentials and endpoints.) Spinnaker Pipeline The pipeline configuration can be found . here 4. Once you add it, the pipeline will look something like this: Continuous Delivery pipeline, triggered by Jenkins, deploying to Staging and Prod. Deep dive into the Pipeline: 1. : This is the stage where you mention the Jenkins endpoint, the job name and expected artifact from the job. In our case trigger. properties. Configuration 2. : The trigger.properties file has an action variable based on which we decide wether we need to trigger a new deployment for the new image tag to patch an existing deployment. The properties file (set in the TAG variable) also tells which version to deploy the patch with. Deploy (Manifest) 3. : Similar to the Deploy stage this stage will check the same variable and if it evaluates to , then the current deployment will be patched. It should be noted that in both these stages, the Kubernetes cluster is being used as a staging cluster. Therefore, our deployments/patches for staging environment will be automatic. Expression validation for Deploy Stage Patch (Manifest) “PATCH” Note: k8-staging-1 under Account Setting : This is a very important stage. It is here where you decide whether or not you want to . This should be approved only when the staging build has been thoroughly tested by various stake holders. 4. Manual Judgement promote the currently running build in the staging cluster to the production cluster 5. : The final stages in both paths are similar to their counterparts in pre-approval stages. The only difference being that the cluster under is a production Kubernetes cluster. Deploy(Manifest) and Patch(Manifest) Account Now you are ready to push out releases for your app. Once triggered, the pipeline will look like this: Pipeline triggered and automated deploy to Staging Cluster Automated Deployment to Staging Environment stage gating the deploy to Production cluster Manual Judgement Once is approved, Pipeline proceeds and deploys to Production cluster Manual Judgement Approval Given to Manual Judgement Section Successful deploy to Production cluster Deployment to Production Successful The sections in have been skipped as the . Once you deploy, you can view the current version and also the previous versions under the Section. Grey ACTION variable did not evaluate “PATCH” Infrastructure Current and Previous Versions of App in the Staging and Production cluster Key Takeaways: Emit as much data as you like to the properties file, and later consume it in a Spinnaker Pipeline. Trigger other Jenkins jobs from Spinnaker and then consume it’s artifacts in the subsequent stages. Spinnaker is quite a powerful tool which helps you perform all kinds of actions like roll-backs, scaling etc., right from the console. Not only deployments but all kinds of Kubernetes resources can be managed using Spinnaker. Spinnaker also provides excellent integration with Slack/Hipchat/Email for pipeline notifications. Conclusion We hope this helps you get started Continuous Delivery to your Kubernetes clusters. Spinnaker is an awesome tool which can deploy easily to multiple environments providing you with a single-pane-of-glass view for all your workloads across different clusters. Feel free to reach out in case you have any questions around the set-up. This article was originally published on https://appfleet.com/blog/continuous-delivery-pipeline-for-kubernetes-using-spinnaker/ .