Overview One of the first things I usually do after spinning up a GKE cluster is to secure HTTP traffic to backends by setting up an API Gateway called . In addition to security, we can use KrakenD to route traffic to different namespaces since GKE’s Ingress doesn’t allow for routing across namespaces at this stage. KrakenD We are also going to see one of the ways of using a single external IPv4 address to handle traffic from multiple domains to a GKE Ingress. Non-existent domains of and will be used throughout this guide. some.domain.dev some.domain.io Please replace these with real domains you own. Before You Begin: Access to GCP. You own a few domains that you can use. Your local machine has command line tool installed. glcoud You are already familiar with KrakenD and it’s configuration file. You have some experience with Kubernetes. Step 1 — Create an external IP address (~ 1 min) Run the following command to create a global external IP address. gcloud compute addresses create my-global-address --global Retrieve the IPv4 address assigned using the following command. gcloud compute addresses describe my-global-address --global Step 2 — Setup domain (~ 5 mins + wait for DNS propagation) Setup and records with you domain provider. A CNAME Example: @               A           1h      <IP_ADDRESS_FROM_PREVIOUS_STEP>
some            CNAME       1h      domain.dev
-------------------------------------------------------------------- @               A           1h      <IP_ADDRESS_FROM_PREVIOUS_STEP>
some            CNAME       1h      domain.io # some.domain.dev # some.domain.io Now we need to wait for the changes to propagate. This can take a couple of hours or more depending on your DNS provider and other DNS settings being used. You can verify propagation using the command. dig dig some.domain.dev
dig some.domain.io Proceed to the next step only if you see the domain resolving to the IP address created in Step 1. Step 3 — Setup a GCP Project and spin up a GKE cluster (~ 5 mins) This is a relatively straightforward step and can be achieved either via the GCP user interface or the command line tool. glcoud After the project and cluster have been setup, make sure you run to connect your local machine to your project. gcloud init Take note of your and the GCP PROJECT ID GKE cluster name Step 4 — Install a service mesh called LinkerD (~5 mins) is a light weight, easy to use and easy to install service mesh. A service mesh has many benefits but the features we are interested in for this exercise are: LinkerD Traffic shifting: during a rolling update of containers we need to move traffic away from containers that are terminating to new containers, thus resulting in zero downtime between deploys. Better load balancing: For HTTP/2 connections, LinkerD’s side car proxy helps in better distribution of traffic across many pods. You can read more about this . here Follow the to install LinkerD in your GKE cluster. LinkerD setup guide Step 5 — Setup a k8s Namespace + LimitRange (~ 3 mins) Create a file called with the following contents: gke-ingress/namespace.yaml apiVersion: v1 kind: Namespace metadata: name: gke-ingress Organising your work in is good practice. Namespaces Create a file called with the following contents: gke-ingress/limitrange.yaml apiVersion: v1 kind: LimitRange metadata: name: gke-ingress-limitrange spec: limits: - max: cpu: "2000m" memory: "1Gi" min: cpu: "10m" memory: "10Mi" default: cpu: "500m" memory: "256Mi" defaultRequest: cpu: "100m" memory: "128Mi" type: Container Briefly, a LimitRange defines how much cpu and memory is assigned to a container by default. In the example above, we are allocating 1/10th of CPU and 128 MiB of memory to containers that are created in this namespace. You can learn more about . LimitRanges here Then run the following commands: gcloud container clusters get-credentials <YOUR_GKE_CLUSTER_NAME> --project <YOUR_GCP_PROJECT_ID>

cd gke-ingress

kubectl apply -f namespace.yaml

kubectl apply -f limitrange.yaml -n gke-ingress Step 6 — Create a KrakenD configuration for your domains (~5 mins) Create a file called with the following contents: gke-ingress/krakend-some-domain-dev/krakend.json { : , : , : { : { : , : , : , : }, : { : , : }
    }, : , : , : , : [
        { : , : [
                { : , : [ ]
                }
            ]
        }
    ], : } "version" 2 "name" "some.domain.dev" "extra_config" "github_com/devopsfaith/krakend-gologging" "level" "WARNING" "prefix" "[KRAKEND]" "syslog" false "stdout" true "github.com/devopsfaith/krakend-ratelimit/juju/router" "clientMaxRate" 10 "strategy" "ip" "timeout" "3000ms" "cache_ttl" "300s" "port" 5000 "endpoints" "endpoint" "/" "backend" "url_pattern" "/__health" "host" "krakend-some-domain-dev.gke-ingress:5000" "output_encoding" "json" The above config is for and will have KrakenD running on port . We are also creating a route to KrakenD’s health check. This is needed for the Ingress health check to pass. More on this in Step 10. some.domain.dev 5000 Create another file called with the following contents: gke-ingress/krakend-some-domain-io/krakend.json { : , : , : { : { : , : , : , : }, : { : , : }
    }, : , : , : , : [
        { : , : [
                { : , : [ ]
                }
            ]
        }
    ], : } "version" 2 "name" "some.domain.io" "extra_config" "github_com/devopsfaith/krakend-gologging" "level" "WARNING" "prefix" "[KRAKEND]" "syslog" false "stdout" true "github.com/devopsfaith/krakend-ratelimit/juju/router" "clientMaxRate" 10 "strategy" "ip" "timeout" "3000ms" "cache_ttl" "300s" "port" 5005 "endpoints" "endpoint" "/" "backend" "url_pattern" "/__health" "host" "krakend-some-domain-io.gke-ingress:5005" "output_encoding" "json" The above config is for and will have KrakenD running on port . We are also creating a route to KrakenD’s health check. This is needed for the Ingress health check to pass. More on this in Step 10. some.domain.io 5005 Step 7 — Build a KrakenD container for your domains (~5 mins) Create 2 files: gke-ingress/krakend-some-domain-dev/Dockerfile gke-ingress/krakend-some-domain-io/Dockerfile with the following contents: devosfaith/krakend FROM COPY krakend.json /etc/krakend/krakend.json Then build containers and push to Google container registry (replace below with a real project id). <YOUR-GCP-PROJECT-ID> gcloud auth configure-docker gcr.io

cd gke-ingress/krakend-some-domain-dev

docker build -f Dockerfile -t gcr.io/<YOUR-GCP-PROJECT-ID>/gke-ingress-krakend-some-domain-dev:v1 .

docker push gcr.io/<YOUR-GCP-PROJECT-ID>/gke-ingress-krakend-some-domain-dev:v1

cd ../krakend-some-domain-io

docker build -f Dockerfile -t gcr.io/<YOUR-GCP-PROJECT-ID>/gke-ingress-krakend-some-domain-io:v1 .

docker push gcr.io/<YOUR-GCP-PROJECT-ID>/gke-ingress-krakend-some-domain-io Step 8 — Deploy your KrakenD containers (~5 mins) Create a filed called with the following contents (replace below with a real project id): gke-ingress/krakend-some-domain-dev/k8s.yaml <YOUR-GCP-PROJECT-ID> apiVersion: apps/v kind: Deployment
metadata:
  name: krakend-some-domain-dev
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: maxUnavailable: selector:
    matchLabels:
      app: krakend-some-domain-dev
  replicas: template:
    metadata:
      annotations:
        linkerd.io/inject: enabled
      labels:
        app: krakend-some-domain-dev
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - krakend-some-domain-dev
                topologyKey: terminationGracePeriodSeconds: containers:
        - name: krakend-some-domain-dev
          image: gcr.io/<YOUR-GCP-PROJECT-ID>/gke-ingress-krakend-some-domain-dev:v ports:
            - containerPort: imagePullPolicy: IfNotPresent
          command: [ ]
          args:
            [ , , , , , ,
            ]
          env:
            - name: KRAKEND_PORT
              value: readinessProbe:
            httpGet:
              path: /__health
              port: initialDelaySeconds: periodSeconds: livenessProbe:
            httpGet:
              path: /__health
              port: initialDelaySeconds: periodSeconds: ---
apiVersion: v kind: Service
metadata:
  name: krakend-some-domain-dev
spec:
  type: NodePort
  ports:
    - name: http
      port: targetPort: protocol: TCP
  selector:
    app: krakend-some-domain-dev
---
apiVersion: autoscaling/v beta kind: HorizontalPodAutoscaler
metadata:
  name: krakend-some-domain-dev
  namespace: gke-ingress
spec:
  scaleTargetRef:
    apiVersion: apps/v kind: Deployment
    name: krakend-some-domain-dev
  minReplicas: maxReplicas: metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 1 1 0 2 100 "kubernetes.io/hostname" 60 1 5000 "/usr/bin/krakend" "run" "-d" "-c" "/etc/krakend/krakend.json" "-p" "5000" "5000" 5005 5 5000 5000 15 20 1 5000 5000 2 2 1 2 4 80 80 The above k8s config is for domain and setups the following: some.domain.dev A with 2 pods that are meshed by LinkerD and has KrakenD running on port . deployment 5000 A to accept traffic on port . service 5000 A horizontal pod autoscaler (HPA) that automatically scales the deployment as traffic changes. Deploy as follows: cd gke-ingress

kubectl apply -f krakend-some-domain-dev/k8s.yaml -n gke-ingress Check if the pods are running: kubectl get pods -n gke-ingress Create a file called with the following contents (replace below with a real project id): gke-ingress/krakend-some-domain-io/k8s.yaml <YOUR-GCP-PROJECT-ID> apiVersion: apps/v kind: Deployment
metadata:
  name: krakend-some-domain-io
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: maxUnavailable: selector:
    matchLabels:
      app: krakend-some-domain-io
  replicas: template:
    metadata:
      annotations:
        linkerd.io/inject: enabled
      labels:
        app: krakend-some-domain-io
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - krakend-some-domain-io
                topologyKey: terminationGracePeriodSeconds: containers:
        - name: krakend-some-domain-io
          image: gcr.io/<YOUR-GCP-PROJECT-ID>/gke-ingress-krakend-some-domain-io:v ports:
            - containerPort: imagePullPolicy: IfNotPresent
          command: [ ]
          args:
            [ , , , , , ,
            ]
          env:
            - name: KRAKEND_PORT
              value: readinessProbe:
            httpGet:
              path: /__health
              port: initialDelaySeconds: periodSeconds: livenessProbe:
            httpGet:
              path: /__health
              port: initialDelaySeconds: periodSeconds: ---
apiVersion: v kind: Service
metadata:
  name: krakend-some-domain-io
spec:
  type: NodePort
  ports:
    - name: http
      port: targetPort: protocol: TCP
  selector:
    app: krakend-some-domain-io
---
apiVersion: autoscaling/v beta kind: HorizontalPodAutoscaler
metadata:
  name: krakend-some-domain-io
  namespace: gke-ingress
spec:
  scaleTargetRef:
    apiVersion: apps/v kind: Deployment
    name: krakend-some-domain-io
  minReplicas: maxReplicas: metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 1 1 0 2 100 "kubernetes.io/hostname" 60 1 5005 "/usr/bin/krakend" "run" "-d" "-c" "/etc/krakend/krakend.json" "-p" "5005" "5005" 5005 5 10 5005 15 20 1 5005 5005 2 2 1 2 4 80 80 The above k8s config is for domain and setups the following: some.domain.io A with 2 pods that are meshed by LinkerD and has KrakenD running on port . deployment 5005 A to accept traffic on port . service 5005 A horizontal pod autoscaler (HPA) that automatically scales the deployment as traffic changes. Deploy as follows: cd gke-ingress

kubectl apply -f krakend-some-domain-io/k8s.yaml -n gke-ingress Check if the pods are running: kubectl get pods -n gke-ingress Proceed to the next step only if all pods are in the running state. Step 9 — Create HTTPS certificates for your domains (~3 mins) Create a file called with the following contents: gke-ingress/some-domain-dev-cert.yaml apiVersion: networking.gke.io/v1 kind: ManagedCertificate metadata: name: some-domain-dev-cert spec: domains: - some.domain.dev Create another file called with the following contents: gke-ingress/some-domain-io-cert.yaml apiVersion: networking.gke.io/v1 kind: ManagedCertificate metadata: name: some-domain-io-cert spec: domains: - some.domain.io Then apply as follows: cd gke-ingress

kubectl apply -f some-domain-dev-cert.yaml -n gke-ingress

kubectl apply -f some-domain-dev-io.yaml -n gke-ingress Move to Step 10 as fast as possible. Step 10 — Create Ingress (~ 5 mins + ~30+ mins of waiting) Before we create the Ingress, all the previous steps must have completed successfully with no errors. Steps 2 and 8 are critical for the Ingress creation. Create a file called with the following contents: gke-ingress/ingress.yaml apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: krakend-ingress annotations: kubernetes.io/ingress.global-static-ip-name: my-global-address networking.gke.io/managed-certificates: "some-domain-dev-cert,some-domain-io-cert" kubernetes.io/ingress.allow-http: "false" spec: rules: - host: some.domain.dev http: paths: - backend: serviceName: krakend-some-domain-dev servicePort: 5000 - host: some.domain.io http: paths: - backend: serviceName: krakend-some-domain-io servicePort: 5005 Then apply as follows: cd gke-ingress

kubectl apply -f ingress.yaml -n gke-ingress If all goes well, in about 30 mins you will be able to access your domains using a web browser at some.domain.dev and some.domain.io and you should see the message from KrakenD’s health check: { : } "status" "ok" : Sometimes certificates can take longer than 30 mins to be issued and your web browser may display a certificate error. Use the following command to track status of your certificates. Note kubectl get managedcertificates -n gke-ingress There is a reason for doing things in this order. For an Ingress to begin routing traffic, each backend in the file needs to respond with a . However the ingress cannot be setup before the objects defined in the files are created. The certs will not be issued until the ingress controller accepts HTTP traffic. ingress.yaml HTTP 200 ManagedCertificate *cert.yaml So how can we overcome this circular dependency? Well, managed certificates are not issued instantly after the command. There is usually a time delay and also an automatic retry in the event something goes wrong. Same goes for the ingress creation. There is usually a time delay between the command and the ingress actually being created. kubectl apply kubectl apply From what I have noticed, the ingress is usually up in less than 10 mins and the certificate process takes about 20+ mins. In a happy path, this is what happens: Ingress is up in under 10 mins and looks for the HTTP 200 response from the backend. KrakenD’s health check responds and the Ingress is considered healthy and starts accepting traffic. Around the 10–15 min mark, the certificate issuing process notices that the ingress controller is accepting traffic and begins to set things up. In the event you run into an error with cert management and ingress creation, run the following commands: kubectl delete -f gke-ingress/ingress.yaml -n gke-ingress

kubectl delete -f gke-ingress/some-domain-dev-cert.yaml -n gke-ingress

kubectl delete -f gke-ingress/some-domain-dev-io.yaml -n gke-ingress kubectl apply -f gke-ingress/some-domain-dev-cert.yaml -n gke-ingress

kubectl apply -f gke-ingress/some-domain-dev-io.yaml -n gke-ingress

kubectl apply -f gke-ingress/ingress.yaml -n gke-ingress # Use these commands only you run into an issue if # Wait about 1 min Step 11 — Create a sample backed in another namespace (~ 5 mins) First create a namespace: kubectl create ns hello Let’s deploy a simple, secure and light weight Golang HTTP server that prints . Hello! Create a file called with the following contents: hello/hello.yaml apiVersion: apps/v1 kind: Deployment metadata: name: hello spec: replicas: 1 strategy: type: RollingUpdate rollingUpdate: maxSurge: 1 maxUnavailable: 0 selector: matchLabels: app: hello template: metadata: annotations: linkerd.io/inject: enabled labels: app: hello spec: terminationGracePeriodSeconds: 60 containers: - name: hello command: ["/golang-hello-server"] image: registry.gitlab.com/ownageoss/golang-hello-server/golang-hello-server:latest imagePullPolicy: Always ports: - containerPort: 5050 protocol: TCP env: - name: HELLO_SERVER_PORT value: "5050" readinessProbe: httpGet: path: /health port: 5050 initialDelaySeconds: 5 periodSeconds: 10 livenessProbe: httpGet: path: /health port: 5050 initialDelaySeconds: 15 periodSeconds: 20 resources: requests: memory: "10Mi" cpu: "10m" limits: memory: "16Mi" cpu: "20m" --- apiVersion: v1 kind: Service metadata: name: hello spec: type: NodePort selector: app: hello ports: - name: http protocol: TCP port: 5050 targetPort: 5050 Apply as follows: kubectl apply -f hello/hello.yaml -n hello Step 12 — Update the krakend.json file to create a route to the new backend (~3 mins) Modify the following files created in Step 6: gke-ingress/krakend-some-domain-dev/krakend.json gke-ingress/krakend-some-domain-io/krakend.json Add the following JSON to the array: endpoints { : , : [
        { : , : [ ]
        }
    ]
} "endpoint" "/hello" "backend" "url_pattern" "/" "host" "hello.hello:5050" To route traffic to another namespace, we simply use <servicename>.<namespace>:<serviceport> syntax in the host value. Step 13 — Build and deploy KrakenD with the new config (~5 mins) Repeat and replacing the image value of with Step 7 Step 8 v1 v2 After the deployment visit and and you will be greeted with the following message: some.domain.dev/hello some.domain.io/hello Hello! Tear Down (~ 5 mins) Make sure you release resources to save costs. kubectl delete ns hello

kubectl delete ns gke-ingress

gcloud compute addresses delete my-global-address --global Finally, delete the GKE cluster. Summary We have secured HTTP traffic to backends using an API gateway called KrakenD. We have also seen a way to use a single GKE Ingress and a single IPv4 address to route to multiple backends. There is a limit to the number of backends that can be used with a single ingress. You can read more about it . Note : here We have generated HTTPS certs for our domains. We have routed traffic across Kubernetes namespaces. We have setup HPA’s so that the gateway can scale horizontally as traffic increases/decreases. Also published here .

Too Long; Didn't Read

Making security fun one episode at a time

How to Successfully Set Up KrakenD on GKE: A Step by Step Guide

Too Long; Didn't Read

Vijay Savanth

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

How to Successfully Set Up KrakenD on GKE: A Step by Step Guide

Too Long; Didn't Read

Vijay Savanth

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

RELATED STORIES