The Art of the Helm Chart: Patterns from the Official Kubernetes Charts

Helm charts package up applications for installation on Kubernetes Clusters. Installing a helm chart is a bit like running an install wizard. So helm chart developers face some of the same challenges faced by developers producing installers:

What assumptions can be made about the environment that the install is running into?
Is the application meant to be able to interact with other applications?
What configurations need to be made available to the user and how should they be offered to the user?

But these questions come with helm-specific twists. To see why, let’s start with a picture of what happens when a user runs a`helm install`. Then we can move on to see how some of the official Kubernetes charts deal with these questions.

A Picture of a Helm Install

I want to install mysql in my cluster. But I don’t want the mysql version that the stable/mysql chart sets in its values.yaml file in the official charts repo. So I create my own values.yaml file called mysql-values.yaml with just one line:

imageTag: “5.7.10”

And then I run helm install stable/mysql --values=mysqlvalues.yaml. Helm generates a unique release name for me (`ignorant-camel`) and mysql is deployed in my cluster. The output of kubectl describe pod ignorant-camel-mysql-5dc6b947b-lf6p8 tells me that my chosen imageTag has been applied.

Actually I didn’t need to install anything in my cluster to see that my selection for imageTag would be applied. I could have run helm install stabe/mysql --values=mysqlvalues.yaml --dry-run --debug and helm would have simply shown me the content of kubernetes deployment descriptor yaml it generated without installing anything.

The process of getting to the generated kubernetes deployment descriptors can be better understood by thinking of the structure of a helm chart:

├── Chart.yaml├── README.md├── templates│   ├── NOTES.txt│   ├── _helpers.tpl│   ├── deployment.yaml│   ├── secrets.yaml│   └── ...more yaml...└── values.yaml

When the user runs helm install stable/mysql then the entries from the values.yaml in the chart and the helm release information (such as unique release name) get injected into the templated yaml resource descriptors as the template is evaluated and rendered into pure Kubernetes deployment descriptors. When the user runs helm install stable/mysql with parameters or a values file, then the parameters or values file will be overlaid on the one in the chart. Effectively the in-chart values file sets defaults that can be overridden.

So the values.yaml is the main interface between us as chart developers and our users. What we expose in the values.yaml determines what users can and can’t do with our charts.

Our charts and our values.yaml files also need to satisfy another kind of scenario. Another chart developer might like our app and decide they want to include it inside a package they want to release. So they add our chart to the requirements.yaml of a new chart they create, along with a bunch of other charts. They override some of our defaults by setting their values.yaml. And then they distribute it to users who do helm install with that chart.

So our chart goes into other charts and they feed onto users in a charts food chain. Seeing how to satisfy our consumers in this food chain is one of the key challenges of writing helm charts.

The Challenges of Writing Helm Charts

Applications often have lots of configuration options and Kubernetes Clusters can be configured in lots of different ways. So when writing a helm chart we naturally face questions like:

What if I forget to expose a configuration parameter in the values.yaml? What if my application parses configuration dynamically so that I can’t say upfront what all the possible parameter names will be?
How can I allow users to use resources defined not directly in my chart but in a chart in which my chart is used as a subchart?
What if another chart developer is using my chart in their own and they need to add significant sections inside a resource that I defined (e.g. add an extra container to a pod or a custom initialization script)?
How can I allow my users to expose the app externally in the various ways that they might want?

We can learn a lot on how to deal with these kinds of problems from the charts in the official helm charts repository. Let’s take a look at some of the patterns that those charts use, so that we can understand how to use the same patterns in our own charts.

Before diving in though it’s worth keeping in mind that the public charts are rather advanced. If you’re worrying about some of the above questions when developing your first helm charts, try to park the more difficult worries to begin with. Start by familiarising yourself with the helm documentation and try to build a chart that works for just the simplest cases first. You can then add advanced options later.

So let’s try to get a snapshot of patterns from the official charts. This won’t be comprehensive and I expect new options to become available and new patterns to emerge when helm 3 adds scripting with Lua. But it will give us look at the current state of the art.

1 Patterns for Exposing Configuration Parameters

Let’s say we’ve defined a Deployment resource in our template and we’ve configured the env section to allow some environment variables to be set from the values.yaml:

- name**:** ENV_VAR1value: {{ .Values.var1 }}- name**:** ENV_VAR2value: {{ .Values.var2 }}

Now users of our chart can override these values in their values.yaml or with —set var1=foo . But what will our users be able to do if we’ve overlooked one? Or even worse, what if our application can dynamically parse configuration options (e.g. it might read ENV_VAR1 and create an internal variable called var1 )? Then there isn’t even a finite set of configuration option names to expose. So how do we let users set the variable names as well as the values?

As the helm charts developer guide says, we could create a configmap with a range function. A nice example of this is in the stable/unbound chart. It contains a configmap that defines its unbound.conf file. It mounts this file into Pods created by its Deployment. Inside the configmap it has entries such as:

{{- range .Values.localRecords }}local-data: "{{ .name }} A {{ .ip }}"local-data-ptr: "{{ .ip }} {{ .name }}"{{- end }}

And its values.yaml lets the entries in localRecords be set as a list e.g.:

localRecords:- name: "fake3.host.net"ip: "10.12.10.10"- name: "fake4.host.net"ip: "10.13.10.10"

The sonarqube chart applies a similar approach directly to environment variables, defining some explicitly and allowing further variables to be set using an extraEnv collection:

{{- range $key, $value := .Values.extraEnv }}— name: {{ $key }}value: {{ $value }}{{- end }}

So that variables would be set in a values.yaml like:

extraEnv:ENV_VAR1: var1ENV_VAR2: var2

Many of the official charts define an extraEnv but in slightly different ways. The buildkite chart defines it differently, without using range. Instead it takes the values.yaml section and simply puts it in the template:

So this means that instead of setting extraEnv entries in the values.yaml as simple pairs we would also need to name each of the keys (name) and values (value) in the pairs like:

extraEnv:— name: ENV_VAR1value: "var1"— name: ENV_VAR2value: "var2"

The keycloak chart does this differently again:

So it treats extraEnv as a string, sends it through the tpl function so that it is evaluated as part of the template and ensures that the correct indenting is applied. This is interesting as it allows values to be set like:

extraEnv: |— name: KEYCLOAK_LOGLEVELvalue: DEBUG— name: HOSTNAMEvalue: {{ .Release.Name }}-keycloak

Normally it wouldn’t be possible to use a template directives like {{ .Release.Name }} inside the values.yaml but here we can because the content will go through tpl. This could be a big advantage in cases where we’re including the chart inside a parent chart that we’re developing and we need to refer to a service that is part of the same parent chart. (More on this in the next section.) The disadvantages are that it’s a bit more abstract and that the content in the values.yaml is treated as a string there so formatting issues aren’t necessarily found by your editor.

2 Referencing Mutually-deployed Resources

Typically resources deployed through helm are prefixed with a release name, so that multiple releases can be installed from the same chart (and in the same namespace) without naming conflicts between the installed resources. This means that template directives need to be used to prefix the resource names when one resource with a chart needs to refer to another, or to resources within the same parent chart.

A common case where a mutually-deployed resource needs to be referred to is a database secret. The xray chart, for example, includes within it the option to deploy a postgres database. It sets up its indexer component’s Deployment with credentials by pointing to the postgres secret (which itself is part of the chart as a subchart and therefore also prefixed with the same release name):

{{- if .Values.postgresql.enabled }}— name: POSTGRES_USERvalue: {{ .Values.postgresql.postgresUser }}— name: POSTGRESS_PASSWORDvalueFrom:secretKeyRef:name: {{ .Release.Name }}-postgresqlkey: postgres-password— name: POSTGRESS_DBvalue: {{ .Values.postgresql.postgresDatabase }}{{- else }}...

Here the xray chart knows which database it needs to connect to since it is itself including the postgres chart. But what if we were developing a chart for an application where the user can choose to supply a database? Then it might be necessary to allow the user to include our chart inside a parent chart alongside a database of their choosing. How would we allow the user to set database configuration then?

One option would be the extraEnv approach we saw already with the keycloak chart. Then the user could then set our chart’s extraEnv to point to postgres even if we’ve not defined it explicitly in our original chart. The values.yaml would then have an entry like:

extraEnv: |- name: POSTGRES_USERvalue: {{ .Values.postgresql.postgresUser }}— name: POSTGRESS_PASSWORDvalueFrom:secretKeyRef:name: {{ .Release.Name }}-postgresqlkey: postgres-password— name: POSTGRESS_DBvalue: {{ .Values.postgresql.postgresDatabase }}

The | is there because the section is treated as a string to be passed explicitly to the tpl function.

A similar question emerges if a release name prefix needs to be used within an entry in a file that is loaded into a configmap. A typical way of loading a whole file into a configmap is to use .Files.Get. However, like the values.yaml, the contents of files loaded into this way are then not part of the template and so template directives can’t be used in them. What does allow template directives to be used inside a file is to load the file content with .Files.Get and pass that into [tpl](https://stackoverflow.com/questions/47595295/how-do-i-load-multiple-templated-config-files-into-a-helm-chart/52009992#52009992). Then the data section of the configmap would have entries like:

conf_file1: {{ tpl (.Files.Get "files/conf_file1") . | quote }}

Or to load into a Secret we need to encode the content in base64:

conf_file1: {{ tpl (.Files.Get "files/conf_file1") . | b64enc | quote }}

Instead of just one file can with load a set of files into a ConfigMap using .Files.Glob :

Though we can’t quite apply the same approach with AsSecret as then the content would be encoded before going through tpl . To load a set of files into a secret we could use [Glob](https://github.com/helm/helm/blob/master/docs/chart_template_guide/accessing_files.md#glob-patterns) to find the files and [Get](https://github.com/helm/helm/blob/master/docs/chart_template_guide/accessing_files.md#glob-patterns) to load them:

{{ range $path, $bytes := .Files.Glob "files/*" }}{{ base $path }}: '{{ tpl ($root.Files.Get $path) . | b64enc }}'{{ end }}

3 Empowering Our Fellow Chart Developers

The use of extraEnv in the keycloak chart suggests a possible pattern for dealing with other kinds of definitions that might need to be injected into a chart. For example, the keycloak chart needs to allow a user to package the keycloak chart inside a parent chart that also supplies a user-defined a json file that the user needs to be able to mount into keycloak Pods. The chart supports this by exposing an extraVolumes:

And extraVoumeMounts:

      volumeMounts:  
        - name: scripts  
          mountPath: /scripts

The user can then point to their secret (which contains the json file) and mount it through the values.yaml:

extraVolumes: |— name: custom-secretsecret:secretName: custom-secretextraVolumeMounts: |

name: custom-secretmountPath: "/realm/"readOnly: true

Effectively the configuration of volumes and volumeMounts is externalized to the values.yaml. The chart also applies this pattern to allow users to inject further resources into the template such as initContainers and even adding entire additional containers (or ‘sidecars’). This pattern gives chart users a lot of power and flexibility, especially for other chart developers using our chart.

The pattern can be especially powerful when combined with the keycloak chart’s exposing of other parameters, such as a preStartScript variable that is used within the chart’s initialization script:

{{- with .Values.keycloak.preStartScript }} echo 'Running custom pre-start script...'{{ . | indent 4 }}{{- end }}

The user can inject whatever shell script content they like into .Values.keycloak.preStartScript in their values.yaml. So this approach makes it possible for users to not just set their own configuration paramters but also load their own files and run their own custom scripts using those files.

4 Exposing Externally in Different Ways

The starter helm chart generated by helm create includes a Service specification but not an Ingress. Many of the public charts do define an Ingress resource. This needs to be configurable as users might not want to use ingress. So the rabbitmq chart, like many others, wraps its whole Ingress resource definition with:

{{- if .Values.ingress.enabled }}...{{-end}

This is in line with the review guidelines for the official charts repo, which recommends that ingress should be disabled by default.

The rabbitmq chart, for example, needs to provide the option whether to expose it by setting a host-based ingress rule:

rules:{{- if .Values.ingress.hostName }}

host: {{ .Values.ingress.hostName }}http:{{- else }}
http:{{- end }}

(Actually the user might want multiple hosts to route to the service so the review guidelines suggest using a range function for hosts. The rabbitmq chart just offers a single host.)

The rabbitmq chart also allows for the host to be not set (the else condition above). In that case it’s likely that the user will instead override the path (so that rabbitmq is exposed on a unique route, distinct from other exposed services):

- path: {{ default "/" .path }}backend:serviceName: {{ template "rabbitmq.fullname" . }}servicePort: {{ .Values.rabbitmq.managerPort }}

It’s also especially important that the user has flexibility to set the annotations on an ingress resource, since these are used to control routing configuration options. The review guidelines suggest supporting this with toYaml:

{{- with .Values.ingress.annotations }}annotations:{{ toYaml . | indent 4 }}{{- end }}

This allows the user of the chart to set annotations in the values.yaml such as:

annotations:kubernetes.io/ingress.class: nginxnginx.ingress.kubernetes.io/rewrite-target: /

So the user can set whatever annotations they need to on the ingress through the values.yaml without the chart needing to be aware of what those annotations might be. Just exposing these annotations can amount to giving the power to apply script-like configuration on the ingress routing. For example, with an nginx ingress a user can apply rules to set headers in custom ways through configuration snippets:

annotations:kubernetes.io/ingress.class: nginxnginx.ingress.kubernetes.io/rewrite-target: /nginx.ingress.kubernetes.io/configuration-snippet: |more_set_headers 'Access-Control-Allow-Origin: $http_origin';

What We’ve Learned About the Art of the Helm Chart

This guy is just chillin because his charts are so good

A good helm chart needs to anticipate what level of flexibility is needed by its users. More flexibility tends to mean either more complexity or greater abstraction or both. This can hamper chart readability and put more burden on chart users. The challenge is to choose the tools that best fit with what users need for that particular chart.

There are other concerns to balance too that we’ve not touched on, such as testing and security. This has just been a look at a particular slice of the official charts. I’ve tried to focus on patterns that seem particularly helpful to me in making sure that users can do what they need to do with your charts. The official Kubernetes charts have been extremely helpful to me in my experience of working on the helm charts for the Activiti project. Hopefully the explanation in this post can help encourage others to dive into the official repo and take inspiration from its charts.