Helm V2 CRD Management

With the last feature release of Helm V2 just published, and Helm V3 on the horizon, we’re still stuck with managing CRDs. CRD management in helm is, to be nice about it, utterly horrible.

The Problem with helm

When you have a chart that both installs a CRD and installs a resource that requires that CRD, there’s a race condition where the resource installation can fail because the CRD is not yet recognized as defined by the API server. To work around this, helm added a hook.

apiVersion: ...
kind: ....
metadata:
  annotations:
    "helm.sh/hook": "pre-install"
...

This hook can only be used at install, not upgrade, not delete, and will still fail if the CRD is already defined.

Why this is a problem

If you want to reinstall a chart and helm delete --purge the release, the CRD is not uninstalled. This is a good thing. Uninstalling CRDs can result in data loss and since the helm tool doesn’t know the value of your data, doing the safe thing is smart. The problem comes when you try to helm install again. Because the CRD is defined, the installation fails. The expectation by the helm community is that this is a human interface problem and the human can then choose to run the command with the --no-crd-hook flag.

The expectation that the helm CLI will always be run by a human is flawed. At enterprise scale, almost nothing is done by humans. There are scripts, controllers, configuration management tools that are all doing this work and making runtime choices based on the failure of a command can be complex and should be avoided. That’s why declarative systems (Ansible, Kubernetes, etc) exist. Declare the desired state and the tools should take the actions necessary to produce consistency.

The workaround

Wrap the CRD in commands to only install it if the API server doesn’t have the capability provided by that CRD.

templates/mycrd.yaml

{{- if not (.Capabilities.APIVersions.Has "mygroup.dom/v1beta1/MyResource") }}
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: myresource.mygroup.dom
  annotations:
    "helm.sh/hook": crd-install
...

Now if the API server already has this capability, the template will render an empty file and the error will not occur.

Unfortunately, this means that helm will never manage that CRD ever again. If you add an optional field, or change the additionalPrinterColumns, they will never be updated.

Just use a Job to kubectl apply the CRD

Unfortunately, this won’t work. Yes, you can make the job run before the resource creation that needs the CRD by using pre-install/pre-upgrade hooks, but helm validates the resources that are going to be installed to ensure that the API server can accept all the resources. Since the capability does not exist in any way that helm can see it, it fails validation and won’t install. The crd-install hook solves this by allowing it to run before validation occurs¹.

How to update CRDs that are no longer managed by helm

To solve the CRD abandonment problem, we’ve added a ConfigMap and Job to our templates in addition to the CRD template with the crd-install hook and capability filtering. Admittedly, this is a very hacky approach but the community has been very resistant to solving this problem in helm v2 with all their focus being on helm v3.

To do this in a maintainable way, we use the .Files template command and keep the CRD definition, itself, in a files directory.

Examples

mychart/
  files/
    mycrd.yaml
  templates/
    configmap.yaml
    crd.yaml
    job.yaml
    myresource.yaml

files/mycrd.yaml

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: myresource.mygroup.dom
  annotations:
    "helm.sh/hook": crd-install
...

templates/configmap.yaml

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: myresource-crd
  annotations:
    "helm.sh/hook": pre-install, pre-upgrade
    "helm.sh/hook-weight": "-5"
  data:
    crd.yaml |
      {{.Files.Get "files/mycrd.yaml" | indent 6}}

templates/crd.yaml

{{- if not (.Capabilities.APIVersions.Has "mygroup.dom/v1beta1/MyResource") }}
{{.Files.Get "files/mycrd.yaml"}}
{{- endif }}

templates/job.yaml

apiVersion: apps/v1
kind: Job
metadata:
  name: myresource-crd-apply
  annotations:
    "helm.sh/hook": pre-install, pre-upgrade
    "helm.sh/hook-weight": "-4"
...

The rest of the Job is just a container that runs kubectl apply on the mounted ConfigMap.