Deploying to DigtialOcean Kubernetes using Helm 3 and Gitlab.com

Some years ago, I wrote a post on how to deploy to DigitalOcean's version of Kubernetes (DOKS) using a Gitlab CI pipeline. The solution I used there had some notable drawbacks, as well as some security concerns that I grew unhappy with some time after writing the post.

Recently, I was building a helm-based pipeline to DOKS using Github Actions. That worked out surprisingly well, and I found the resulting solution rather elegant. That got me thinking, can I do the same with Gitlab?

Assuming you're going to be building your Gitlab pipeline to build and deploy a custom container, you're faced with a problem. Building the container is best done on CI systems without relying on the Docker daemon, as it requires root access. Like in the previous article, I decided to use Google's Kaniko to build OCI complaint container images which are capable of being run on Docker as well as Kubernetes. 

A typical Kaniko command not only builds the image, but also tags and pushes it to a remote repository.

Most container registries, like Gitlab Container Registry, require you to authenticate in order to push new images. Kaniko relies on the same authentication filename and format as produced by the docker login command. 

 /path/to/home/.docker/config.json

In the case of the Kaniko image we'll use in our Gitlab pipeline, the home directory is simply /kaniko. While we could write that config.json as a Gitlab Secret and simply pipe it to a file in an pipeline, it's often more flexible to only pipe out the values which make up the file, rather than the file itself. We can do that in a single command in our CI:

This is better, but it still comes with a security implication. The credentials are static, and don't time out after use. If those credentials are leaked, you would need to invalidate them manually in order to prevent abuse or intrusion. 

Instead, it's better to only allocate credentials for a short time. Fortunately, if we're using Gitlab's built in container registry, there are handy predefined variables which give us dynamically allocated and time-bound registry credentials. 

We can put all the above together to make the first stage in our pipeline. It'd look something like this in your gitlab-ci.yml:

Getting Kaniko to work on Gitlab CI requires a few esoteric configurations. 

1. It's essential we use the :debug tag of the Kaniko container image, as it has a shell necessary for it to run on Gitlab.
2. the entrypoint: [""] line is necessary to prevent Gitlab CI executing the image /bin/sh, which Kaniko's image does not possess (see https://forum.gitlab.com/t/how-to-override-entrypoint-in-gitlab-ci-yml/9429 for details).

Note that the only syntax here is an example. You will obviously have your own branch names, patterns, tags, or events on which you want run the container build

The next step in our pipeline is to authorize ourselves to access our DigitalOcean Kubernetes (DOKS) cluster. While this can be done with curl, it's much better to use DigitalOcean's own command line utility doctl. The doctl command let's us get a kubeconfig authorization file from DOKS if we provide it a DigitalOcean Personal Access Token. 

The command we want to use is:

We can pass that our token using the -t switch, but in a pipeline, we really should store that as a Gitlab Secret. Fortunately, the doctl command looks for the DIGITALOCEAN_ACCESS_TOKEN environment variable. Since Secrets are exposed as environment variables to commands in the pipeline, we can create a Gitlab Secret Variable with the name of DIGITALOCEAN_ACCESS_TOKEN and store our token to it without anything more.

Th downside of the above command is that the kubeconfig has a long expiration date. Much longer than we probably want for a CI run, so let's time bound that:

The --expiry-seconds switch let's you specify the number of seconds the Kubeconfig will be valid before expiring. We don't really expect our pipeline to take longer than 10 minutes, so we use 600 seconds here. Adjust this to suit your pipeline's needs.

Combining all of that, we get a gitlab-ci.yml stage like this:

To prevent us from having to create a custom container image to build our image, we're using the doctl image maintained by DigitalOcean themselves, which is hosted on Docker Hub:

docker.io/digitalocean/doctl:latest

You'll notice we also run the /app/doctl auth init command first, before attempting to get our kubeconfig. This is necessary for the doctl command to function correctly.

When complete, the build stage will write a time-limited kubeconfig file for our cluster to /root/.kube/config. 

The above all looks good, until we get to writing the stage where we actually run Helm. The problem is, the kubeconfig file was written to /root/.kube/config in the  docker.io/digitalocean/doctl:latest image. Our next build stage starts a new container image, and it won't have the kubeconfig file we worked so hard to get.

At this point, you could decide to build a custom container image which has both doctl and Helm. The Alpine Linux official repositories do have both of those commands, but we would need to either install those applications on each pipeline, or host a custom container image somewhere with those tools to keep build times low.

A different approach is to leverage Gitlab Job Artifacts to persist the kubeconfig for the duration of the build.

The above adds an artifacts section which specifies a path to a kubeconfig file to preserve from the build. You might wonder why this is .kubeconfig.yml instead of /root/.kube/config. This is actually a workaround for an important limitation of Gitlab Job Artifacts.

Artifacts cannot be saved if outside of the repository directory cloned inside the CI run. So, when we get our kubeconfig, we copy it from the default location, to one in our repository. In this case, we save it as a .kubeconfig.yml file in the root of our in-CI clone of the Gitlab code repository. 

Gitlab Job Artifacts are also permanent, so we need to expire them so that they don't persist (for long) after the build is complete. This is where the expire_in setting comes from. We tell Gitlab to expire the artifact after 600 seconds -- the same as our kubeconfig is valid for.

It's important to note that this expiration is ignored if the Keep artifacts from most recent successful jobs setting is enabled on the project. If you're not using artifacts for the project in any other way (container images aren't artifacts), then you can safely uncheck this value.

Even if you don't, the kubeconfig is time-bound anyways, and DigitalOcean will refuse to use it after the original 600 seconds are up. The auto-deletion is just a nice value-add.

Okay, now we can write the final section of our Gitlab pipeline, calling Helm. Like doctl, we'd rather not maintain a container image just to run helm. Fortunately, we don't. 

The Alpine Helm container is very small image which provides Helm, and all the dependencies necessary to run Helm. We can use that to power the final stage of our pipeline.

There's a lot going on here! The first thing is we use a needs to prevent this stage from executing if our 'Get Kubeconfig' stage fails. Next, we use the alpine/helm:latest image, so we're always pulling the latest version of Helm. At the time of this writing, we're in the Helm 3.x era, so there's no need to have Tiller or any cluster-side resources set up in order to use Helm.

With that done, the first thing we do is download the charts we need using the helm repo command:

In this case, we're using a Helm chart used to deploy a PHP application using the Flightdeck system of containers. This is again, an example. Substitute your chart repository as needed. We could also use a chart which is local inside our project repository if we so choose. If so, there's no need to download any external charts which aren't dependencies.

Next we run our Helm command:

The helm command will perform an upgrade of a release named my-release using a specific chart in the repository we just added. If this is the first time we ever ran the pipeline, that's no problem; we use --install to install the chart if it's not currently present. To give Helm some "CI-like" behavior, we also instruct it to block until deployment completes (--wait) and to automatically roll back changes if something goes wrong (--atomic). We pass it our .kubeconfig.yml file (--kubeconfig) and set a namespace.

We also pass it a explicitly named Values File with --values. While Helm does look for a values.yaml by default, we want to allow our application to mutate with different settings per environment. So, we create separate values files for each env and store them all in a directory in or repository. 

Eagle-eyed readers will notice that the Helm command in the gitlab-ci.yml I presented earlier isn't the same as the one above. There are two additional --set switches missing. What's the deal?

Like many helm charts, we can specify the image repository and tag to use in our values file. While they can be different from chart to chart, it's typical to find them as the first thing defined in the values.yaml:

While we could always rely on the latest tag, Kubernetes sometimes doesn't pull updated images when expected, if ever.

You can set the imagePullPolicy key on specific Kubernetes definitions to Always to encourage k8s to pull images, but this can result in additional network traffic each time a container restarts. Moreover, it doesn't entirely fix the problem as k8s may assume, "I already have latest" and never repull the image. 

A way to fix this is to always specify a different image tag when doing a deployment. Earlier, we used the Gitlab predefined variable $CI_COMMIT_SHORT_SHA when tagging our image in Kaniko. As the git commit ID -- even the short commit ID -- is reasonably unique and also easy to access in the pipeline, it become a handy way to tag our image. We always can look at the tag, and then go to that specific git commit in our project. 

But more importantly, we can use it to force Kubernetes' hand and pull a new image. Our full Helm command does this through two --set switches:

The two --set switches override the repository value under the image key, and the tag value respectively. In Helm, --set has precedence over --values, and --values has precedence over the values.yaml provided with the Helm chart to act as default values. This allows us to override the latest tag with the value of $CI_COMMIT_SHORT_SHA, a reasonably unique tag which forces k8s to pull the image.

Technically, we don't need to override the repository value, provided we set it in the values file we pass to --values. However, doing so makes it more consistent since we already know what the repository is. That is, the value of the $CI_REGISTRY_IMAGE predefined variable we used when building and pushing the image with Kaniko. 

Building and deploying a Kubernetes application from Gitlab.com to DOKS is much easier when employing newer tools such as doctl and Helm. In theory, applications could even be deployed manually by building the container image locally, pushing that to a repository, and then using doctl and Helm commands locally to deploy the application. 

Even better, as these are more standard tools, it makes it easier for other developers to grasp your continuous integration system more easily. Complex configuration management tooling such as Ansible or Terriform need not be involved if Helm is all you need.