Customising Kubernetes Manifests

Using manifest files for deploying to Kubernetes creates a manageable library of resources.

A challenge arises when you have to manage these manifest files across multiple environments.

Imagine you have the following directory structure:

├── dev
│   ├── dev_namespace.yaml
│   └── nginx_pod.yaml
├── prod
│   ├── nginx_pod.yaml
│   └── prod_namespace.yaml
└── test
    ├── nginx_pod.yaml
    └── test_namespace.yaml

Let's say you need to upgrade the image version in all three environments. You would have to edit the pod manifests one by one in each directory.

I'm going to discuss two configuration management tools for tackling such grunt work.

This article aims to provide a general overview of the possibilities offered by these solutions, and is not intended to be a comprehensive guide. The goal is to arm the reader with the knowledge needed to make an informed decision.

Helm

Also called a package manager for Kubernetes. It provides a single location to declare all modifications for the manifests.

It works with charts that can be downloaded from the public Artifact Hub repository (https://artifacthub.io). Modifications are made using the values.yaml manifests, which can alter the original chart.

Let me give a quick introduction on how this works using the Nginx chart.

Download and extract the chart locally

helm pull --untar oci://registry-1.docker.io/bitnamicharts/nginx --version 24.0.0

Edit values.yaml in the root of the downloaded folder, and let's change the replica count for our deployment

replicaCount: 2

Install the release

helm install nginx ./nginx

We can see that three pods were created

kubectl get pods

NAME                               READY   STATUS     RESTARTS   AGE
nginx-5d5ff86bb-827jz              0/1     Running    0          3s
nginx-5d5ff86bb-ksgxj              0/1     Running    0          3s

To give you a little more depth without going into the nitty-gritty details of GO templating, let's take a look at templates/deployment.yaml, which contains the following object.

spec:
  {{- if not .Values.autoscaling.enabled }}
  replicas: {{ .Values.replicaCount }}
  {{- end }}

The curly braces after replicas define a variable, which is taken from the values.yaml file.

Another significant advantage of Helm is the ability to roll back changes made to the charts by utilising the change history. Starting from version 3, Helm not only tracks every change made to the chart, but it also considers the live state of the cluster when doing an upgrade or a rollback in case a resource was manually modified. This is called the 3-way strategic merge patch. You can read more about Helm here: https://helm.sh/docs/intro/using\_helm

This can get quite complicated with the introduction of features such as conditionals, loops, functions and hooks, which are out of the scope of this article.

Kustomize

Despite its simpler approach, it remains a highly effective solution that takes a significant burden off the user's shoulders.

If you need a refresher about the basics, you can check out the documentation on this page: https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/

I'm going to focus on the modification options right away.

Transformers

I created the directory structure with the appropriate Kustomize files.

├── base
│   ├── dev
│   │   ├── dev_namespace.yaml
│   │   ├── kustomization.yaml
│   │   └── nginx_pod.yaml
│   ├── kustomization.yaml
│   ├── prod
│   │   ├── kustomization.yaml
│   │   ├── nginx_pod.yaml
│   │   └── prod_namespace.yaml
│   └── test
│       ├── kustomization.yaml
│       ├── nginx_pod.yaml
│       └── test_namespace.yaml
└── kustomization.yaml

Let's say we want to assign a label to all of the nginx pods.

For this, it is enough to modify the root kustomization file, which will apply this to all of the pods in each environment.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ./base

commonLabels:
  blog: BehrMetal

Patches

Unlike Transformers, Patches provide a targeted approach to modifying specific sections in the resource manifest.

Three parameters need to be specified:

• Type add | remove | replace

• Target Kind | Version/Group | Name | Namespce | Label | Annotation

• Value

Let's start with a quick example.

I added the following to the root kustomization file

patches:
  - target:
      kind: Pod
      name: nginx
      namespace: dev
    patch: |-
      - op: replace
        path: /metadata/name
        value: nginx-dev

After hitting apply, a new pod is created with the name nginx-pod. Notice that the old pod kept running because kubectl does not track resources outside its scope. Or, in other words, it does not monitor the cluster state in a holistic way.
For this, we would need a tool like ArgoCD or Flux.

kubectl get pod -n dev

NAME        READY   STATUS    RESTARTS   AGE
nginx       1/1     Running   0          32h
nginx-dev   1/1     Running   0          2m44s

There are two ways to define a Patch.

Json 6902

patches:
  - target:
      kind: Pod
      name: nginx
      namespace: dev
    patch: |-
      - op: replace
        path: /metadata/name
        value: nginx-dev

Using this scheme, the target needs to be specified with at least a single parameter. The defined patch operation is performed on the manifest by providing the path to the variable.

Strategic Merge Patch

patches:
  - patch: |-
      apiVersion: v1
      kind: Pod
      metadata:
        name: nginx
        namespace: dev
        labels:
            blog: BehrMetal

In this type, we are providing the objects from the regular Kubernetes config that we need to update.

Note that I could not have recreated the previous replace operation just by declaring the new desired name, because there is currently no pod named nginx-dev in the cluster. That's why I chose to add a label instead.

Just as a sidenote: patches can be stored in two ways.

Inline patch

patches:
  - target:
      kind: Pod
      name: nginx
      namespace: dev
    patch: |-
      - op: replace
        path: /metadata/name
        value: nginx-dev

Patch in a separate file

# Kustomization.yaml

patches:
  - path: name-patch.yaml
    target:
      kind: Pod
      name: nginx
      namespace: dev

# name-patch.yaml

- op: replace
  path: /metadata/name
  value: nginx-dev

Additional operations

In addition to replacing a key in a dictionary, we have the following operations at our disposal.

Add a Dictionary

patch: |-
  - op: add
    path: /metadata/labels/environment
    value: development

Adding a new dictionary using the strategic merge patch means merely providing the key value pair in the manifest.

Remove a Dictionary

- op: remove
  path: /metadata/labels/run

To remove a key using the strategic merge patch, set it to null

Replace a List item

- op: replace
  path: /spec/containers/0
  value:
    name: nginx-dev
    image: nginx

The '0' at the end of the path defines the index of the list items starting from 0. So having /1 would correspond to the second item in the list.

Replacing a list item using the strategic merge patch means merely modifying the key value pair to the desired value.

Add a List item

- op: add
  path: /spec/containers/-
  value:
    name: haproxy
    image: haproxy

The dash at the end of the path means "append to the list", so the item will be last in the list. The index can be specified with a number, zero being the first item.

Adding a list item using the strategic merge patch means merely providing the key-value pair in the manifest.

Remove a List item

- op: remove
  path: /spec/containers/1

Again, the number in the path means the index.

Doing the same thing using the strategic merge patch requires the following syntax.

spec:
  containers:
    - $patch: delete
      name: haproxy

Overlays

Overlays enable provisioning of patches on a per-environment basis. The only novelty here is the reference to the base folder.

Our new directory structure now includes the overlays folder.

├── base
│   ├── dev
│   │   ├── dev_namespace.yaml
│   │   ├── kustomization.yaml
│   │   └── nginx_pod.yaml
│   ├── kustomization.yaml
│   ├── prod
│   │   ├── kustomization.yaml
│   │   ├── nginx_pod.yaml
│   │   └── prod_namespace.yaml
│   └── test
│       ├── kustomization.yaml
│       ├── nginx_pod.yaml
│       └── test_namespace.yaml
└── overlays
    ├── dev
    │   └── kustomization.yaml
    ├── prod
    │   └── kustomization.yaml
    └── test
        └── kustomization.yaml

# ./overlays/test/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
  - ../../base
patches:
  - target:
      kind: Pod
      name: nginx
      namespace: test

    patch: |-
      - op: replace
        path: /metadata/labels/run
        value: nginx-test

Overlays also allow us to place custom manifests for the given environment in the corresponding folder, which will be applied accordingly. In this case, the new resource has to be declared in the local kustomization manifest.

Components

Components enable us to create a unique combination of resources for each environment by referencing configuration files from a single library. This way, we can avoid copy-pasting the same manifest to multiple overlays and avoid configuration drift when making changes to these resources.

I upgraded the directory structure to a more refined version.

├── base
│   ├── deployment.yaml
│   ├── kustomization.yaml
│   └── service.yaml
├── components
│   └── database
│       ├── db.yaml     
│       └── kustomization.yaml
└── overlays
    ├── dev
    │   └── kustomization.yaml
    ├── prod
    │   └── kustomization.yaml
    └── test
        ├── kustomization.yaml
        ├── deployment-patch.yaml
        └── namespace.yaml

Now the base directory only contains the template configurations, which are modified by the Overlays for the specific environments. The Components provide additions which are loaded into Overlays on demand.

Let's take a detailed look at our test environment.

# overlays/test/kustomization.yaml

namespace: test
bases:
  - ../../base

components:
  - ../../components/database

resources:
  - namespace.yaml

patches:
  - path: deployment-patch.yaml

# components/database/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

resources:
  - db.yaml

# overlays/test/deployment-patch.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-deployment
spec:
  replicas: 2

The namespace was defined in the overlays kustomization file because I need all resources in this namespace. Note that the namespace manifest is also in this directory, as it applies to the entire overlay.

The components, on the other hand, can be deployed across multiple environments using the same configuration file, which serves as a single source of truth.

Overlays and Components

To understand the meaning and the difference between Overlays and Componentsconsider the following scenario. Imagine that you need to fulfil the following requirements specifically for the test environment:

• The Deployment needs to have 2 replicas.

• You need an additional database with mock data for testing

Because the replica count is environment-specific, its patch belongs in the test overlay. Other environments will require different pod counts, so we use the base template and modify it to fit the specific requirements of each namespace.

The additional database will be pulled in as a reusable component. Because it contains pre-baked data samples, it can be reused as a shared resource across multiple testing stages.

Overlays contain configuration scoped to the whole environment, while using reusable components to enrich their repertoire. If we ever need to make a change to a component, it will be automatically propagated to all of the environments that are using it.