# Customising Kubernetes Manifests Using manifest files for deploying to Kubernetes creates a manageable library of resources. A challenge arises when you have to manage these manifest files across multiple environments. Imagine you have the following directory structure: ```plaintext ├── dev │   ├── dev_namespace.yaml │   └── nginx_pod.yaml ├── prod │   ├── nginx_pod.yaml │   └── prod_namespace.yaml └── test    ├── nginx_pod.yaml    └── test_namespace.yaml ``` Let's say you need to upgrade the image version in all three environments. You would have to edit the pod manifests one by one in each directory. I'm going to discuss two configuration management tools for tackling such grunt work. This article aims to provide a general overview of the possibilities offered by these solutions, and is not intended to be a comprehensive guide. The goal is to arm the reader with the knowledge needed to make an informed decision. ## Helm Also called a package manager for Kubernetes. It provides a single location to declare all modifications for the manifests. It works with charts that can be downloaded from the public Artifact Hub repository (https://artifacthub.io). Modifications are made using the `values.yaml` manifests, which can alter the original chart. Let me give a quick introduction on how this works using the Nginx chart. Download and extract the chart locally ```plaintext helm pull --untar oci://registry-1.docker.io/bitnamicharts/nginx --version 24.0.0 ``` Edit values.yaml in the root of the downloaded folder, and let's change the replica count for our deployment ```plaintext replicaCount: 2 ``` Install the release ```plaintext helm install nginx ./nginx ``` We can see that three pods were created ```plaintext kubectl get pods NAME READY STATUS RESTARTS AGE nginx-5d5ff86bb-827jz 0/1 Running 0 3s nginx-5d5ff86bb-ksgxj 0/1 Running 0 3s ``` To give you a little more depth without going into the nitty-gritty details of GO templating, let's take a look at `templates/deployment.yaml`, which contains the following object. ```plaintext spec: {{- if not .Values.autoscaling.enabled }} replicas: {{ .Values.replicaCount }} {{- end }} ``` The curly braces after replicas define a variable, which is taken from the `values.yaml` file. Another significant advantage of Helm is the ability to roll back changes made to the charts by utilising the change history. Starting from version 3, Helm not only tracks every change made to the chart, but it also considers the live state of the cluster when doing an upgrade or a rollback in case a resource was manually modified. This is called the **3-way strategic merge patch**. You can read more about Helm here: https://helm.sh/docs/intro/using\_helm This can get quite complicated with the introduction of features such as conditionals, loops, functions and hooks, which are out of the scope of this article. ## Kustomize Despite its simpler approach, it remains a highly effective solution that takes a significant burden off the user's shoulders. If you need a refresher about the basics, you can check out the documentation on this page: [https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/) I'm going to focus on the modification options right away. ### Transformers I created the directory structure with the appropriate Kustomize files. ```plaintext ├── base │   ├── dev │   │   ├── dev_namespace.yaml │   │   ├── kustomization.yaml │   │   └── nginx_pod.yaml │   ├── kustomization.yaml │   ├── prod │   │   ├── kustomization.yaml │   │   ├── nginx_pod.yaml │   │   └── prod_namespace.yaml │   └── test │   ├── kustomization.yaml │   ├── nginx_pod.yaml │   └── test_namespace.yaml └── kustomization.yaml ``` Let's say we want to assign a label to all of the nginx pods. For this, it is enough to modify the root kustomization file, which will apply this to all of the pods in each environment. ```plaintext apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - ./base commonLabels: blog: BehrMetal ``` ### Patches Unlike Transformers, Patches provide a targeted approach to modifying specific sections in the resource manifest. Three parameters need to be specified: * `Type` add | remove | replace * `Target` Kind | Version/Group | Name | Namespce | Label | Annotation * `Value` Let's start with a quick example. I added the following to the root kustomization file ```plaintext patches: - target: kind: Pod name: nginx namespace: dev patch: |- - op: replace path: /metadata/name value: nginx-dev ``` After hitting apply, a new pod is created with the name `nginx-pod`. Notice that the old pod kept running because kubectl does not track resources outside its scope. Or, in other words, it does not monitor the cluster state in a holistic way. For this, we would need a tool like ArgoCD or Flux. ```plaintext kubectl get pod -n dev NAME READY STATUS RESTARTS AGE nginx 1/1 Running 0 32h nginx-dev 1/1 Running 0 2m44s ``` There are two ways to define a Patch. **Json 6902** ```plaintext patches: - target: kind: Pod name: nginx namespace: dev patch: |- - op: replace path: /metadata/name value: nginx-dev ``` Using this scheme, the target needs to be specified with at least a single parameter. The defined patch operation is performed on the manifest by providing the path to the variable. **Strategic Merge Patch** ```plaintext patches: - patch: |- apiVersion: v1 kind: Pod metadata: name: nginx namespace: dev labels: blog: BehrMetal ``` In this type, we are providing the objects from the regular Kubernetes config that we need to update. Note that I could not have recreated the previous replace operation just by declaring the new desired name, because there is currently no pod named `nginx-dev` in the cluster. That's why I chose to add a label instead. Just as a sidenote: patches can be stored in two ways. #### Inline patch ```plaintext patches: - target: kind: Pod name: nginx namespace: dev patch: |- - op: replace path: /metadata/name value: nginx-dev ``` #### Patch in a separate file ```plaintext # Kustomization.yaml patches: - path: name-patch.yaml target: kind: Pod name: nginx namespace: dev # name-patch.yaml - op: replace path: /metadata/name value: nginx-dev ``` ### Additional operations In addition to replacing a key in a dictionary, we have the following operations at our disposal. #### Add a Dictionary ```plaintext patch: |- - op: add path: /metadata/labels/environment value: development ``` Adding a new dictionary using the strategic merge patch means merely providing the key value pair in the manifest. #### Remove a Dictionary ```plaintext - op: remove path: /metadata/labels/run ``` To remove a key using the strategic merge patch, set it to `null` #### Replace a List item ```plaintext - op: replace path: /spec/containers/0 value: name: nginx-dev image: nginx ``` The '0' at the end of the path defines the index of the list items starting from 0. So having /1 would correspond to the second item in the list. Replacing a list item using the strategic merge patch means merely modifying the key value pair to the desired value. #### Add a List item ```plaintext - op: add path: /spec/containers/- value: name: haproxy image: haproxy ``` The dash at the end of the path means "append to the list", so the item will be last in the list. The index can be specified with a number, zero being the first item. Adding a list item using the strategic merge patch means merely providing the key-value pair in the manifest. #### Remove a List item ```plaintext - op: remove path: /spec/containers/1 ``` Again, the number in the path means the index. Doing the same thing using the strategic merge patch requires the following syntax. ```plaintext spec: containers: - $patch: delete name: haproxy ``` ### Overlays Overlays enable provisioning of patches on a per-environment basis. The only novelty here is the reference to the base folder. Our new directory structure now includes the overlays folder. ```plaintext ├── base │   ├── dev │   │   ├── dev_namespace.yaml │   │   ├── kustomization.yaml │   │   └── nginx_pod.yaml │   ├── kustomization.yaml │   ├── prod │   │   ├── kustomization.yaml │   │   ├── nginx_pod.yaml │   │   └── prod_namespace.yaml │   └── test │   ├── kustomization.yaml │   ├── nginx_pod.yaml │   └── test_namespace.yaml └── overlays ├── dev │   └── kustomization.yaml ├── prod │   └── kustomization.yaml └── test └── kustomization.yaml ``` ```plaintext # ./overlays/test/kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization bases: - ../../base patches: - target: kind: Pod name: nginx namespace: test patch: |- - op: replace path: /metadata/labels/run value: nginx-test ``` Overlays also allow us to place custom manifests for the given environment in the corresponding folder, which will be applied accordingly. In this case, the new resource has to be declared in the local kustomization manifest. ### Components Components enable us to create a unique combination of resources for each environment by referencing configuration files from a single library. This way, we can avoid copy-pasting the same manifest to multiple overlays and avoid configuration drift when making changes to these resources. I upgraded the directory structure to a more refined version. ```plaintext ├── base │   ├── deployment.yaml │   ├── kustomization.yaml │   └── service.yaml ├── components │   └── database │   ├── db.yaml │   └── kustomization.yaml └── overlays ├── dev │   └── kustomization.yaml ├── prod │   └── kustomization.yaml └── test ├── kustomization.yaml ├── deployment-patch.yaml └── namespace.yaml ``` Now the `base` directory only contains the template configurations, which are modified by the `Overlays` for the specific environments. The `Components` provide additions which are loaded into `Overlays` on demand. Let's take a detailed look at our `test` environment. ```plaintext # overlays/test/kustomization.yaml namespace: test bases: - ../../base components: - ../../components/database resources: - namespace.yaml patches: - path: deployment-patch.yaml ``` ```plaintext # components/database/kustomization.yaml apiVersion: kustomize.config.k8s.io/v1alpha1 kind: Component resources: - db.yaml ``` ```plaintext # overlays/test/deployment-patch.yaml apiVersion: apps/v1 kind: Deployment metadata: name: api-deployment spec: replicas: 2 ``` The namespace was defined in the `overlays` kustomization file because I need all resources in this namespace. Note that the namespace manifest is also in this directory, as it applies to the entire overlay. The components, on the other hand, can be deployed across multiple environments using the same configuration file, which serves as a single source of truth. #### Overlays and Components To understand the meaning and the difference between `Overlays` and `Components`consider the following scenario. Imagine that you need to fulfil the following requirements specifically for the test environment: * The Deployment needs to have 2 replicas. * You need an additional database with mock data for testing Because the replica count is environment-specific, its patch belongs in the test `overlay`. Other environments will require different pod counts, so we use the base template and modify it to fit the specific requirements of each namespace. The additional database will be pulled in as a reusable `component`. Because it contains pre-baked data samples, it can be reused as a shared resource across multiple testing stages. Overlays contain configuration scoped to the whole environment, while using reusable components to enrich their repertoire. If we ever need to make a change to a component, it will be automatically propagated to all of the environments that are using it.