A challenge arises when you have to manage these manifest files across multiple environments.

Imagine you have the following directory structure:

├── dev
│   ├── dev_namespace.yaml
│   └── nginx_pod.yaml
├── prod
│   ├── nginx_pod.yaml
│   └── prod_namespace.yaml
└── test
    ├── nginx_pod.yaml
    └── test_namespace.yaml

Let's say you need to upgrade the image version in all three environments. You would have to edit the pod manifests one by one in each directory.

I'm going to discuss two configuration management tools for tackling such grunt work.

This article aims to provide a general overview of the possibilities offered by these solutions, and is not intended to be a comprehensive guide. The goal is to arm the reader with the knowledge needed to make an informed decision.

Helm

Also called a package manager for Kubernetes. It provides a single location to declare all modifications for the manifests.

It works with charts that can be downloaded from the public Artifact Hub repository (https://artifacthub.io). Modifications are made using the values.yaml manifests, which can alter the original chart.

Let me give a quick introduction on how this works using the Nginx chart.

Download and extract the chart locally

helm pull --untar oci://registry-1.docker.io/bitnamicharts/nginx --version 24.0.0

Edit values.yaml in the root of the downloaded folder, and let's change the replica count for our deployment

replicaCount: 2

Install the release

helm install nginx ./nginx

We can see that three pods were created

kubectl get pods

NAME                               READY   STATUS     RESTARTS   AGE
nginx-5d5ff86bb-827jz              0/1     Running    0          3s
nginx-5d5ff86bb-ksgxj              0/1     Running    0          3s

To give you a little more depth without going into the nitty-gritty details of GO templating, let's take a look at templates/deployment.yaml, which contains the following object.

spec:
  {{- if not .Values.autoscaling.enabled }}
  replicas: {{ .Values.replicaCount }}
  {{- end }}

The curly braces after replicas define a variable, which is taken from the values.yaml file.

Another significant advantage of Helm is the ability to roll back changes made to the charts by utilising the change history. Starting from version 3, Helm not only tracks every change made to the chart, but it also considers the live state of the cluster when doing an upgrade or a rollback in case a resource was manually modified. This is called the 3-way strategic merge patch. You can read more about Helm here: https://helm.sh/docs/intro/using\_helm

This can get quite complicated with the introduction of features such as conditionals, loops, functions and hooks, which are out of the scope of this article.

Kustomize

Despite its simpler approach, it remains a highly effective solution that takes a significant burden off the user's shoulders.

If you need a refresher about the basics, you can check out the documentation on this page: https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/

I'm going to focus on the modification options right away.

Transformers

I created the directory structure with the appropriate Kustomize files.

├── base
│   ├── dev
│   │   ├── dev_namespace.yaml
│   │   ├── kustomization.yaml
│   │   └── nginx_pod.yaml
│   ├── kustomization.yaml
│   ├── prod
│   │   ├── kustomization.yaml
│   │   ├── nginx_pod.yaml
│   │   └── prod_namespace.yaml
│   └── test
│       ├── kustomization.yaml
│       ├── nginx_pod.yaml
│       └── test_namespace.yaml
└── kustomization.yaml

Let's say we want to assign a label to all of the nginx pods.

For this, it is enough to modify the root kustomization file, which will apply this to all of the pods in each environment.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ./base

commonLabels:
  blog: BehrMetal

Patches

Unlike Transformers, Patches provide a targeted approach to modifying specific sections in the resource manifest.

Three parameters need to be specified:

• Type add | remove | replace

• Target Kind | Version/Group | Name | Namespce | Label | Annotation

• Value

Let's start with a quick example.

I added the following to the root kustomization file

patches:
  - target:
      kind: Pod
      name: nginx
      namespace: dev
    patch: |-
      - op: replace
        path: /metadata/name
        value: nginx-dev

After hitting apply, a new pod is created with the name nginx-pod. Notice that the old pod kept running because kubectl does not track resources outside its scope. Or, in other words, it does not monitor the cluster state in a holistic way.
For this, we would need a tool like ArgoCD or Flux.

kubectl get pod -n dev

NAME        READY   STATUS    RESTARTS   AGE
nginx       1/1     Running   0          32h
nginx-dev   1/1     Running   0          2m44s

There are two ways to define a Patch.

Json 6902

patches:
  - target:
      kind: Pod
      name: nginx
      namespace: dev
    patch: |-
      - op: replace
        path: /metadata/name
        value: nginx-dev

Using this scheme, the target needs to be specified with at least a single parameter. The defined patch operation is performed on the manifest by providing the path to the variable.

Strategic Merge Patch

patches:
  - patch: |-
      apiVersion: v1
      kind: Pod
      metadata:
        name: nginx
        namespace: dev
        labels:
            blog: BehrMetal

In this type, we are providing the objects from the regular Kubernetes config that we need to update.

Note that I could not have recreated the previous replace operation just by declaring the new desired name, because there is currently no pod named nginx-dev in the cluster. That's why I chose to add a label instead.

Just as a sidenote: patches can be stored in two ways.

Inline patch

patches:
  - target:
      kind: Pod
      name: nginx
      namespace: dev
    patch: |-
      - op: replace
        path: /metadata/name
        value: nginx-dev

Patch in a separate file

# Kustomization.yaml

patches:
  - path: name-patch.yaml
    target:
      kind: Pod
      name: nginx
      namespace: dev

# name-patch.yaml

- op: replace
  path: /metadata/name
  value: nginx-dev

Additional operations

In addition to replacing a key in a dictionary, we have the following operations at our disposal.

Add a Dictionary

patch: |-
  - op: add
    path: /metadata/labels/environment
    value: development

Adding a new dictionary using the strategic merge patch means merely providing the key value pair in the manifest.

Remove a Dictionary

- op: remove
  path: /metadata/labels/run

To remove a key using the strategic merge patch, set it to null

Replace a List item

- op: replace
  path: /spec/containers/0
  value:
    name: nginx-dev
    image: nginx

The '0' at the end of the path defines the index of the list items starting from 0. So having /1 would correspond to the second item in the list.

Replacing a list item using the strategic merge patch means merely modifying the key value pair to the desired value.

Add a List item

- op: add
  path: /spec/containers/-
  value:
    name: haproxy
    image: haproxy

The dash at the end of the path means "append to the list", so the item will be last in the list. The index can be specified with a number, zero being the first item.

Adding a list item using the strategic merge patch means merely providing the key-value pair in the manifest.

Remove a List item

- op: remove
  path: /spec/containers/1

Again, the number in the path means the index.

Doing the same thing using the strategic merge patch requires the following syntax.

spec:
  containers:
    - $patch: delete
      name: haproxy

Overlays

Overlays enable provisioning of patches on a per-environment basis. The only novelty here is the reference to the base folder.

Our new directory structure now includes the overlays folder.

├── base
│   ├── dev
│   │   ├── dev_namespace.yaml
│   │   ├── kustomization.yaml
│   │   └── nginx_pod.yaml
│   ├── kustomization.yaml
│   ├── prod
│   │   ├── kustomization.yaml
│   │   ├── nginx_pod.yaml
│   │   └── prod_namespace.yaml
│   └── test
│       ├── kustomization.yaml
│       ├── nginx_pod.yaml
│       └── test_namespace.yaml
└── overlays
    ├── dev
    │   └── kustomization.yaml
    ├── prod
    │   └── kustomization.yaml
    └── test
        └── kustomization.yaml

# ./overlays/test/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
  - ../../base
patches:
  - target:
      kind: Pod
      name: nginx
      namespace: test

    patch: |-
      - op: replace
        path: /metadata/labels/run
        value: nginx-test

Overlays also allow us to place custom manifests for the given environment in the corresponding folder, which will be applied accordingly. In this case, the new resource has to be declared in the local kustomization manifest.

Components

Components enable us to create a unique combination of resources for each environment by referencing configuration files from a single library. This way, we can avoid copy-pasting the same manifest to multiple overlays and avoid configuration drift when making changes to these resources.

I upgraded the directory structure to a more refined version.

├── base
│   ├── deployment.yaml
│   ├── kustomization.yaml
│   └── service.yaml
├── components
│   └── database
│       ├── db.yaml     
│       └── kustomization.yaml
└── overlays
    ├── dev
    │   └── kustomization.yaml
    ├── prod
    │   └── kustomization.yaml
    └── test
        ├── kustomization.yaml
        ├── deployment-patch.yaml
        └── namespace.yaml

Now the base directory only contains the template configurations, which are modified by the Overlays for the specific environments. The Components provide additions which are loaded into Overlays on demand.

Let's take a detailed look at our test environment.

# overlays/test/kustomization.yaml

namespace: test
bases:
  - ../../base

components:
  - ../../components/database

resources:
  - namespace.yaml

patches:
  - path: deployment-patch.yaml

# components/database/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

resources:
  - db.yaml

# overlays/test/deployment-patch.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-deployment
spec:
  replicas: 2

The namespace was defined in the overlays kustomization file because I need all resources in this namespace. Note that the namespace manifest is also in this directory, as it applies to the entire overlay.

The components, on the other hand, can be deployed across multiple environments using the same configuration file, which serves as a single source of truth.

Overlays and Components

To understand the meaning and the difference between Overlays and Componentsconsider the following scenario. Imagine that you need to fulfil the following requirements specifically for the test environment:

• The Deployment needs to have 2 replicas.

• You need an additional database with mock data for testing

Because the replica count is environment-specific, its patch belongs in the test overlay. Other environments will require different pod counts, so we use the base template and modify it to fit the specific requirements of each namespace.

The additional database will be pulled in as a reusable component. Because it contains pre-baked data samples, it can be reused as a shared resource across multiple testing stages.

Overlays contain configuration scoped to the whole environment, while using reusable components to enrich their repertoire. If we ever need to make a change to a component, it will be automatically propagated to all of the environments that are using it.

The need for this solution was conceived during the search for a storage solution for an RKE2 Kubernetes Cluster. Since the whole cluster was designed to run on virtual machines provisioned on VMware vSphere, using the storage providers already attached to this virtualisation platform was a convenient design choice.

At first, we used a simple NFS server running on a VM, which was sufficient for demonstration purposes, but we already knew that a more robust solution was needed. Without going into too much detail, this architecture has numerous disadvantages. Just to mention a few: traffic has to pass through the virtual network stack and the operating system of a non-redundant machine, which therefore stands as a single point of failure in the pipeline. Also, not to mention how cumbersome it would be to utilise different storage technologies.

VMware offers an enterprise-grade storage solution, known as the Cloud Native Storage (CNS). It enables the Kubernetes cluster to directly request container volumes from an interface communicating with the hypervisor. The system comprises two fundamental components. The CNS itself lives in vCenter and its counterpart, the vSphere volume driver, as a pod in Kubernetes. CNS creates a platform through which stateful Kubernetes workflows can be run. Such a performant data path is mature enough for high-reliability architectures suitable for enterprise environments. The main advantage of this solution is that you can incorporate multiple different storage types (vSAN, VMFS, NFS, vVOLS) into a single plane and create policies to control their allocation.

The Architecture

CNS has two fundamental constituents: the vCenter component and the vSphere Volume Driver in Kubernetes, also known as the Container Storage Plug-in.

vCenter

CNS is the control plane for managing storage volumes' lifecycle (CRUD operations), which is independent of the virtual machine.

It relies on these three essential resources from vSphere.

• First Class Disk (FCD): This is the virtual disk, or in fact the .vmdk image file, which represents the storage that can outlive the pods and being a managed object, its state can be traced.

• vSAN File Services: a built-in block storage file server enabling shared volumes for situations where multiple pods are using the same volume

• Storage Policy Based Management (SPBM): make decisions about the utilisation of the datastore based on pre-defined policies

Through an example:
A PVC was created requesting 100GB of multi-pod (RWX) gold storage

SPBM looks at the gold rule, which instructs it to use the SSD storage provider. vSAN FS spins up an NFS share to satisfy the RWX requirement. It will appear as an FCD among the vCenter resources.

CSI

The Container Storage Plug-in, running as a set of pods in a Kubernetes cluster, utilises two main components.

• CSI Plug-in: provides volume mounts for pods by acting as a StorageClass for the volume claims. Its functionality is outsourced into two types of pods.

  • Controller (control plane): manages the lifecycle and provisioning of volumes. Acts as an interface for Kubernetes to create/delete/attach/detach volumes to the specific nodes.

  • Node (data plane): performs the mounting and formatting of volumes on the host OS. It runs as DaemonSet in the cluster.

• 

  Syncer: communicates the volume claims and their associated metadata with the CNS control plane, which can be displayed under the Monitor → Cloud Native Storage → Container Volumes on the vSphere dashboard. The syncer is useful when various components (including during etcd restoration from backup) experience downtime, and the metadata needs to be re-synced.

Installation manual

I was using the following component versions for the PoC:

• RKE2 v1.34.3+rke2r1

• vCenter v8.0.3

• vsphere-csi-driver-v3.3.1

• 24.04 LTS (Noble Numbat) for the cluster nodes

Prerequisites

Since I was running the RKE2 cluster using vCenter VMs, which will be hosting the pods utilising the volumes, I needed to satisfy the following requirements.

• Enabling the external cloud provider

  This should be as simple as adding a single line in the config and restarting the service.

    /etc/rke2/config.yaml
    cloud-provider-name: external
  
    # On control node
    systemctl restart rke2-server
    # On the worker node
    systemctl restart rke2-agent
  

  When you have an already working cluster with running applications on it, you will probably have to drain the nodes before restarting the service or resetting them. The latter is no big deal on a worker node, but resetting the control node can cause issues, especially if you are only running a single one.

  Anyhow, you should be seeing the following output.

  (You will have provider IDs starting with rke2:// before registering the external provider.)

    kubectl get nodes -o custom-columns=NAME:.metadata.name,PROVIDER_ID:.spec.providerID
    NAME                           PROVIDER_ID
    control-node                   vsphere://4219ef7e-d8e7-e264-7b45-9e7cde0b4678
    worker-node                    vsphere://4219f724-4281-d937-8dfe-8212e7cc877a
  

• Installing VMware Tools on each VM

  • You can find a guide for that here^: https://knowledge.broadcom.com/external/article/315363/how-to-install-vmware-tools.html

• Enabling the DiskUUID

  • This can be done by opening the hardware settings of the VM, going to the Advanced Parameters and setting disk.EnableUUID to TRUE. The machine has to be stopped while setting this parameter.

  • 

Installing the Cloud Provider Interface (CPI)

Download the manifest with the following command.

VERSION=1.22 # use the Kubernetes version currently running
wget https://raw.githubusercontent.com/kubernetes/cloud-provider-vsphere/release-$VERSION/releases/v$VERSION/vsphere-cloud-controller-manager.yaml

Open vsphere-cloud-controller-manager.yaml in your favourite editor.

Upon examining the file, you will see two resources defined for storing your vCenter credentials. This could be confusing at first, because you only need one of them. Using a Secret would be the more secure approach, but only if you encrypt it separately. You can use SOPS, for example (https://github.com/getsops/sops). I have a guide about the encryption process here: https://github.com/behrlevi/kube_cluster?tab=readme-ov-file#secrets-management-with-sops.

For simplicity's sake, I went with the ConfigMap for the PoC defined in the following fashion.

apiVersion: v1
kind: ConfigMap
metadata:
  name: vsphere-cloud-config
  labels:
    vsphere-cpi-infra: config
    component: cloud-controller-manager
  namespace: kube-system
data:
  vsphere.conf: |
    # Global properties in this section will be used for all specified vCenters unless overriden in VirtualCenter section.
    global:
      port: 443
      # set insecureFlag to true if the vCenter uses a self-signed cert
      insecureFlag: true
      datacenters:
        - <DATACENTER_NAME>

    # vcenter section
    vcenter:
      "<IP OR HOSTNAME>":
        server: <IP OR HOSTNAME>
        user: <vCenter service user's name>
        password: <vCenter service user's password>
        datacenters:
          - <DATACENTER_NAME>

Notice that this configmap is mounted as a volume in the DaemonSet like so.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: vsphere-cloud-controller-manager
...
      volumes:
        - name: vsphere-config-volume
          configMap:
            name: vsphere-cloud-config

Installing the Cloud Storage Plug-in (CSI)

• Create a namespace

    kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/vsphere-csi-driver/v3.3.0/manifests/vanilla/namespace.yaml
  

  On a side note, you can also generate the .yaml manifest by adding the following at the end of this command

    --dry-run=client -o yaml > vsphere-csi-namespace.yaml
  

  Using this approach lets you record each resource as a file, which you can apply to the cluster and store for future reference.

• Taint the control node

  You need to taint the control plane node with the node-role.kubernetes.io/control-plane=:NoSchedule parameter using the following command.

    kubectl taint nodes <k8s-primary-name> node-role.kubernetes.io/control-plane=:NoSchedule
  

  Verify with

    $ kubectl describe nodes | egrep "Taints:|Name:"
    Name:               <k8s-primary-name>
    Taints:             node-role.kubernetes.io/control-plane:NoSchedule
    Name:               <k8s-worker1-name>
  

• Create a Secret

  Create a config file and fill in the required credentials

    $ cat /etc/kubernetes/csi-vsphere.conf
    [Global]
    cluster-id = "<cluster-id>"
    cluster-distribution = "<cluster-distribution>"
    ca-file = <ca file path> # optional, use with insecure-flag set to false
    thumbprint = "<cert thumbprint>" # optional, use with insecure-flag set to false without providing ca-file
  
    [VirtualCenter "<IP or FQDN>"]
    insecure-flag = "<true or false>"
    user = "<username>"
    password = "<password>"
    port = "<port>"
    datacenters = "<datacenter1-path>, <datacenter2-path>, ..."
  

  You can leave the cluster-id and cluster-distribution empty, as these will be auto-generated.

  In my case, the datacenter sits in the root directory of the vSphere instance, so I only entered the its name in the last key. If yours is located inside a subdirectory, then you should provide the full path here.

  Apply the secret with the following command

    kubectl create secret generic vsphere-config-secret --from-file=csi-vsphere.conf --namespace=vmware-system-csi
  

• Install the CSI driver pod

  Download the manifest with the following command.

    curl -o vsphere-csi-driver-v3.3.1.yaml https://raw.githubusercontent.com/kubernetes-sigs/vsphere-csi-driver/v3.3.1/manifests/vanilla/vsphere-csi-driver.yaml
  

  There is a caveat, though, because the images defined in the manifest are no longer available in the Google Artifact Registry. This could be corrected in the future, so you should check the file first.

  You can change the URLS in the manifest with the following command.

    sed -i 's|gcr.io/cloud-provider-vsphere/csi/release|registry.k8s.io/csi-vsphere|g' vsphere-csi-driver-v3.3.1.yaml
  

If all goes well, you should see the following pods in a healthy Running state inside your cluster.

The CSI Deployment is configured with 3 pod replicaset by default for fault tolerance, but this can be changed in the manifest here.

---           
kind: Deployment
apiVersion: apps/v1
metadata:     
  name: vsphere-csi-controller
  namespace: vmware-system-csi
spec:
  replicas: 3

Storage policy

The policies work based on tags, so you should first create those on the datastores.

When you are done, open Policies and Profiles from the side menu and create a new VM Storage Policy.

Select an appropriate name for the policy that you will later use in the StorageClass manifest. Create a rule for the chosen tag.

The matching datastores will be displayed on the next page (Storage Compatibility).

You can create more rules and fine-grained policies for your specific needs. As for the PoC, I went with the simplest setup.

Testing the storage

You test the solution by creating a testing pod and mounting a volume inside it provided by the CSI. This can be achieved by defining the following resources.

StorageClass

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: vsphere-storage
  annotations:
    storageclass.kubernetes.io/is-default-class: "true" # Optional: Makes this the default
provisioner: csi.vsphere.vmware.com
parameters:
  # This MUST match the policy name in vSphere EXACTLY (Case Sensitive!)
  storagepolicyname: "k8s-storage-policy"
allowVolumeExpansion: true

VolumeClaim

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-vsphere-pvc
spec:
  storageClassName: vsphere-storage
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

Pod

apiVersion: v1
kind: Pod
metadata:
  name: vsphere-test-pod
spec:
  containers:
  - name: test-container
    image: busybox
    command: [ "sleep", "3600" ]
    volumeMounts:
    - mountPath: "/data"
      name: my-vsphere-vol
  volumes:
  - name: my-vsphere-vol
    persistentVolumeClaim:
      claimName: test-vsphere-pvc

You should see the container volume appear on the vSphere Monitor Dashboard.