Immutable ConfigMaps and Secrets in Kubernetes: A Complete Guide

ConfigMaps and Secrets are resources in Kubernetes that are used to govern configuration for applications. ConfigMaps maintain non-sensitive data whereas Secrets hold sensitive data like credentials and passwords.

In Kubernetes 1.21, a feature for Immutability of ConfigMaps and Secrets was GA. What exactly is this concept and its purpose? This article has the answers.

Concept

Kubernetes 1.21 was released back in 2021, and it provided an option to mark ConfigMaps and Secrets as immutable. This meant that once created, the data in these objects could not be altered.

Purpose

Some would question the need for this feature. Since ConfigMaps and Secrets deal with configuration, and configuration is subject to change, why would we need such a feature?

Performance

It is important to understand that the kubelet on each node watches over each ConfigMap and Secret and periodically syncs its information to the Pods via the kube-apiserver. Large scale production deployments can have tens of thousands of such ConfigMaps and Secrets and obviously there is a cost associated with continuously keeping a watch for changes.

ConfigMaps and Secrets containing data that is constant over time or seldom changes can be marked as immutable. The kubelet does not watch over these resources. This in turn reduces the load on the kube-apiserver, saves compute resources and effectively improves the performance of the cluster.

Security and Reliability

As the data for immutable ConfigMaps and Secrets cannot be altered, it can protect the sensitive and critical configuration from accidental (or unwanted) updates that could cause application outages.

It provides an additional layer of security against malicious actors attempting to manipulate data.

Consistency

Immutability is ideal for scenarios where you want to ensure a consistent configuration across multiple Pods and Deployments.

Definition

Immutable ConfigMaps and Secrets are defined by setting the immutable field to true.

apiVersion: v1
kind: ConfigMap
metadata:
  ...
data:
  ...
immutable: true

apiVersion: v1
kind: Secret
metadata: 
  ...
data: 
  ...
immutable: true

Example

Let's check an example of an Immutable ConfigMap that is mounted as a Volume in a Pod.

We will also demonstrate how to update configuration driven by an Immutable ConfigMap.

WARNING: Immutable ConfigMaps and Secrets are especially used for configuration which is constant and is not expected to change over time. In a production use-case, it is advisable to not change the contents of Immutable ConfigMaps and Secrets unless absolutely necessary.

Prerequisites

1. A local Kubernetes cluster such as minikube, kind, K3s or microk8s.

2. The kubectl command line tool to interact with the Kubernetes cluster.

An example manifest for an Immutable ConfigMap is shown below:

apiVersion: v1
data:
  season: "summer"
kind: ConfigMap
immutable: true
metadata:
  name: season

Create the Immutable ConfigMap:

kubectl apply -f https://raw.githubusercontent.com/adityasamant25/courses/main/kubernetes/configmaps/examples/immutable-configmaps.yaml

Below is an example of a Deployment manifest with the Immutable ConfigMap season mounted as a volume into the Pod's only container:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: immutable-configmap-volume
  labels:
    app.kubernetes.io/name: immutable-configmap-volume
spec:
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/name: immutable-configmap-volume
  template:
    metadata:
      labels:
        app.kubernetes.io/name: immutable-configmap-volume
    spec:
      containers:
        - name: alpine
          image: alpine:3
          command:
            - /bin/sh
            - -c
            - while true; do echo "$(date) My preferred season is $(cat /etc/config/season)";
              sleep 10; done;
          ports:
            - containerPort: 80
          volumeMounts:
            - name: config-volume
              mountPath: /etc/config
      volumes:
        - name: config-volume
          configMap:
            name: season

Create the Deployment:

kubectl apply -f https://raw.githubusercontent.com/adityasamant25/courses/main/kubernetes/configmaps/examples/deployment-with-immutable-configmap-as-volume.yaml

Check the Deployment:

kubectl get deployment immutable-configmap-volume

You should see an output similar to:

NAME               READY   UP-TO-DATE   AVAILABLE   AGE
configmap-volume   3/3     3            3           19s

Check the pods for this Deployment (matching by selector):

kubectl get pods --selector=app.kubernetes.io/name=immutable-configmap-volume

You should see an output similar to:

NAME                                          READY   STATUS    RESTARTS   AGE
immutable-configmap-volume-5c677f6494-n7cg5   1/1     Running   0          30s
immutable-configmap-volume-5c677f6494-sdhkp   1/1     Running   0          30s
immutable-configmap-volume-5c677f6494-vsszf   1/1     Running   0          30s

The Pod's container refers to the data defined in the ConfigMap and uses it to print a report to stdout. You can check this report by viewing the logs for one of the Pods in that Deployment:

# Pick one Pod that belongs to the Deployment, and view its logs
kubectl logs deployments/immutable-configmap-volume

You should see an output similar to:

Found 3 pods, using pod/immutable-configmap-volume-5c677f6494-n7cg5
Tue Mar  5 11:38:27 UTC 2024 My preferred season is summer
Tue Mar  5 11:38:37 UTC 2024 My preferred season is summer
Tue Mar  5 11:38:47 UTC 2024 My preferred season is summer
Tue Mar  5 11:38:57 UTC 2024 My preferred season is summer
Tue Mar  5 11:39:07 UTC 2024 My preferred season is summer

Edit the ConfigMap:

kubectl edit configmap season

In the editor that appears, change the value of key season from summer to winter.

Here's an example of how that manifest could look after you edit it:

apiVersion: v1
data:
  season: winter
immutable: true
kind: ConfigMap
# You can leave the existing metadata as they are.
# The values you'll see won't exactly match these.
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"season":"summer"},"immutable":true,"kind":"ConfigMap","metadata":{"annotations":{},"name":"season","namespace":"default"}}      
  creationTimestamp: "2024-03-05T11:36:22Z"
  name: season
  namespace: default
  resourceVersion: "3840"
  uid: 10cdc5e6-47b8-4473-a9f3-ed401c2073f9

Attempt to save the changes. You should see that the save fails and a temporary manifest is opened which contains the reason for failure:

# reopened with the relevant failures.
#
# configmaps "season" was not valid:
# * data: Forbidden: field is immutable when `immutable` is set
#

Without any further change, save the temporary manifest. You should see an output similar to:

A copy of your changes has been stored to "/var/folders/nn/bb6y1j8d69n9lhqppjlrfthm0000gn/T/kubectl-edit-139284375.yaml"
error: Edit cancelled, no valid changes were saved.

Once a ConfigMap is marked as immutable, it is not possible to revert this change nor to mutate the contents of the data or the binaryData field. In order to modify the contents, the only option is to delete and recreate the ConfigMap.

Delete and recreate the ConfigMap season by using the temporary manifest as follows:

# Use the appropriate path to the temporary manifest
kubectl replace --force -f /var/folders/nn/bb6y1j8d69n9lhqppjlrfthm0000gn/T/kubectl-edit-139284375.yaml

Tail (follow the latest entries in) the logs of one of the pods that belongs to this Deployment:

# As the text explains, the output does NOT change
kubectl logs deployments/immutable-configmap-volume --follow

Notice that the output remains unchanged, even though you recreated the ConfigMap:

Tue Mar  5 12:02:07 UTC 2024 My preferred season is summer
Tue Mar  5 12:02:17 UTC 2024 My preferred season is summer
Tue Mar  5 12:02:27 UTC 2024 My preferred season is summer
Tue Mar  5 12:02:37 UTC 2024 My preferred season is summer

CAUTION:Although the ConfigMap has been recreated, the existing Pods maintain a mount point to the deleted ConfigMap. To force an update, you would need to clean up every existing reference to the old ConfigMap. In this scenario, it equates to deleting the Deployment and recreating it. Exercise caution before executing these steps in a production environment.

A simple restart of the existing Deployment using the kubectl rollout restart command causes the Pods to be recreated, but is insufficient to refresh the contents of the ConfigMap.

Delete the Deployment:

kubectl delete deployment immutable-configmap-volume

Wait for all the Pods to terminate. Verify that the Pods have terminated using the following command:

kubectl get pods --selector=app.kubernetes.io/name=immutable-configmap-volume

You should see an output similar to:

No resources found in default namespace.

Recreate the Deployment:

kubectl apply -f https://k8s.io/examples/deployments/deployment-with-immutable-configmap-as-volume.yaml

Next, check the Deployment:

kubectl get deployment immutable-configmap-volume

You should see an output similar to:

NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
immutable-configmap-volume   3/3     3            3           6s

Check the Pods:

kubectl get pods --selector=app.kubernetes.io/name=immutable-configmap-volume

You should see an output similar to:

NAME                                          READY   STATUS    RESTARTS   AGE
immutable-configmap-volume-5c677f6494-dxjwg   1/1     Running   0          31s
immutable-configmap-volume-5c677f6494-k54fc   1/1     Running   0          31s
immutable-configmap-volume-5c677f6494-w9b4c   1/1     Running   0          31s

View the logs for a Pod in this Deployment:

# Pick one Pod that belongs to the Deployment, and view its logs
kubectl logs deployment/immutable-configmap-volume

You should see an output similar to the below:

Found 3 pods, using pod/immutable-configmap-volume-5c677f6494-dxjwg
Tue Mar  5 12:40:43 UTC 2024 My preferred season is winter
Tue Mar  5 12:40:53 UTC 2024 My preferred season is winter
Tue Mar  5 12:41:03 UTC 2024 My preferred season is winter

Summary

Once a ConfigMap is marked as immutable, it is not possible to revert this change nor to mutate the contents of the data or the binaryData field. You can only delete and recreate the ConfigMap. Because existing resources maintain a mount point to the deleted ConfigMap, it is necessary to ensure that every existing reference to the old ConfigMap (not just Pods, any reference) has been cleaned up.

Cleaning up

Delete the resources created during the tutorial:

kubectl delete deployment immutable-configmap-volume
kubectl delete configmap season