Managing Application Environment Configuration with Kubernetes

Over the past several months we’ve been in the process of moving all of our sites into an AWS-hosted kubernetes cluster. To anyone who’s tried to admin a kubernetes cluster, it will come as no surprise that it’s been a learning experience.

One of the things that has been a bit mystifying to me until recently is the best way to manage application configuration via shell environment variables. Kubernetes provides a few ways to handle it, but I’m going to explain the way I’ve landed on handling it. This is what feels the most straightforward to me.

For most applications, you’ll use a Deployment to manage your application Pods. A Deployment is a Kubernetes object that manages a collection of Pods which are replicas of one another. A Pod manages a set of containers as a single discrete virtual host. A simple Deployment’s spec defines a number of replica Pods to run and a template for building them. There are additional configuration options available for supporting more complex deployment scenarios. You can configure a Deployment to load data into its Pods’ shell environment from ConfigMaps and Secrets.

Using ConfigMaps

A ConfigMap is a type of Kuberenetes object meant for storing arbitrary configuration values. Values in a ConfigMap are stored as plain text. A ConfigMap can be mounted into a Pod in a number of ways, but in this example we’ll load them directly into the Pod’s environment. Creating a ConfigMap is simple using kubectl (see kubectl create configmap --help for more examples):

$ kubectl create configmap example-configmap --from-literal=exampleKey=exampleVal
configmap/example-configmap created

$ kubectl get configmap example-configmap
apiVersion: v1
data:
  exampleKey: exampleVal
kind: ConfigMap
metadata:
  name: example-configmap
  namespace: default
<output truncated>

# for more examples, etc
$ kubectl create configmap --help

A Secret is functionally similar to a ConfigMap in most ways. The main differentiating factor is that data stored in a secret is base64 encoded.

It’s worth noting that encoding is not the same and encrypting, though kubernetes can be configured to encrypt stored secrets at rest. These features are meant to add enhanced privacy and security.

Using Secrets

Creating secrets works similarly to creating ConfigMaps:

$ kubectl create secret generic example-secret --from-literal=exampleKey=exampleVal
configmap/example-secret created

$ kubectl get secret example-secret -o yaml
apiVersion: v1
data:
  exampleKey: ZXhhbXBsZVZhbA== # see it's base64 encoded
kind: Secret
metadata:
  name: example-secret
  namespace: default
  <more output redacted>
type: Opaque

# for more examples, etc
$ kubectl create secret --help

Now that you understand how ConfigMaps and Secrets work, I’ll explain how to use them to set environment variables for your applications within their Pods.

You can choose to store your application config in Secrets, ConfigMaps, or both at the same time. It should be fairly obvious, but because of the added security and privacy features of Secrets, they’re best to use for passwords and other info that you’d prefer to keep under wraps. All other data can be stored in ConfigMaps. For simplicity you could choose to store all your data in Secrets, but dealing with the encoding can be tedious.

Configuring Deployments

Configuring your Deployment to use your ConfigMaps and Secrets is easy by following this example:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: env-example
  labels:
    app: env-example
spec:
  replicas: 1
  selector:
    matchLabels:
      app: env-example
  template:
    metadata:
      labels:
        app: env-example
    spec:
      containers:
        - name: app
          image: nginx
          ports:
            - containerPort: 80
          envFrom:
          - configMapRef:
              name: example-configmap
          - secretRef:
              name: example-secrets

Here we build an nginx deployment that mounts the key values in example-configmap and example-secretwhich are respectively a ConfigMap and a Secret that we created in the earlier. Any value stored in one of those objects will be available in the Pod’s shell environment.

The final caveat is that changing the data in your Secrets or ConfigMaps won’t automatically change the shell environment in the Pods that mount them.

After you change Secret or ConfigMap data, you’ll need to roll out any Pods that are mounting them. You can do this simply by deleting them one at a time (be careful, if you delete them all at once you’ll take your application down). Alternatively, instead of adding data to your existing Secrets or ConfigMaps, you can copy the old ones and add the new data to the copies, then change the Deployment configuration to use the new copies. This will allow you to utilize Deployments to manage the Pod updates and also allow you to keep backups of known good environment configurations.

These are simply the workflows that work the best for us. Kubernetes is very flexible and there are some other ways to manage environment configuration, so let us know what you’re doing in the comments!

Kubernetes is pretty new so we’re always excited to hear how other teams are using it!

More Posts by Adam Clarke: