Tuesday, March 31, 2020

Debugging K8S applications with Ephemeral Containers

It's always CRITICAL to pack a Container image with the minimal binaries required as this makes the surface area of attack minimal, upgrading the image and testing also becomes easier as there are less variables to be addressed. Distroless Docker images can be used for the same. In the above diagram Container (A) has only the application and the dependent binaries and nothing more. So, if there are no debugging tools in the Container (A) nor any way to check the status of the process then how do we debug any problem in the application? Once a pod is created, it's even not possible to add Containers to it for additional debugging tools.

That's where the Ephemeral Containers come into picture as in the Container (B) in the above picture. These Containers are temporary that can be included in the Pod dynamically with additional debugging tools. Once a Ephemeral Container has been created, we can connect to it as usual using the kubectl attach, kubectl exec and kubectl logs commands.

Here are the steps for creating an Ephemeral Container. The assumption is that kubeadm has been used to create the K8S Cluster and the Cluster is using K8S 1.18 version or newer which is the latest as of this writing.

Step 1) The Ephemeral Containers feature has to be enabled before creating it. More about enabling K8S features here.

Edit the below files to add '- --feature-gates=EphemeralContainers=true' in the command section. there is no need to restart the Pods as the kubelet process continuously monitors these files and restarts the Pods automatically. Run the 'kubectl get pods --all-namespaces' and notice the changes to the AGE column.

/etc/kubernetes/manifests/kube-apiserver.yaml
/etc/kubernetes/manifests/kube-scheduler.yaml

Step 2) Edit the '/etc/systemd/system/kubelet.service.d/10-kubeadm.conf' to pass the  '--feature-gates=EphemeralContainers=true' parameter to kubelet as shown below.

ExecStart=/usr/bin/kubelet --feature-gates=EphemeralContainers=true $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

Step 3) Restart the kubelet service for the above changes to take place into effect.

sudo systemctl daemon-reload
sudo systemctl restart kubelet

Note that Step 2 and 3 have to be run on all the Control Plane Nodes and the Worker Nodes.

Step 4) Now that the feature has been enabled, lets create a Pod with an application. For sake of simplicity, a pause Pod is created using the below command.

kubectl run ephemeral-demo --image=k8s.gcr.io/pause --restart=Never

There is no shell or any other debugging tools in this container, so the below command will fail.

kubectl exec -it ephemeral-demo -- sh

Step 5) Let's start an Ephemeral Container with the below command. Note that this particular feature is in alpha release of K8S, so the parameter alpha in the kubectl command. Once the feature graduates to stable, we don't need this any more. The below command opens a terminal to the Ephemeral Container and from where we can start debugging the main application.

kubectl alpha debug -it ephemeral-demo --image=busybox --target=ephemeral-demo

Step 6) Open a new terminal and run the below command to get the details of the Ephemeral Container.

kubectl describe pod ephemeral-demo

Hope you had fun learning about Ephemeral Containers, more about here and here.