It's always CRITICAL to pack a Container image with the minimal binaries required as this makes the surface area of attack minimal, upgrading the image and testing also becomes easier as there are less variables to be addressed. Distroless Docker images can be used for the same. In the above diagram Container (A) has only the application and the dependent binaries and nothing more. So, if there are no debugging tools in the Container (A) nor any way to check the status of the process then how do we debug any problem in the application? Once a pod is created, it's even not possible to add Containers to it for additional debugging tools.
That's where the Ephemeral Containers come into picture as in the Container (B) in the above picture. These Containers are temporary that can be included in the Pod dynamically with additional debugging tools. Once a Ephemeral Container has been created, we can connect to it as usual using the kubectl attach, kubectl exec and kubectl logs commands.
Here are the steps for creating an Ephemeral Container. The assumption is that kubeadm has been used to create the K8S Cluster and the Cluster is using K8S 1.18 version or newer which is the latest as of this writing.
Step 1) The Ephemeral Containers feature has to be enabled before creating it. More about enabling K8S features here.
Edit the below files to add '- --feature-gates=EphemeralContainers=true' in the command section. there is no need to restart the Pods as the kubelet process continuously monitors these files and restarts the Pods automatically. Run the 'kubectl get pods --all-namespaces' and notice the changes to the AGE column.
/etc/kubernetes/manifests/kube-apiserver.yaml
/etc/kubernetes/manifests/kube-scheduler.yaml
/etc/kubernetes/manifests/kube-scheduler.yaml
Step 2) Edit the '/etc/systemd/system/kubelet.service.d/10-kubeadm.conf' to pass the '--feature-gates=EphemeralContainers=true' parameter to kubelet as shown below.
ExecStart=/usr/bin/kubelet --feature-gates=EphemeralContainers=true $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
Step 3) Restart the kubelet service for the above changes to take place into effect.
sudo systemctl
daemon-reload
sudo systemctl
restart kubelet
Note that Step 2 and 3 have to be run on all the Control Plane Nodes and the Worker Nodes.
Step 4) Now that the feature has been enabled, lets create a Pod with an application. For sake of simplicity, a pause Pod is created using the below command.
kubectl run ephemeral-demo --image=k8s.gcr.io/pause --restart=Never
There is no shell or any other debugging tools in this container, so the below command will fail.
kubectl exec -it ephemeral-demo -- sh
Step 5) Let's start an Ephemeral Container with the below command. Note that this particular feature is in alpha release of K8S, so the parameter alpha in the kubectl command. Once the feature graduates to stable, we don't need this any more. The below command opens a terminal to the Ephemeral Container and from where we can start debugging the main application.
kubectl alpha debug -it ephemeral-demo --image=busybox --target=ephemeral-demo
Step 6) Open a new terminal and run the below command to get the details of the Ephemeral Container.
kubectl describe pod ephemeral-demo
