Tuesday, May 19, 2020

Accessing private resources using AWS Client VPN

AWS VPC supports creating public and private subnets. Any EC2 in the public subnet will have public and private IP and so front end or customer facing applications like web applications are installed on this EC2. Also, this EC2 can be reached from outside the Cloud using the public IP for any maintenance.

The private subnet EC2 will have only private IP and no public IP, backend applications like databases ore installed on this EC2. This EC2 can't be reached directly from outside the Cloud as it doesn't have any public IP. So, how do we connect to it for activities like updating databases, creating tables etc? This is where Jump box or the Bastion box comes into play.

To connect to the EC2 in the private subnet we need to launch a Bastion box in the public subnet, connect to it and from there connect to the EC2 in the private subnet. This corresponds to step (1) and (2) in the below diagram. Step (2) pretty much like connecting to remote server, but in this case both the EC2s are in the same

The problem with the above approach is that the Bastion box does have an public IP and there is a probability of someone trying to access it over the public IP. We can get rid of the Bastion Box and use AWS Client VPN to connect the EC2 in the private subnet although the EC2 doesn't have an public IP as shown below.


I have created a new VPC with public and private subnet using the VPC wizard. And then created an EC2 in the private subnet. Note that there is no public IP and just the private IP. But, still we want to connect to it for doing the maintenance work on the EC2.

I followed this blog and created a VPN Endpoint and connected to the EC2 in the private subnet. Note the AWS Client VPN on the left side and connection to the EC2 from my Laptop via the Putty on the right side. Notice that the IP address in the Putty matches with the private IP address of the EC2 in the above screen.

Friday, May 8, 2020

Sticking to the AWS free tier and building K8S cluster using kops

In one of the previous blog we have seen how to setup K8S on AWS using t2.micro EC2 instance and MicroK8S. This falls under the AWS free tier. The same can be done with kops also. This blog is all about the same. Again we will try to stick to the free tier.

The first step is create an Ubuntu EC2 instance and install kops and AWS CLI on it. THen on this EC2 we would be running a bunch of commands to create and tear down the K8S cluster as shown below. The steps mentioned in this blog would be creating a K8S cluster of 1 master and 2 slaves and this would allow to try how K8S behaves for worker failures and also scheduling across multiple workers. But, this configuration requires a total of 34 GB of EBS volumes out of which 30GB is covered within the AWS free tier, rest need to be paid for. To strictly fall under the free tier only one worker node can also be created, but we won't be able to try out the different K8S features.

Once the cluster has been created, we can see the nodes in the ready state as shown below.

Steps for setting up the K8S on EC2 using kops

Step 1: Create an Ubuntu 18.04 EC2 instance (t2.micro) and connect to it via Putty and execute the below commands.

-- Install kubectl and Python3, AWS CLI and kubectl.

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update

sudo apt-get install -y python3-pip apt-transport-https kubectl

pip3 install awscli --upgrade

export PATH="$PATH:/home/ubuntu/.local/bin/"

Step 2: Install kops (used for installing K8S on AWS)

curl -LO https://github.com/kubernetes/kops/releases/download/$(curl -s https://api.github.com/repos/kubernetes/kops/releases/latest | grep tag_name | cut -d '"' -f 4)/kops-linux-amd64

chmod +x kops-linux-amd64

sudo mv kops-linux-amd64 /usr/local/bin/kops

Step 3: Check if aws, kops and kubectl commands are there in the path or not

Step 4: Create a AWS IAM User (with API, SDK access) with the below policy and get the Access Keys for this user.

-- AmazonEC2FullAccess
-- AmazonS3FullAccess
-- IAMFullAccess
-- AmazonVPCFullAccess

Step 5: Configure the Access Keys and AWS Region on the EC2 instance using the `aws configure` command. Use us-east-1 or some other region. For the Output format, leave it as empty.

Step 6: Export the keys. Note to replace the keys after the $ symbol in the below commands.

export AWS_ACCESS_KEY_ID=$(aws configure get aws_access_key_id)
export AWS_SECRET_ACCESS_KEY=$(aws configure get aws_secret_access_key)

Step 7: Generate the ssh keys

ssh-keygen -f .ssh/id_rsa

Step 8: Export the name of the cluster and the S3 bucket name. Make to sure it ends with k8s.local to use the gossip protocol or else we need to work with Route53.

export NAME=myfirstcluster.k8s.local
export KOPS_STATE_STORE=s3://praveen-kops-cluster

Step 9: Create S3 bucket

aws s3api create-bucket --bucket praveen-kops-cluster --region us-east-1

Step 10: Create a cluster configuration. Note that the cluster is not built. Also, as mentioned earlier we need to pay for the additional 4GB of EBS volume. To make sure we fall under the free tier, specify the node-count as 1, to create one worker node.

kops create cluster --name=$NAME --state=$KOPS_STATE_STORE --zones=us-east-1a --node-count=2 --node-size=t2.micro --master-size=t2.micro  --master-volume-size=8 --node-volume-size=8

Step 11: Change the EBS volume size for etcd main and events attached to the master. By default multiple 20GB EBS volumes are created and the total storage doesn't fall under the free tier (1). Edit the cluster configuration by executing the below command.

kops edit cluster myfirstcluster.k8s.local

and add the below after "name: a" in two different locations.

volumeType: gp2
volumeSize: 1

Step 12: Build the cluster

kops update cluster $NAME --yes

Step 13: Check the status of the cluster. The cluster and the related resources will take around a few minutes for it to be created. Also execute `kubectl get nodes` to get the status of the cluster.

kops validate cluster $NAME

Step 14: Create deployments on the cluster which we created. Create a file deployment.yaml with the contents from K8S documentation here.

-- Create a deployment using the below command.

kubectl apply -f deployment.yaml

-- Once the deployment has been created, check the status of the pods using the below command.

kubectl get pods

-- Delete the deloyment

kubectl delete -f deployment.yaml

Step 15: Delete the cluster. Again the cluster deletion will take a few minutes.

kops delete cluster --name $NAME --yes

Step 16: Delete the s3 bucket.

aws s3api delete-bucket --bucket praveen-kops-cluster --region us-east-1

Step 17: Terminate the Ubuntu EC2 instance which was created manually earlier.

Step 18: Delete the IAM User


Following the above steps a K8S cluster can be created within a few minutes on AWS and the good thing is that it falls within the AWS free tier. This is not production ready, but is good enough to get started.

One more thing to note, with the kops installation we have complete control over the K8S cluster. But, the user has to manage the master and the worker nodes like upgrading, patching, securing K8S on them. And so, the Cloud vendors provide different managed K8S installations like AWS EKS, GCP GKE and Azure AKS.

Friday, April 24, 2020

How K8S authentication and authorization works in kubeadm?

To get started and practice a few things around K8S, I have setup a K8S cluster on my laptop as shown below using kubeadm. It gives me the utmost flexibility to try different things around K8S. A few days back I have posted a blog on how a K8S cluster gets bootstrapped. This gives an idea on how the different K8S process start and then we would be able to change some of the K8S process startup parameters to try out new features in K8S.

In the above K8S cluster, the default user (kubernetes-admin) created during the cluster setup has admin privileges to the cluster. This time I was curious on how the authentication and authorization work in K8S for this user to have full access to the cluster. This will enable me to be create additional users with different privileges, authentication and authorization mechanisms. It took me some time to get my mind/thoughts around it, but it's all interesting. This blog is all about the same.

Note that the cluster has been setup using kubeadm, for kops and other the below varies a little bit. And also, kubeadm cluster setup default used X509 certificates for authentication. Authentication Providers are not built into K8S and so has to be integrated with external systems like Google Accounts, Active Directory, LDAP etc.

1) By default with kubeadm installation, the user and cluster details are stored in the .kube/config file on the client side. This file helps kubectl command to connect and authenticate to the K8S cluster. The client-certificate-data and client-key-data fields in this file are used for authentication.

2) The X509 certificate from the .kube/config can be extracted and encoded. Finally the user details can be extracted with the opensll using the below commands. Note the output of the openssl command, the user name (CN) is kubernetes-admin and belongs to the system:masters organization (O).

grep "client-certificate-data" config | sed 's/    client-certificate-data: //g' > encoded.txt

base64 --decode encoded.txt > kubernetes-admin.crt

openssl x509 -noout -subject -in kubernetes-admin.crt

3) Now from a K8S cluster perspective. The cluster-admin (a ClusterRoleBinding object) binds the cluster-admin (a ClusterRole object with admin privilages) to the system:masters group. Any user of this group will have full cluster access.

kubectl describe clusterrolebindings.rbac.authorization.k8s.io cluster-admin

kubectl describe clusterrole cluster-admin

4) So, kubernetes-admin which belongs to the system:masters group as seen in the output of the openssl command has full access to the cluster.


K8S with the different concepts is a bit intimidating and it takes a bit of digging and research into the concepts to get a good understanding of the concepts. So, below is the mindmap for the above concept.

Note that (7 in the above diagram) maps the user details the organization name (system-masters) in the certificate with the group name (system-masters) in the ClusterRoleBinding. This is what gives the kubernetes-admin user full access to the K8S cluster.

As mentioned earlier, any user part of the system-masters group will have full access to the cluster.This Bitnami article (1) talks about the adding K8S user called 'employee' with X509 certificates. The same with some tweaks can be used to create another user besides the kubernetes-admin part of the system:master group.

Friday, April 10, 2020

MicroK8S - Easiest way to get started with K8S for those familiar with AWS

AWS provides different ways of running K8S in the Cloud via EKS, ECS and with/without Fargate. These are production ready setups for deploying K8S applications. In this blog we would look at microk8s, one of the most easiest way to get started with K8S. Here the assumption is that you already know how to create an EC2 instance and login to it.

Steps for setting up the K8S on EC2 using microk8s

-- Create an Ubuntu EC2 t2.micro instance with the default Security Groups. This Security Group allows all the traffic inbound and outbound. This is OK non-production purpose. Usually depending on the core services, add-ons installed the appropriate ports have to be opened in the Security Group.

-- Login to the EC2 via Putty or some other SSH client and execute the below commands.

#Install the microk8s. The latest release can be got from here.
sudo snap install microk8s --classic --channel=1.18/stable

#Create an alias in the .bashrc file
alias kubectl='microk8s kubectl'

#Add the user part of the microk8s group and change the ownership of the ~/.kube
#Exit and login back into the EC2 for these changes to take place into effect
sudo usermod -a -G microk8s $USER
sudo chown -f -R $USER ~/.kube

#check the status of the microk8s installation
microk8s status --wait-ready

-- Create a file deployment.yaml with the below content. Run 'kubectl create -f deployment.yaml' to create a Deployment with 3 Pods with the nginx Containers. BTW, kubectl is the CLI for interacting with K8S.

apiVersion: apps/v1
kind: Deployment
  name: nginx-deployment
    app: nginx
  replicas: 3
      app: nginx
        app: nginx
      - name: nginx
        image: nginx:1.14.2
        - containerPort: 80

-- 'kubectl get deployments' and 'kubectl get pods' will give the list of deployments and pods created in the above step.

--Now lets try to install the K8S Dashboard, which is a graphical way of interfacing with K8S. Execute the below commands in the EC2.

#Enable the K8S Dashboard
microk8s enable dashboard

#Get the token for the default user
token=$(kubectl -n kube-system get secret | grep default-token | cut -d " " -f1)
kubectl -n kube-system describe secret $token

#Port forward  the K8S Dashboard port 443 to port 10443 onto the EC2.
microk8s kubectl port-forward -n kube-system service/kubernetes-dashboard 10443:443 --address

-- Now the K8S Dashboard is ready to be accessed.

#Open the below URL in the browser. Note to replace the ec2-ip with the Public IP of the EC2.
#Provide the token got from the previous step
#The Deployments and Pods created earlier can be seen.


In the above setup a t2.micro instance was used which provides 1 vCPU and 1 GB of RAM. The only reason for using t2.micro is because it falls under the AWS Free Tier. Once the Deployment and Pods are started, the CPU is reaching the peak while some memory left is as shown below. With the t2.micro I have noticed that microk8s behaves a bit slow especially the Dashboard. Try to pickup a bigger instance for trying out various features of K8S.

The above steps are for creating a single node K8S cluster, multi node K8S cluster is also possible with microk8s as mentioned in the documentation here.

Setting up K8S with proper configuration especially for the Production setup is not an easy task. But, microk8s, K3S and Minikube have made it easy to get started with K8S.

Wednesday, April 8, 2020

How does the K8S cluster gets bootstrapped?

Although the different Cloud vendors provide managed service like AWS EKS, GCP GKE, Azure AKS and others, nothing beats running K8S on the local machine. Not only we can get started quickly, it's absolutely free and provides the ultimate freedom for any experimentation. Here is the setup I have using Oracle VirtualBox on Windows 10 OS.

It's a 3 node K8S cluster with one Control Plane node to do the orchestration and scheduling of the containers and two Worker nodes for the execution of the the containers. Recently, I upgraded to the latest version of K8S (1.18 as of this writing) and was able to try some of the new features.

The command 'kubectl get pods --all-namespaces -o wide' gives the Pods in all the namespaces. Below is the screenshot listing the Pods on the Control Plane and the Workers node. I was curious on how the Pods get started, this will give us a chance to check out the initialization parameters, tweak them and also to enable/disable features. This blog is all about the same. Note that the instructions are specific to installation using kubeadm and differ a bit for other installation process.

I was poking around in the K8S cluster and this StackOverflow Query (1) helped me. Below is the workflow on the how the K8S cluster gets bootstrapped. It all starts with the kubelet getting started automatically as a systemd service on all the Control Plane and the Workers nodes, which starts the minimum required static Pods for K8S to start working. Once the K8S Cluster boots up, additional Pods are started to get the K8S cluster into the desired state as stored in the etcd database.

Here is the workflow in a bit more detail

- Below is how the kubelet gets started as a systemd Service. /etc/systemd/system/kubelet.service.d/10-kubeadm.conf has the command to start the kubelet process and also the initialization parameters. Note that one of the parameter is /var/lib/kubelet/config.yaml location.

- In the /var/lib/kubelet/config.yaml file the staticPodPath variable is set to /etc/kubernetes/manifests path which has the yaml files for the static Pods to be started once the kubelet starts. As of now the apiserver, scheduler and etcd haven't started yet. So, kubelet starts them and manages them. Although these Pods are visible to the apiserver later, the apiserver doesn't manage them, kubelet is the one which manages them.

- In the /etc/kubernetes/manifests folder we have the yaml definitions for the etcd, apiserver, contoller-manager and the scheduler. These files will help us to understand how the K8S system Pods are initialized.

- OK, how about the coredns, flannel and proxy Pods getting started? The coredns and the proxy Pods are created by kubeadm during the K8S cluster initialization phase (1, 2) using kubeadm. The flannel Pods were created by me manually while setting up the K8S cluster (1). The details of these are stored in the etcd database as with any other user created K8S objects and K8S will automatically start them once the cluster starts.

Mystery solved, now that we know on how the K8S cluster gets bootstrapped, more can be explored. For using and administering K8S it is required to know the K8S bootstrap process in a bit more detail.

Tuesday, April 7, 2020

How does Key Pair work behind the scenes for Linux EC2 authentication?

Different  ways of authenticating against Linux EC2

Once a Linux EC2 instance has been created, the same can be accessed via Putty or some other SSH client. To access the EC2, first we need to authenticate the user. Either the Username/Password or the KeyPair can be used for authentication. There are pros and cons of each of them. AWS has chosen to go with the KeyPair way of authentication by default. If required this can be disabled and the Username/Password can be enabled.

How does KeyPair work behind the scenes for Linux EC2 authentication?

A picture conveys more than words, so is the above workflow. A KeyPair consists of a Private Key and a Public Key. The Private Key goes onto the Laptop and the Public Key automatically goes into the EC2 instance. Go through the above workflow to get to know what happens behind the scenes.

Note that the Private Key never leaves the laptop, this is one of the advantages of using the Key Pairs. Also, the way we never share the passwords with anyone, we should never share the Private Key with anyone. This would allow them to access the EC2. Also, the way we never use the same password across multiple services, never we should use the same Key Pair across multiple EC2 instances for the obvious reasons.

Also, it's always better to create a different set of Key Pairs for multiple users accessing the same EC2 instance, very similar to different passwords. Here is a small write-up from the AWS site on the same (1).

>> If you have several users that require access to a single instance, you can add user accounts to your instance. For more information, see Managing User Accounts on Your Linux Instance. You can create a key pair for each user, and add the public key information from each key pair to the .ssh/authorized_keys file for each user on your instance. You can then distribute the private key files to your users. That way, you do not have to distribute the same private key file that's used for the AWS account root user to multiple users.

Here are a few articles on some of the regularly performed tasks around the combination of EC2 and Key Pairs.

1) How do I enable a password login instead of a key pair when logging into my EC2 instance using SSH? (1, 2)

2) How do I add new user accounts with SSH access to my Amazon EC2 Linux instance? (1, 2)

3) Connecting to your Linux instance if you lose your private key (1)

4) Rotating the Key Pairs (1) - Note that there are many better ways, but this is the easiest.

There is lot more to Key Pairs and how the authentication works, but this article gives a basic gist on what happens BTS (behind the scenes) when we use KeyPairs to access an EC2 instance. I like to keep things clear and simple, this helps me to get my concepts clear and express myself better.

Monday, April 6, 2020

AWS AMI vs Launch Templates

Very often I get asked about the differences between AMI (Amazon Machine Images) and EC2 Launch Templates, although both of them have different purpose. This blog is about getting these concepts clear.

What is AMI?

There might be a requirement where we need to install softwares and applications on hundreds of EC2s. It's not practically possible to login to each of the EC2 and perform the task as it is time consuming and also prone to errors. We can automate these tasks using AMIs. Below is the workflow to start working with the AMIs.

Once an EC2 has been created, the appropriate software/applications have to be installed along with the configurations and then an AMI has to be created. The AMI has to OS and all the software/applications on top of it. The AMI usually is a couple of GB, based on the original EC2 which was created and is immutable (no changes can be made to it).

With this AMI, additional EC2s can be created and each one of the EC2 will have the original software automatically installed as in the case of Apache2. This makes getting the EC2 ready much easier/quicker.

Different ways of launching EC2 Instance

AWS provides multiple ways of launching EC2 instances via the SDK, CLI, CloudFormation and the EC2 Management Console (Web UI). From the EC2 Management Console again there are two ways, one is is using the Launch Instance wizard and other is via the EC2 Launch Templates as shown below.

Both these approaches lead to launching an EC2 and takes the required parameters for the same like the AMI which was mentioned above, the EBS volume size/type, SecurityGroup, KeyPair to be used and a few other details.

In the EC2 creation via the Wizard approach we need to select each and every time the AMI, Instance Type, Network Settings, Storage, Security Groups, pricing model etc from the different options. In the below screen the AMI has to be selected from the available ones. This is OK when we launch an EC2 a few times, but it's more of a routine task and is also time consuming.

This is where the EC2 Launch Templates come to the rescue. We can create a template and predefine all the EC2 properties and reuse the same to launch an EC2 instance. This way we don't need to select the EC2 properties again and again. Below is the template with the AMI, instance type, KeyPair, SecurityGroup predefined. Using this template we should be able to create an EC2 instance using Option (6).

CloudFormation Templates vs EC2 Launch Templates

While the EC2 Launch Templates can be used for the automation of the EC2 Instance creation. CloudFormation Templates are much more than that. It's possible to create many of the AWS resources via the CloudFormation Templates and connect them together, watch the drift (changes to the AWS resources) and much more. Here is the list of AWS Resources that can be created by CloudFormation.

Tuesday, March 31, 2020

Debugging K8S applications with Ephemeral Containers

It's always CRITICAL to pack a Container image with the minimal binaries required as this makes the surface area of attack minimal, upgrading the image and testing also becomes easier as there are less variables to be addressed. Distroless Docker images can be used for the same. In the above diagram Container (A) has only the application and the dependent binaries and nothing more. So, if there are no debugging tools in the Container (A) nor any way to check the status of the process then how do we debug any problem in the application? Once a pod is created, it's even not possible to add Containers to it for additional debugging tools.

That's where the Ephemeral Containers come into picture as in the Container (B) in the above picture. These Containers are temporary that can be included in the Pod dynamically with additional debugging tools. Once a Ephemeral Container has been created, we can connect to it as usual using the kubectl attach, kubectl exec and kubectl logs commands.

Here are the steps for creating an Ephemeral Container. The assumption is that kubeadm has been used to create the K8S Cluster and the Cluster is using K8S 1.18 version or newer which is the latest as of this writing.

Step 1) The Ephemeral Containers feature has to be enabled before creating it. More about enabling K8S features here.

Edit the below files to add '- --feature-gates=EphemeralContainers=true' in the command section. there is no need to restart the Pods as the kubelet process continuously monitors these files and restarts the Pods automatically. Run the 'kubectl get pods --all-namespaces' and notice the changes to the AGE column.


Step 2) Edit the '/etc/systemd/system/kubelet.service.d/10-kubeadm.conf' to pass the  '--feature-gates=EphemeralContainers=true' parameter to kubelet as shown below.

ExecStart=/usr/bin/kubelet --feature-gates=EphemeralContainers=true $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

Step 3) Restart the kubelet service for the above changes to take place into effect.

sudo systemctl daemon-reload
sudo systemctl restart kubelet

Note that Step 2 and 3 have to be run on all the Control Plane Nodes and the Worker Nodes.

Step 4) Now that the feature has been enabled, lets create a Pod with an application. For sake of simplicity, a pause Pod is created using the below command.

kubectl run ephemeral-demo --image=k8s.gcr.io/pause --restart=Never

There is no shell or any other debugging tools in this container, so the below command will fail.

kubectl exec -it ephemeral-demo -- sh

Step 5) Let's start an Ephemeral Container with the below command. Note that this particular feature is in alpha release of K8S, so the parameter alpha in the kubectl command. Once the feature graduates to stable, we don't need this any more. The below command opens a terminal to the Ephemeral Container and from where we can start debugging the main application.

kubectl alpha debug -it ephemeral-demo --image=busybox --target=ephemeral-demo

Step 6) Open a new terminal and run the below command to get the details of the Ephemeral Container.

kubectl describe pod ephemeral-demo

Hope you had fun learning about Ephemeral Containers, more about here and here.

Monday, January 13, 2020

Prajval in '32nd South Zone Aquatic Championship – 2019'

My son SS Prajval got Silver Medal in the finals of ‘4x50 mts Medley Relay – Group III – Boys’ 32nd South Zone Aquatic Championship– 2019. This was the first time Prajval represented in the finals of the South Zone Competitions and he was able to grab a Silver Medal in the same.

The District and the State level competitions were held at ‘B V Gurumurthy Memorial MCH Swimming Pool, Secunderabad, while the finals were held at ‘GMC Balayogi Athletic Stadium, Gachibowli, Hyderabad’. These events were conducted by the Telangana Swimming Association. The event was well covered by all the major media outlets.

GMC Balayogi Athletic Stadium, Gachibowli, Hyderabad

In the finals Swimmers from Andhra Pradesh, Telangana, Kerala, Karnataka, Tamil Nadu and Puducherry (Southern States of India) represented. It was nice to see kids really motivated with the Swimming. The final events were held on 3rd/4th/5th of January 2020. We had been to the events on the all three days and it was an enriching and motivating experience for Prajval and us.

Receiving the Silver Medal

With his Swimming Coach Mr Satish Balasubramanian (NIS Coach)

With his buddies - BEFORE the competition

With his buddies - AFTER the competetion


Deccan Chronicle

It's a tough balancing act between studies, swimming and peer pressures. But it's good to see that the kids are able to balance and prioritize them. Wish him luck for all the future competitions.