Wednesday, August 12, 2020

Connecting to S3 Service via VPC Gateway Endpoint

Lets say we are building a image processing application using ML which gets the images from S3 and identifies the action performed (sitting, standing etc) in those images . By default the network data flows from the application to S3 over the internet as shown in the left image which is not really that efficient and secure. AWS provides VPC Gateway Endpoint feature and this all the data will be within the AWS network only.

 
VPC Endpoints provides Gateway Endpoints for S3 and DynamoDB services, while Interface Endpoints are for the rest of the services. In this blog we will explore the VPC Gateway Endpoints.
 

Step 1: Create a VPC as mentioned in the previous blog and connect to the EC2 in the Private Subnet. By default the route table for the Private Subnet has route for 0.0.0.0/0 and so any EC2 in the Private Subnet will have internet connection.

 
The same can be verified by pinging google.com or some other host.

Step 2: In the Putty execute the below commands to install the AWS CLI.

sudo apt-get update
sudo apt-get install python2.7 python-pip -y
pip install awscli --upgrade
export PATH="$PATH:/home/ubuntu/.local/bin/" 

Step 3: Create an IAM Role with AmazonS3ReadOnlyAccess and attach it to the EC2 instance in the Private Subnet.

Step 4: Lets remove the internet connection for the EC2 instance in the Private Subnet. For this select the Routing Table for the Private Subnet and click on Edit Routes. Delete the route for 0.0.0.0/0 and click on "Save routes".

 

The Route Table will be updated as shown below.

 
Step 5: There is no need for NAT Gateway and the ElasticIP as we have removed the internet connection for the EC2 in the Private Subnet. Make sure to remove them. You can also keep it, but there is a cost associated with it.


Test out the internet connectivity (ping google.com") and also try to get the list of files in S3 (aws s3 ls). Both the commands should fail. Press Ctrl+C to come out of the commands.

Step 6: In the VPC Management Console, go to "Endpoints" and click on "Create Endpoint".

Search for S3 in the Service Name and select "com.amazonaws.us-east-1.s3". Make sure to select the VPC which was created in the previous step and select the Private Subnet.


Rest of the default options are good enough. Click on "Create endpoint" and the Endpoint will be created in a few minutes.

Step 7: Go back to the Route Table of the Private Subnet and note that a Route has been automatically added to the VPC Endpoint.

 
Step 8: Go back to the Putty session and execute the below commands. Notice that there is no internet connection, but still we are able to get the number of buckets in the AWS S3. This is because we have setup the AWS Gateway Endpoint and all the traffic remains with the AWS network only.
 
ping google.com
aws s3 ls | wc -l 

Conclusion

By default when we consume any AWS service from an EC2 instance the network traffic goes through the internet, which is not really secure. And there is an additional cost for NAT and Internet Gateway. By using the VPC Endpoint Gateway we noticed that the network traffic remains within the AWS network only. This makes it easy for migrating the applications to AWS and also make sure they are compliant.

Tuesday, August 11, 2020

Creating a VPC and connecting to the EC2 in the Private Subnet

An AWS VPC (Virtual Private Cloud) is a logically isolated network for isolating different environments like Production, QA, Development. VPC can also be used to isolate applications like CRM, HR and others. Applications is one VPC by default won't be able to communicate with applications in another VPC. A VPC Peering Connection has to be explicitly setup for communication to happen between two VPC.


In this blog we will setup a new VPC. In this VPC, we will create a Public and a Private Subnet. The way we will configure them is that any EC2 in the Public Subnet will have a Public and Private IP address, while any EC2 in the Private Subnet will have only a Private IP address associated with it. We can connect to the EC2 in the Public Subnet as it has a Public IP, but how do we connect to the EC2 in the Private Subnet as it doesn't have a Public IP? This might be required for making any changes to that particular EC2 instances to perform tasks like installing/upgrading databases etc.

One way to it to setup a VPN connection between the Laptop and the VPC as mentioned in the previous blog. This way the Laptop and the EC2 in the Private Subnet will appear as though they are in the same network and so we would be able connect from the Laptop to the EC2 in the Private Subnet. Another way is to connect to the EC2 in the Public Subnet using the Public IP and from there connect to the EC2 in the Private Subnet using the Private IP. This is what we would be exploring in this blog.

Step 1: Go to the EC2 Management Console and make sure the "New EC2" experience is selected.


Step 2: Click on the "Key Pairs" and click on "Create key pair". Enter the Key pair name and make sure the ppk format is selected and click on "Create key pair".

The Key pair would be created as shown below.


Step 3: Click on "Elastic IPs", click on "Allocate Elastic IP address" and finally click on Allocate. The Elastic IP address is required for NAT instance which will be automatically created while creating the VPC later.

An Elastic IP address will be created as shown below.

Step 4: Go to the VPC Management Console and click on "Launch VPC Wizard".

Step 5: Select "VPC with Public and Private Subnets" Option and click on Select.

Step 6: Enter the name of the VPC as MyVPC and select the Elastic IP created in the previous steps. Finally, click on "Create VPC". Rest of the default options are good enough.

The VPC creation process process takes a few minutes and the VPC screen will be updated as shown below.

Step 7: Any EC2 created in the Public Subnet, will only have the Private IP, we need to make sure it also has a Public IP address. For this to happen, from the list of Subnets select the Public Subnet, Actions -> "Modify auto-assign IP settings".

Make sure to check "Auto-assign IPv4" and click on Save.

Step 8: Click on the Security Groups and click on "Create security group". Enter the name as AllowSSH, give some description and add an inbound rule as displayed below to allow Port 22/ssh inbound. Make sure to select the VPC which has been created in the previous steps. Click on "Create security group". This one will be associated with the EC2 instances later.



Step 9: Create Ubuntu EC2 instances with t2.micro as the instance type in the MyVPC, one each in Private and Public Subnets. Make sure to select MyVPC and the appropriate Subnet in 'Configure instance' options while creating the EC2 instance. Also, select the AllowSSH Security Group and the KeyPair which has been created in the previous steps, this allows us to connect to the EC2 via SSH using the Key pair.

Step 10: Name the EC2 instances as MyVPC-PublicSubnet and MyVPC-PrivateSubnet appropriately. Also notice that the EC2 in the Public Subnet has both Private and Public IP address, while the EC2 in the Private Subnet has only Private IP address as shown below.


Step 11: Download putty.exe and pagent.exe from here. There is no need to install any of these softwares, simply download them. Start the pagent.exe and add the ppk file which has been downloaded in one of the previous steps.


Step 12: Open putty.exe and in the Host Name field enter the username ubuntu and the Public IP address of the EC2 in the Public Subnet separated by the symbol @ as shown below.

Step 13: Go to Connection --> SSH --> Auth and make sure to select "Allow agent forwarding" and click on Open to connect to the EC2 instance in the Public Subnet. There is no need to specify the Private key as it has been specified in the pagent.exe.


Step 14: From the Putty session execute the below command to connect to the EC2 in the Private Subnet. Make sure to replace 1.2.3.4 with the Private IP address of the EC2 in the Private Subnet. Note that there is no need to specify the Private key this time also as it is from the pagent.exe. Now on this EC2 instance we should be able to install any back-end applications like database, business logic and so on.

ssh ubuntu@1.2.3.4

 
Step 15: Finally the cleanup process

    - Terminate the EC2 instances
    - Delete the NAT Gateway
    - Wait for a few Minutes
    - Delete ElasticIP
    - Finally delete the VPC
    - Delete the Key pair
 

Conclusion

An EC2 instances in the Private has only a Private IP and is primarily used to host back-end applications like Database, Business Logic and so on. Since it doesn't have an Public IP, we won't able to connect to out from outside the Laptop using the Private IP. We have seen how to connect to it using the EC2 instance in the Public Subnet as a Bastion or Jump box.

For this we have used pagent.exe which stores Private Key in the memory and is not a safe approach. Another approach is to copy the Private key to the Bastion or Jump box which is also not a safe approach. Both the approaches are easy to use, but are not safe and is recommended in the non-production environments. For the production environments using the Client VPN is the preferred approach, in this the Bastion or Jump Box is altogether avoided and all the communication between to the AWS Cloud is encrypted using IPSec protocol.