Friday, May 19, 2017

AWS and Big Data Training

I have been into training line (along with consulting and projects) for a few years, both online and classroom, using a Wacom Tablet which I bought 5 years back. The Tablet helps me to convey the message better and quick, whether the participant is in some other part of world or in front of me. I am really passionate about the trainings, as it helps me think about a particular concept from different perspectives.

So, here is an AWS demo on how to create a ELB (Elastic Load Balancer) and EC2 (Linux Servers) in the Cloud. This particular combination can be used for the sake of High Availability or Scaling Up as more and more users start using your application.


Similar, below is the Big Data Demo on creating a HBase Observer, which is very similar to a Trigger in RDBMS. BTW, those who are new to HBase, it's a NoSQL Database of Columnar type. There are a lot of NoSQL databases and HBase is one of them.


If you are interested in any of these trainings, please contact me at praveensripati@gmail.com for further details. Both Big Data and Cloud are on the rage, so don't miss the opportunities around them.

See you soon !!!

Wednesday, May 3, 2017

Attaching an EBS Disk to a Linux Instance

In the previous blog, we looked the sequence of steps to create a Linux instance and log into it. In this blog, we will create a new hard disk (actually an EBS) and attach it to the Linux instance. I am going with the assumption that the EC2 instance has already been created.


1. Goto the EC2 management console and click on the Volumes in the left pane. And then click on `Create Volume`. Change the volume size to 1GB and click on Create.


2. It takes a couple of seconds, but there will be two EBS volumes. An 8 GB volume (in-use) which was automatically created at the the time EC2 creation. Another 1 GB volume (attached) which was created in the above step.

 
3. Select the 1 GB volume. Click on Actions and then Attach Volume. We have created a Linux EC2 instance and an EBS volume, now we need to attach these two.


4. Move the cursor to the instance box and select the EC2 instance to which the EBS has to be attached and then click Attach. The state of the 1GB instance should change from available to in-use.


5. Get the ip address of the EC2 instance and login to it using Putty as mentioned here.

6. Change to root (sudo su) and get the list of the partition tables (fdisk -l). The commands are mentioned in the parentheses. Note that the device name is /dev/xvdf from the below commands. It may vary, so the previous mentioned commands have to be run.


7. Build the Linux filesystem (mkfs /dev/xvdf), create a folder (mkdir /mnt/disk100) and finally mount the file system (mount /dev/xvdf /mnt/disk100). The commands are mentioned in the parentheses. Note that, I have chosen to create disk100 folder, you can replace with any other folder name.


Now the device has been mounted to /mnt/disk100 folder, the data in this folder will be written to the 1 GB EBS volume which we have created in one of the previous step. Even after stopping the EC2 instance, the data will be still there in the EBS instance. This EBS can be attached to another EC2 instance also.

Note that in the case of AWS, an EBS volume cannot be attached to multiple EC2 instances. But, the same thing can be done in the Google Cloud Platform.

Don't forget to terminate the Linux EC2 instance and delete the EBS volume. In the next blog, we will attach an EBS volume to an Windows EC2 instance, which is a bit more easier.

Wednesday, April 26, 2017

Scratch programming for kids

It's summer holidays and I had been helping my kid (8 years) get started with programming. So I bought the Computer Coding for Kids book by Carol Vorderman. The books covers initially Scratch (from MIT) and then Python from the basics. Scratch is a visual programming language which is easy for those who are just getting started with programming. No need for typing code, more of drag and drop.

So, here is his first program (link here), a small and cute game in Scratch. Click on the flag on the top right, to get the game started. He was all excited to get it published in this blog. One thing to notice is that the execution of the game requires Adobe Flash, which is slowly getting out of fashion with the different browsers because of the vulnerabilities and stability issues. For some reason, I was able to get it run only in IE and not in Edge/Chrome/Firefox browsers.


Scratch looks a bit basic, but I have seen some interesting programs developed in it. I would recommend the above mentioned book for anyone who wants to get started with programming. Carol has written a few more books, which I plan to buy once we complete this book.

Monday, April 24, 2017

Creating a Linux AMI from an EC2 instance

In one of the earlier blog, we created a Linux EC2 instance with a static page. We installed and started the httpd server, created a simple index.html in the /var/www/html folder. The application in the EC2 is too basic, but definitely we can build complex applications.

Lets say, we need 10 such instances. Then it's not necessary to install the softwares, do the necessary configurations multiple times. What we can do is create an AMI, once the softwares have been installed and the necessary configuration changes have been made. The AMI will have the OS and the required softwares with the appropriate configurations. The same AMI can be used while launching the new EC2 instances. More about the AMI here.


So, here are the steps

1. Create a Linux EC2 instance with the httpd server and index.html as shown here.

2. Once the different softwares have been installed and configured, the AMI can be created as shown below.


3. Give the image name and description and click on `Create Image`.


4. It takes a couple of minutes to create the AMI. The status of the same can be seen by clicking on the AMI link in the left side pane. Initially the AMI will be in a pending state, after a few minutes it will change into available state.


5. On launch a new EC2 instance, we can select the new AMI which was created in the above steps in the `My AMIs` tab. The AMI has all the softwares and configurations in it, so there is no need to repeat the same thing again.


6. As we are simply learning/trying things and not anything in production, make sure to a) terminate all the EC2 instances b) dregister the AMI and c) delete the Snapshot.



Now that we know how to create an AMI with all the required softwares/configurations/data, we will look at AutoScaling and ELB (Elastic Load Balancers) in the upcoming blogs.

Creating an alarm in CloudWatch for a Linux EC2 instance

AWS CloudWatch does a couple of things as mentioned in the AWS documentation here. But, one of the interesting thing is that it can gather the metrics (CPU, Network, Disk etc) for the different Amazon resources.

In this blog, we will look how to look at the CPU metrics on a Linux EC2 instance and trigger an Alarm with a corresponding action. To make it simple, if the CPU Utilization in the Linux instance goes beyond 50% then we will be notified through an email. We can watch different metrics, we will use CPU Utilization for now. CloudWatch also allows to create custom metrics, like the application response time.

I will go with the assumption that the Linux instance has already been created as shown in this blog. So, here are the steps.

1. In the Linux instance make sure that the detailed monitoring is enabled. If not then enable it from the EC2 management console as shown below.


2. Go to the CloudWatch management console and select Metrics on the left pane. Select EC2 in the the `All Metrics` tab.


3. Again select the `Per-Instance Metrics` from the `All Metrics` tab.


4. Go back to the EC2 management console and get the `Instance Id` as shown below.


5. Go back to the CloudWatch management console. Search for the InstanceId and select CPUUtilization. Now the line graph will be populated as shown below for the CPUUtilization. We should be also able to select multiple metrics at the same time.


6. The graph is shown with a 5 minute granularity. By changing it to minute granularity, we would be getting a much smoother graph. Click on `Graphed Metrics` tab and change the Period to `1 minute`.


7. Now, lets try to create an Alarm. Click on Alarm in the left pane and then on the `Create Alarm` button.


8. Click on the `Per-Instance Metrics` under EC2. Search for the EC2 instance id and select CPUUtilization as shown below.


9. Now that the metric has been selected, we have to define the alarm. Click on `Next`. Specify the Name, Description and CPU % as shown below.


10. When the alarm is breached (CPU > 50%), then the notification has to be done. In the same screen, go a bit down and specify the notification by click on `New list` as shown below.


11. Click on `Create Alarm` in the same screen. To avoid spamming, AWS will send an email with the confirmation link which has to be clicked. Go ahead and check your email for confirmation.


12. Once the link has been clicked in the email, we will get a confirmation as below.


13. Now, we are all setup to receive an email notification when the CPU > 50%. Login to the Linux instance as mentioned in this blog and run the below command to increase the CPU on the Linux instance. The command doesn't do anything useful, but simply increases the CPU usage. There are more ways to increase the CPU usage, more here.
dd if=/dev/urandom | bzip2 -9 >> /dev/null

14. If we go back to the Metrics then we will notice that the CPU has spiked on the Linux instance because of the above command.


15. And the Alarm also moved from the OK status to the ALARM status as shown below.


16. Check your email and there would be a notification from AWS about the breach of the Alarm threshold.

17. Make sure to delete the Alarm and terminate the Linux instance.

The procedure is a bit too long and it took me good amount of time to write this blog, but once you do it a couple of times and understand whats being done, then it would be a piece of cake.

Friday, April 14, 2017

Got through `AWS Certified Developer - Associate`

Today, I got through the AWS Certified Developer - Associate. This is the second certification around AWS (first certification here). It had been a bit more tough than I expected, but it was fun. Anyway, here is the certificate.


There were 55 questions which had to be solved in 80 minutes. It was a bit tough and I did spend till the last minute solving and reviewing the questions. As usual, nothing beats the practice to get through the certification.

The next certification I am planning is `AWS Certified SysOps Administrator - Associate` and then the `AWS Certified Big Data - Specialty` Certification. The Big Data Certification is in Beta and I am not sure when Amazon would make it available to the public. This is the certification which I am really interested, as it is an intersection of Big Data and the cloud.

Thursday, April 13, 2017

Which EBS type to use to meet our requirements?

In the previous blog, I pointed to the approach to figure out the best way to pick the best EC2 instance to meet our requirements. Here is a video to pick the best EBS type. EBS (Elastic Block Storage) is like a Hard Disk Drive which can be attached to the EC2 instance.


The way AWS provides different EC2 instances, there are different EBS types which are basically categorized into SSD and Magnetic. Depending upon if we are looking at high IOPS or throughput, the appropriate EBS type can be picked. Here is a flow chart in the same video to figure out the type of EBS to pick.

One interesting thing to note about EBS is that the EC2 instance and EBS need not be on the same physical machine. That is, the processing and the disk need not be on the same machine. There is a low latency and high throughput between the EC2 and EBS machines.

A good amount of the AWS videos are really good and some of them are so-so. I would be promoting the videos which are really interesting in this blog for the readers.