In the previous blog, we looked on how to start a AWS EMR cluster and run a Hive Script. Once the cluster has been started, it does provide a web console to check the status of the cluster and also to see the progress of the different data processing tasks. By default, the web consoles are blocked for the sake of security.
Below are the URLs of some of the web consoles.
Step 1 : Start the EMR cluster as shown in the previous blog.
Step 2 : Setup a ssh tunnel to the master using local port forwarding using the below command. Here the local port 8157 is being forwarded to the remote port 8088. The port 8157 can be replaced by any free local port and 8088 is the port on which the YARN console is available. Port 8088 can be replaced by the port of the Web console which we want to access.
a) the path of the key pair
b) the DNS name of the master node (twice)
Step 3 : Access the YARN console from the browser from the same machine the Step 1 has been performed.
An alternate for the above step is to make changes to the master Security Group and allow inbound 8088, which is the YARN Web Console port number.
Below are the URLs of some of the web consoles.
YARN ResourceManager http://master-public-dns-name:8088/ Hadoop HDFS NameNode http://master-public-dns-name:50070/ Spark HistoryServer http://master-public-dns-name:18080/ Zeppelin http://master-public-dns-name:8890/ Hue http://master-public-dns-name:8888/ Ganglia http://master-public-dns-name/ganglia/ HBase UI http://master-public-dns-name:16010/ YARN NodeManager http://slave-public-dns-name:8042/ Hadoop HDFS DataNode http://slave-public-dns-name:50075/In this blog, we will be exploring on how to access the web consoles. The AWS documentation for the same is here.
Step 1 : Start the EMR cluster as shown in the previous blog.
Step 2 : Setup a ssh tunnel to the master using local port forwarding using the below command. Here the local port 8157 is being forwarded to the remote port 8088. The port 8157 can be replaced by any free local port and 8088 is the port on which the YARN console is available. Port 8088 can be replaced by the port of the Web console which we want to access.
ssh -i /home/praveen/Documents/AWS-Keys/MyKeyPair.pem -N -L 8157:ec2-54-147-238-2.compute-1.amazonaws.com:8088 hadoop@ec2-54-147-238-2.compute-1.amazonaws.comIn the above command replace the following
a) the path of the key pair
b) the DNS name of the master node (twice)
Step 3 : Access the YARN console from the browser from the same machine the Step 1 has been performed.
An alternate for the above step is to make changes to the master Security Group and allow inbound 8088, which is the YARN Web Console port number.
Your command is right.I want to access web consoles using URLs.Your steps helps me to access very easily.
ReplyDelete