Thursday, January 12, 2012

Getting started with HDFS client side mount table

With HDFS federation it's possible to have multiple NameNode in a HDFS cluster. While this is good from a NameNode scalability and isolation perspective, it's difficult to manage multiple name spaces from a client application perspective. HDFS client mount table makes multiple names spaces transparent to the client. ViewFs more details on how to use the HDFS client mount table.

Earlier blog entry detailed how to setup HDFS federation. Let's assume the two NameNodes have been setup successfully on namenode1 and namenode2.

Lets map
/NN1Home to hdfs://namenode1:9001/home
and
/NN2Home to hdfs://namenode2:9001/home
Add the following to the core-site.xml
<?xml version="1.0"?>
<configuration>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/home/praveensripati/Installations/hadoop-0.23.0/tmp</value>
    </property>
    <property>
        <name>fs.default.name</name>
        <value>viewfs:///</value>
    </property>
    <property>
        <name>fs.viewfs.mounttable.default.link./NN1Home</name>
        <value>hdfs://namenode1:9001/home</value>
    </property>
    <property>
        <name>fs.viewfs.mounttable.default.link./NN2Home</name>
        <value>hdfs://namenode2:9001/home</value>
    </property>
</configuration>
Start the cluster with the `sbin/start-dfs.sh` command from the Hadoop Home and make sure the NameNodes and the DataNodes are working properly.

Run the following commands
bin/hadoop fs -put somefile.txt /NN1Home/input
Make sure that somefile.txt is in the hdfs://namenode1:9001/home/input folder from NameNode web console.
bin/hadoop fs -put somefile.txt /NN2Home/output
Make sure that somefile.txt is in the hdfs://namenode2:9001/home/output folder from NameNode web console.

4 comments:

  1. Hi, Great post! Where is the core-site.xml file that your refer to located? Is it on the name nodes? Thanks!

    ReplyDelete
  2. Great..So with this..Client can access both the directories simultaneously under different name space right?

    ReplyDelete
  3. hi
    I am getting an exception URISyntax exception: viewfs:///

    Let me know if that makes sense to you.

    Thanks

    ReplyDelete