HDFS Federation

We understand that DataNode provides storage service in hadoop cluster by storing blocks on the local file system and allows read/write access. Whereas NameNode is responsible to manage entire file system namespace in HDFs cluster including block locations. We can summarize responsibilities of NameNode in two categories.

1. Namespace Management
1. Manage directories, files and block locations
2. Support filesystem operations like create, delete, modify and list directories.
2. Block storage Management
1. Provide DataNode a membership by handling registration and periodic heart beats.
2. Processes block reports and maintain block locations.
3. Support block related operations create, delete, modify and get block location.
4. Manages replica placement, replication of a block for under replicated blocks and deletes blocks that are over replicated.

The prior HDFS architecture allows only a single namespace for the entire cluster. Since NameNode keeps this entire information in memory, there are two specific issues which remain unaddressed.

• For a very large cluster, memory becomes a limiting factor. Storage at data nodes can scale horizontally but memory at NameNode is vertically scalable. Performance of file system operations will suffer with this memory limitation. The only method to handle this performance issue is to add more memory at NameNode.
• You may want to utilize your cluster for mix of purposes for example you may want to test some experimental applications on your production cluster. In that situation, you may want to isolate your experimental namespace from actual production namespace. Requirements for isolation may come from various operational reasons but HDFS architecture doesn’t allow this. This requirement even exists for small clusters.

HDFS Federation addresses these limitations of the prior architecture by allowing you to have multiple NameNodes in a single hadoop cluster. It is just like a large file system is broken into smaller pieces and distributed amongst multiple NameNodes. Each NameNode will manage one or more parts of the entire file system. For example /user and /tmp is managed my Node-1 while /projects is managed by node-2.  HDFS federation was introduced in hadoop 0.23 release and is part of hadoop 2.x.

It is important to understand following key points about HDFS federation.

1. Federated NameNodes are independent and don’t require coordination with each other. If one NameNode goes down, it doesn’t impact namespace managed by other NameNode.
2. DataNodes are common for all NameNodes that means a DataNode can store blocks from any of the NameNode.
3. DataNodes will also send heartbeat/ block-report to all NameNodes and will handle commands from all NameNodes.
4. All the blocks which are part of a single namespace and is managed by its NameNode and called block pool of the NameNode. So a NameNode and its block pool together is one namespace in HDFS terminology.
5. HDFS federation mandates use of a ClusterID to identify all the NameNodes in the cluster. This helps to extend namespaces to a multi cluster environment.

Implementation

Once we understand basics of HDFS federation, next question comes, how it is implemented for use? There are two parts of the problem for implementing HDFS federation in a cluster.

• How do we configure multiple NameNodes – This part of the problem is quite simple, we can configure and start multiple NameNodes just like we do single. We may have to address some issues like giving a unique id for each NameNode, configuring their RPC and HTTP address etc. all such configuration is placed in hdfs-site.xml file.
• How do we map different namespaces with individual NameNodes – This part will have another implicit problem. This new problem is “How do we use file system after breaking it into smaller part and mapping it with different NameNodes”.

This second problem along with implicit third problem is solved by ViewFS. We will discuss about it but let’s take them in sequence.

Let’s assume that we want to configure two NameNodes in a cluster. To be able to do this, we will have to define a new property in hdfs-site.xml file as dfs.nameservices. A value for this property is a list of NameServiceID. Each NameServiceID is a unique id for a NameNode. An example is give below.

 <property>

    <name>dfs.nameservices</name>

   <value>ns1,ns2</value>

 </property>

In above example, ns1 and ns2 are two logical NameServiceIDs which represents two NameNodes. In the next step, we will have to map these logical ids with physical NameNode addresses. This part is also defined in hdfs-site.xml. An example is given below.

<property>

   <name>dfs.namenode.rpc-address.ns1</name>

    <value>node-1.mycompany.com:50070</value>

 </property>

<property>

   <name>dfs.namenode.rpc-address.ns2</name>

    <value>node-2.mycompany.com:50070</value>

 </property>

In the above example, ns1 is mapped with a node-1 at port 50070 and similarly ns2 is mapped with node-2 at port 50070.

Once these configurations are specified, you may need to format your new NameNodes using hdfs namenode -format [-clusterId <cluster_id>]. Cluster id is optional, if you don’t provide cluster id, a new id is provided automatically and you can use this id while formatting other NameNodes in same cluster. After formatting your new NameNodes, you should start them and refresh DataNodes to pick up new NameNodes. There are several other changes implemented to support operations like balances has been changed to work with multiple NameNodes, a Cluster Web Console is added in federation to monitor the federated cluster at http://<any_nn_host:port>/dfsclusterhealth.jsp. This completes our first problem.

With the knowledge of your NameNode URI (scheme://authority) as specified in configuration, you can use them for file system operations for example create directory and create files etc. Creating a new directory will be as simple as using hdfs  dfs  -mkdir with fully qualified path name URIs as scheme://authority/path.

At this stage, there is no global namespace defined. For a client, each NameNode looks like a separate HDFS cluster. To perform any operation, client needs to know NameNode URI. This makes life for client programs more difficult and less portable. This was our second problem. Hadoop attempts this problem using ViewFS.

ViewFS

The View File System (ViewFS) provides a way to manage multiple NameNodes in hadoop cluster and hence multiple namespaces in HDFS Federation. In the old HDFS implementation, a cluster has a single NameNode. The core-site.xml of the cluster has a configuration property that sets the default file system and maps NameNode of the cluster. An example of such property is shown below.

<property>

   <name>fs.defaultFS</name>

   <value>hdfs://NameNodeOfClusterX:port</value>

</property>

On such a cluster where core-site.xml is set as above, a typical path name will be something like /projects. Alternatively you can also specify fully qualified names while referring to HDFS location for example hdfs://NameNodeOfClusterX:port/projects. While both are valid, you should prefer relative name /user/hue because it makes your code portable to different cluster and independent from the underlying NameNode.

In a federated HDFS implementation, you have multiple NameNodes. We have two problems to be solved. First problem is to assign parts of file system to individual NameNodes. Second is to use relative file paths like /projects without any need to specify specific NameNode in the URI.

Hadoop solves this problem in similar using a mount table mechanism similar to Unix/Linux mount tables. In Linux mount table implementation, we specify a logical name to a mounted file system to be used by clients and mapping between logical mounted names and actual physical locations are stored in a mount table. Similar mechanism is used in ViewFS.

The first part is to specify that we are using ViewFS for specifying mount tables and second step is to create mount tables to resolve logical names with actual physical locations. Specifying that we are using ViewFS is done in core-site.xml file by changing value of fs.defaultFS property. In a non-federated setup, this property used to store NameNode URI but in a federated implementation it will store mount table name for the cluster. An example of such configuration is shown below.

<property>

   <name>fs.defaultFS</name>

   <value>viewfs://ClusterX </value>

</property>

In above example viewfs is the scheme which indicates that HDFS federation and ViewFS is implemented for this cluster and available NameNodes can be found in a mount table. So client will look further for mount table information within cluster configuration file. The authority ClusterX in above example is mount table name. In a multi cluster environment, it is recommended that you name mount tables with cluster names. In case, you just have a single cluster, you can leave authority unspecified which indicates that default mount table should be used. For a default mount table, an entry will look like <value>viewfs: /// </value>.

Next part is to define mount tables. The mount tables can be described again in core-site.xml however you can define it in a separate xml file and include it into your core-site.xml. An example of a mount table is given below.

<property>

   <name>fs.viewfs.mounttable.ClusterX.link./home</name>

   <value>hdfs://node-1.mycompany.com:50070/home</value>

</property>

 <property>

   <name>fs.viewfs.mounttable.ClusterX.link./tmp</name>

    <value> hdfs://node-1.mycompany.com:50070/tmp</value>

 </property>

 <property>

   <name>fs.viewfs.mounttable.ClusterX.link./projects </name>

    <value> hdfs://node-2.mycompany.com:50070/projects </value>

 </property>

In this example, there are two NameNodes, node-1 manages /home and /tmp while node-2 manages only /projects. Clients can directly use relative path names like /home, /tmp and /projects which gets resolved using configuration file.

If you are using default mount table, configuration will look like as below.

 <property>

   <name>fs.viewfs.mounttable.default.link./projects </name>

    <value> hdfs://node-2.mycompany.com:50070/projects </value>

 </property>

You can use fully qualified names like viewfs://ClusterX/projects if you are outside cluster. However, if you are within the cluster, you should avoid such naming convention and prefer relative paths like /projects. This will make your code portable from one cluster to another cluster.

Using HDFS File system

In one of my previous posts, we understand HDFS as a distributed file system where we can do everything we can do with other file systems like create directory, create files, copy them, move them, rename them, delete them and many more operations. If you have your portable hadoop setup as explained in one of my previous posts, it’s time to experience HDFS using file system commands. HDFS file system commands are based on Linux file system commands because HDFS is primarily build on Linux platform.

In a typical hadoop installation, you will have three types of machines as NameNode, DataNode and client machine. You already understand NameNode and DataNode (if not read this post). Client machine is nothing but your local or remote computer having hadoop client components installed and configured to interact with NameNode and DataNode. Ideally your client machine does not run any NameNode or DataNode daemons.

In our current setup of portable Hortonworks hadoop sandbox, everything is running on single virtual machine. My windows laptop can’t act as a client machine in such setup because I do not have hadoop client components installed on my windows machine. But that’s not a worry to demonstrate this concept as our sand box can act like a client machine as well for us.

For our tutorial purpose, I will move one large file into sandbox local file system from my windows laptop. File will still not be in HDFS. That situation is exactly same as you have a file in your client machine and you are ready to move that file into a HDFS file system to take advantage of the power of HDFS.

As we all know easiest method for moving a file from a windows machine to a Linux machine is through an SSH client, I will use open source WinSCP for this purpose. If you have any other tool, you can use it or download WinSCP and use as demonstrated below.

1. Start you’re SandBox VM and you will see screen similar to below.

2. This means, you can ssh your VM using 127.0.0.1 at port 2222. Your VMs actual IP address is different than shown in above screen and ssh in your VM is actually running at port 22. But you can access it at 127.0.0.1 due to one of the Oracle VirtualBox features called port forwarding. Using this feature, you access your localhost ports using loopback interface and VirtualBox will forward it to VM on preconfigured ports. This feature will allow you to connect your laptop to any network and still keep your sandbox working.
3. Let’s start WinSCP and connect to your Linux virtual machine. Select values as shown in below image and click login.

4. On successful login, you will see WinSCP showing files on your local machine and remote machine side by side. In screen shown below, left side shows SampleLargeFile.tsv on my local machine and on right side is /root directory on my VM. Transferring file from my local machine to VM is now just a matter of drag and drop. Once you moved your file into /root directory of your VM, you are done with WinSCP so just close it for now.

5. Now you need to go back to your VM window and press ALT+F5. You will get a Linux login screen. Login to your VM using root user name and password as hadoop. After successful login, execute clear screen command to get a clear screen.

6. You can issue ls command to see your file is there. In my case it is SampleLargeFile.tsv. This file is available in my VM machine’s local file system. It is still not available in HDFS. This situation is exactly same as you have a file in your client machine and you are ready to move that file into a HDFS file system to take advantage of the power of HDFS. Now you are ready to use HDFS commands so let’s start.
7. First command I will use is hadoop fs –ls /user

All hadoop file system commands follow general syntax as hadoop fs <args>in above example; I am using ls command to list contents of /user directory in HDFS. Please note that /user is an HDFS directory and you will not find it in local file system. If you have doubt, issue ls /user command and you will get an answer as “no such file or directory”.

8. Next command I am going to use mkdir to make a directory under /user/hue directory and finally I will use put command to copy my SampleLargeFile.tsv into hadoop directory /user/hue/myfiles. You can see these syntaxes and results in below screen.

9. You can get more details on various available file system commands from apache hadoop documentation and practice them in similar way I demonstrated some of them in here.
10. We have seen hadoop file system shell interface in action. But that’s not the only interface offered by HDFS for users and developers to access and work with HDFS. Most basic at the bottom are Java APIs. Using those APIs open source community has developed a web interface to make it simple for end users. Let’s explore this web interface.
11. Start your favorite browser while your VM is up and running and type in this url http://localhost:8888/

After completing one time registration process, you will see Hortonworks sandbox home page as below.

12. Click on Go TO SNADBOX link and you will reach hadoop web interface called hue. We will talk about hue some other time. For now just click on file browser icon from the hue home page. It opens HDFS file browser and you should be able to see myfiles directory which we created in this tutorial. If you navigate through myfiles directory, you will find the file which we created in that directory.
13. Hue file browser provides and easy interface to upload files directly into HDFS without following all those steps we used earlier to create our file in HDFS. You can perform all your day to day file system operation using hue file browser.

 

In a typical HDFS installation, NameNode and DataNode each run an internal web server in order to display basic information about the current status of the cluster. With the default configuration, the NameNode front page is at http://namenode-name:50070/ but In case of our sandbox it will be http://localhost:50070/. Screen look like below image.

It shows some basic reports about HDFS, NameNode and a link to DataNode. You can browse to DataNode to get reports for DataNode. You can browse the file system using link on the page. It did not worked for me until I change url again replacing actual IP address of my NameNode with localhost.

This ends a basic introduction to HDFS, The distributed file system. Next key concept of hadoop ecosystem is MapReduce process which actually makes hadoop powerful and purposeful.

Geting a portable hadoop environment

Before we start learning individual components of hadoop ecosystem, it is good to get your portable hadoop environment. There are various options available, but after playing with some of them, I personally find hortonworks sandbox virtual machine image for Oracle VirtualBox as easiest for getting started on hadoop ecosystem.

Everything you need is freely available and fairly simple to setup. Follow these steps you are novice in downloading and installing software.

1. Download Oracle VirtualBox from here.

I have Microsoft Windows 7 on my laptop so I need VirtualBox for windows hosts as shown in below image.

 

 
 

2. In previous step you downloaded VirtualBox-4.3.0-89960-Win.exe file. Now you just need to execute it and follow on screen instructions. In fact you just need to click next button three times, Yes button once, install button once and finally finish button. It’s done. You have powerful virtualization software on your laptop.
3. After installation Oracle VirtualBox should start automatically which looks like image shown below. You can close it now and when you need to start it again, you can find it in your start menu. 

4. After installing Oracle VirtualBox on Microsoft Windows 7, I found a new network connection setup. Presence of this network stopped me to connect through internet using my data card. If you face similar problem, you need to disable this network. Follow instructions below if you want to disable this network.

Start Control Panel->Network and Internet->Network and Sharing Center->Change Adapter Settings. You should see a network connection as below. Right click and disable it. You may have to reconnect your internet once again.

5. Now you are ready to setup virtual machine into Oracle VirtualBox. For our purpose, you need to download hortonworks sandbox appliance from here. 

6. You downloaded Hortonworks+Sandbox+2.0+VirtualBox.ova file.
7. Start Oracle VirtualBox and select File->Import Appliance as shown in image below.

8. You will see a dialog box, click browse button and select Hortonworks+Sandbox+2.0+VirtualBox.ova file which you have downloaded from hortonworks website.

9. Click Next button and then Import button when new dialog box appears. Your Virtual machine import will start and complete in few minutes.

10. After completion you will see Hortonworks Sandboc 2.0 listed in your Oracle VirtualBox as shown below. 
11. You will notice that your virtual machine is configured with 2048 MB memory. This means your guest OS once started will consume 2 GB of your RAM. If you have 3 GB or more RAM on your laptop, you don’t need to worry. If you have less than that, you can click on the System button just above and reduce memory it to 1 GB.
12. To start your virtual machine click Start Button. Once started, you will get your portable Hadoop running which is a familiar Linux terminal interface.

13. To shut down your virtual machine, Click ACPI Shutdown as shown in below screen.