Hadoop HDFS Overview
Hadoop HDFS is an open-source distributed file system that is part of the Hadoop framework, used for reliable and scalable data storage. HDFS is designed to handle large amounts of data and provide fault-tolerance through replication and auto-scalability. It consists of multiple data nodes spread across a cluster to store files.
HDFS Architecture:
There are six types of nodes in the HDFS architecture:
– NameNode: Maintains file system metadata and monitors DataNode health.
– Secondary NameNode: Creates checkpoints for NameNode.
– DataNode: Stores data and sends heartbeats to the NameNode.
– Checkpoint Node: Takes regular snapshots of the file system metadata.
– Backup Node: Provides warm standby for the active NameNode.
– Blocks: The Hadoop file system breaks up files into blocks, which are distributed across all of the machines in the cluster.
Features of HDFS:
– Built for handling big data: HDFS can store large amounts of data and handle write once, read many times (WORM) workloads.
– Fault tolerance: HDFS automatically replicates data across multiple DataNodes to ensure data availability.
– Scalable: HDFS can scale horizontally by adding more DataNodes to the cluster.
– Directories and files: HDFS provides a hierarchical structure of directories and files.
Replication Management in HDFS:
When a file is written to HDFS, it is split into blocks and stored across multiple DataNodes in the cluster. HDFS stores multiple replicas of each block to ensure data availability. The number of replicas can be configured based on the desired level of replication.
Write Operation:
When a file is created in HDFS, it is split into one or more blocks and each block is replicated based on the replication factor. The NameNode assigns a DataNode to host each block. The client then writes the blocks to the assigned DataNodes.
Read Operation:
When a client wants to read a file from HDFS, it requests the blocks from the nearest DataNode, and if that DataNode fails to respond, another replica is used. The client reads the file from the sequence of blocks.
Advantages of HDFS Architecture:
– Handles big data effectively.
– Provides fault tolerance and high availability.
– Designed for commodity hardware.
– Scalable storage and processing.
Disadvantages of HDFS Architecture:
– Not suitable for small files.
– High latency for real-time data processing.
– Requires significant expertise to manage.
Conclusion:
Hadoop HDFS is the core storage system of the Hadoop ecosystem, providing a reliable and scalable data storage solution for big data applications. With its built-in replication and scalability features, HDFS is a well-rounded solution for handling large volumes of data while providing fault tolerance and high availability.
Additional Resources:
– Apache Hadoop official website: https://hadoop.apache.org/
– Hadoop Distributed File System: https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html
Understanding Hadoop HDFS
Hadoop HDFS is a software platform that handles the distributed storage and processing of big data sets. It comprises several open-source applications such as Yarn, MapReduce, and HDFS. Although Hadoop serves various purposes, its primary functions are Big Data analytics and NoSQL database administration. The Hadoop Distributed File System (HDFS) is a decentralized storage system that saves data on multiple computers within a cluster. This structure is perfect for large-scale storage as it distributes the load across several machines, reducing pressure on each device. MapReduce is a programming model that enables users to compose code once and execute it on various servers simultaneously. When integrated with HDFS, MapReduce processes colossal data sets by dividing the job into smaller chunks, which are then executed in parallel.
Understanding HDFS Architecture
HDFS is a powerful open-source component of the Apache Software Foundation for efficient data management. Its vital features include scalability, availability, and replication. The HDFS architecture comprises of Name nodes, Secondary name nodes, Data nodes, Checkpoint nodes, Backup nodes, and Blocks. HDFS ensures fault tolerance through replication, and files are distributed through the cluster systems, coordinated by the Name node and Data nodes. Hadoop is a non-relational data store, whereas Apache HBase works as a non-relational database. In a master-slave architecture, HDFS includes the following elements:
Code:
“`
– Name node
– Secondary name node
– Data node
– Checkpoint node
– Backup node
– Blocks
“`NameNode
All about NameNode in Hadoop
NameNode acts as a master node to handle all the blocks and performs various functions in a Hadoop cluster, such as regulating DataNodes, storing blocks on DataNode, and much more. There are two file types in NameNode: FsImage and EditLogs. FsImage stores all the details about a filesystem in a hierarchical format. EditLogs record all modifications made to files in the filesystem. For instance, in case of a DataNode failure, it can be replaced with another.The Role of Secondary NameNode in Hadoop
The Secondary NameNode in Hadoop performs various tasks when the primary NameNode is unable to. Here’s what it does:
– It stores the transaction log data from all source databases into one location for easy replaying. It is then replicated across servers.
– Metadata about the Hadoop cluster nodes is stored in all data nodes to determine where to send and what type of data it is.
– FsImage creates new replicas of data to scale it up. It also helps in backup and recovery operations.
Datanode
A DataNode is a slave machine that stores data in ext3 or ext4 file format. It stores all the data and handles requested file operations such as creating new data and reading file content. DataNodes also perform tasks such as scrubbing data and establishing partnerships.
Checkpoint Node
The Checkpoint Node creates checkpoint nodes in FsImage and EditLogs from the NameNode at certain intervals, and merges them to produce a new image. This generates a checkpoint in the HDFS and delivers it to the NameNode. The directory structure of the checkpointed image is always the same as that of the NameNode, ensuring availability.
Backup Nodes for High Availability
Backup Nodes ensure high availability of data. If an active NameNode or DataNode fails, the Backup Node can be used as the new active node, with the previous active node switching to the Backup Node. However, Backup Nodes cannot be used to recover from failures of active NameNode or DataNodes. Instead, replica sets of data are used for recovery. Data Nodes store data and generate FsImage and editsLogs files for replication. They connect with replica sets of data to enable replication. Data Nodes do not provide high availability.
HDFS DataBlock Architecture
Code:
The block size of HDFS can be set between 32 and 128 MB to enhance
performance. Data is always written to DataNodes on any user change
and added to the end of the data. HDFS replicates DataNodes to ensure data
consistency and fault tolerance, and if a Node fails, the system can
recover the data and replicate it on remaining healthy Nodes. Since DataNodes
do not store data directly on hard drives, it uses the HDFS file system
to horizontally scale as the number of users and data types increase.
The block size is increased when the file size increases, and bigger data
is added to the next block. For instance, if the data is 135 MB and the
block size is 128 MB, the first block will be 128 MB while the second block
will be 7 MB.Advantages of HDFS
HDFS provides several advantages:
– Multiple replicas of files can be created for data redundancy, ensuring accessibility even if one replica fails.
– HDFS can horizontally and vertically scale to store massive amounts of data, up to 5PB in a single cluster.
– Data is stored on HDFS, not local filesystems and is accessed through client tools like Java or Python client, or the CLI.
– Replication improves data reliability by storing data on multiple nodes to make it quickly available even if one server fails.Replication Management in HDFS Architecture
HDFS ensures resilience to crashes and data corruption through duplication and replica availability. Every block has at least three replicas stored on multiple DataNodes. The nameNode uses maintenance techniques to keep track of block replication and add or delete copies as needed.File Writing and Assembly in Hadoop
After the file is written to all DataNodes, they confirm the location of the last block. The NameNode uses this information to reconstruct the file and confirm the completion of the job. File segmentation optimizes storage, fault tolerance, and availability. The client divides the file into segments, and each DataNode receives and passes on the segment to the next until all segments are received and the file is reassembled.
File Reading Operation in Hadoop Architecture
In Hadoop architecture, the file is sent from the client to the Replicator, which doesn’t have a copy of it. The Replicator reads the data from another source and sends it to the DataNode in the background. The DataNode contacts other data nodes to get the actual data. After receiving the data, the Replicator sends it to the Reducer, which has a compressed version of the data.HDFS Architecture Advantages
HDFS is a scalable and reliable file system, ideal for data-intensive applications, particularly analytics. The benefits of Hadoop include flexibility, ease of implementation, and the ability to easily increase the size of the cluster. Specialization reduces data movement overhead, and data replication and logging help detect and respond to failures.HDFS Architecture Disadvantages
HDFS architecture has its downsides that include:
– Lack of backup and security strategy can put company data at risk causing high costs of downtime.
– Vulnerability to hacking when data only exists in one location. It is advised to backup data to a remote location to restore it in case of a disaster.
– Data migration process or manual copying can be done to access and analyze data in the local environment.
Understanding Hadoop Distributed File System (HDFS)
HDFS is a reliable and scalable distributed storage system, widely popular across the world. It offers several benefits, one of which is efficient data storage and retrieval across a machine cluster without compromising disk reliability. HDFS also facilitates easy addition of new nodes to expand the cluster as required. A three-tier architecture is used for data storage on HDFS, with a single server storing the data in the first tier, copies of data stored on each node in the cluster in the second tier, and a copy on HDFS in the third tier, allowing fast accessibility to data even if a node is offline.
Additional Resources
Check out the following resources to learn more about Hadoop:
