Lambda Architecture: Combining Batch and Real-Time Data Processing
Lambda Architecture combines batch and real-time data processing for efficient data access and analysis. It is a popular architecture model used by businesses to become more data-driven and event-oriented in order to handle large volumes of rapidly generated data.
This article will cover the following topics:
- What is Lambda Architecture?
- Data Sources
- Batch Layer
- Serving Layers
- Speed Layer
- Query
- How Does Lambda Architecture Work?
- Features of Lambda
- Typical Lambda Applications
- Advantages of Lambda Architecture
- Disadvantages of Lambda Architecture
- Conclusion
Lambda Architecture: A Brief Explanation
Lambda Architecture is a powerful solution for big data ingestion and processing that can handle historical data. It deals with three layers of problems: the Batch layer, Speed layer, and Serving layer.
The Batch layer is responsible for batch queries and analytics using Hadoop or a similar system. It manages the master dataset and pre-computes batch views. Its analytics are referred to as “analytics” and handle past data.
The Speed layer carries out real-time analytics on recent data, using data up to an hour old. It can compensate for the high latency of updates to the Serving layer.
The Serving layer indexes batch views to make them low-latency and fast for ad-hoc queries. With a lambda architecture, data is processed by the batch and speed layers simultaneously, ensuring that the most recent data is used to answer incoming queries.
Data Sources
To conduct data analysis using the Lambda Architecture, various data sources can be utilized. Apache Kafka serves as an intermediate layer between the original data source and the batch layer, storing data in batches at high speeds. The data is then processed simultaneously by both the batch and speed layers to increase indexing speed.
BATCH LAYER
To prepare for indexing, the system creates a model to resemble the changes made in a system of record. This is similar to a change data capture (CDC) system generating a change record (CR). The model is saved in a comma-separated values (CSV) file, and the data is stored as immutable append-only records during a CDC system. Apache Hadoop is commonly used to process the data.
Serving Layers
The Erlang Cache has multiple layers, and one of them is responsible for indexing the latest batch of views. It also creates different indexing schemes for specific purposes and fixes any coding bugs. This layer, known as the serving layer, must process data in a highly parallelized manner to minimize indexing time. Any new data will be queued for the next indexing job if no index currently covers them.
SPEED LAYER
The speed layer complements the serving layer by indexing new data that has been recently added to the system or taking longer to be indexed by the serving layer. Real-time data processing technologies like Apache Storm, Hazelcast Jet, Apache Flink, and Apache Spark Streaming are usually used for indexing incoming data. This helps reduce the latency and narrow the gap between the most recent data and the most recent indexed data in the serving layer.
User Query Processing in Serving and Speed Layers
In this component, the serving and speed layers handle queries from end-users. The component consolidates the results of these processes to produce near real-time analytics.
How Does Lambda Architecture Work?
Lambda Architecture consists of batch layers and speed layers. In batch layers, data is indexed in batches while in speed layers, new data is continuously indexed in real-time. The combination of both batch and speed layers offers a comprehensive and up-to-date view of data.
When a batch indexing job ends, its indexed data is ready to be queried. The speed layer’s copy of the same data and indexes is then deleted, and the serving layer begins indexing the most recently produced data that was already indexed by the speed layer for quick querying. Through this process, all data is accessible for querying while keeping the latency low.
The serving layer eliminates the speed layer’s unnecessary data copies when the serving layer finishes indexing.
Benefits of Lambda Architecture
The Lambda architecture offers several advantages:
- No software management necessary, eliminating installation, maintenance, and administration steps.
- Automatic or manual capacity scaling to fit application needs.
- Built-in fault tolerance and availability for minimal human errors or outages.
- Enables business agility to respond to changing market conditions.
Typical AWS Lambda Applications
Lambda functions and triggers are essential components while developing on AWS Lambda. Lambda functions are used to define code and runtime while triggers are used to invoke them. Below are some examples:
1. A photo-sharing application that stores images in an S3 bucket and creates thumbnails for display on user profiles. A Lambda function can be used to automatically create a thumbnail or manually create one. The function can retrieve an object from S3, create a thumbnail version and save it to a new S3 bucket.
2. A custom DynamoDB table can store and analyze data. You can create a Lambda function that writes, updates, or deletes items in the table using DynamoDB streams to publish item update events. You can then aggregate raw data using the function to create custom metrics.
3. A Lambda function can be created in response to events produced by a mobile-based application. You can configure a Lambda function to handle clicks within your custom mobile application.
Advantages of Lambda Architecture
Lambda Architecture offers many benefits including:
- High fault tolerance and robustness as the batch layer has the entire data set and can restore all data from the point of corruption forward.
- Ability to easily scale using more machines in the distributed top layers.
- Flexibility to compute batch views or perform speed layer computations in various ways.
- Extensibility for new data to enter the system and expand resources for new views.
- Possibility for ad hoc queries through the batch layer despite high latency.
- Ease of maintenance using Apache Hadoop for the batch job layer and ElephantDB for the serving layer.
- Debuggability through simpler debugging of computations and queries.
- Real-time queries provided by the low latency reads and updates of the speed layer.
Drawbacks of Lambda Architecture
Lambda architecture suffers from several drawbacks, which include:
- Requires extensive coding due to the intensive processing involved.
- In some scenarios, every batch cycle is re-processed, causing inefficiency.
- Modelling data with Lambda architecture can be challenging when data needs to be moved or reorganized.
- The complex nature of Lambda architecture results in the maintenance of two separate code bases for batch and streaming layers, leading to difficult debugging.
// sample code here
Additional comments or further explanations can be added here.
Understanding Lambda Architecture
Lambda Architecture is a software design pattern that allows for the loose coupling of logic and data. The primary goal is to reduce dependencies between the two, which aids in writing testable and maintainable code. Lambda Architecture can be applied to various areas of software development, including web frameworks, databases, and web services.
That said, Lambda Architecture has its limitations and benefits. It is essential to understand these thoroughly before implementing it in a project. Overall, Lambda Architecture promotes a separation of concerns that can be beneficial for developing robust and flexible applications.
