In this experimental study we try to compare the most popular Big Data frameworks. We divided the experiment study into two parts: (1) batch mode processing and (2) stream mode processing. The experimental study covers the performance, scalability, impact of several configuration parameters on the performance and resource usage study of frameworks such as SPARK, HADOOP, FLINK, SAMZA and STORM. We have chosen WordCount, Kmeans and PageRank workloads with real and synthetic data sets to evaluate the frameworks in batch mode and an Extract, Transform, Load (ETL) process to evaluate the studied frameworks in stream mode.

All the experiments were performed in a real cluster called GALACTICA (https://galactica.isima.fr) and hosted in the LIMOS laboratory at the Clermont Auvergne University. The cluster is composed of 10 machines operating with Linux Ubuntu 16.04. Each machine is equipped with a 4 CPU, 8GB of main memory and 500 GB of local storage. For our tests, we used HADOOP 2.6.3, FLINK 1.1.0, SPARK 1.6.0, SAMZA 0.10 and STORM 0.9. Both Spark and Flink have been deployed in the stand-alone mode, while Hadoop and Storm have been deployed respectively with zookeeper and YARN cluster manager. For all the evaluated frameworks, we used the default values of their corresponding parameters. We also varied these parameters in order to analyze the impact of some parameters on the performance.

We consider two scenarios according to the data processing mode. For each scenario, we measure the performance of the presented frameworks.

- Batch Mode: In the Batch Mode scenario, we evaluate HADOOP, SPARK and FLINK while running our workloads, in the WordCount application we have used a set of tweets as a data set. The used tweets were collected by Apache Flume and stored in HDFS. The motivations behind using Apache Flume to collect the processed tweets is its integration facility in the HADOOP ecosystem (especially the HDFS system). Moreover, Apache Flume allows collecting data in a distributed way and offers high data availability and fault tolerance. We collected 10 billions tweets and we used them to form large tweet files with a size on disk varying from 250 MB to 100 GB of data. For KMeans application we generated a synthetic datasets containing between 1000 and 50 millions learning examples. For PageRank workload we have used four real graph datasets with different number of nodes and edges. Table 1 shows more details of the used datasets. The above presented datasets have been downloaded from the Stanford Large Network Dataset Collection (SNAP)and formatted as plan files in which each line represents a link between two nodes.

- Stream Mode: In the Stream Mode scenario, we evaluate real-time data processing capabilities of STORM, FLINK and SPARK. The Stream Mode scenario is divided into three main steps. The first step is devoted to data storage. Do to this step, we collected 1 billion tweets from Twitter using Apache Flume and stored in HDFS. Those data are then sent to KAFKA, a messaging server that guarantees fault tolerance during the streaming and message persistence .

To allow monitoring resources usage according to the executed jobs, we have implemented a personalized monitoring tool as shown in the Figure below.

Here, we present snapshots of the codes used to implement the WordCount workload with the studied frameworks

Batch Mode: Scalability

This experiment aims to evaluate the impact of the size of data on the processing time. In this experiment, we used two simulations according to the size of data: (1) simulation with small datasets and (2) simulation with big datasets. Experiments are conducted with various datasets with a size on disk varying from 250 MB to 2 GB for the first simulation and from 1 GB to 100 GB for the second simulation.

Batch Mode: Iterative processing

In this experiment, we tried to evaluate our frameworks in the case of iterative processing with both workloads Kmeans and PageRank.

Batch Mode: Impact of some configuration parameters

In order to understand the impact of the configuration parameters on the performance and the quality of the results, we try in these experiments to study some configuration parameters mainly related to the RAM and the number of threads in each framework.

Apache Flink : In order to evaluate the impact of some configuration parameters on the performance of Flink, we first executed our workloads while varying the number of slots in each TaskManager. Then, we varied the amount of used memory by each TaskManager.

Batch Mode: Resource usage (WordCount workload with 50 GB of data)

Here, we measure the CPU, RAM, disk I/O usage as well as and bandwidth consumption of the studied frameworks in a Batch Mode scenario.

Stream Mode: Events processing

The goal here is to compare the performance of the studied frameworks according to the number of processed messages within a period of time. In the first experiment, we send a tweet of 100 KB (in average) per message and in the seconds experiment we duplicated this message fives time (500 kb in average).

Stream Mode: Resource usage (with 100 KB per message use case)

Here, we measure the CPU, RAM, disk I/O usage as well as and bandwidth consumption of the studied frameworks in a Stream Mode scenario.

Batch Mode scenario

Flink projects (Jar files and source code): WordCount | Iterative workload

Hadoop projects (Jar files and source code): WordCount | PageRank | Kmeans

Spark projects (Jar files and source code): WordCount | Iterative workload

Stream Mode scenario

Flink project (Jar file and source-code) is downloadable here

Storm project (Jar file and source-code) is downloadable here

Spark project (Jar file and source code) is downloadable here

Samza project (Jar file and source-code) is downloadable here

Monitoring tool

Our python script (deployed in every machine of our cluster to collect CPU, RAM, Disk I/O, and Bandwidth history) is available in this github repository.

For more details about running our scripts, please follow the steps in this readme file file.