Case file CS-04DSC761Individual project
Hive vs Spark SQL: Benchmarking Table Configurations and File Formats on Hadoop
A benchmark of Hive against Spark SQL on a Hadoop cluster, comparing file formats and table configurations on Stack Exchange comments and GDELT event data.
- Headline
- 35,826Stack Exchange comment records
- Course
- DSC761 Advanced Data Science Technology
- Format
- Individual project
- Programme
- Master of Data Science, UiTM
// 01 · Problem
The problem
On a Hadoop cluster, the same query can behave very differently depending on the engine, the file format and how the table is laid out.
This project benchmarks Hive against Spark SQL across those choices: TextFile vs Parquet, internal vs external tables, and partitioned and bucketed tables.
// 02 · Approach
The approach
STEP 01
Cluster and data
Ran on a Hadoop cluster (HDFS, YARN) using Stack Exchange comments (35,826 records) and GDELT event data.
STEP 02
Table configurations
Created internal and external tables, plus partitioned and bucketed variants, in both TextFile and Parquet formats.
STEP 03
Two engines
Ran the workloads in Hive and in Spark SQL, with PySpark scripts for the Spark side, and compared the results across configurations.
// 03 · Results
The results
2
Query engines
Hive and Spark SQL
2
File formats
TextFile and Parquet
35,826
Comment records
Stack Exchange
- Parquet reduced storage substantially compared with TextFile.
- Columnar formats sped up analytical queries.
- Bucketing helped joins.
// 04 · Tools
Tools and methods
- Hadoop
- HDFS
- YARN
- Hive
- Spark SQL
- PySpark
- Parquet
// 05 · Role & team
Role and team
Individual project for DSC761 Advanced Data Science Technology, Master of Data Science, UiTM.