Software: Uniffle 0.2.0 Hadoop 2.8.5 Spark 2.4.6

Hardware: Machine 176 cores, 265G memory, 4T * 12 HDD, network bandwidth 10GB/s

Hadoop Yarn Cluster: 1 * ResourceManager + 6 * NodeManager, every machine 4T * 10 HDD

Uniffle Cluster: 1 * Coordinator + 6 * Shuffle Server, every machine 4T * 10 HDD

Spark's configuration

spark.executor.instances 100
spark.executor.cores 4
spark.executor.memory 9g
spark.executor.memoryOverhead 1024
spark.shuffle.manager org.apache.spark.shuffle.RssShuffleManager
spark.rss.storage.type MEMORY_LOCALFILE

Shuffle Server's configuration

rss.storage.type MEMORY_LOCALFILE
rss.server.buffer.capacity 50g

We used spark-sql-perf to generate 1TB data.

Uniffle is a little 9% slower than vanilla Spark. Because the amount of shuffle is tiny.

We generate 1TB data, we use the code of the repo

Overall Time: Write Time: Read Time:

Overall Time: Write Time: Read Time: Uniffle is 30%+ much faster than vanilla Spark when there is a large shuffle.