Improving Scalability of MapReduce-MPI at Large Scale using Multithreading and Advanced MPI Features

Boyu Zhang
Seminar

MapReduce-MPI (MR-MPI) provides support to run big data applications on HPC clusters. However the data shuffling stage that involves all to all communication becomes the performance bottleneck at large scale. We propose two main optimizations to improve the performance scalability of MR-MPI. The first optimization is to use multithreading to take advantage of intra node parallelism and to reduce the number of MPI processes that participate in all to all communications. The second optimization is to use non-blocking MPI collectives to overlap communication and computation. This talk presents the performance problem with MR-MPI at large scale, the redesign of MR-MPI library in order to optimize performance scalability, and the preliminary evaluation results of our threading MR-MPI using the word count benchmark on Fusion.