An Earlier Experiences Towards Optimizing Apache Spark Over Frontera Supercomputer.

Apache Spark has become a very popular computing engine that allows distributing computing tasks on a compute cluster. However, the current approaches lack necessary optimizations and often lead to non-optimal resource usage. This paper briefly describes our earlier experiences on the Frontera supercomputer towards deployment and optimizing the Apache Spark components and presents our approach to running Apache Spark jobs efficiently. We ran the simulation at different scales to understand the performance trend of the proposed implementation.