Open Source Summit + Embedded Linux Conference + OSPOCon 2021

We will discuss many exciting challenges and opportunities for HPC, AI, and Data Science researchers by highlighting recent advances in AI, Data Science, and HPC technologies to improve the performance of deep neural network training (DNN) and data science/HPC workflows/applications on modern HPC systems. Traditionally, DL & data science frameworks have utilized a single GPU to accelerate their performance. However, approaches to parallelize them are being actively explored. We will provide an overview of emerging trends in DL frameworks from an architectural/performance standpoint, and evaluate new high-level distributed frameworks like DeepSpeed, Horovod, Dask, & cuML. We'll highlight new challenges for message-passing interface (MPI) runtimes to efficiently support DNN training and communication backends for Dask and CuML. We will present performance evaluation results from multiple HPC clusters and demonstrate the efficiency of MPI-based backends using micro-benchmark results and applications like sum of cuPy array with transpose, cuDF merge, K-Means, Nearest Neighbors, Random Forest, and tSVD. Finally, we scale DNN training for very large pathology images using model-parallelism to 1,024 NVIDIA V100 GPUs.