Open Source Summit + Embedded Linux Conference + OSPOCon 2021

Tensorflow & Pytorch have become de facto tools to build Deep learning models. These frameworks provide abstractions to build the networks and manage the data transport from disk to GPUs. For real world applications, a single machine (with one or more GPUs) is enough to train the model. In practice, the amount of available data greatly outweighs the size of the model. DataParallel training scales model training to multiple machines, often requiring sharing weights after one batch of training data. Horovod (LF AI & Data project) simplifies synchronization across multiple machines. On the other hand Spark is used to extract, preprocess and partition the data. Chaining these pieces together involves complexity and managing disparate infrastructure for each framework. Flyte (LF AI & Data incubating project) provides an innovative and user-friendly way to manage each of these pieces. Flyte simplifies the process of connecting the data output of a Spark transform to a training system using Horovod, while ensuring high utilization of GPU’s. We will describe how we use Flyte to manage and orchestrate data parallel training workflows end to end. We will discuss the benefits and challenges of integrating distributed data science tools in a cost-efficient and developer-friendly way.