Predict deep learning training through graph neural networks

What is it about?

This research introduces Driple, a novel approach using graph neural networks (GNNs) to predict the resource consumption of diverse workloads in distributed deep learning systems. The study addresses the crucial challenge of accurately estimating resource needs for training deep learning models, considering various settings like GPU types and workloads. Unlike previous attempts, Driple accommodates a wide range of combinations of settings and workloads. By leveraging GNNs and transfer learning, Driple efficiently predicts resource consumption, including GPU utilization, memory usage, and network throughput, contributing to better management of training time and associated costs.

Featured Image

Photo by julien Tromeur on Unsplash

Photo by julien Tromeur on Unsplash

Why is it important?

In the context of the rapidly evolving field of deep learning, where models are becoming larger and more complex, predicting resource consumption is vital. Driple addresses the uncertainties users face when configuring execution settings for distributed training, such as GPU types and the number of GPUs. Existing challenges stem from the diverse choices in training settings (devices, parameter servers, etc.) and the variability in workloads (models, datasets, hyperparameters). Driple's contribution lies in its ability to accurately predict resource consumption for a broad spectrum of scenarios, aiding users in optimizing their deep learning model training and reducing the time and effort required to tailor predictions for different settings. Key Takeaway: Driple, utilizing graph neural networks, effectively predicts resource consumption for a variety of distributed deep learning workloads and settings. It introduces transfer learning to adapt predictions to different settings, significantly reducing the time needed for tailored predictions. With its ability to predict GPU utilization, memory usage, and network throughput, Driple offers a comprehensive solution for users to estimate resource needs, ultimately enhancing the efficiency and cost-effectiveness of distributed deep learning model training.

Perspectives