Abstract:
Distributed machine learning (ML) workloads, particularly the training of deep neural networks (DNNs) and large language models (LLMs), rely heavily on efficient network communication to achieve scalability. However, network limitations, such as bandwidth bottlenecks, latency, and traffic imbalances, often limit the linear scalability of distributed training systems. This report investigates the network characteristics and communication behaviors of distributed ML workloads through simulation, with a focus on identifying bottlenecks and deriving insights for improving network infrastructure. Using Astra-Sim, a distributed ML workload simulator integrated with NS-3 for network modeling, we simulate various scenarios, including the introduction of network jitter and variations in NIC parameters and congestion control protocols. The experiments analyze the impact of topology, de lay, and protocol selection on job completion times (JCT), revealing key trade-offs and performance bottlenecks. The results demonstrate that network jitter significantly affects performance, particu larly in hierarchical topologies, while lower-latency NICs and efficient congestion control protocols such as DCQCN and DCTCP provide substantial improvements. Looking ahead, this work proposes the simulation of large-scale models across high-dimensional topologies involving thousands of GPUs to analyze network characteristics at scale. These insights aim to guide the design of next-generation network protocols and infrastructure, addressing the challenges of scaling distributed ML workloads.
