Enhancing Goodput in Large-Scale LLM Training with Nonuniform Tensor Parallelism

NVIDIA has developed Nonuniform Tensor Parallelism (NTP), a framework that enables large-scale LLM training jobs to dynamically adapt to transient GPU unavailability, ensuring sustained Goodput and minimizing computational stalls and throughput loss. NTP incorporates efficient tensor resharding techniques and dynamic power boosting to maintain high Goodput even in the presence of hardware fluctuations.

1sec.ai