This repository provides a comprehensive guide and practical examples for training deep learning models using PyTorch across various parallelism strategies. Whether you are working on single-GPU training or scaling to multi-GPU setups with Distributed Data Parallel (DDP) these examples will guide you through the process.

• Foundational concepts of deep learning and PyTorch.
• HPC Environment Setup:
  • Using SLURM for job scheduling: Submitting and managing training jobs.
  • Loading necessary modules: Configuring PyTorch and CUDA on an HPC cluster.

• Efficiently training models on a single GPU.

• Scaling models across multiple GPUs using torch.nn.DataParallel.
• Key Considerations:
  • Understanding inter-GPU communication overhead.
  • Differences between DP and DDP for better performance.

• Leveraging torch.nn.parallel.DistributedDataParallel for efficient multi-GPU training.
• Setting up process groups and distributed samplers
• Advantages of DDP Over DP:
  • Lower communication overhead.
  • Better scalability across multiple nodes.

• Running PyTorch training using NVIDIA Enroot and NGC Containers on HPC.
• Topics Covered:
  • Importing and running NGC PyTorch containers with Enroot.
  • Running single and multi-GPU PyTorch workloads inside containers.
  • Using SLURM to launch containerized PyTorch jobs on GPU clusters.

• PyTorch Documentation
• PyTorch with Examples
• Data Parallel
• Distributed Data Parallel (DDP)
• Hugging Face Transformers
• NVIDIA Enroot
• NGC PyTorch Containers

• If you are already familiar with deep learning with PyTorchand HPC, you can skip 01. Introduction to Deep Learning and go directly to 02. Single-GPU Training.