Generative AI and LLMs Quiz

4GB

24GB

32GB

16GB

Distributed Data Parallel

Data Distribution Protocol

Dynamic Data Processing

Distributed Data Processing

Reduces overall GPU memory utilization

Increases model accuracy

Speeds up training time

Simplifies model architecture

To synchronize gradients across all GPUs

To update model parameters

To split input data into mini-batches

To perform forward and backward passes

NCCL

MPI

HTTP

TCP/IP

Memory usage and communication overhead

Training speed

Model accuracy

Ease of implementation

Shards both model parameters and optimizer states

Uses a single GPU for training

Increases model size

Reduces the number of parameters

To store model shards for reconstruction

To increase training speed

To reduce memory usage

To synchronize gradients

Need to split the model across multiple GPUs

Increased training time

Higher accuracy requirements

More complex model architectures

To split input data into smaller subsets

To increase model size

To reduce training time

To synchronize gradients