Single-Node Multi-GPU

Properties:

Pros:

• All GPUs are in the same machine
• Communicate with each other by NVLink or PCIe
• Super high Bandwidth(300-600GB/s)
• Super low latency(<1 us)

Cons:

• Limited number of GPU
• High Cost to configure one machine

Examples:

• DGX A100
• 4 RTX 4090 Workstation