Distributing over 4 GPUs

#10
by BigDeeper - opened

I have 4 Nvidia GPUs with 12.2GiB each.

Which workflow, nodes can I use to distribute the models over 4 GPUs' VRAM?

AFAIK there's a few solutions, but I think for that usecase your best bet would probably be using ComfyUI-MultiGPU
The other alternative would be this PR, but with 12GB per card you may run into issues.
As a disclaimer, I don't currently have a proper multi-gpu setup so I can't really test how well these scale.

Nvidia Triton is with dcgm an dcgm-exporter running on each node - this will allow your prometheus and grafana (if you have them) to monitor your distribution nodes performance.
Nvidia has other tools on their developer page, its a very deep subject.

Your need to confirm your account before you can post a new comment.

Sign up or log in to comment