Distributing over 4 GPUs
#10
by
BigDeeper
- opened
I have 4 Nvidia GPUs with 12.2GiB each.
Which workflow, nodes can I use to distribute the models over 4 GPUs' VRAM?
AFAIK there's a few solutions, but I think for that usecase your best bet would probably be using ComfyUI-MultiGPU
The other alternative would be this PR, but with 12GB per card you may run into issues.
As a disclaimer, I don't currently have a proper multi-gpu setup so I can't really test how well these scale.
Nvidia Triton is with dcgm an dcgm-exporter running on each node - this will allow your prometheus and grafana (if you have them) to monitor your distribution nodes performance.
Nvidia has other tools on their developer page, its a very deep subject.