Running Multiple GPUs on Oddball Hardware Is a Mess
83 posts across 7 communities in the last 30 days; 84% from r/LocalLLaMA.
“A bit of backstory : i got lucky in August and was able to grab a CMP 170 HX 8GB for around 500$ on alibaba. I tried later to get a second one for around 1k$ but my order got cancelled and prices went to the moon... I unlocked it with https://github.com/amoghmunikote/cmpunlocker…”
“> better invest in pcie switch, vllm and a bit of research I've done quite a bit of research. vLLM doesn't support half of the features I need and can't run the number of slots/context size on 6 24GB cards on this model. Nvidia doesn't publish support info for P2P on commonly…”
“Hey everyone, I have a workstation that originally had four RTX 3090s in it, and I’m hoping someone here has dealt with a similar power issue, especially with the same Supermicro power distribution board. The system has an ASUS Pro WS WRX80E-SAGE SE WIFI motherboard, a…”
These users are grappling with the limitations of their existing hardware (motherboards, power supplies, PCIe slots) when trying to leverage multiple GPUs or higher-end GPUs for demanding AI workloads, particularly large language models (LLMs). They're weighing the performance trade-offs of dual-GPU setups versus single, more powerful cards, and struggling with VRAM constraints and PCIe bandwidth limitations.
SOURCES (60)
“I have both, rtx 5000 is x2 faster than the 3090, you have nvfp4 support , much better power consumption, hope you got a good price [re: First time GPU buyer. Got a RTX 5000 Pro. Was it a bad decision compared to two 3090s?]”
“pc freeze during strata startup, as i mentioned it may be related to a fact that my second card is on chipset pcie4.0 x4 or its just related to that card or maybe RDNA4? idk but memory locking over multiple gpus is unstable in my case also you have to use "--resident-experts" [re: Strata - Windows - working dual AMD GPU setup with VISION !]”
“any old or newish DDR4 or DDR5 (typically makes little difference for the most part) gaming PC with two x8 slots or one x8 slot and one x4 slot and two 16GB cards and you really have a beast [re: You don’t need that expensive of a build for local llms.]”
“Does it work with model split over 2 GPUs, or is it optimized only for single one? [re: You don’t need that expensive of a build for local llms.]”
“As long as you can run each card at 16x you will be fine. Ideally I like to have a 1:2 VRAM to RAM ratio, but try to look into used dual socket motherboards [re: 3xrtx8000 48gb... what to do?]”
“I have this gr aphics card: ASUS DUAL NVIDIA GeForce RTX 4070 Ti SUPER, 16 GB GDDR6X VRAM [re: Use chatgpt for roleplay or interactive stories. (I'm sad)]”
“maybe take a look at the rtx 5000 pro blackwell if you want expand your multi-gpu rig with something that comes with a better price ratio than the 5090 or 6k pro [re: Add GX10 to 5090 + 2x3090?]”
“You need to install the cards, the drivers and then run the models you want on each of them with the tasks you would run on your local LLM. Check tokens per second and other numbers. [re: Can I self-host local AI (LLMs + image generation) alongside Jellyfin and Immich on my hardware?]”
“You are comparing three different risk profiles, not three hardware options. The used 3090 is cheap now and expensive later, the failure is sudden and silent. The modded 3080 pair is the highest risk in the stack, two used boards with a history you did not write. The R9700 is the only one with a warranty and a predictable floor. If the budget is the binding constraint, buy the one card and add the second when the workload demands it. A single clean 32GB card that you trust beats a 60GB cluster t”
“Hi peeps, this question might come up quite often but could someone advise me on what hardware to get to run some gpu's. I would like to run 4x cards at pcie 4.0 if possible and with DDR4 ram as it's cheaper. Don't mind running at x8 as I believe it should still be quick. I want to run it in an open air case so size and space is not important. I see the epyc systems mentioned frequently and I am considering this. If I get a used server can I just remove the mobo and cpu and place it”
“vLLM-2080Ti-Definitive [re: RTX 2080 Ti 22GB modded card for local LLM inference – anyone tested one?]”
“Not to be the erm actually guy, I mod 2080tis myself and run a small gpu lab. I dont know what someone would consider a "proper bga rework station" but I bought all my gear from ali express and has been working well for me the past couple years. You are right too much temp will cook the chip. Also you do not want lead free solder. Those balls are prone to cracking and require much higher temps to get to melting point and therefore put the memory through more heat stress which can cause”
“Current setup is a Z890 Max Gaming WIFI7 motherboard, RTX5070ti, Ultra 7 265KF, 32GB DDR5-6000 CL30 RAM, 2TB SSD, 1000w Apevia PR1000w PSU. Should have 2 PCIe-4.0x16 (x4) slots open. Also considering getting another identical 2x16gb set for 4 sticks, but they’re $530 rn, so I may hold off since I probably gotta replace the PSU, which is 150-200 as well on top of GPU. submitted by /u/tableball35 [link] [comments]”
“Yes, speed will depend heavily on your memory bandwidth. With those CPUs and memory, it's going to be pretty slow, but it'll work. Note that a typical modern GPU has around 1.8 TB/s of memory bandwidth, your system theoretically has about 136 GB/s, but realistically quite a bit less, so you can expect maybe 3-5% of the speed you'd get with a GPU. [re: Is Running local LLMs on a homelab with a ton of RAM but no dedicated GPU Possible?]”
“Hardware -- RTX 4070 on PCIE 4.0 x16 V100 sxm2 on PCIE 4.0 x4 (limited to PCIE 3.0 x4 cause of pcie adapter) They both are limited to 150 watts, and randomly under load v100 falls of the bus one clean restart fixes it, now I do not know what causes this, and I have no prior experience with ubuntu or dual gpu, so any help would be much much appreciate. submitted by /u/UNO10100f [link] [comments]”
“You intentionally choose to ignore the point. 600W in a desktop or workstation card is definitely possible as the 5090 and 6000 prove. [re: MI350P vs RTX Pro 6000 (Blackwell)]”
“I’m all ears. Educate me. [re: 4x R9700 builds with $127 used EPYCs: $10.7k all-in per machine. I built three (384GB VRAM)]”
“Strata already supports "or AMD Radeon RX 7900 XT / XTX, RX 7800 XT / 7700 XT, RX 9060 XT, RX 9070 / 9070 XT, Radeon AI PRO R9700 or RX 6800 / 6900 series."”
“one thing to check once it runs: the display card only protects you if the desktop and the apps actually render on it. on our box the monitors hang on another gpu, but firefox still rendered through the nvidia card, and every time vllm filled that card the browser crashed with an out of memory in the driver. nvidia-smi shows you which desktop processes sit on the inference cards. pin them to the display card or leave vllm some headroom. [re: eGPU on blackwell for inferencing/display?]”
“GPU- i have asymetric architectures - Titan RTX(a 2018 card released same time as 2080ti) and i have a RTX 3090. Both of these cards are 24gb the TitanRTX does not have flashAttention-2 only standard flash attention SystemRam - i have about 48gb of Free SystemRam to spare. the rest is used up by my chrome VsCode and docker instances. 6000mhz ramspeed. Im getting 32tok/s on qwen3.8-27b running on a single 3090 at 300k context the Titan Rtx is pretty much empty and just sitting there. but im wonde”
“I have a 3rd gen Threadripper paired with this board, which annoyingly has only three PCIe slots. https://rog.asus.com/motherboards/rog-strix/rog-strix-trx40-e-gaming-model/ I am currently running 3x 3090's at PCIe 4.0 x16 each and I get around 100 t/s using QFN on Strata. Now I wish to add a 4th 3090 to run other models and I am not sure whether to use bifurcation or upgrade the motherboard, or the entire system (mobo+cpu). I had a look at the c-payne SlimSAS cards to bifurcate to x8/x8 but”
“The 5090 is a consumer card. The RTX6000 is a workstation card with a maximum of two in a system. The RTX6000 Max-Q edition allows for up to four cards in a system. This is a workstation/ local server part and is not typically sold retail. And this is the comparison people are looking for. The MI350P goes into servers . It does not go into your PC. These are not comparable products. [re: MI350P vs RTX Pro 6000 (Blackwell)]”
“I saw the 4x R9700 build for a 10-person startup here a few days ago and wanted to share my results, since I opted to spend most of the money on the cards. I went with a super cheap used EPYC instead of a Threadripper. In the end it came to about $10.7k per box all-in, and I built three of them for a total of 384 GB of VRAM. As a long time lurker on my other account, I'm always frustrated that people never show how they actually use the models, so after a lot of back and forth I decided to s”
“Yeah, PCIe lanes matter a lot, I have a 5070ti on a 5.0x16 and a 3060 on a 4.0x4 with 64GB DDR5-6400, and I actually get worse performance when tying to use them both. Currently using the 3060 for my monitors and vision module, but I think even that's probably not as efficient as just letting Strata load/unload it like an expert (an option I think I saw get added recently?). [re: Strata performance finetuning - try it out]”
“They are ok, technically the gpu generation is compatible AMD ai software. It uses navi2 and that the oldest GPU compatible. I use the rx6600m with Qwen and is pretty good [re: Someone is running Qwen 27B at ~20 tok/s on a jailbroken PS5's GPU]”
“Hmm. Any reason why 2 cards with some load on it running pcie 2.0 x16? Ive never seen that. The other 2 are on 5.0 as I would expect. [re: 4x 6000 Pro for the company]”
“Whoah, I needed the two R9700s I just bought, for... [re: Got 2x DGX Sparks. Excited and confused!]”
“so what have you settled on? what settings and what performance target metrics? have you disassembled your card yet? i bought a few of these cards when they were very fairly priced but not crazy cheap. low and behold, i discover the absolute worst thermal paste applied after doing basic testing and memory validation to eliminate DOA or paypal refunds etc. i have never seen thermal paste become crystalized, rock hard, bonded onto the IHS like that. also very cheap pads, and, actually quite a few”
“CUDA would not be better optimised if you reverse engineered it and put it on AMD cards though. The stack is optimised for NVIDIA who also has the best hardware for the job. If people want to optimise AMD performance the solution is to improve things like llama.cpp and Strata on AMD ROCm not trying to reverse engineer CUDA. [re: Anyone feel like reverse-engineering CUDA?]”
“I live in 240V land, but I would still need a way bigger circuit to run it off even for me lol [re: Intel "Crescent Island" GPU Gets D-Series Name, Supports Up to 480 GB of LPDDR5X]”
“Dual A100X PCIe (link at Gen4 x8, weird DPU cards) in a 2U EPYC 7402P box, 256GB DDR4. Proxmox with the GPU's passed through to debian 13, 610.43.02 drivers. llama.cpp, qwen3.8 flash next UD-Q4_K_XL, Q8_0 MTP head, draft max 2, mmproj enabled. -sm layer -fa on --parallel 7 --ctx-size 1835008 -ub 2048 --cache-ram 32768 (7 slots x 262k), q8_0 K/V Real world single user performance sits around 50-70tk/s. When bumping things up to 2-3 users/sessions, generation drops like a rock, 10-15tk/s. Seek”
“What about the software ? The harness and the tools? What are you using ? [re: Building a 4x R9700 setup for a 10 person startup]”
“On multi GPU systems, multimedia frameworks default to the primary GPU. It is currently difficult to run Fluster tests against a secondary GPU. We need the ability to configure and run tests against explicit decoders on the same machine, such as in GStreamer:”
“I get it. I tried to be all local. Frontier is frontier for a reason though and I'm doing complex 3d and spatial work. Nothing local comes close at the moment. [re: Building a 4x R9700 setup for a 10 person startup]”
“I'm trying to figure out the best long-term upgrade path for my setup. Right now my GPUs are spread across several consumer PCs: Main: 5900X, 32GB DDR4, 2× RTX 3090 24GB. Secondary: i3-10320, 16GB DDR4, 2× RTX 3060 12GB. Daily driver: 5800X3D, 64GB DDR4, RTX 3080 10GB. Other hardware: Another RTX 3060 12GB (potentially to be put into main as a separate 3rd GPU and an i7-9700 mini-PC running various Docker services) Currently trying out Swift1.5 Qwen3.8-27B FP8 @ vLLM on the 3090s at speeds I”
“Not worth it imo, these GPUS power capped perform within 1-3% of max and do not require a special circuit to be installed so can do with a regular 15a or 20a outlet. [re: Building a 4x R9700 setup for a 10 person startup]”
TRX50 SAGE [re: Building a 4x R9700 setup for a 10 person startup]
“In a couple of years, 99% of those start ups are going to go the way of start ups has been going and all that hardware is gonna be there waiting for me on eBay [re: Building a 4x R9700 setup for a 10 person startup]”
“Concurrent usage might be bandwidth constrained but it's not that kind of bandwidth. The stress for 10 users at ounce would be on the GPUs, not network [re: Building a 4x R9700 setup for a 10 person startup]”
“Just wanted to share a build I am doing for a client. $18k. Specs: Threadripper 9970x 128gb DDR5 ECC 5600 4x AMD Radeon R9700 AI Pro 32gb(128gb total VRAM) 1600W PSU(GPUs are undervolted and under 210w each) Engine : using a fork of Radiance to serve Qwen 3.8 Next flash/27b and DSV4 Flash Results: 16 concurrent sessions Qwen 3.8 27b MXFP4 6.3-6.8k aggregate prefill tok/s 900 t/s - 80t/s aggregate decode(range: 4k - 128k context each) Qwen 3.8 Next flash 48K/user | 6,880 tok/s | 538.5 tok/s aggre”
“That's not entirely true. Nvidia is slower for prompt processing than apple, for instance. [re: No more RTX 5090]”
“You can now with radiance And much faster and better On the flip side, I think you should help port V620 on radiance, and you’d be flying with performance boost [re: $2800 rig with 8x Radeon Pro V620 (256 GB VRAM) + custom vLLM fork = Qwen3.8-Flash-Next at 60 to 100 t/s decode and 3000+ t/s prefill]”
“Nice build! And for relatively cheap! I am curious: which PCIe risers did you use? And which Mobo+CPU? [re: $2800 rig with 8x Radeon Pro V620 (256 GB VRAM) + custom vLLM fork = Qwen3.8-Flash-Next at 60 to 100 t/s decode and 3000+ t/s prefill]”
“A bit of backstory : i got lucky in August and was able to grab a CMP 170 HX 8GB for around 500$ on alibaba. I tried later to get a second one for around 1k$ but my order got cancelled and prices went to the moon... I unlocked it with https://github.com/amoghmunikote/cmpunlocker without any issue to 64GB and restored compute capabilities. I also got a framework desktop 128GB last year (was 2.2k$ at that time...) that i use mainly for linux at home and ai workload (i went from gpt-oss-120b to qwe”
“Yes, that is a great case — 3 cards means 36 GB VRAM total and 300W capped, so both the electricity line and the VRAM fit change. I have queued the preset with the per-card cap in the watts math, so your 170 vs 100W result (11% prefill, 0.6% decode) is the right data to feed it :) [re: How much does your local LLM server really cost? (power draw + tok/s, self-hosted vs cloud API, free in-browser) :)]”
“I have 2x r9700, and 2x 128gb strix halo. The strix halo are both off, the 9700 are on but mostly idle. I use chatgpt and Claude more than anything for actual building, have the local setups for playing around. From that standpoint the strix halo lets you play with more but it's not as usable. I have tried to get a local chat setup a few times, strix halo was always too slow, any other smaller GPU (20-24 GB) seemed to trip over itself because of small context, the dual 9700 is the most usabl”
“Seems 1-slot design, so you could cram in quite a lot into a case submitted by /u/Mr_Moonsilver [link] [comments]”
“Let's preface this with the fact I have 3090, 4090, 2080ti - and lots of stable long running compute heavy workloads. I recently got a 5070ti (I am not willing to pay ridiculous money for 5090) And it's been working reasonably well, until I left it running on a 6 hour CUDA job. Near the end, it died, with system journal message: NVRM: krcWatchdog\_IMPL: RC watchdog: GPU is probably locked! Notify Timeout Seconds: 7 So I tried to reproduce, but no success. My code is fine. Then I investig”
“...without slowing everything to a halt? Edit : I realize this might not be clear, but I'm refering to the iGPU within a consumer CPU, eg Ryzen 9950x, rather than Halo. For example, is there a kind of buffer or operation that could be safely and specifically offloaded to iGPU's RAM? And how? submitted by /u/ParaboloidalCrest [link] [comments]”
“> better invest in pcie switch, vllm and a bit of research I've done quite a bit of research. vLLM doesn't support half of the features I need and can't run the number of slots/context size on 6 24GB cards on this model. Nvidia doesn't publish support info for P2P on commonly available PCIe switches for 3090s. I wonder where is such advice coming from? Are you running this model for daily work? On what hardware/config? How many slots, how big context, how many simultaneus user”
“https://preview.redd.it/s5vu5iiiy4uh1.png?width=1280&format=png&auto=webp&s=813d2bd0c9fa29fe0e2cc41432f95432d1d3e404 > Then two of my cards are watercooled and inside the PC case =) the first switch in the chain is connected via x4 PLX88096 were relatively cheap, but I also spent 500-600 usd on the cables and slots. [re: Qwen3.8-Flash-Next on 6x3090 / 6x4090 without NVLink: prefill 8-10x faster and long-context decode 2-3x faster than stock llama.cpp, binaries included]”
“I bought a Framework Desktop motherboard, and put it inside a Phanteks Enthoo Pro case. I have a spare RTX2060 graphics card. I managed to get it working with the Framework Desktop motherboard, after making it go through two PCIe risers, and mounting it on a vertical GPU bracket. What do I do with my RTX2060? Should I use it as a subagent? I currently configured it as a PCIe passthrough device for my Windows VM. I very occasionally use it for Windows gaming using Looking Glass. I am thinking tha”
“Upgrade Question If we say the "budget max" is $1700-1800, and this could include upgrading to a Taichi motherboard: With a new motherboard , no need to bifurcate 1. Would you add a second 5060Ti (16GB)? Someone is selling one for around $500 locally; 2. Buy a used 7900 XTX (24GB VRAM); local seller, $850 All-in on the GPU , I'd have to make the current motherboard work for my use-case 3. Or, just go for an R9700? (No budget for Motherboard upgrade) Current system - 96GB system RAM”
“I got dual 7900xtx running on a z390 (pcie3 8x) running the 27b. I've been thinking upgrading the motherboard to a x570 (pcie4 8x) or a x870 (pcie5 8x) improving performance with TP, and then wait to buy medusa or spark with lpddr6 and 256GB (apple isn't an option for me). However seeing the gorgon halo price... I'm wondering how much those would cost and if i would pay that much, and if it will be better to go with a wrx80 route just now. I'm not really looking to add more GPUs,”
“Environment Hardware : 2x ASUS GX10 (NVIDIA GB10, 128 GB) connected directly with 1x 200G QSFP cable on Port 0 OS : Ubuntu 24.04 LTS (Kernel 6.8 / 7.0) Atlas Image : ghcr.io/enntity/atlas sparkglm:f047a0c5e796 (latest main ) Problem Description On official NVIDIA DGX Spark units, the ConnectX 7 port uses PCIe bifurcation ("both PCIe halves of the port"), where both rocep1s0f0 and roceP2p1s0f0 bind to the same physical cable. However, on OEM hardware like ASUS GX10 , the ConnectX 7 is wired as tw”
“I'm considering the Gigabyte AORUS RTX 5090 AI BOX (external GPU, 32GB GDDR7, connects via Thunderbolt 5/USB4) as an alternative to building a desktop PC with an internal RTX 5090, specifically for running local LLMs. Does anyone have real-world tokens/sec numbers comparing the AI BOX vs. a desktop RTX 5090 for popular models at various quantizations? submitted by /u/Paco7575 [link] [comments]”
“Thinking of repurposing an old X79 PC for Strata / on my old X79: - i7-3930K -56 GB DDR3 (32gb matched but I have a few 4GB sticks and 1x8GB so they wouldn't match but maybe they work.) - RTX 2080 Ti 11 GB + RTX 3060 Ti 8 GB - CachyOS headless Would Flash-Next IQ3_XXS work well on this? Do I need to go lower? I was also thinking of using an M4 32 GB as a coordinator/router with GLM-4.7-Flash, plus another machine with a 9070 XT running 27B. I tried Gemma 4 26B it 4b JANG, asked it through He”
“The power limit override results in a 370w power limit lock in game on my 5080 Astral OC even though xOCD displays whatever that the power limit is set higher than the max 450w stock limit. So that 370w limit in game is irrespective of whether I set the power limit percentage slider to max or don’t touch it at all after enabling the “Nvidia Extended NVIDIA Limits” seemingly successfully according to the settings enabled confirmation. Before opening xOCD I need to reset everything to default in a”