VRAM Runs Out Before the Model Fits
236 posts across 15 communities in the last 30 days; 86% from r/LocalLLaMA.
“I am building my first local LLM station and considering to start by picking a good graphic card. The V100 is significantly cheaper, but also apparently no longer supported for future CUDA releases. How big of a problem is that considering the current trend of continuous…”
“Hello guys, hoping you're doing fine. Today I want to share something that may affect your cards (? I have released 3 tools: A way to modify VBIOSes, so you can increase power limit, voltage, NVVDD offsets, MSVDD offsets (Blackwell 2.0), OCPs, fan channels, power per fan, core…”
“The overflows thing is wild, most people would just give up and blame the drivers. 20 t/s on 4 cards isn't blowing the doors off but having it actually stable across multiple B70s is a bigger deal than people realize, that platform has been a nightmare to get consistent results…”
Users are consistently frustrated by the limitations of their GPUs when running large AI models. They're grappling with insufficient VRAM, seeking workarounds like using multiple GPUs or offloading display tasks, and desiring better tools to estimate VRAM requirements. This impacts productivity and experimentation with cutting-edge AI applications.
SOURCES (60)
“i ran a setup like this for about a year, 12700k and a 3080 passed through to a windows vm with the monitors plugged right into the gpu. feels 100% like bare metal for gaming and desktop use, i was playing competitive shooters with no…”
“tbf DS41f quantized runs on a rtx3090: https://github.com/lxe/ds41 [re: 72 hours, four big open weights drops. The only one that fits in 24GB came from NVIDIA, of all companies.]”
“How did you find info on how to repad? [re: PSA for 3090 FE owners: Running LLMs overheats your VRAM]”
Is there one? submitted by /u/yehiaserag [link] [comments]
“Yes this setup (but maybe ddr4) is the point at which any upgrades in performance begin to require accelerating increases in investment, in my opinion which is pulled right out of my head which hasnt really doone much local ai recently so probably pretty worthless if you ask me [re: You don’t need that expensive of a build for local llms.]”
“AMD is a company that somehow fail forwards every time lol idk what goes on over there [re: 72 hours, four big open weights drops. The only one that fits in 24GB came from NVIDIA, of all companies.]”
“Gosh, and here I am feeling like a big shot with 4 3060 12gb cards 😄 [re: 72 hours, four big open weights drops. The only one that fits in 24GB came from NVIDIA, of all companies.]”
“Time to learn about MoE and why the 35B is poor's man 27B. Don't be a confident idiot 😉 [re: 72 hours, four big open weights drops. The only one that fits in 24GB came from NVIDIA, of all companies.]”
“I came here to say this. here nvidia, let me help you with your naming: NVIDIA-Nemotron-3.5-Bolt-74B-A9B-wontfit24GB-GGUF [re: 72 hours, four big open weights drops. The only one that fits in 24GB came from NVIDIA, of all companies.]”
“Some memory is more equal than others. I run 35b a3b on a machine with 64gb of DDR5 and an 8gb GPU with 96k context in Q4, and it runs at around 25 t/s which is acceptable for a laptop that isn't geared to LLM's imho. If it's a better model at E2B then that means more context for people to use that have smaller GPU vram. And cool kids think that's cool. [re: 72 hours, four big open weights drops. The only one that fits in 24GB came from NVIDIA, of all companies.]”
“yeah i know that and i think they know thet too but for some people like 32gb ram and no gpu anything that is not a moe is basically not gonna be usable especially in agents and lfm actually does well there [re: 72 hours, four big open weights drops. The only one that fits in 24GB came from NVIDIA, of all companies.]”
“Those speeds on a 16GB GPU are pretty impressive. [re: You don’t need that expensive of a build for local llms.]”
“Nice reminder that you don't need a massive GPU setup. [re: You don’t need that expensive of a build for local llms.]”
“Probably can work without any vram. It’s just not very usable. I use Q2 with 64 GB of VRAM. Down from Q4 as that was too slow. At least with larger context. I have ddr4 though. Probably Q4 would be possible with strata, but getting a performant solution would be a project in itself and not something I would do in a wide benchmark. [re: Nine models, five coding tasks, one 16GB GPU]”
“Perfect for M5 Ultra 512 GB or 4 x RTX 6000 Pro. I just wish my parents weren't poor :-( [re: unsloth/DeepSeek-V4.1-Flash-1017-GGUF · Hugging Face]”
“I need to study this more - but I'm seeing about 115 t/s on average. That's brilliant honestly for my needs. When I run setup.sh it discourages me from using this particular model saying it needs 64gb but I only have 59gb available, but I just push it through and it seems to be working fine at 256k context. [re: Basalt: NVFP4 + faster prefill for Qwen3.8 Flash-Next on Blackwell (single 5090)]”
“So, before this comment the whole thread was "24 GB with CUDA could become a lot more valuable soon". Fair point, something could happen there. Now suddenly the 3090, which has roughly half the compute units and memory bandwidth of a 5090, should become more valuable. Unlocking 64 GB on a 8 GB card worked for crypto cards, not regular gaming GPUs. The question then remains: What secret thing does a 3090 have that a 5090 doesn't? A less protected BIOS maybe. [re: Everyone go buy RTX”
“I mostly agree with you, but my general recommendation is as least as much memory as vRAM. Sounds kinda weird to say it that way, but I have some workstation cards with 48 GB and ran into issues on a 32 GB host. That said most of those issues were PyTorch related. [re: You don’t need that expensive of a build for local llms.]”
“put all three in one box. with 144gb you can run a much higher quant fully in vram, and llama.cpp splits layers across cards fine without nvlink, pcie bandwidth barely matters for layer split when it's just you using it. check the gguf size for q5 or q6 and leave room for kv cache at the context you want. one thing with rtx 8000s is they're turing, so no bf16 and some newer kernels in other engines expect ampere, llama.cpp is the low drama option. power is the other thing, three of those”
“Yeah, I nearly bought 3 RTX 8000's also, but my bank blocked the transaction due to lack of funds (for the first one, for the other two they just laughed). [re: 3xrtx8000 48gb... what to do?]”
“I accidentaly bought 3 rtx8000 48gb (yeah, i swear). I am using 1 right now with Strata and Qwen Next IQ3 and runs very good. Should I put 3 on a PC and runb Strata on a higher Q? What do you suggest? Thanks! submitted by /u/evillarreal86 [link] [comments]”
“I ran five coding tasks against nine models on one RX 9060 XT (16 GB). Each run gets a single spec file in an empty directory and has to produce solution.py and test_solution.py , run its own tests, and stop. Grading happens with a checker the model never sees: fixed cases from the spec, randomized cases against a brute-force oracle, and a performance gate where the spec names one. A cell is capped at 1750 seconds; if the model is still working when the clock runs out, whatever is on disk gets g”
“Thanks for sharing this - I see the model size itself is 131 GB. So 2 questions: How does it differ from the NVFP4 model used in strata here ? I only have 5090 + 64GB and I'm able to run the NVFP4 model above. But I guess I'm out of luck with yours? [re: Basalt: NVFP4 + faster prefill for Qwen3.8 Flash-Next on Blackwell (single 5090)]”
“It actually does do CPU experts, it's just tuned for my box by default. Experts that live in RAM are split between copying over PCIe and computing on the CPU pool, and --pcie sets that share (0.75 = 75% go to the GPU). On my 2-channel DDR4 the CPU expert path and PCIe both top out around 23 GB/s and share the memory bus, so leaning on the GPU won. On 8-channel RAM you'd want a lower --pcie so more runs on the CPU. The FreeToken paper has a nice rule for it (roughly PCIe bandwidth / CPU e”
“I am building my first local LLM station and considering to start by picking a good graphic card. The V100 is significantly cheaper, but also apparently no longer supported for future CUDA releases. How big of a problem is that considering the current trend of continuous optimizations and findings such as n-grams for MoE, or diffusion models and so on? Still learning here, and genuinely overwhelmed by the pace I have to keep to have even a grasp of what’s going on out there. Thank you in advance”
“Hello guys, hoping you're doing fine. Today I want to share something that may affect your cards (? I have released 3 tools: A way to modify VBIOSes, so you can increase power limit, voltage, NVVDD offsets, MSVDD offsets (Blackwell 2.0), OCPs, fan channels, power per fan, core offsets more than 1000, VRAM offsets more than 3000, lower power limits (below 400W on a RTX 5090), lower clocks, etc. But as you know, these can't be flashed directly on the cards because falcon, right? A custom V”
“https://preview.redd.it/r29oiw12szvh1.jpeg?width=2040&format=pjpg&auto=webp&s=dc8514e4532d5e7c234a66890a8ebcec293d0ab4 4x AMD MI50 32GB 2x Epyc 7532 16x 16GB 2133@2667mhz DDR 4 [re: Show me your rack]”
“The overflows thing is wild, most people would just give up and blame the drivers. 20 t/s on 4 cards isn't blowing the doors off but having it actually stable across multiple B70s is a bigger deal than people realize, that platform has been a nightmare to get consistent results on [re: vLLM finally working on 4 x Intel B70, QFN 3.8 FP8]”
“Hardware----------------- 4070 12gb - gen 4.0 x16 v100 16gb - gen 3.0 x4 32gb 3200mhz ram it5-12400f ssd read - 3.6gb ssd write - 2.7gb -------------------------------- quant- rco iq3_xxs -------------------------------- the prefill on both single and dual gpu is near 200, never around 1k like people show and say, I thought its cause of v100 pcie, but even on 4070 its very slow. I would love any and every tip I can get, or flags I should use for dual gpu, thanks a bunch. ------------------------”
“I hope I'm wrong but don't get your hopes up - typically Nvidia models aren't as parameter-efficient as Qwen or even Gemma. The larger models like the 120B and 74B look pretty good though. [re: Why do we have models under 15b so rarely anymore?]”
“96gb VRAM is not that far off from ~112gb of unified memory on the sparks. And with 96gb+ of DDR5 memory as well that can be used for slight offloading whereas the sparks can’t add more memory [re: 2x3090 to 4x3090 Upgrade Vs DGX Spark]”
“Just like with memory bandwidth, they can simply be clocking it higher. Clock the chip a little faster, it has a little more compute. It's not night and day. It's about the same difference as in memory bandwidth. [re: RTX Spark is faster than DGX Spark]”
“I don’t think anyone would. After seeing his setup of 8 RTX pro 6000s only hitting 156 tok/s on Deepseek V4 flash, his “benchmarks” lost all credibility in my eyes. [re: RTX Spark is faster than DGX Spark]”
“Alex Ziskind posted a review of a RTX Spark laptop. It's faster than DGX Spark. He also says there's no problem with throttling. Go watch his review on youtube. submitted by /u/fallingdowndizzyvr [link] [comments]”
“Everything is in the title. Does this computer worth it? submitted by /u/dh7net [link] [comments]”
“Planning for projects: science and research first, coding more for drawing up a good plan. Maybe 10tks min speed on any engine. What are you guys running? submitted by /u/GodComplecs [link] [comments]”
“To host bigger models, upgrade the existing 2x3090 rig to 4x3090 (with a threadripper perhaps) or buy a dgx spark or equivalent? what makes more sense overall? Factors? - distributed vram vs unified memory - high combined vs low memory bandwidth - 1200-1400w vs <200w - ampere vs blackwell submitted by /u/Dapper_Chance_2484 [link] [comments]”
“Considering an ASUS GX10 to 'future proof' my setup, and of course to run big models when i already have 5090 and 2x3090 based boxes. 3.8 27b runs pretty decent on both, can try 3.8 flash next, maybe, what else? Million $ q: should I buy Asus GX10 or Nvidia gb10? submitted by /u/Dapper_Chance_2484 [link] [comments]”
“What is the issue? Environment OS: Windows 10 (10.0.19045) Ollama: 0.40.5 GPU: AMD Radeon Pro 5700 XT VRAM: 16 GB Backend: Vulkan Model: gpt oss:20b Model ID: f38aa0c53da5 Format: GGUF Quantization: MXFP4 Model size: 20.9B parameters Ollama correctly detects the GPU and uses the Vulkan backend. Problem gpt oss:20b loads successfully and performs inference on the GPU, but returns an empty response. The model generates tokens, but the returned response from /api/generate and message.content from /”
“Worth the price is the wrong frame. The question is whether it was worth the alternative. If the next-best card would have run your models at 4-bit with the context cut in half, you did not buy a card, you bought headroom. Price it against the model you will be running in twelve months, not the one on your desk today. If that answer is yes, the R9700 was cheap. Hardware questions resolve only against a future workload, never against the spec sheet in front of you. [re: Was my R9700 Worth the pri”
“The X299 platform is the constraint you are not going to outgrow by adding cards, it is the limit you have already hit. A fourth 3090 on that board will trade stability for a number that is only slightly bigger. If the workloads are staying at current 70B and below, the three cards are already the ceiling the platform supports cleanly. Sell the extras into the 5090 market and buy a current platform with room for one larger card. Bigger models are going to win on a single fat GPU, not on four nar”
“Start with the workload then build the system to do it. That's all you have to do. This post seems so contrived to me. [re: 4 R9700's vs the 2 5090s I am currently using.]”
“Do you power limit your cards? [re: **DeepSeek-V4-Flash-0731 on 4x Radeon AI PRO R9700 with affinity: the "refused" 4-card config works, ~79 tok/s on real cards**]”
“or RTX discrete GPUs (32GB+ / 48GB+); I don't like when quants are basically eating all the vram on 24GB [re: nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB · Hugging Face]”
“Recently NextUI officially introduced H700 support for RG SP, 35xx SP, etc. Is it possible to add too?”
“Same GPU setup with 64GB of DDR5 RAM and a 98K context window. Using dFlash it averaged 139 t/s. I kind of like it and can def find a place for it in my setup, especially when I need something reasonably capable and fast as hell. Results from llama.cpp UI on the Chessboard SVG test that was kicking around here a while ago: Context·29.26K / 98.30K 30% used 69.05K remaining Token usage details Across all turns Prompt tokens evaluated 941 tok 302 reused from KV cache Tokens generated 28,015 tok Thi”
“Now I’m wondering if I should go get a 4th card or try to get this working with my TP3 setup I have right now [re: **DeepSeek-V4-Flash-0731 on 4x Radeon AI PRO R9700 with affinity: the "refused" 4-card config works, ~79 tok/s on real cards**]”
“I have 288GB of combined VRAM and I still run Qwen 3.8 27B for nearly all coding tasks, which could easily be run on 48GB. [re: anyone running good open weight models for development in 48GB, 64GB or 96GB of RAM?]”
“You need to give us more details. Is this a system with a GPU or without a GPU? What GPU, if so? Is this DDR3? DDR4? DDR5? Dual channel? Single channel? Quad channel? What CPU? There are reasons I ask this - different types of models perform differently in system ram versus vram versus a mix of the two. (Largely it's about active parameters versus total parameters, but there's also related stuff like NGrams in Qwen3.8-Flash-Next) [re: anyone running good open weight models for developmen”
“192gb for nvfp4 with 1.8M kv cache! [re: Introducing Muse Spark Open: the most capable US open-weight model to date]”
“We’ve all repeated the same rule on here for the last two years: never go below Q4_K_M unless you enjoy watching your model hallucinate its own name. Between today's Nemotron 3.5 Bolt 74B-A9B drop and Liquid AI’s LFM2.5-24B-A2B yesterday, I think we officially have to retire that rule—or at least put a massive asterisk next to it when Quantization-Aware Training (QAT) and hybrid architectures are involved. While Meta was busy locking Muse Wonder behind a $4/$20 API this week (RIP Llama open-”
“Its useful for people with worse hardware. Theres lot's of people still running qwen 35b because they dont have 24gb of vram, and an moe runs way faster off of ram. [re: nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB · Hugging Face]”
“I'm not sure if this information helps but the 5080 24GB is rumored to release in Q1 2027. https://videocardz.com/newz/nvidia-rtx-5080-24gb-reportedly-launching-in-q1-2027-board-partners-still-waiting-for-details [re: Open or not?]”
“Boring, but this is the right answer. [re: Best Model for 48gb Unified RAM users?]”
“ah fair enough - I might end up upgrading my gaming PC with a 12700k to a r9700 and running strata on that machine [re: Best Model for 48gb Unified RAM users?]”
“Swift fixed this for me entirely [re: nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB · Hugging Face]”
“Hi, I have the following specs: 110 Gigabytes of DDR4 RAM Two intel Xeon E5-2630 V4 No dedicated GPU, but fastfetch shows a nVidia geforce 210 discrete GPU. My goal in this project is to be able to run a modal equivalent or slightly better than gemini 3.8 flash, this is the only model I use currently for coding so if I am able to replace it with something local that would be great for having governance over my tools. Since this model will be used for coding only, logically a dedicated GPU wouldn”
“When loading a model across 2 GPUs, the layers are split evenly, but the GPU memory usage is quite a bit higher on the first GPU: This seems to be due do CUDA0 compute buffer size being 1GB higher than the CUDA1 equivalent. When using text generation webui & llamacpp I'm able to specify a 50,51 split which results in the second GPU getting a layer or two more, thus balancing the memory usage, and allowing bigger models to run (or more layers to get offloaded). Does this exist? If not, is it poss”