VRAM Runs Out Before the Model Fits
Users are consistently frustrated by the limitations of their GPUs when running large AI models. They're grappling with insufficient VRAM, seeking workarounds like using multiple GPUs or offloading display tasks, and desiring better tools to estimate VRAM requirements. This impacts productivity and experimentation with cutting-edge AI applications.
SOURCES (60)
“https://preview.redd.it/0nv83e02idnh1.png?width=1080&format=png&auto=webp&s=3fdf90ddd184a2de44c5778e5c680e951268fc62 Hello everyone, I'm wondering if this is a normal speed for a model with 10 billion parameters and on RTX Pro 6000 video cards, this is a normal speed. I'm interested in talking about this. submitted by /u/zemondza [link] [comments]”
“Feature Summary 常驻显存从 66.3GB 降到 40.3GB（社区实测），净省 25GB，无画质损失。 Detailed Description MiniMax H3 的 33B Omni Transformer 中，约 13B 参数（约 40% DiT）位于 AdaLN 分支（50 个 block，每个 adaln proj 是 [96768, 2688]，输入只有时间步嵌入 timestep embedding）。 这些投影的输入与序列内容无关，只依赖时间步。因此纯推理时可以： 1. 采样开始前，用时间步 t 一次性算出各层 scale/shift； 2. 缓存结果（仅 745MB）； 3. 丢弃 13B AdaLN 权重，后续采样步只保留主干 20B。 预期收益：常驻显存从 66.3GB 降到 40.3GB（社区实测），净省 25GB，无画质损失。 对 24GB 消费级卡的现实意义 / Why it matters 在 24GB 显卡（如 RX 7900 XTX）上跑 MiniMax H3，当前只能依赖 stream layers（权重驻 RAM、按段搬回显存）。”
“A potato can create a very cool RPG in 24 minutes Laptop I3 8gb RAM 0gb VRAM, Windows 11 llama-server.exe --host 0.0.0.0 --port 8080 -m models\qwen3.6-35b-Instruct\Qwen_Qwen3.6-35B-A3B-IQ2_XXS.gguf -c 8192 -n 8192 -tb 4 -b 512 -ub 512 --cache-type-v q8_0 --cache-type-k q8_0 -fa auto -ngl 0 --temp 0.0 --reasoning off MODEL: https://huggingface.co/bartowski/Qwen_Qwen3.6-35B-A3B-GGUF/resolve/main/Qwen_Qwen3.6-35B-A3B-IQ2_XXS.gguf (I use the Instruct model to avoid so many reasoning tokens) RESULT:”
“Any VRAM is useful. I have 4x3090 now, but I had 3x3090 some time ago and adding a single 3060 was useful for some use cases.”
Yep. Rule of thumb: vram poor? if yes, you dont get choices
“Describe the feature I hope to add a community benchmark sharing feature: Users can submit their real‑world test data, including: 1. Machine hardware config: GPU model, VRAM, CPU, RAM 2. Model name & quant version running on FreeToken 3. Real measured performance: token/s prefill/decode speed 4. Max context length achieved, stable or limitation These submitted benchmark records can be browsed/searchable for other users. Use case Users who are making hardware purchase decisions can directly look”
“It doesnt fit thats the thing. 4070ti has 12G VRAM and this quant with this much context and 8bit KV is well over 20GB”
“we're in this weird period of time where qwen and gemma are way too good for their respective strengths and beats out bigger models unless you go ~250b+. so around 32gb vram is the first jump, then around 48 gives you good quants and context. then more vram means more subagents lol until you get to the, idk, 150gb+ free ram+vram regime. coding will feel like ass unless everything is on vram.”
“It feels to me like there are three tiers of local inference. The first begins with a single GPU and your options are open to a number of dense models or small MoEs. Increasing RAM or VRAM can afford you more context, but you won't really see a valuable jump until you hit the 128-256 GB RAM/VRAM mark. This would be the second tier, gaining you access to mid-sized MoE models or small dense models with maximum context length or higher precision. The last tier is 512 GB+ systems, which allows y”
“complement no. sell it and use that money to build something with at least 48GB VRAM -- preferably 64 to accommodate multi-agent workflows.”
“Oh that's great I absolutely do not want someone elbows deep in python and cuda. Can I ask what GPU you have”
“I did say that sparks are great for MoE models, so I'm not disagreeing with you. For 27B there is little question GPUs are better unless you take a spark cluster and use it for large concurrent context”
“NVIDIA is publicly supportive of open weights, so why don't they release the weights of DLSS 5 for it to be fine-tuneable? It's probably an extremely small model considering that it has to run at 60 fps on RTX 50XX hardware, which would make it easily and quickly fine-tuneable for anyone. It's optimized to run on only those cards anyway, so probably making it open weights would not be a competitive disadvantage. In fact, it would make it more competitive once the community builds the”
“Hey guys, Just bought a PC mainly for local experimenting with AI/coding + plus the occasional gaming sesh and it hasn't even arrived yet 😅 9950X, 7900 XTX 24GB, B650 Tomahawk, 32GB DDR5 (likely going 64GB+), planning to run Linux/llama.cpp. I've noticed used RX 6800 16GB (cannot afford more )cards are still accessible, which got me curious about adding one eventually for LLMs. That would give me 40GB VRAM across the two GPUs, but the second PCIe slot on my board is only PCIe 4.0 x2. I”
“I have an EPYC 7742 with 512 GB of DDR4 and 2x Nvidia RTX Pro 4500 GPUs. My numbers are consistently higher than the equivalent DGX setup.”
“I'm planning to upgrade my workstation (linux with 5700X/64GB DDR4) for local inference and pytorch training. I'm trying to decide between: 2× AMD Radeon AI PRO R9700 32GB 2× AMD Radeon PRO W7800 48GB I already have an RTX 3090 24GB , so the final system would have 3 GPUs . My motherboard has two PCIe 4.0 x8/x8 slots available for the two AMD GPUs. The RTX 3090 would have to move to a PCIe 3.0 x4 slot. My workload looks like: 1. Local GGUF inference : Mainly coding/reasoning models and m”
“One GPU drops off immediately when vllm is started and the other throws CUDA errors on start submitted by /u/cantgetthistowork [link] [comments]”
“MI100 does not have enough critical in the community for good support. R9700 is newer and easier to obtain while MI50 is much cheaper and more available, so these two are getting all the AMD fans.”
“Totally not jealous or anything, yeah 32GB vram, totally nothing *cries in gtx 1060*”
“Ollama's automatic GPU/CPU allocation generally works well, but there is an issue when multiple agents use different models. If GPU VRAM is partially occupied by another model and there isn't enough remaining VRAM to load a new model entirely, Ollama may partially load the new model into GPU VRAM and put the rest in system RAM. This can be extremely inefficient when the CPU is much slower than the GPU. For example, with a 32 GB GPU: Model 1: 8 GB Model 2: 30 GB Current behavior: Model 2 partiall”
“But at what quant? My numbers are for the unsloth Q8_K_L at full context without any q8kv cache. A 5070ti is faster, but a single one does not have the vram to not have to sacrifice something else.”
“My current workstation has 32gb ram and I have three 16gb cards of all different brands laying around. With a riser and some Lego I managed to wedge the AMD xt 7900 and Nvidia Quadro 5000 card into my case (of the top of my head) With llama.cpp I get about 20t/s on Qwen 3.8 27b q4_k_m with 128k context fully GPU resident. It's totally viable as a coding model, just too slow... Of course you can spend 10k on a machine which is not happening. I'm wondering if there is like a 1k upgrade tha”
“This was the question I started my master's research with ~4 years ago. I was working on GPU algorithms for graphs with billion edges (500 600GBs) while having access to just 40GB GPU. The obvious answer was "use a larger GPU", except, ahem, govt institute, ahem budget 🙂 That's where I started my exploration on how to utility the available resources better. What initially started as a GPU Programming / engineering problem quickly became an algorithm + architecture + memory + d”
“Flash Next on a mixed 5070 Ti + R9700 box — anyone got this running? I just lost 2 days figuring out the newest amd pro driver has an unload bug that freezes the system... 🤪 I'm trying to get Flash Next running locally and I'm stuck. Claude first told me my VRAM is too small. After I pointed out that it's a MoE, it said the model still wouldn't run properly. My setup: RTX 5070 Ti (16 GB) plus a Radeon AI PRO R9700 (32 GB), 96 GB system RAM ddr5. I also got a nvme gen 5 with 14gb”
“I am seeking information regarding parameters for GPU monitoring for an internal tool. Specifically, I would like to identify the key metrics for monitoring GPUs and their associated agents, including relevant queries and performance indicators. Additionally, I am interested in exploring any open-source tools designed for this specific purpose. submitted by /u/technicalfault_ [link] [comments]”
“Got this 32GB VRAM card (basically 2x WX7100 in one card) for 100 USD. Gigasteal. I want to be running local LLMs for the most part but i was wondering what else can i do with it? Do you have any other uses submitted by /u/sentry_ahead [link] [comments]”
“The fact that MTP slows you down at 80% acceptance tells you the bottleneck is host RAM bandwidth, not compute. On a model fully in VRAM that acceptance rate would be a net positive, but with experts spilled to RAM the draft tokens eat the same bandwidth your real decode needs.”
“Your setup sounds beastly with 3 GPUs. Do you have a single power supply running all of this? I’m pretty sure I have the clearance in the case if I get creative, but my power supply is 1000W which I don’t think will suffice for my 5090+3080”
“I'm adding a 2nd 5060ti. Already have it but waiting for a mb that has x8x8 pci. Running q4kxl. 200k context but when it gets 3/4 full it chugs down to 10 tok/s”
“IQ4_XS will fit but you'll have to sacrifice some context and quantize the KV cache. Here's my llama.cpp preset for my 9070XT, but bear in mind my baseline vram usage is <1G so YMMV. In my case it fits juuuust barely [qwen3.8-27b] hf = unsloth/Qwen3.8-27B-GGUF:UD-IQ4_XS ctx-size = 100000 cache-type-k = q5_0 cache-type-v = q4_1 ubatch-size = 1024 cache-ram = 8192 mmproj-offload = 0 threads = 8 threads-batch = 8 temperature = 1.0 top-p = 0.95 top-k = 20 min-p = 0.0 presence-penalty = 0.”
“I heard that Nvidia GPUs with 16 gb of vram can run it (4 bit quant of it) through some nvidia only tech ? I am not sure. Edit: 4 bit quant or closest I mean, the largest quant/size possible. submitted by /u/Badhunter31415 [link] [comments]”
“Gotcha. If it helps, I'm getting the prefill numbers mentioned in the post using a 5700x and running ddr4 3200.”
“I was thinking OP can do 2 x r9700 into a separate box, and then leave the existing 5090 in another box. The 64GB VRAM box for LLM and the 5090 box for media. But the problem is the r9700 would run LLM slower than OP current box. I'm drooling at the idea of 64GB VRAM to run 27B at full precision and serving multiple streams in parallel though.”
“Nice. How’s your 4 card performance? Are you running them on a server or workstation board? I just bought an epyc 7443 and and board that will give me 4x PCIe x16 lanes that I plan to run them in”
“I already have a 5090 that I use got Hermes and coding mostly. My only jealously is trying models that don’t fit in my VRAM. I do want to get into some more media creation (the 5090 would be better for it, I know) and I was thinking I could use a spark for coding too (give it some problems that a smarter model could benefit from or just for more local horse power in general). submitted by /u/DustNearby2848 [link] [comments]”
“the generation gap kinda gimps both cards. you're limited by what the 3090 supports with older cuda libraries. it'll still be better than the 3090 alone but the 5060 ti alone would be a tad more capable.”
“if a high end rubin based gpu comes for gaming, it will have less vRAM than 5090, granted. At most, it will be a 24GB card. Even more likely, a 20GB. NVIDIA is just about AI cards and realized that 5090 can decently run 27b models costing one third (original MSRP) than a 6000 pro. With almost the same speed...”
“for me it seems same, and I cannot think about any mechanism that should alter math operations just by doing them slowly. Setting power to 290-300W seems to be quite optimal for my case.”
“I wanted to try using the larger models on my computer (32GB RAM, RTX 5080, Gen5 NVMe), but the best I could do was around 30B. So I started with the idea that it might be possible by taking advantage of the fact that MoE models use only some of the experts rather than all of them.MoE-Direct essentially uses the three layers of SSD, RAM, and VRAM instead of residing entirely in memory, caching only the necessary experts in RAM and making the model usable even with resources far smaller than requ”
“i'd test a brand-new 2 minute project before spending another dollar on hardware. if that edits cleanly, you've isolated it to that project/media instead of Chrome or RAM.”
“As a preview of what people might expect running the q4 in 512GB. submitted by /u/burritoresearch [link] [comments]”
“Hello, newbie and long time lurker here. I want to build a budget AI rig for my upcoming Datascience and AI Masters and found 2 Tesla T10 16GB (TU102) GPUs (200€ each) and am not sure if they’re worth getting. If I get them I’d also build a cheap X99 Xeon system around them to support both at PCIe 3.0 x16 because I’d like to also experiment with Tensor Parallelism. I am also not sure if they’re worth getting, because I found Mi50 16GB selling for 175€ each. Their power draw may be double the T10”
“I really love FreeToken’s goal of running large MoE models on consumer grade hardware. However, when using long contexts (e.g. 32K+ tokens), my GPU VRAM is quickly exhausted. After looking into it, I found that the KV cache is one of the biggest memory bottlenecks. Projects like LM Cache solve this by offloading part of the KV cache to SSD, which significantly reduces VRAM usage. Since my PC has 32GB of RAM and a fast NVMe SSD, this approach fits my setup very well. What I’d like to see: Suppor”
“🐛 Describe the bug I ran Inferact/Qwen3.8 2.4T A95B NVFP4 tp2 dp4 ep on 4 4GB300 and saw after such estimation I got fail for sure I had around a dozen of such standalone sweeps and every time got estimation 118.65 GiB. With VLLM MEMORY PROFILER ESTIMATE CUDAGRAPHS=0 and gpu memory utilization 0.85 the endpoint ran successfully. I suspect that something is too pessimistic in CUDA graph memory profiling estimation. PS I saw similar pessimistic estimation on another tp/dp configs but here it is m”
“What is the issue? Today, Ollama prompted me to update when I tried to run Qwen3:14b. I ran the oneline install command ( curl fsSL https://ollama.com/install.sh | sh ), and during the install, it said "Nvidia GPU installed.", even though I run an AMD card. Now, whenever I try to run a model, it won't use my gfx card. Right before updating, I observed my GPU being utilized with radeontop . I watched the VRAM fill up, etc. Now, I can see that my Graphics card isn't being utilized at all in radeon”
“It would be helpful if there was an API endpoint to see the hardware details that the Ollama server is running on. Especially in research/accademic computer clusters, it can be a little bit tough to ensure that the right resources are being used. Having the option to dump those details would help a ton!”