Local LLMs Require Significant VRAM
10 posts in the last 60 days, all from r/LocalLLaMA.
“People running local LLMs with Nvidia HW are looking for vLLM in Linux. I mean I understand what it is, but I can't believe they didn't research a bit who the users were. The laptop is a separate thing, this is about the dev box. It it not clear if it's even cheaper than a…”
“What is the issue? Description There is a significant discrepancy between the VRAM usage reported by the ollama ps command and the actual hardware allocation reported by rocm smi when running an LLM with an additional "mtp accelerator" model. While ollama ps reports only size 23…”
“It's going to get worse before it gets better. More companies are now discovering it's better to use local LLMs than pay through the nose to Anthropic and OpenAI for simple stuff like conversational interaction with databases. I snagged an RTX 3090 to complement my thinking…”
Running local large language models demands substantial video RAM (VRAM), limiting accessibility for users with less powerful hardware. Trimming model size, like removing multi-language support, is a workaround but introduces compromises.
SOURCES (13)
“People running local LLMs with Nvidia HW are looking for vLLM in Linux. I mean I understand what it is, but I can't believe they didn't research a bit who the users were. The laptop is a separate thing, this is about the dev box.…”
“It's going to get worse before it gets better. More companies are now discovering it's better to use local LLMs than pay through the nose to Anthropic and OpenAI for simple stuff like conversational interaction with databases. I snagged an RTX 3090 to complement my thinking setup (2x DGX) to route simple agentic work to before prices are completely insane.”
“my computer have rtx3050 8gb r5 5500 and 16gb 3200mhz ram i run these model: gemma4 e4b qwen3.5 9b ornith1.5 9b bonsai 27b (all of them run at 128k context lenght) i run my models on lm studio whit rag and web resarch plugin they have enough "brain" for me they can talk turkish very good and fast(slowest is 30tps fastest is 50tps) and have good tool calling(gemma is bad at tool calling but still works) and good reasoning and very good vision. i dont code(only think code related i do is”
“Hardware: CPU: 7600x System RAM: 64 GB DDR5-6000 CL30 GPU0: AMD 9070 16 GB VRAM | GPU1: AMD 6700XT 12 GB VRAM VRAM TOTAL: 28 GB Models: Gemma-4-26B-A4B-it-qat-UD-Q4_K_XL, 256K context, KV Cache Q8 Quant, FA on Qwen3.8-27B-Q5_K_M, 128K context, KV Cache Q8 Quant, FA on Gemma-4-31B-it-UD-Q4_K_XL, 128K context, KV Cache Q8 Quant, FA on”
“What is the issue? Description There is a significant discrepancy between the VRAM usage reported by the ollama ps command and the actual hardware allocation reported by rocm smi when running an LLM with an additional "mtp accelerator" model. While ollama ps reports only size 23 GB the actual VRAM consumed by the runtime fills the hardware to over 90 GB (consisting of gemma4 31B BF16 + gemma4 assistant 0.5B BF16 ) This suggests that ollama ps is not accounting for additional memory overhead such”