Local AI Rigs Cost More Than Most People Expected
65 posts across 12 communities in the last 30 days; 71% from r/LocalLLaMA.
“I work at B3IQ, we sell and host NVIDIA GPU systems, so I’m on the owning side of the buy-vs-rent thing a lot. The one part I still can’t give anyone a clean answer on is depreciation. When you own the hardware, whatever the card loses while you hold it is on you. The problem is…”
“Well, the models could be better. QFN is pretty good, but it takes multiple repeats to get stuff working. And I run at 4-bit because I can't afford 8-bit let alone 16-bit versions of this model, so perhaps it's more confused than it should be. I am happy to hold what I got. QFN…”
“No I'm just capable of choosing a model that is sufficient for completing the task at hand. If you think needing frontier models from closed labs for basic tasks is a flex, it isn't. You're probably one of those people that can't wait for a new model release, excited to ask it…”
Several posts express frustration and financial strain related to the high costs of running local AI setups. Users discuss expensive hardware, power consumption, and the difficulty of justifying the investment, highlighting a disconnect between aspiration and affordability.
SOURCES (60)
“Stop fucking around and pay the 200$ lol. All that hassling makes no sense, when you get top notch AI for 200$. In 2 years top models will run on gtx 1080, so there no point of investing in those. Models has become smarter and…”
“Got lucky with a 5070 Ti for $750 AFTER TAX (might be the best decision I've ever made tbh) back in November 2025. If only I had known I'd be getting into local llms at the time...anyways. I recently decided to start messing with local AI. I began getting comfortable in Powershell and having local models help me write random automation scripts. Naturally, that spiraled into wanting bigger models, which meant hunting for more VRAM. I had my old 3070 sitting around, so I mounted it outside”
“Been sitting on this for a few days and it's finally worth writing up now that most bugs are fixed. A fellow degenerate called nullmoth has vibe coded this solution for us! I am waiting for an NVME drive to test this. The project on GitHub (NullMoth) has an open-source Metal driver for NVIDIA GeForce cards on macOS Sequoia. Not a translation layer bolted on top. Actual Metal 3: argument buffers tier 2, ray tracing, mesh shaders, MetalFX path. Turing and newer is supported, which means GTX 16”
“I feel like this is good starting point for a meta post in this sub to perhaps start a discussion on the rule here. submitted by /u/esiy0676 [link] [comments]”
“what a clown that reptilian dude. basically it's probably a good model that only 1% will be able to run locally. [re: Unsloth Muse Spark Open is out!]”
“I, for one, believe the current overinflated prices of GPUs are fully warranted, given what is to come, and what is already possible today. Serious coders spend hundreds of dollars on Claude every month. Many go far beyond the $200/month plan. Meanwhile, an RTX 5090 equipped with ample DDR5 RAM is turning out to be a highly capable inference machine. It can run Qwen3.8-27b of course, and even Qwen3.8-Flash-Next at lower quants, which is suitable to meet most coding needs for basing things like w”
“It might be a silly question, but why people are spending tens of thousands of Euros to run models locally? I can understand doing it for privacy, but I don't think that's the reason that drives most people to do it. When GPUs can be rented online for a fraction of the cost, a situation that becomes even worse for models, and when hardware improves at a breakneck pace making current generation quickly obsolete, what makes people think that investing in a local system is the best choice?”
“Expand the replies to this comment to learn how AI was used in this post/project. [re: What self-hosted inference server actually shares one GPU across models?]”
“I work at B3IQ, we sell and host NVIDIA GPU systems, so I’m on the owning side of the buy-vs-rent thing a lot. The one part I still can’t give anyone a clean answer on is depreciation. When you own the hardware, whatever the card loses while you hold it is on you. The problem is I don't think anyone can predict that number very well upfront. From what I've seen, two cards of the same age can end up with very different resale values depending on the model, condition, and whether anyone st”
“I've been reading about all these modders using AI to reverse-engineer games and software. I've been trying out photocraft and gridcraft and have just watched a GTA5 mod featuring Sonic. It's pretty damn impressive. As a struggling AMD Radeon owner I'd love to see AI generated open source CUDA implementation. Not only do I dislike NVIDIA, but there's delicious irony in using CUDA cores to do it. Anyone up for the challenge? submitted by /u/Odd-World4925 [link] ”
“We rented GPUs for our inference for a long time, mostly because it’s the obvious way to start. There is no capital committed, and if the workload changes, you can just stop renting. What changed for us was the workload going steady. Once the card was running all day, every day, the monthly rental bill started making less sense. If usage is bursty or part-time, renting still wins easily, and owning a card that's idle is just wasted money. So the real question is whether my usage is steady en”
“ah here. "A regular gaming pc" ffs. [re: The MoE tradeoff seems bad for local hosting]”
“Indeed the hardware-inefficiency of local inference is in principle good, not bad, for hardware vendors. But until the manufacturing capacity exists to ship local-inference systems with enough VRAM to enough customers then inference is going to keep taking place at (more-)efficient centralised providers and nVidia will have to keep trying to sell what it makes to them to remain an AI player.”
“Tommorow's world might be different though. I doubt investors will continue pouring unlimited amounts of money into closed AI labs blackholes. [re: Building a 4x R9700 setup for a 10 person startup]”
“Can you explain how running local models on hardware I already own is somehow going to become more expensive? Educate me, please. [re: Another peaceful day in mathematics.]”
“Hi everyone! I'm very new to AI and have spent the past few days doing some research on running AI locally. I only have my personal PC with 32GB of RAM and an RTX 3060 Ti. Based on what I've learned so far, I'm planning to go with this setup: LM Studio → Qwen 3.5 9B (or 8B) → Hermes My expectations are pretty simple. I just want a personal AI assistant that can help me manage my schedule, write simple texts, give it permission to view some Notion page, and do some quick research. I m”
“Yeah, there is no way I have the cash to build a gpu farm (a bit hyperbolic, but you get me) big enough to host a local model for doc review and research, masking personal data or find and replace, and some easy meeting summary stuff, sure.”
“Well, the models could be better. QFN is pretty good, but it takes multiple repeats to get stuff working. And I run at 4-bit because I can't afford 8-bit let alone 16-bit versions of this model, so perhaps it's more confused than it should be. I am happy to hold what I got. QFN is already usable, in theory good enough to get anything done that I need. I'm one of those CRUD app web monkeys and QFN definitely has me served. What I want is mostly either higher quality, or smaller size,”
But in this voice: https://m.youtube.com/watch?v=F2y5q31E1gQ
“I thought the same initially, but for me it's actually the prefill speed that makes 3-bit worth using. With 64 GB of VRAM you can fully fit the model (excluding the PLE embedding table) into VRAM at IQ3_XXS with 120k context at Q8. That takes my prefill from around 500 tok/s to about 1,800 tok/s. I do wish that I could the the same with the full context length at native precision and dynamic 4-bit weights, but the price just isn't right for a third GPU currently.”
Oh sorry yes. I have quite a strong PC, a 7900 XTX and 5800x3D
“No Agree - you can run. I have tested it as well (in 4 GB Vram with very low context) . The key challenge is the performance drops hard in q1/q2 and you are left with very little ram to run other programs like vscode/harness etc. Anyway, good job and have fun.”
“Hello everyone, I'm completely new to running AI models locally and would appreciate some guidance. my laptop specs GPU:Nvidia RTX3050 6gb VRAM CPU:Intel13th gen i5- 13450HX RAM:16GB DDR5 I wanna run an AI model locally to help me with cybersecurity in general because any other public agent wont do what i ask for like any hacking question submitted by /u/Plastic_Artichoke153 [link] [comments]”
“That sounds like a really practical split: laptop for portable offline classification, desktop rig for the heavier models. Did you actually fine-tune the 9B model on the M1 Pro, or was that inference only? I'm trying to separate "a model runs" from genuine local adapter training.”
“That's a great real-world local stack. Running a main model, embeddings, and speech simultaneously offline is exactly where a desktop with plenty of VRAM wins. Are all three staying resident on the 4090, or are you offloading any part to system RAM?”
“Fair point. When you tried it, was the main limitation VRAM, thermals, training speed, or just the price-to-performance? Tailscale into a desktop is definitely compelling if portability isn't essential.”
“Love the little sim you made! I got a GPU earlier in the year specifically for AI (to run Qwen 3.6 27b) because at the time I thought sparks were slow, didn't want 2 when I was just starting out, and didn't want to manage more hardware (OS, updates, etc). Now I sort of wish I would have gone with the sparks, they seem pretty capable. I haven't been in local LLMs too long but since I heard of it, GLM 5.x was always a 'dream' of mine, it just comes across as top-tier. It may be”
“This is an easy fix: bash sudo bash -c "systemctl set-default multi-user.target && reboot" Then use the AI rig remotely and have all the systems' resources ready for AI. If you want to use it as gaming rig again, run bash sudo bash -c "systemctl set-default graphical.target && reboot" and enjoy. That's how I do it. edit: Before anyone follows my advice, obviously make sure that you can SSH into the rig”
“Disclosure: I run Kitani.AI , an open model inference provider. Hey guys so we've been experimenting a lot with AMD GPUs lately, and I'm starting to think the "gap" everyone says between AMD and NVIDIA for inference is a lot smaller than people assume once the software stack is actually optimized. We've been working on optimized kernels and serving configs for some of the newer MoE/open models. The economics have gotten good enough that we're currently serving MiMo V2.6”
“I have drastically overbuilt computers so I end up just hosting whatever I can or want. I idle at ~40GB of RAM on Jupiter alone. During busy moments that peaks to about 100. Good on you for making 8GB of RAM work, that is a challenge I'm sure, but it also makes you more deliberate and come up with creative ways to accomplish something. For me self-hosting is about control and privacy as well as "for fun" so the line is really on how much I want some third-party to potentially have”
“Tbh I don't think we should complain about it the more people attempt their take on spending up llms on consumer hardware, the better the chances we get a genuinely good breakthrough.”
“Once my new job hits I’ll make the move to dedicated hardware. Got a lil bump in salary taking a new position so I’m holding out on my lil 4090 until I can get some real VRAM budget onsite”
“I'm keeping a list of newer engines for RTX3090+ here: https://github.com/tomByrer/rtx3090-AI/tree/main#inference-engines But the ones Tac mentioned will give you access to most models, my list focuses on very few models.”
“Absolutely not a universal truth, but maybe truthful for many ( if not, the significant majority ). And it’s not about comparing quality between frontier models and 20b-30b models. It’s about value. And with my personal investment in AI hardware and my specific use for that hardware, I feel confident in my local AI financial value being greater than I’d get in frontier models.”
“I’m on Intel. So any of my performance is like half of your 3090. You probably already have really nice quants for Intel like nvfp4 or whatever though. I had to make my own…. https://huggingface.co/Wrapzii/Swift-1.5-Qwen3.8-27b-GPTQ-Int4-baked-v1-embed-int8 https://github.com/Wrapzii/k8v4-xpu”
“Basically a DGX Spark minus the ConnectX-7 ports. I’ve seen expected initial pricing from like $1800 to $2900. Not sure what configurations are actually at those price points. It’s all Internet hearsay until we actually see these things ship, but it’s nice to know that it’s potentially around the corner next week, especially given DGX Spark’s insane price increases lately. Sadly, you can’t cluster them, but getting an entire computer + GB10 equivalent chip for a little over half the price of a u”
“This just seems like total nonsense to be honest, why would I have an AI generate software on demand when there almost certainly exists faster, secure, audited alternatives to what I want. It's not like the hallucination's are ever going away. Maybe if you don't care about the quality or assurance of the software, and it doesn't talk to other machines, sure. Almost all the software on my computer does not fit that standard.”
I'm making a videogame and I need the llm start in the background, I'm using Ollama, what I should use? Thanks
“I have an rtx3060. Running 100% raises my power consumption from 200w to about 400W.”
“What do you actually do with AI that helps you? Not a dig, just genuinely curious as to how people are using it for j assume monetary gain?”
“Haven't dug into it yet, but curious how this is different to the Burn rust project, which can target WebGPU for it's output kernels (after optimization steps in an IR)”
“Im 100% sure they were just trhowing money at it to get to AGI or ASI, but they dont know they are just making the perfect tool for coders or boost the productivity of people”
“No I'm just capable of choosing a model that is sufficient for completing the task at hand. If you think needing frontier models from closed labs for basic tasks is a flex, it isn't. You're probably one of those people that can't wait for a new model release, excited to ask it to one shot minecraft.”
“Been lurking on all the "My Homelab" posts this week, so here's mine. Hardware is nothing special: a 3-node Proxmox cluster with Home Assistant, a Minecraft server for friends and the usual pile of self-hosted services. The weird part: the whole lab is run by a crew of Claude AI employees. Richard is CEO, Gilfoyle does security, Dinesh does network and DNS, Big Head does storage (and somehow still keeps the backups alive). Each one owns its own services and has a runbook, a memory”
“I've spent time tuning my GPU power curve down, This really helps. I got my 2x3090Ti from 600W to 200W with literally no loss in performance.”
“Most definitely better to use blender through API. Just thought it was worth highlighting that computer use is becoming mature on the local models. Slowly but surely.”
“For small tasks like that then sure, but as soon as you go a bit bigger into reasoning models then cloud wins again. DS4.1F hits 350-400 t/s generation speed, nearly impossible locally without some genuinely insane hardware”
“You know how much it costs to run your rig 24-7, if your project runs amuk and you use 2t tokens its fine you were always going to pay the power bill anyway, if its now a seperate bill you no longer know what llm usage costs you.”
“Newer models are generally trending more powerful at the same compute cost, so running older models just because doesn't make economic sense.”
“Your GPU idles at less than 10W High end machines idle at far more than 10W. The only high-end local AI machine that is power efficient AFAIK is the Mac Studio. Idles at 5W, consumes 150W at the wall for inference on the M2/M3 Ultras. By my math, it's by far the best local power efficiency you can have. Because even at 5 tokens per second, the electricity cost beats the API rates of most frontier models by an order of magnitude. I don't know about the M5 Ultra yet, but the math probably”
“Computer doesn't get cheaper just because more data centers come online. The cost of electricity and GPUs continue to go up. Training current models and new models get more and more intense and compute hungry. Inference scales liberally with use.”
“Where are you finding 3-bit benchmarks on that site? I've got no idea where those are located. It's also a benchmark rather than real world usage, but even those benchmark numbers you gave are really far apart. Qwen 3.8 Flash Next is good, but I wouldn't put it at Opus 5 levels.”
“it could go both ways tbh. Most people will probably just use cloud AI because it is convenient, but I can definitely see local AI becoming more common for people who care about privacy or just want something that works without an internet connection”
“Well I'm glad to tell ya that your 9700xt is worth ~1K now, let's end it hear on such a positive note. After all you can run a 2B on almost everything so let's call it an hobby and in case let's query an API. :)”
“The idea that Frontier AI prices will soon skyrocket is often an excuse to justify buying local hardware. Frontier price cannot become unaffordable, because of Chinese competition, unless they become way better then Chinese ones, which they probably won't. And they cannot lose either, because the US administration will not accept it. So the goverment will support them regardless of their losses. So no, prices are not going up significantly anytime soon. But local AI is here to stay and more”