Local LLMs Struggle With Limited Hardware Capacity
17 posts across 4 communities in the last 30 days; 82% from r/LocalLLaMA.
“Hey r/SideProject , If you ask any PC forum if you can pool RAM across old computers, the answer is always an emphatic "No." And for gaming or operating systems, they are right- memory bus speeds cannot cross motherboards over Ethernet. However, for local LLMs, transformer…”
“We need hardware recycle, like, right now. Millions of phones are thrown out every years. In them contain the precious memory. I see no reason why we can't just take out the RAM/nand/mmc and recycle it. What we are lacking is tools and document, which manufactures intentionally…”
“The only reassuring thing I am able to tell myself is that it's not a raw material supply issue or physics limitation, it's just a capacity issue. This tech is so powerful it's going to be a hardware focus for decades to expand ram capacity and optimize for AI compute. The fabs…”
Users are experiencing limitations in running Local LLMs due to hardware constraints, specifically RAM capacity. They anticipate that expanding RAM and optimizing AI compute will be a focus for hardware development in the coming years.
SOURCES (25)
“So on paper ram speed I've managed 110gbs! With 28k context a3b slows to about 45 on the fill but I'm working on that and still getting gains. I have a theory on this that there is improvement available but it IS the base snapdragon...…”
“Hey r/SideProject , If you ask any PC forum if you can pool RAM across old computers, the answer is always an emphatic "No." And for gaming or operating systems, they are right- memory bus speeds cannot cross motherboards over Ethernet. However, for local LLMs, transformer layers process sequentially. Weights only load once into RAM at startup, and during token generation, only the small intermediate activation vector (~650 KB/token) crosses the network. To solve AI memory walls withou”
“I'm actually interested to know why those are important to you. Aside from the requirement of having a GPU. It's been my experience that these models are far more RAM bound that compute bound. Nearly any open model you're going to be running is going to be quantized and not even using the FP capabilities on the GPU. And the speed of the RAM is really only going to affect your token speed. Not model capability. [re: anyone running good open weight models for development in 48GB, 64GB”
“I really want to find a way to get frontier-level intelligence on local hardware.I think some really cool ways to get this have been done already like the technique of spreading an LLMs weights across a fast-enough SSD and then only serving up an expert at a time on the GPU/SoC.I have this idea to make a tool that lets people download several small open-weight models and they act like experts but are different very task specific models with a harness designed to keep track of their thoughts & wo”
“Yeah, that's actually one of the reasons I'm curious about this comparison! It's pretty impressive that an older eight-channel DDR4 platform can still offer that much memory bandwidth. I appreciate bandwidth isn't everything, particularly with Strata doing a mixture of CPU and GPU processing. I'm just wondering how much that extra bandwidth actually helps compared with the significantly faster Zen 5 cores and AVX-512 on the 9950X3D. I might be misunderstanding how Strata bala”
“Soooo I currently have 4*8Gb of DDR4 3200mhz cl16 RAM modules. I usually run Swift Qwen 27B in IQ4 quant from Bartowsky (the goat himself) on my 7900XT (40tks with mtp) for daily stuff, researches and code when I'm out of work token for work related stuff. I would love to be able to run a more efficient and more accurate model like an MoE in Q8 like Qwen Flash Next that utilizes all my hardware instead of solely my 7900XT because 20Gb of vram is just not enough for higher accuracy quants for”
“Depends on how many memory channels they give it. 1.5TB/sec per the article isn't bad, although that's 20-24 channels at conventional speeds. Until RAM prices come back to earth, being datacenter only hardly matters. That base 160GB model is going to be what, $5k worth of RAM? [re: Intel "Crescent Island" GPU Gets D-Series Name, Supports Up to 480 GB of LPDDR5X]”
“It's not weird, faster memory controllers always needed more die area, they have to massively parallelize stuff or something. I guess you could give it 20000GB/s but you will NOT be paying for it, so they give it like 200GB/s bandwidth HBMs get like 1TB/s of 32GB per die, but those get tons of rejects and only cheap in 10 year old surplus form [re: Sipeed announces MaixStation, comparable to Strix Halo and DGX Spark]”
“I wonder what the engineers will come up with. Id be ecstatic if I could run a 100b param model on my desktop at anything over 10 tps [re: Qualcomm CEO reveals AI companies want phones running 100-billion-parameter models continuously by 2028.]”
“Exactly, and it would solve the barrier they have in building new data centers. Public opinion becomes a break. Concentrated energy production too. However they won't find enough RAM in the short time [re: Qualcomm CEO reveals AI companies want phones running 100-billion-parameter models continuously by 2028.]”
“Well, if we can get a working 4080ish 24gb performance for a couple thousand, I wouldn't mind a spot of deception. Odds are that we wouldn't be that lucky, though. I wouldn't be surprised after the AI bubble, such falsified frankenboards would be restitched into much better and informed versions. [re: Be careful fam - New RTX 4090 fakes hit the market, with GPUs and memory almost indistinguishable from real chips]”
“with what ram? [re: Qualcomm CEO reveals AI companies want phones running 100-billion-parameter models continuously by 2028.]”
“Great, I'll have the ram for like a 10k context window [re: Qualcomm CEO reveals AI companies want phones running 100-billion-parameter models continuously by 2028.]”
“Yeah, I think it is possible. I’m using the defaults the ai setup script generated for my hardware. according to the docs I should be able to go higher”
“I'm actually surprised by the number of people telling you to wait in the local subreddit as all signs point to hardware getting more expensive not les Prices WILL improve and LLM's will become easier to run. Prices may not go back to 2020 prices but they won't be nearly as bad as they are now. So I don't think it's bad advice to wait.”
“RAM is in high demand but also in high production and coming from factories that are not allowed any downtime if they want to be profitable. If suddenly the industry realize that OpenAI is not going to use even half the RAM it reserved, prices can go down.”
“We need hardware recycle, like, right now. Millions of phones are thrown out every years. In them contain the precious memory. I see no reason why we can't just take out the RAM/nand/mmc and recycle it. What we are lacking is tools and document, which manufactures intentionally hide from us(under NDA)”
“The businesses which don't trust OpenAI ZDR can trust neoclouds. At the current prices, on-premise AI servers are only feasible for the largest companies who will enjoy benefit of scale. If the prices start to fall tho, then more and more businesses will compete with consumers for the hardware. In any case, the GPU-poor are screwed”
“Is Micron not aware of CXMT or are they actually not set to flood the market with memory?”
“Doesn’t really matter. China consumes 30% of global Ram so if that starts to slow significantly it still creates reduced global demand.”
It's good for lots of specific tasks, depending on the model.
“My solution a while back was to make my LLM box a dedicated LLM box so it doesn't have to do anything but LLM's, and then I just access it over my local network. Right now it's running 64GB DDR5 w/ an RTX 4090, and IQ3_S runs great on it. The system ram hits 55-57GB usage, but it doesn't matter because the box doesn't have to do anything else (that said, I am considering adding a 2x16GB kit to it so maybe I can try UD_Q4_K_XL). My daily driver desktop is more modest hardware”
“The only reassuring thing I am able to tell myself is that it's not a raw material supply issue or physics limitation, it's just a capacity issue. This tech is so powerful it's going to be a hardware focus for decades to expand ram capacity and optimize for AI compute. The fabs will come online, the purpose built hardware will hit the market.”