Hobbyists Fighting Slow Local LLM Inference
Users are grappling with the challenges of running large language models (LLMs) locally, specifically Qwen models. They're facing issues like slow inference speeds, looping, formatting errors, and difficulty optimizing models for specific tasks like coding and tool calling, despite experimenting with different quantization methods and hardware configurations. The desire to utilize these powerful models locally is hampered by the technical complexities and resource limitations.
SOURCES (60)
“If you have a large model, you can distill it to a smaller one with significantly less effort than it takes to train the small model from scratch.”
“Right, sure, but I'm running them both at the same quant? My question is why, in an apples-to-apples comparison, does the higher-performing model seem to do a substantially worse job at basic things like "following instructions"? This, of course, makes me very hesitant to use it for long-running agentic work.”
“M3 is ~428B-A23B - https://huggingface.co/unsloth/MiniMax-M3-GGUF - some of the lower tier quants fit - IQ3_XXS would be a reasonable choice. StepFun AI's models are actually pretty good despite silly name. I find them to be... more creative? Like they aren't going to beat any frontier model at code or whatever, but it occasionally surprises me by doing something in code or prose that is like "Huh, that's neat." Oh, also, there are some 397B fine tunes that I haven't tr”
“I am using the script below. Please advise on what i am doing wrong. GLM5.2 is crawing at 5-9 t/s decode and 60 t/s prefill. The machine is a EPYC 9654 with 768 GB of DDR5 4800 MHz RAM (~460Gbps bandwidth theoretical). This is CPU only inference. I do have a RTX pro 6000 Max Q on the machine, but that's only 96 GB vRAM. #!/bin/bash # ============================================================================== # FILE: start_glm52_ultimate.sh # HARDWARE: AMD EPYC 9654 (96 Cores, SMT OFF, NPS”
“I like to use the MoE modles qwen3.6-35B and Gemma-4-26B. I noticed differences in result quality between versions from different providers, like bartowski, unsloth, lm-studio, google, etc. My tests dont give me a clear answer to though. Is there a rule if thumb which provider to prefer for 1) coding and 2) chatting and general world knowledge? submitted by /u/scubid [link] [comments]”
“This is too specific to individual models and quants. Sometimes Q4 works better that Q5, maybe due to shaving off something that was not working well due to quantization anyway. I've moved on to UD Q8 only and now planning to move to BF16, but hardware requirements will be obviously higher. I've got one coding test which 3.6 27B fails, it prefers to provide single argument callbacks for event callbacks where two arguments are passed (ko library). I expect to see less of such borderline f”
“The biggest single lever I found with smaller coders is shrinking the job. Things that transferred for me across harnesses: ask for unified diffs against one named file instead of whole-file rewrites, keep one task per context and restart instead of continuing a long session, and wire the loop so the model sees compiler or test output after every change. A model this size recovers from errors surprisingly well when the actual error text is in front of it, and surprisingly badly when it has to im”
“So I know this is going to incur the wrath of the Qwen cult but after a month of using 27b q8_0 as my primary coding agent in 6+ agent coding workflow with GPT-5.5 as the orchestrator, I got very frustrated with the amount of back and forth I was doing just to always end up in the same place without a resolution to a simple bug. Out of desperation and lack of progress on a medium complexity project, I switched to Gemma4-31b q8 as the main coding agent and 27b as the QA and Reviewer agents, 35b a”
“I just saw that these had dropped. Still very much early days, but nice to see a new locally runnable foundation model, along with a couple of thinking preview versions. Anyone taken a look at this yet? Am intrigued to see how it holds up compared to Qwen 3.6 and Gemma 4 (my current stack) submitted by /u/Graemer71 [link] [comments]”
“Hey everyone, have you noticed this issue too? In a scenario where a model fails to load when it should for multiple people but the issue persist for a while without being looked into, is it appropiate to ping the maintainers? submitted by /u/Kahvana [link] [comments]”
“This is very interesting, maybe LMStudio + OpenCode is faulty after all, but I can't run Qwen 3.6 27B for agentic tasks with KV quantization at all.”
“How do you configure the quant step-down? I have primarily used ollama which just has simple context config. Is this in llama.cpp?”
“Originally wrote Qwen3-VL-30B off as too slow for document extraction. Running it on a 5090 showed that it was actually stuck repeating the same valid JSON object, so I wrote up what I tried and how Qwen and Gemma behaved across 37 pages. https://coles.codes/posts/grammar-constrained-repetition-trap/ submitted by /u/mattjcoles [link] [comments]”
“benchmarks as of july 16 2026 -- Apache 2.0 Thinking Machines Lab: Inkling , July 15, 2026 -- MIT DeepSeek V4 Pro, April 24, 2026 Xiaomi MiMo-V2.5-Pro , April 22, 2026 GLM-5.2 , June 13, 2026 -- Custom / restricted license MiniMax M3 , June 1, 2026 Kimi K2.7 Code , June 12, 2026 -- Modified MIT license NVIDIA: Nemotron 3 Ultra submitted by /u/Atagor [link] [comments]”
“Don't be surprised if Qwen MoE starts breaking down with even a little bit of a quant. 27B has been steadfast in dealing with quants but when I do the same with MoE, especially with KV quants, they start looping breaking down FAST”
“Have you tried unsloths q4 version? It retains more accuracy than googles qat but might be a little slower. I benchmarked gemma 12b qat and gemma 12b q4 on rag and agentic coding and unsloths q4 beat it.”
Hi sir, are you still using this or you moved on to some other llm?
“What are you doing StepFun model? /s Sorry that model name just makes me unable to resist. Also I'm genuinely curious if you mean just big model. Or should we be crunching our 8-35B local models down that far now?”
“Tries default non-ternary bonsai 27b dflash and it did not load due to it not being supported in llama.cpp yet. For the ternary version I remember they have a fork that supports ternary but not sure about the dflash part.”
i could live with slow generation as long as the processing is fast
“Title says it, is anyone having any luck with the Ternary Bonsai 27B DFlash? I have been playing around with it but have seen no speed up whatsoever. If anything, I've been seeing a slowdown. Running a RX 7900 XT with ROCm. Is this working better on CUDA for Nvidia users, or METAL for Apple users? submitted by /u/linuxid10t [link] [comments]”
“This is about coding. Let's say I have 2 servers. One can host a large model but is slow (like 5-15 t/s slow). The other can host a small model, but is quick. As a general question, how can I use both models to code up an application. Or a specific question, can I have a coding agent attached to each model and have 1 agent drive the other agent with specific smaller coding tasks (like create a method that does X) that it could handle, but the larger model would drive the design. submit”
“GLM-5.2-Int4-Int8 on 8× GB10: ~1,200 t/s prefill, 33–54 t/s avg decode (generic - coding/structured) and memory remaining to run also a Mimo 2.5 in parallel for image/audio input, both tp 8. https://x.com/i/status/2077123292352204943 submitted by /u/ciprianveg [link] [comments]”
“Very impressive release by the PrismML team. 1-bit quantization shrinks it from 54GB to just 3.8GB (-93%), while retaining 90% of its intelligence. - Collection on Hugging Face: https://huggingface.co/collections/prism-ml/bonsai-27b - Demo link: https://huggingface.co/spaces/webml-community/bonsai-webgpu-kernels submitted by /u/xenovatech [link] [comments]”
“In the 200-450B class you have models like DeepSeek v4 Flash, Hy3, MiniMax M3, Qwen 3.5 397B-A17B, and probably others I’m missing. They’re all noticeably better than Qwen 3.6 27B. Qwen 3.6 27B does seem a bit better than Qwen 3.5 122B-A10B at agentic coding tasks, but they’re quite similar aside from the preserve thinking support on 3.6.”
“Qwen3's native window is ~32,768. People keep shipping "1M context" GGUFs of the same checkpoint. What's going on isn't a new brain. It's YaRN (Yet another RoPE extensioN): stretch the position map the model already learned. The only equation that matters: new_len = original_max × factor factor 4 → ~131K (what Qwen actually validates) factor 32 → 1,048,576 (the viral community jump) In config terms you're usually setting something like: rope_scaling.type = yarn rope”
“They can decrease clock speed drastically to decrease power usage and you would get 100 tokens per second when running on battery. When you plug in, say to 50W power bank, you get 500 tokens per second, when you plug in to its adapter, you get 14000 t/s.”
“Hey everyone, I recently switched from DS4 Flash to Qwen3.5-122B on my M3 Ultra Mac Studio for long-context agentic coding. While the model fit better, I hit a wall where follow-up messages took 3-5 minutes to start generating (cold fills) despite having a "warm" context. Turns out the issue wasn't the model, but three specific bugs in my serving stack (qMLX fork of rapid-mlx): Prompt Instability: A unique message ID in the system prompt broke byte-exact KV cache matching, forcing”
Hey all, here are two new high performance qwen3.5 gguf sets I created using a new state of the art technique for optimizing mixed precision called Voodoo Quant. https://huggingface.co/voodooquant/Qwen3.5-0.8B-MTP-Voodoo https://huggingface.co/voodooquant/Qwen3.5-2B-MTP-Voodoo Voodoo Quant…
“I generally found Qwen 3.5 122B A10B a fair bit superior to Qwen 3.6 27B although granted I was using NVFP4 quants. DeepSeek V4 flash beats both - there's Antirez's hybrid 2 bit quant available through DS4 server.”
“I rarely get looping when I max out context, but other than those loops I have had no issues with the model comprehending long contexts of conversations history. Running fp8 kv with the club 3090 dual setup. Pretty sure not quanting kv is your best option.”
“How far can i stretch the context window with Qwen 3.6 27B (using Q8_0) before it gets too unreliable? I am at 100k right now and i am not quite statisfied. Other than not quantizing KV cache, is there anything else that can be done to make the model more stable over longer CTX? submitted by /u/milpster [link] [comments]”
FP8 and NVFP4 are more (and way more) lossy than Q8 or Q6 quants.
“Been testing out the ONNX models to see how far I can push the CPU to take on ASR and TTS, so the GPU is completely free for running the LLM. The video attached shows me testing latency on a 2022 Macbook M2 and an AMD Ryzen 9 7900. This is just running the regex fast commands, so most of the latency (apart from grabbing the spotify music) should be from the ASR and TTS. These ONNX models are really great. The M2 is mostly usable, the Ryzen 9 is blazing fast. The two models I am running are: Daum”
“Lol. You can even run this on 8gb ram. But the speed is probably 1 token per 10 years”
“My experience has been that Qwen3.6 GGUFs Q6_K and Q8_0 are the smallest sizes that work in agentic harness, and I'm not sure Q6_K is good enough, but it is definitely hard to spot if it's mildly confused. The lower quants work fine at first, but after about 100k tokens in context, they start to make mistakes. Typically, they cite file paths incorrectly, display general confusion at longer context where they no longer seem to be able to separate their own messages from my messages, and u”
“Not OP, but what made the biggest difference for me in terms of prefill speed on mi50 was moving from a K quant to Q5_0. 200 to like 600 (2 16GB gfx906 in pipeline parallel) on qwen 27b.”
“Great point on separating weight quant from KV cache quant, most people lump them together and blame Q4 for long context issues when it's really the cache doing the damage. The ctk/ctv trick is something we want to surface directly in the data. I'm planning to use RULER for the context sweep too since it separates retrieval from multi hop cleanly.”
“Yeah exactly. KLD looking good on paper doesn't mean much when the model falls apart halfway through a long coding session that's what I'm trying to actually measure.”
“It is about what I would expect from a netbook running off a glorified SD card and a badly cooled atom processor.”
“I've been running some systematic tests on a few models comparing FP16 vs various GGUF quant levels, and instead of looking at one aggregate benchmark score, I broke it down by capability: math (GSM8K), code (HumanEval), reasoning (ARC-Challenge), and knowledge recall (MMLU-Pro). The results are way more nuanced than "Q4 loses X% quality." For example on one 27B model, Q4_K_M barely moved the needle on conversational/knowledge tasks (under 2% degradation) but dropped multi step mat”