Hobbyists Fighting Slow Local LLM Inference
Users are grappling with the challenges of running large language models (LLMs) locally, specifically Qwen models. They're facing issues like slow inference speeds, looping, formatting errors, and difficulty optimizing models for specific tasks like coding and tool calling, despite experimenting with different quantization methods and hardware configurations. The desire to utilize these powerful models locally is hampered by the technical complexities and resource limitations.
SOURCES (60)
“For testing you should be able to add {"reasoning_effort": "medium"} to the model's settings ("Chat Template Kwargs").”
“xhigh was used, yes. Unfortunately I've not found a way for the benchmark client to pull the used reasoning effort from oMLX.”
“Q8. Q4 doesn't save a bit but waste as model failed to utilize poor quality kvcache.”
“with the same prompt, I obtained less tokens, so i think the gain will be the same, it's all about using or not using extra parameters (otherwise they are not used)”
“How good/bad is it against comparable MoEs the same size? How does it compare against Qwen 3.6 35BA3B? Since we don't have 3.8 35B this seems like an upgrade if we look at some benchmarks like terminal bench, but they don't have SWE bench pro on the benchmarks table, and i don't really know anything about this lab, I'm wondering if it trades blows with models like tiel coder or if it's some benchmaxxed model like ornith? At a single glance it looks really decent but haven'”
“I’d probably choose Qwen 3.5 2B, but I wouldn’t expect much from a 2B model for coding beyond simple tasks. Between Q8 and Q16, I’d stick with Q8. Q16 uses significantly more memory and may run slower, while the quality improvement is likely to be small. If you have enough RAM for Q16, I’d rather use that memory for a larger Q8 model.”
“Is using their q8 version fine or will i get better results on q16? submitted by /u/Charming_Barber_3317 [link] [comments]”
“Managed to get this small model to run on the $250 MSRP SoC board level computer. The inference speed is kind usable. Used 4-bit quant, q8 kv cache, 7.4 GiB memory supports 128K context length. Device tops at 25W power, and idle less than 10W. Quite suitable for a simple agent running 24/7. Needle in a haystack test pass at 128K context length. 2046 needles passed out of 2048 needles. - **2048-needle (fully random unique word+number pairs, seed 20260902): 2044/2048 (99.8%) @ 90K prompt**, finish”
“Goal Run Qwen 3.x 27b locally for agentic coding - I'd also run other models of similar or smaller size for other uses Would this hardware be appropriate (for starters) or would I hit a point of frustration pretty quickly? Specs - MSI B650 tomahawk motherboard (included the info b/c I know you can't really run 2 GPUs in here, but I could swap this for another AM5 that can handle x8/x8, something like the X870E?) - Gskill 64gb of memory at 6000mhz and 36cl - I've read offloading some”
“Which KV cache do you use F16 or BF16 for Q6 quant? What is the difference between them? submitted by /u/esw123 [link] [comments]”
“I'm on M3Max 64GB. Qwen-3.8-27B is amazing, but pretty slow. What's the best MoE model under 40B for agentic use? Is it still Qwen-3.6-35B? Ability to reliably use tool calls would be important for agents. No hope for Qwen-3.8-35B? submitted by /u/chibop1 [link] [comments]”
“Well i tried running the local model after downloading it , its Qwen3.8 27B UD Q4 K M here is nous link that i generated https://support.nousresearch.com/diagnostics/0a798cd3 3400 4959 b5a3 23d32b1e8d93 and here is the error i copied from session ── Hermes error details ── time: 2026 09 03T10:59:14.528Z layer: provider code: server error retryable: true provider: llamacpp model: Qwen3.8 27B UD Q4 K M error: HTTP 500: model name=Qwen3.8 27B UD Q4 K M failed to load”
“I'm setting up the AI integration in Paperless-ngx v3 that allows it to connect to an Ollama instance , and I'm trying to find out what models other self-hosters are successfully running, particularly on low-power Intel systems with integrated graphics. My hardware: UGREEN DXP4800 Plus Intel Pentium Gold 8505 Intel integrated GPU 16 GB RAM — single 16 GB module Second RAM slot is currently empty Ollama running in Docker Paperless-ngx v3 My main requirements are: Document classification T”
“Pretty please, these are some of the most performant quants (quality and speed) for their size I have ever tested. exl3 Qwen 3.8 Flash Next: https://huggingface.co/turboderp/Qwen3.8 Flash Next exl3 ExLlamaV3: https://github.com/turboderp org/exllamav3 In my setup, the 4.05bpw model fits comfortably within 36GB VRAM + 64GB system RAM with decent context. When trying to load exl3 .safetensors:”
“You don't want an MoE model quantized to 4_0 bits. The routers are too sensitive to quantization, so it ruins the models capacity to select the best expert for each token. You get a huge drop-off in output quality. The reason you get all these different sizes is due to experimental methods of quantization that search for what tensors harm the output when quantized. Those vulnerable tensors end up preserved at 8-bit or full precision, while the more resilient ones get quantized fully to 4-bit”
“Don't go back to Q4_0? Those are compatibility quants, not good quants. And it's not apples, oranges, and mangoes, it's gotten to the complexity of a spice blend at this point. Why isn't it as easy as "the good old days?" Because people figured out a naive, equal, blanket level quantization is not as effective as targeted quantization. Different models are sensitive in different ways to quantization. There aren't really any standards anymore, most of the labels are”
“Not even sure if that plea should be addressed at llama.cpp but, look at the Q4KM quants of Qwen3.8-Flash-Next: Quantizer Size Unsloth: q4km is non-existent but the closest is q4kxl 111 GB Lmstudio 119 GB AtomicChat 94.5 GB Bartowski 120 GB AesSedai 135 GB Ggml. The goat itself. Only Q8 but not far from the mean 163 GB mradermacher. No Q4KM but only iq4xs 97 GB What does Q4 even mean if the variance in size is 150% ? What are we comparing here if we have apples, oranges and mangoes? Shall I succ”
“That's a good point, but it aces all kinds of coding benchmarks, at xhigh, for a reason. For me, xhigh has been worth it for discovering edge-cases and push-back against my requirements, but within a very strict yet obvious constraint: The given task has to be small enough to be finished within a session. If it has to go back to me so I steer/correct then it's a lost effort since the 256k context simply can't hold that much thought tokens.”
“I'm just saying qwen is extremely sensitive and it sounds like the same issue I had.”
“I do wonder whether if it was an issue with the harness or with the model. Haven’t used Pi in a while, but IIRC you should be able to see what happened by viewing the session logs, stored in jsonl. It could’ve been that the harness silently failed reading the file, and because the model saw that it read a file, it just assumed it had that info now. The fact that there was a visible tool call but no preview of that content is a bit weird. Could also be that Qwen just went off the rails, but I’d d”
“How many people here are using 3.8 in less than Q8 though? I mean I’m not sure I really count it as using 27b properly if you’re doing something like Q4”
“I've seen a lot of people (and people in videos) mention that bypassing the default extra-high reasoning effort of both the qwen 3.8 models currently released is overall better. Is this your experience? Or do you leave it as default or set to medium? EDIT: Forgot to mention this is purely for agentic coding (pi coding agent) For llama-cpp, the way I'm doing this is by using this: --chat-template-kwargs '{"reasoning_effort":"low"}' submitted by /u/J”
“https://preview.redd.it/6h4xc5o8l6nh1.png?width=1158&format=png&auto=webp&s=b65074b6baaa1faa2347e5259229c8ba803bcd4b Here is link to repo: https://github.com/perplexityai/pplx-garden/tree/main/lily It's optimized for just one model to get best perf on apple silicon submitted by /u/Specter_Origin [link] [comments]”
“Quite surprised to be beating other high quality quants. It took a lot of benchmarking to get here and we are quite pleased with these, hope they are useful to the community. It required a modified way of measuring KLD with a new dataset, since the NGRAM got in the way by remembering basically all of wikipedia. We tried to not only go for high precision, but also keep prefill performance in mind. Full model card here https://huggingface.co/agentionai/Qwen3.8-Flash-Next-AP-GGUF Let us know if the”
“I will take a small 4 - 8b model that calls tools reliably and an OpenZIM MCP server over a larger model while relying on internal knowledge any day of the week. A picture free version of the entire English wikipedia is smaller than most 30b-tier models in unquantized form, and I'd sooner have a model that reliably explores the knowledge over a model that tries to answer from internal training weights. I have had good luck using ling-3.0-tiny for the OpenZim MCP server for example, that mode”
“Any model from qwen 3.5 series would do. Id suggest starting with 9b and going smaller until you feel results are fading. Or on opposite end, if you habe resources, you can go larger. My general purpose model for visual file sorting was qwen 3.5 122b, since i tend to clutter my system a lot with multiple format of files. Audio is not one of them so qwen was all rounder. Now its qwen 3.8 flash next.”
“Yes, but again - this chart is comparing the old unsloth UD2 quants, not the new UD3, which are much better and smaller, as seen in the above posted chart.”
“You're free to browse their HF for more: https://cdn-uploads.huggingface.co/production/uploads/6a54dea4f19f5386700504da/dmMjNbYfWXrhVGAVeBroC.png Only so much "dynamic" you can do with a dense model. They publish metrics which is nice.”
“I've honestly heard about much faster speeds with my setup but I'm yet to get MTP working. It usually is around the same speed rather than being the significant speedup. It's part of why I'm wondering what other people are using for their setups.”
“Extended reasoning spoiled me. I can't trust anything without it anymore. Qwen 3.8 Max is 100% correct with any challenge I throw at it, with the downside of taking hours before it can find the correct answer”
“Your current environment <details <summary The output of <code python collect env.py</code </summary </details 🐛 Describe the bug Title [Performance]: Qwen3.8 Flash Next long prefill workload periodically starves active decode for 3 7 minutes on 2 node DGX Spark TP2 Current environment field Replace the placeholder below with the complete output of the latest collect env.py executed inside the running vLLM container. Describe the bug Summary On a two node NVIDIA DGX Spark deployment running Qwe”
“That said, FP8 and NVFP4 are horribly space inefficient quantisation methods. Using 8 bits to get a quality equivalent to ~5.5 bits per weight. And 4 bits to be equivalent to 3. https://quesma.com/blog/qwen-quantization-quality/#kullback-leibler-divergence”
“Yes that's right. You can see exactly which tensors in the HuggingFace web interface. https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/blob/main/Qwen3.8-27B-UD-Q4_K_XL.gguf Scroll down. Notice even though this is a "Q4" quant, parts of the weights are Q8 or even F32.”
“imatrix q4 beats a static q4 at the same size. bartowski or unsloth q4_k_m / q5_k_m is fine. kv at q8 usually moves the needle more than swapping another weight file.”
“There's soooo maaany options to choose from, AutoRound from Intel even, Unsloth, bartowski, etc ... which one is the closest to BF16 in Q4/Q5 range ? submitted by /u/soyalemujica [link] [comments]”
“Kaitchup just posted results of his benchmarks for Qwen3.8 27B for quants from different labs, Q4 to Q1, . All the details are hidden behind the paywall, but high level result is visible and looks like for people with 16GB cards UD Q3_K_XL is a winner - it has accuracy of 100% and size is only 12.8GB. submitted by /u/ColorsOfCosmos [link] [comments]”
“What is the issue? Running qwen 3.8 27b q4 from ollama on a nvidia 5090 with 32gb. When I turn on MTP from the python API using the same prompts and context size, performance goes down, as layers are ending up on the CPU. PS from ollama with MTP off: NAME ID SIZE PROCESSOR CONTEXT UNTIL qwen3.8:27b q4 K M 25b843619e94 30 GB 100% GPU 200000 59 minutes from now PS from ollama with MTP of 2: NAME ID SIZE PROCESSOR CONTEXT UNTIL qwen3.8:27b q4 K M 25b843619e94 20 GB 19%/81% CPU/GPU 200000 59 minutes”
I only started using some vLLM forks recently in a 4 x 48GB 4090 system. DS4F - ~5000pp/180tg (DSpark) Qwen3.8 Flash next - ~7500pp/135tg (MTP) This is amazing, like having the API…
“I kept seeing guys in this sub saying how AtomicChat's Qwen3.8-Flash-Next quant is so good, fits in their machine when unsloth's can't, runs faster than other quants etc, so I went check out what's happening there. First thing I noticed was that AtomicChat's Q4_K_M quant is suspiciously small when the ngram table is removed (only ~56GB), it seems like most of the tensors in this quant are IQ2_S instead of the usual Q4_K, Q5_K and Q6_K that you usually find in Q4_K_M quants, t”
“Yeah, looks like it’s a community Gemma 4-based MoE, not an official Google 120B release. Probably worth pointing that out.”
“Somewhere in the last 22 upstream commits something broke the Patch 13 block's MoE support for 2+ GPU's. It is unclear yet if it's the Block 13 work extraction itself that introduced a regression (the WIP MoE work was fine built on top of mainline 22 commits back), or if it's an upstream commit that contributed. Currently bisecting and working on the solution. In the meantime, avoid using the Block 13 MoE patch if you are striping an MoE model across multiple R9700 GPUs.”
“I used a famous Simon Willison's pelican riding a bicycle prompt on the biggest local LLMs that can run on 128GB Apple Silicon. U used quantizations by Unsloth. Qwen3.8 Flash-Next gives a lot of details. DeepSeek V4 Flash is strangely underwhelming. Qwen3.8 27B still rocks, and I like its consistent minimalism. Is Qwen3.8 27B still large at 31GB? It is! But for this tasks 2-bit quantizations (at around 12GB) will give the same results. For more complicated coding, 4-bit are more than enough.”
“Found this a few months back for more compute-efficient (relative to memory size) MoE models. Nowadays Qwen3.6-35B-A3B exists along with Qwen3.8-27B but MoE SLMs might not get another around. Maybe Marco can be recycled somehow along with the other two Qwen models to make things extra fast? https://www.reddit.com/r/LocalLLaMA/comments/1sgzt0p/marcomini_173b_086b_active_and_marconano_8b_06b/ And this is not just finetuning and knowledge distillation, there has to be some way of recycling individu”
“More new massive updates from turboderp: - CPU offload of MoE experts - Qwen-3.8-Flash-Next ngram disk offload - GLM-5.3-Flash - New self-calibrated optimization technique - Countless other optimizations and improvements If you have an NVIDIA card and haven't tried it lately, you might be missing out. The attached cat image was made with Qwen-3.8-Flash-Next-3.05bpw-exl3 and this prompt: Create a detailed SVG image of a cute kitten riding a magic turtle into space. Come join the crew at the e”
“If you use llama.cpp, Use biggest quant you can fit, just probably it should be bellow UD-IQ4_NL, after that you just get slower decode speed for no additional quality gain If you want to use ablirated models, make sure ypu chose proper one, most of them have same damage as q2-3”
“The sad reality is that there is no good NVFP4 quants at all, but world is not ready for this discussion.”
“I used llama-perplexity to check Qwen3.8 27B NVFP4 GGUF models vs Unsloth as a baseline. Perplexity (lower better) is a measure of quality and is only comparable for the same model family. https://preview.redd.it/x8kdh1d0ysmh1.png?width=1806&format=png&auto=webp&s=64c3403367e63cebf058fd0e721258b40f1f4a0c As you can see for the Unsloth quants, perplexity increases with smaller file sizes. What is disappointing is all the tested NVFP4 GGUF models have worse perplexity for their file si”
“i prefer result quality over speed i use 3.8-27B as body model, for brain - DS4 flash 0731 Use Q8 if you have enough space with ctx”
“I just tried it, I think since the last time people were trying it, they added something to system prompt so when you ask it, it will refer to itself as GLM 5.3, but when you refer to it as Claude it just goes with thinking it's an Anthropic model. In so much of the underlying data it is referred to itself as Claude, that some system prompt is hopeless to stop it from happening. Similar to how when you talk to literally any chatbot about chatbots/AI and humanity, it will inevitably at some p”
“Did you like the latest ui update? Cleans it up a bit I think! 3b performance isn’t great as it lacks codecs for video processing like the zero 2w does :)”
“Hello, I read a lot about Qwen here in the last weeks and months. Many people seem to like it, but I'm curious whether anyone here has actually used it for anything else than coding and development. I am looking for an LLM that can help a non-profit org deal with everyday paperwork. It should be able to: Summarize documents, mails, letters etc. and help draft answers Use a specified document set for RAG, i.e. answer questions like "How many letters did we get regarding xy, and what did”
“🚀 The feature, motivation and pitch As per title. Seeing mi300 1: Quantization: 125 min mi355 1: Quantization: 60 min Especially, test online quantization is too slow, and may result in: from e.g. https://buildkite.com/vllm/amd ci/builds/11661/list?sid=019fcd74 48e6 44ff a48b 41b436ef8f84&tab=output Alternatives No response Additional context From: https://github.com/vllm project/vllm/blob/a3b86752fbcd2fb1a9adfe44634ac0d955228e2a/.buildkite/test amd.yaml L2401 L2412 and https://github.com/vllm”
“Your current environment Collecting environment information... ============================== System Info ============================== OS : Debian GNU/Linux 13 (trixie) (x86 64) GCC version : (Debian 14.2.0 19) 14.2.0 Clang version : Could not collect CMake version : version 3.31.6 Libc version : glibc 2.41 ============================== PyTorch Info ============================== PyTorch version : 2.10.0+cu130 Is debug build : False CUDA used to build PyTorch : 13.0 ROCM used to build PyTorch”