Hobbyists Fighting Slow Local LLM Inference

“Qwen3.5 9B runs smooth on 12G of vram, but it is limited. Qwen3.6-35B-A3B also works on 12G of vram, if you have enough ram, but it is slower than 9B [re: A world model that can run realtime on 12Gi of VRAM]”

“Currently I have a Claude pro plan and qwen 3.8 27B locally. I use Opus whenever I hit a bug qwen can’t fix, or to do create a plan and/or do an adversarial review for a more complex/vital part of software architecture. I’ve been debating if there is a more optimal pairing given my use case? Part of me likes the idea of a Qwen/kimi sub so I can still be open weight, but I do hear Claude and OpenAi are quite good deal. I have also debated between grok and Gemini… Gemini 4 is actually pretty good.”

“See also: https://github.com/huggingface/huggingface.js/pull/2587 [re: New quant types suggested: IQ2_K, IQ3_K, IQ4_K, IQ5_K and IQ6_K]”

“Decide for yourself: http://ciar.org/h/1227692.txt [re: What quant do you actually use day to day? Anyone happy with 2-bit or 3-bit?]”

“whats the vibe check when u compare it coding btw 27b version? Is there a noticeable difference in terms of smartness btw the two? [re: Qwen 3.8 Flash Next optimized for two CMP 170hx GPUs]”

“But GLM 5.3 Flash is totally insane on Max, it can spend half an hour just "but wait!"-ing itself. And if you try to constrain that with a million instructions, aren't you kind of back to High anyway? [re: DeepSeek V4.1 Flash (1017) Now Leads Open-Weight Models]”

“https://forum.level1techs.com/t/ultimate-2x-r9700-guide-for-qwen-3-8-27b-mxfp4-200t-s-c-1-500t-s-at-c-8/257789 idk if you're interested in trying to debug but this thread seems to have a lot of different folks chiming in w/ some common gotchas around configuration leading to low tok/s ... I am curious how it performs on decent context size as I'd only invest in the setup for local coding [re: 72 hours, four big open weights drops. The only one that fits in 24GB came from NVIDIA, of all c”

“I usualy try to not go under Q4 and highest i run is Q6 [re: What quant do you actually use day to day? Anyone happy with 2-bit or 3-bit?]”

“Curious what everyone runs in practice: - What quant do you use most, and for what (chat, coding, agents)? - 2-bit: do you use it? Is it actually usable or too broken? - 3-bit: good enough for coding and more complex stuff, or do you go back to 4-bit? - If you look at KLD: what number do you call "basically lossless"? Is 0.1 already good, or do you want something like 0.01-0.03? Real experience over benchmarks. Thanks! (English isn't my first language, so I used an LLM to help tran”

“I can confirm you can run QFN on 64GB Strix Halo. You're probably going to want to use ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF , you'll need to use --lazy-mode on , and you'll have to make some tricky decisions on quantization (the 2 bit quants fit comfortably, whereas the 3 bit ones will need you to either configure the VRAM beyond the 48GB max, take a haircut on context length, or both), but it's doable and a bit faster than Q3.8 27B. At those quantization levels, I'm n”

“Hi all. Saw JetBrains’ Mellum2 on Hugging Face and was wondering if anyone here has actually used it. Worth trying, or nothing special? submitted by /u/groovy-sky [link] [comments]”

“Looks quite interesting. Lots of numbers presented, but for a single model only. Highlights: File sizes are the same as Q-quants. KLD improves substantially over Q-quants. Performance suffers slightly Creating IQ quants is somewhat slow. Asterisks apply... https://github.com/ggml-org/llama.cpp/pull/30715 https://huggingface.co/eaddario/IQ2_K-IQ6_K-Test-Models Appologies to Mr. Addario if I mangled anything. submitted by /u/ethertype [link] [comments]”

“Qwen can take a walk here. Decode is memory bandwidth bound, and even on a single card the V100 will win, because it has 50% more memory bandwidth, which also answers your feed forward comment. The comment I was answering to was talking specifically about prefill, which is where the R9700 wins because it has more compute. Nobody is saying 4 bit quanta are useless. But heavy quantization of the attention layers is the only scenario where the R9700 would win by a large margin vs V100 [re: Given al”

“Qwen4s are hopefully coming pre-Xmas… there was all the announcements recently but no exact release date. I’m running qwen flash next 3.8 at about 28tokens per second and gotta be real, most of those go into thinking. Thinking less with the same or higher intelligence would be the biggest upgrade I could think of regardless of hardware or inference smarts. submitted by /u/turtleninja99 [link] [comments]”

“I use nvfp4 with qwen 27B swift on ninfer. swift keeps the overthinking down, and ninfer is blazing. I use heavily shared prefixes by design so all tools and isntructions are shared. It has been rock solid for me. It lands code for me at about 90% of haiku 5.5 rates. I haven't tried lower quants since I don't need to, and don't want to risk it. kv is 8bit since I found serious compounding issues going lower on KV, especially with longer context or agentic chains... so don't do th”

“Ah I see, yes text only does "work" I was able to run inference on these quants -- but not sure how stable it is [re: DeepSeek V4.1 Flash 1017 [GGUF + Abliterated]]”

“In my experience at least, it's too chatty and gets stuck in circles. I went back to glm 5.3 flash [re: DeepSeek V4.1 Flash (1017) Now Leads Open-Weight Models]”

“Indeed. But there's still an advantage, i was running a q3_xxs before at around 65k, and now I can run Q4 around 95k at pretty much the same speed. Roughly 36tps empty down to 25sh filled. [re: Qwen 27B (IQ4_XS, Q8/Q4 KV) at full 262k context on a 16GB RX 9070 XT: 80+ t/s decode with Adaptive KV Streaming]”

“I actually just ran a benchmark on one of my projects yesterday. Told qwen 3.8 flash next low, glm 5.3 flash low and opus 5.5 high to search for bugs and issues. Then I did the same with sol 6.1 high and let it be the judge of the 3 reviews and rate them for accuracy, false positives, findings etc. Max points was 100. The interesting part here is that qwen got 84, opus 75 and glm 71. This is when they was running on low and opus on high. Sure, they took a bit longer to run than opus, but it is s”

“Going by AA's index: MiMo-V2.6-Pro is 46 (1T), GLM-5.3 45 (753B), Step 5 Preview 44 (600B) and K3 44. GLM-5.3-Flash is 42, and AA lists it at 320B. It's one index though, so "caught" is fair to argue. [re: Kimi K3, three months after the hype: too big for almost anyone to run locally, and smaller models have already caught it]”

“1) delete it and download qwen 3.8 27B instead. [re: 10 Things you maybe didn't know you can do with Qwen 3.6 27B right now. (For Noobs)]”

“I was using 3.6 with claud code harness. And now I'm uding 3.8. it's night and day. 3.8 is way more smarter and doing less stupid things. [re: 10 Things you maybe didn't know you can do with Qwen 3.6 27B right now. (For Noobs)]”

“It is still neatly 2x what I was getting on ik_llama.cpp when using thinking parameters. But yeah, it dropped about 5% speculative deciding acceptance in my tests. [re: I don't get the qwen 3.8 flash next hype]”

“I mean, UD-Q4 is quantized/pruned into the ground. Quantization affects long context tasks disproportionately. This isn’t a Qwen problem, this is a UD-Q4 problem. Your observations are valid and I agree but your title is extremely missing. You’re condensing a 360gb model into ~40gb. Idk what you expect. Of course Qwen 27b is going to outperform it. [re: I don't get the qwen 3.8 flash next hype]”

“For coding task, the quant is fine. Plus the Q1 on the model card s based on size not on actual quantization. The prefill for qwen 3.8 27b is probably like 30tk/s if i remember at q4. So this atleast makes the gpu viable. [re: Strata performance on Quadro M6000 / Tesla M40]”

“Yep some qat etc. it isn't just exl3, it is a 3 step process that takes multiple days. This isn't a mainline guff though, they are custom kernels. Look at my 27b quants to see me beating ista on mainline quants. [re: Qwen Flash Next potential SOTA Q2 nobody talks about]”

“I was running Unsloth IQ4_XS for a while and while it provided me with a healthy 96K context on 20GB vram, I did notice all the issues it had during draft and all the time and local tokens wasted trying to fix said issues. I decided to see what would happen if I used less context and a higher quant and oh boy was I in for a surprise. The end result was even better than I thought. Having higher context is great as it allows you to run more complex things without compaction and with less interrupt”

“The Nemotron-3.5 releases are particularly not so good at the pandas/numpy and DDDM (data-driven decision-making) sections of my domain-specific tests. Medium thinking on Bolt does worse than Nemotron nano. High thinking marginally better (not to a statistically significant level) but so much slower. The super-vl had 3 truncations due to default max token budget (16k) per test. If that's raised or uncapped, it might potentially solve them and get better score. If you have the hardware for su”

“I really appreciate that they didn't have to pretrain a foundation model from scratch for this. They just take Qwen weights, slap on a 10M parameter adapter, and train the model to coordinate mode switches using a couple of anchor tokens [re: New paper from Stanford, Together AI, UIUC, and Meta introduces Hybrid Reasoning Agents. These AI models think mostly in a hidden internal way (faster and cheaper), and only use normal text when they need tools or give the final answer.]”

“Defect (solver input): the committed L03 ship benchmark golden deck writes the mesh format as "Wamit dat" ( golden/L03 ship benchmark.yml:81 ). OrcaWave would refuse that at load: the repository's own native OrcaWave files for these meshes use "Aqwa dat" (for example the L03 orcawave 001 ship raos rev2.yml:85 ). The control surface map at orcawave backend.py:77 82 also still writes "Wamit dat" and "STL". Found by the second provider review of 2605, which refuses an ambiguous plain dat but offers”

“i’ve found qwen 3.8 27b does a good job. smaller than that can give a lot of false positives atm hoping we can get smaller than 27B in the next gen of local models [re: Self-hosted AI code review for Gitea, using your own local model]”

“Check out davidau's fable cold fusion turbo Qwen3.8. Cuts the reasoning down to like, three bulleted lists instead of 3000 tokens of "maybe need reconsider meaning of life" Makes it actually useable on my 3050ti laptop [re: nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB · Hugging Face]”

“The model is a Qwen 3.6 35B finetune, maybe I can save someone else the search. The visuals are quite nice, though. [re: Single shot response given by a simple model on a "potato" system.]”

“Is your feature request related to a problem? Please describe. With the qairt plugin, a vision language model keeps its whole embedding table in private memory as float32. For qualcomm/Qwen3 VL 4B Instruct:W4A16 that is 151,936 x 2,560 x 4 B = 1.56 GB , nearly all of the GenieX process's private memory (1.59 GB measured) and about a quarter of its working set ( 5.8 GB). On a 16 GB Snapdragon laptop (15.6 GB visible to Windows) that is a lot to hold for a table the NPU only ever reads as uint16,”

“I was curious how quantization affects Finnish language ability and why Q38FN’s Q8 quantization got such poor accuracy numbers. So I set the machine running in the background. The Finnish corpus is three books from Project Lönnrot, and the English one is the same books from Project Gutenberg. Wiki is wiki. The first observation is that Finnish really does take a hit from quantization. If KLD is said to become noticeable after 0.02, then even at Q8 we’re far from that. This measurement doesn’t sa”

“I would consider Nemotron 3 series to have worse arch than Qwen3.5. Key point of the Mamba2 layers is the decay rule, Gated Delta Net has it, with additional delta rule. Trough Nvidia did post a paper about Gated Delta Net 2, which seems to be better than Kimi Delta Attention [re: NVIDIA AI: So many models launching this week! And we have one of our own coming...]”

“wasted a whole evening on this with qwen2.5-coder 32b, kept bumping quant levels thinking the model was dumb. turned out the gguf was packaged with a generic chat template with zero tool support lol. now I always run --verbose-prompt once and eyeball what the template actually renders before trusting a single tool call [re: No model is better than another its the god damn tools templates are broken & some tools need clean filenames]”

“Been using GLM 5.3 flash reap144 coder from Qwen next flash q4_m_k haven’t looked back, it’s smarter yes slower on my hp omen 17 16GB vram and 32gb ram but so much smarter 3 hours down to 1 hour and that’s at 3token/sec write 15tk/s read. [re: Did someone use/try the Qwen flash 3.8 next "coder" model on some work?]”

“I calculated KLD and perplexity for Qwen3.8 27B models that fit 16GB VRAM: See comments for the benchmarks. Reddit filters keeps deleting the post. Conclusion: q5_1/q5_1 is almost always better than q8_0/q4_0 or q4_0/q8_0. There is very little difference between q8_0/q4_0 and q4_0/q8_0 q4_0/q8_0 is sometimes better than q8_0/q4_0. So it's safer to just go q5_1/q5_1. Going up a model quant and running at q4_0/q4_0 is almost always better than running the quant below at f16/f16 in terms of KLD”

“Curious if you have any different opinions after a couple of weeks. I have been using 27b Q8 a little over a month and has been a pretty solid experience. The main thing is I give it plenty of long running tasks, full stack environment where it has to interact with lots of tools etc. Really good experience. I have started to mess with QFN, and while I can say it is capable it has already required more steering out of context ditches. I have downloaded another quant for additonal testing (still Q”

😂 [re: My local Qwen3.8FN agent is significantly cheaper than cloud]

“Let me know how GLM 5.3 Flash runs. I've only got a single 170hx :( [re: Chicken-coop shed AI PC]”

“Looking for people who have tried this model and was able to use it on real-world tasks. It's what Strata list as "coder". What was your experience like? On my setup I am not really sure if it is worth more than iq3 submitted by /u/JsThiago5 [link] [comments]”

“Q4....*shakes head* NVFP4 if you are going to run a quant. did you also quantize KV... keep KV at bf16 [re: Engineer / developer observations of Gemma4-31B, Qwen3.8-27B, and 6.1-Sol for software engineering work]”

“Depends on what I’m doing, I have QFN uncensored in Vllm and it’s my main model. For personal assistant I use it in Hermes and for coding I use OMP. But I swap to opus 5.5 In Hermes to plan out a project and it delegates and supervises OMP witch is still on QFN. I want a few auxiliary models running on a spare 3060 I have but none of them really get used for much (breeze tts2, qwen image& z image turbo, minimax h3, minicpm something and qwen 3.5 9b and 3.6 35b) all of these are served throug”

“Nice one! I started down same path (for privacy and tinkering value). It was solid but I was disappointed with decode speed on the bigger models. Switched to 2x sparks so I can spend more time tinkering the stack. Wish prices were less insane so I could do both! [re: 4x R9700 builds with $127 used EPYCs: $10.7k all-in per machine. I built three (384GB VRAM)]”

“In my setup, Qwen3.8-27b achieves about 3.5x to 4x faster prompt processing and text generation speed than DeepSeek V4 Flash 0731. However, on the exact same self-contained prompt to do a task, DeepSeek V4 Flash 0731 finishes in 3 hours while Qwen 3.8-27b still takes 2.5 hours. Qwen3.8's high raw speed doesn't translate into significant time savings. That minor 30-minute speed advantage isn't really worth the trade-off in quality in my use cases. [re: Qwen 3.8 Flash Next, DeepSeekV4”

“Yup, it's great. Pairs wonderfully with Qwen3.8 27B. [re: LiquidAI/LFM2.5-24B-A2B]”

“Just trying to figure out what to recommend to people running ESP32 voice frontends (xiaozhi-esp32, M5Stack AtomS3R, ESP32-S3 + I2S mic/speaker, WiFi to a local LLM). The wall every voice thread in this sub hits: 8B class (Llama 3.1, Qwen3.5 3B, Gemma): fast enough (40-60 tok/s on a 12GB card) but the conversation quality is bad. Repeats itself,loses the thread after 10 minutes, personality falls apart exactly where a companion needs it.70B+ quantized: smart, but 10-25 tok/s. A 30-token answer t”

“I have been working on a side project the past week. I am running a single R9700, Qwen3.6-35B-A3B-UD-Q6_K with a 220k context size. I use the claude cli as my harness and I have been testing out the AWS AIDLC 2.11 workflow. Stats from Claude : Sessions: 46 Longest session: 13h 37m 31s Active days: 5/7 Longest streak: 3 days Most active day: Oct 14 Current streak: 3 days Input 20.7m · Output 3.2m · Cache read 1.4b · Cache write 0 A few notes... Qwen 3.60 35B has not been perfect. As context grows”

“tl;dr: new qwen finetune that simultaneously pushes the frontiers of small model capability and reasoning efficiency. Hey everyone, We publish two post-trained versions of Qwen 3.8 27B called Sabi-Noema and Sabi-Noema-MLA . These models push the capability frontier on <30B models with 2x reduction in verbosity, and are exclusively trained on open source environments. These models have improved capabilities on coding, agentic and tool calling tasks, as well as hallucinating lesser. Sabi-Noema”

“I have a 5090 and I use GPT-6 Luna 100x more often than a local qwen model. I'm sure some people derive real value from local inference. But if the rumored discontinuation of the 5090 means the used market goes up 25% I'm selling this sucker and buying billions of cheap API tokens from smarter models [re: What is the true value of a local AI machine?]”

“Thanks for sharing! It's nice to hear of experiences in addition to Qwen (nothing against Qwen, I've enjoyed it). How are you using Mellum? Coding or something else? [re: JetBrains Mellum 2.1 12B A2.5B MoE is a very good model.]”

“This model is a 74B parameter, 9B-active-parameter mixture-of-experts (MoE) model, packed into 24GB using 2-bit quantization with advanced quantization-aware training techniques to retain quality. It aims to achieve the optimal balance between speed and intelligence, being faster than most popular dense models while smarter than most popular MoE models with a similar memory footprint. Nemotron 3.5 Bolt E4B 4GB releases Oct. 23 Additional detail submitted by /u/CulturalKing5623 [link]”

“Yes. I don't think there's any way to make the full model fit even with aggressive quant [re: Got 2x DGX Sparks. Excited and confused!]”

“There are several methods to get quants below 1 bit. NanoQuant from Samsung, BTC-LLM, TurboQuant from Google.... [re: Someone is running Qwen 27B at ~20 tok/s on a jailbroken PS5's GPU]”

“i gave up on ornith after the original one, seems very benchmaxxed and fragile KAT coder seems to be consistently good, but nothing even comes close at all to qwen3.8-27B [re: LiquidAI/LFM2.5-24B-A2B]”

“Ornith1.5 35b-a3b was an upgrade for my tasks over Qwen3.6. Give it a shot - shorter reasoning traces with higher output quality (for my uses, at least). [re: LiquidAI/LFM2.5-24B-A2B]”

“I bought exactly the same b60 for the home server. It works fine and Di can run Qwen 3.8 27b q4 with 100k context and ~20-40tps and 500-2000prefil. It has working sriov out of the box which I plan on using once this card is old and not enough for llms. I like it's power consumption is peanuts at idle and it is barely audible under load. I wish it had more performance and VRAM tho, but I'm not paying for b70. The card is 270mm long but the power connector stick out another 10-15mm. Standa”