Devs Tuning Local LLMs for Speed
AI developers are experiencing significant challenges in optimizing large language models (LLMs) like Qwen for speed and performance. They are experimenting with different quantization methods, hardware configurations, and speculative decoding techniques to achieve desired throughput (tokens/second) and balance power consumption, often encountering unexpected performance variations and seeking community solutions.
SOURCES (60)
“One thing worth reconsidering is the "-np 1" restriction. For long context serving, kv streaming has a benefit beyond fitting one huge context, it could decouple concurrency capacity from vram capacity. On a 32g gpuwith a ~20g model, a ~200k context leaves room for at…”
Btw, the "GPU"s that they use for llms don't have video output
“this is a whole lot of work when you could have just bought two r9700s and been doing 710 tok/sec decode with 10 concurent streams.... https://preview.redd.it/xwxzc3i5menh1.png?width=1420&format=png&auto=webp&s=571c034822e4c782f67a669c23ad5834bf3a2811”
“This is a follow-up to my post from yesterday (17 -> 25-29 t/s with the expert cache PR). Same box: 2x RTX 3090 on PCIe 3.0, dual Xeon E5-2696 v4, 188 GB DDR4-2133 LRDIMM, llama.cpp, full 261k context, f16 KV, all 48 expert layers in host RAM, everything else on the GPUs. Since then I switched quants, stacked MTP on top of the cache, fixed the load time, found a bug in the cache PR and found out my RAM was thermal throttling (Now i gotta buy an additional case fan lol). Numbers are all from t”
“Quality: 81.1 → 87.7 (+8%) Speed: 35 → 29 tok/s (−16%) Runtime: 8m51s → 44m39s (5x longer) Output tokens: 18K → 78K (🤯) Noticeably better quality, but you're paying for it with tokens and time. Full benchmark results (all hardware, all quants): llm-bench.io Qwen3.6-27B Vs. Qwen3.8-27B submitted by /u/DerTomsn [link] [comments]”
“I would like to thank ChatGPT for the following: Benchmark K2-Horizon-MoVA-36B-A4B Qwen3.8-27B Qwen3.6-35B-A3B τ³-Banking 26.8 48.0 9.3 Terminal-Bench 2.1 58.6 79.8 44.9 SciCode 38.9 44.7 35.8 Humanity's Last Exam (HLE, no tools) 25.2 33.9 22.2 GPQA Diamond 80.8 90.5 84.1 CritPt 2.1 5.4 0.3 AA-LCR 66.3 77.3 66.7 AA-Omniscience Accuracy 18.8 15.6 18.8 AA-Omniscience Non-Hallucination Rate 69.2 69.7 49.5”
“I have dual 3090s with no nvlink and I get around 40-50tok/s with Qwen3.8 27b q8 with f16kv and 200k context.”
“Spent the last week getting Qwen3.8-27B (bf16, no quant) running on Kaggle's free TPU v5e-8. It works, and it's very fast: ~130 tok/s single stream (MTP), ~78 without 10,300 tok/s prefill, a 105k-token prompt takes ~10 seconds Full native 262,144 context. A 225k-token prompt prefills in ~28s ~540 tok/s aggregate at 8 streams ~20 min from hitting run to a live OpenAI-compatible endpoint (Cloudflare tunnel Repo: https://github.com/ARahim3/kaggle-tpu-lab Kaggle Notebook: https://www.kaggle.”
“I would be really curious about this especially on unified memory devices. submitted by /u/giveen [link] [comments]”
“what workload gets expensive enough that being thousands of times faster actually changes the business outcome? pick that bottleneck first, otherwise youre selling a benchmark instead of a product”
“ik_llama.cpp merged qwen4exp MTP support yesterday (PR #2369, mine, reviewed and tested by four other people on their own hardware). It's on main now, no fork or patch needed. Posting since the last couple threads had people saying MTP for this model only exists as an unsloth fork PR... there's another path. Flash-Next ships a 2.6B MTP head that the public converters were dropping. With it loaded the model drafts its own next tokens and then verifies them, so output is identical to runni”
“Useful numbers. I see the same thing on a single 16GB card, just at a smaller scale. After the experts move to host RAM, decode hits a bandwidth limit, and every tweak comes down to how much VRAM remains after KV. Since available VRAM per GPU determines your cache size, I'd like to know whether Q4 gets a similar hit rate to your Q6. The routed-expert stream should be about the same, but the slots are smaller. Does this improvement also work with far less host RAM, or can 188GB achieve someth”
“This is really interesting and I'm still learning all this stuff, so dumb question lol. I'm building a local assistant and we've been doing something a little different with the supervisor. Instead of having another LLM act as the manager, we're keeping the supervisor boring/deterministic and letting the main model do the actual thinking and planning. The supervisor just handles task state, permissions, execution, verification, logging, etc. Seeing your results made me wonder tho”
“Further inference tuning for higher context window with 0 hallucination/memory loss? It’s been almost 3 days of constantly tuning my model, unsloth MTP Qwen3.6 35B-A3B Q4\\\_K\\\_M, with the help of Claude, and we’ve finally come to a quick conclusion I’m still hoping there are more commands or settings I haven’t tried yet that could potentially squeeze out some extra token speed with 0 hallucination/memory loss \*\*for coding specifically\*\* .\\\\llama-server.exe -m "C:\\\\Users\\\\br”
“Same. And it only barely runs with the Q4 on my 5080 15GBVRAM at 16 tokens / s. However I havent found any other model that I can run and has better reasoning.”
“Yes AI was used to give the post structure and formatting after I had provided it with all the info required.”
“Testing tabby more today with real tasks I am seeing tg peaks up to 99 tok/sec. Seeing what AWQ and this EXL3 SC can do, it feels like imatrix is very old and GGUF number formats are limiting. I bet llama.cpp could do more with better number formats.”
“Just curious, what are some use cases you have for 6 q4 agents as opposed to running 2 q8 agents?”
“Sharing some numbers because most posts on this model are either using a single 3090 or unified systems from what I've seen. My current setup: 2x RTX 3090 (PCIe 3.0), dual Xeon E5-2696 v4, 188 GB usable (192GB) DDR4-2133 LRDIMM, llama.cpp, unsloth UD-Q6_K_XL. All 48 expert layers pinned in host RAM, everything else on the GPUs. Full 261k context, f16 KV. Before: ~17 t/s decode, ~350 t/s prefill on a 26k prompt, 12 t/s decode at 131k depth. Now: 25-29 t/s decode short and mid context, ~17 at”
“One of y'all were getting some crazy inference speeds on dual R9700s, so me with my single card wanted to try. They mentioned the "official AMD MXFP4", which was... https://huggingface.co/amd/Qwen3.8-27B-Quark-AWQ-MXFP4 Can't load safetensors in llama.cpp, so off I went to download this: https://huggingface.co/magiccodingman/Qwen3.8-27B-MXFP4-MagicQuant-GGUF But it doesn't load in llama.cpp. Is there no support for MXFP4 in llama.cpp yet? submitted by /u/mailto_”
“M5 Max 128. It's plenty of memory but the bandwidth is only 614G/s which is rough for dense models.”
“First, I don't buy their plan and won't buy. (But got some free token from them today 😂 ) OK. Today I tried to add qwen3.8 27b to ZCode's app. It was faster (maybe less token) than my other harness CLIs, dsh, copilot, etc, and got better results. (wrote more code on same task and no issues). However, I don't understand why. As this app is free, I think I need to raise this and let you to have a try. submitted by /u/fbms2 [link] [comments]”
“Am I understanding it right that Ngram in Unsloth Studio should be ON in settings 100% of the time for intelligence? With only MTP I have like +10% performance in tg but token usage jumped from 20K to 66K with extra thinking? submitted by /u/esw123 [link] [comments]”
“Yes https://llm-stats.com/benchmarks/mrcr-v2-(8-needle) Seems all current models solved it. It was quite bad few moths ago yet.... Even GPT 5.5 was badly struggling yet but 5.6 getting over 92%”
“This came from another thread or comment. I forgot exactly where, but the basic idea was to replace the 51B N-gram layer in Qwen 3.8 Next with a much higher precision version. Someone running a 5090 replaced the N-gram portion of their Qwen 3.8 UD Q4 model with BF16. Since I'm already running IQ4_XS, I wanted to try something similar. I don't have enough storage for the BF16 N-gram weights right now, so I replaced the lower-precision N-gram portion with Q8 instead. I'm still testing”
“https://preview.redd.it/74bmvel9b5nh1.png?width=1602&format=png&auto=webp&s=0d0c1adaa016a486ffd97c4c466e980dc611b139 I've only recently started looking deeper into vLLM after running llama.cpp for a good while. Initially vLLM (official repo) was terribly slow on my four R9700s (tried that one with two as well), however after trying radiance everything changed. Prefill 17636 - TG at that time was 36,6 That prefill spike was two agent profiles working on different tasks simultaneou”
“Hm. I've not tested qwen side by side, but ran gemma 31 bf16 literally today and at 128k lcpp was doing ~1610 and vllm ~1520 and gewell 2055 but it is not production ready yet because its still batch-1 only”
“This was my first serious attempt at tuning a local LLM. I started because Qwen3.8-27B IQ3 was fast on my RTX 5080 but the coding quality disappointed me, and the Q4 profiles I tried in LM Studio were much slower than reports here. Hardware: RTX 5080 16 GB i5-14600K 64 GB DDR5-5600 (4 DIMMs) Windows Final model/runtime: Unsloth Qwen3.8-27B UD-Q4_K_M, unmodified (16.46 GB) official llama.cpp b10760 CUDA 13.3 build 65,536 context, one slot Q4_0 K/V cache, Flash Attention medium thinking, text only”
“That's a theoretical benefit ut there's no performance (as in “intelligence”) benefits from dLLMs so far, quite the opposite. Where they shine is in speed on local hardware, because you move from being bandwidth-bound to compute-bound, which means that for token generation you end up with the kind of speed you usually get for prefill.”
“I've been using Groq and really like the speed, but I'm currently running into an issue where Developer tier upgrades are temporarily unavailable due to high demand . I'm building for production, so relying on the free-tier limits isn't really practical. My main workload is: GPT-OSS 120B GPT-OSS 20B Whisper / Whisper Large V3 for speech-to-text I'm looking for a Groq alternative that offers: Reliable production API Higher/scalable rate limits Good uptime Competitive pricing G”
“Hi, "simple" question, what you would suggest that is price effective to run big models like Qwen 3.8 Flash next , or even Qwen3.8 27B efficiently at Q8 ? Price is the biggest point, target performance for 27B Qwen3.8 around 20 tokens /sec at least I was looking at huawei ascend 310 series with 96GB memory, but they are completely out of stock everywhere ... AMD Instinct Mi50? Garbage or viable? submitted by /u/Snoo-2768 [link] [comments]”
“I'm not sure. The problem with the cloud llm's is they are complete black boxes that change frequently and randomly day by day.If you run Qwen 3.8 on your own hardware, every single day, it's the exact same model running in the exact same way.Yes, it's nowhere near as "smart" as the cloud based models. But it's consistent.So the workflows and "ways of working" you create will work mostly similar day to day.With Claude/OpenAI you frequently find days where the models are useless, and days when th”
“you're getting 30t/s on the mobile 5090? do you have MTP on? that sounds suspiciously low, should be at least double that, no? I remember mobile 4090s were getting 50~60t/s on 3.6 dense with mtp.”
“I came across Ling-3.0-flash through an NVIDIA Developer Forum thread about running it on one DGX Spark. The thread kept changing shape: first it was an A5B speed story, then an INT4/MXFP4 recipe hunt, then a promising tool benchmark that was retracted, followed by reports of long-context slowdown and one coding workload growing into OOM. That made me less interested in whether one short benchmark looked fast and more interested in the smallest end-to-end loop I could actually inspect. I reviewe”
“Ninfer emits logs. I ingest them and draw. Qwen and cursor have made me this. Full runtime metrics from my rigs”
That's about what I expected, similar to my NVIDIA Laptop 5090 24GB.
“Interesting. I don't have any GPU that has native FP block below 16-bit, so I can't grasp how better or worse at low-precision floating point weight quantization. (Thanks to the Strix Halo, which uses a very outdated architecture. Thank you AMD. An AI-focused product that can't even run FP8 natively at all. lol) But most GGUF quantization aren't truely uniform that throwing it into a matmul block, then you got a result in a cycle compared to K-quants which do microscaling heavily”
“They are, running 3 agents of DS V4 flash simultaneously at around 50 tokens per second each for around 100 watts”
“> "Qwen 3.8 is a damn good coder, but a terrible collaborator" It modifies SO many things in my scripts for what should be a 2 line PR, I get a 100 line linter style mess that looks like a high school freshmen got his hands on vs code and pushed code in. It can't maintain a common style, if I give it a script with a certain naming and layout, 3.6 one shots small changes that fit into my structure. 3.8 will add code that looks much more advanced, handling, strict type checks . Bu”
The MXFP4 numbers are impressive! How well does it run with FP8?
I am running this on Linu, Fedora as I started with donatto’s toolbox
Hi, you are running the above in Windows or Linux? Thanks. :-)
“Follow-up to my Qwen3.8-27B post. This time the target is Qwen3.8-Flash-Next: 512 experts per layer, 36 layers of gated DeltaNet, 12 layers of top-k sparse attention, a 26.8 GiB n-gram table and a built-in MTP draft head. unsloth UD-Q4_K_XL, 103.69 GiB. It fits in the Strix Halo's unified memory and nowhere else on a consumer box. Numbers first, caveats after. Hardware: AMD Ryzen AI MAX+ 395 (Strix Halo, 128 GB, 64 GiB carve-out for the iGPU) + RTX 3090 Ti on a PCIe x4-class eGPU link. One l”
“Following this post I decided to try GLM 5.3 Flash on a 8x3090 setup and I can now run a Q4 with surprising speed; 700tk/s prefill & 42tk/s decoding! (lcp & vllm do not allow me to get that). Was afraid about quality but > 30m tokens with DSH and no issue (did not test vision yet, but looks supported). Just to say that I am really grateful to Turboderp and we should really support as much as possible others projects even if they do not comply with all our needs yet and not rely only o”
“2 Months ago I had made a post how I was working on my dual R9700's. It's wild to look back at where we were then and where things now stand. Since then after many users commenting and complaining about developers doing the same thing. I threw out a discord link and expected maybe 5 other developers to join which I thought would be fun. The community has now grown to 1,200 users (mostly developers) and a ton of collaboration happening. A few weeks ago I started working on building suppor”
“Based on https://huggingface.co/hardware , the RTX 3090 is the second most used GPU by LLM enthusiasts. Because RTX 3090 has native INT8 tensors cores, it can provide better performance with INT8 W8A8. However people seems to default to FP8 or smaller quants anyway. I suppose I am missing information that explains why ? submitted by /u/TheOnlyBen2 [link] [comments]”
Gemma 4.20 69B DA6-7 Dynamic active parameters. 6-7 at a time.
“I'm running the same A40 card and was wondering what you found the best settings to be. I've tried llama.cpp, Ninfer and a few others, but I'm currently running llama.cpp with a Q6 model and it seems ok, around 35 tok/s, but not great. I can squeeze in f16 cache and 262144 context though, so that's good.”
“I did some benchmarking for embedded engineering use cases and Gemini has caught up with sonnet completely, even did better than it. These are actual cases where even compilation and output of received C code is actually tested. Most cases are embedded Linux and zephyr. I like how muse glimmer is really doing good for its size EmbedEval by Ecro Benchmark — Methodology Dataset: 219 embedded-firmware tasks across Zephyr RTOS (158), embedded Linux (51), ESP-IDF (5), STM32 HAL (4), FreeRTOS (1), cov”
“I was browsing HF for small LLMs and run into this model. It does not seem to be a fine tune - the model has its own architecture. https://huggingface.co/XHToken/Spark-X2.5-1.7B https://huggingface.co/XHToken/Spark-X2.5-4B There are 4B/1.7B versions - the benchmark is quite interesting (4B is neck and neck with Qwen 3.5 9B). The HF page claims both models support native 1M context size . Currently does not run out of the box on llama.cpp - pending this PR: https://github.com/ggml-org/llama.cpp/p”
“Hey, wondering if anyone's seen this issue themselves? I'm using a 16gb 9060XT on a proxmox LXC, llama-server via docker on the Vulkan backend, and it's been serving me fantastically - 40-50t/s on most modesl with MTP, even 25t/s with the IQ2 or IQ3 of 3.8 27B with around 100k ctx! But I've been experimenting with Gemma 4 12B QAT and I'm noticing something odd: despite being only ca~ 12gb in VRAM with 262k context and vision enabled, I get around 33t/s though I would expect t”
“I'm gonna try the same with qwen-flash. I run it at... 14~20/tok it should take a bit.”
“Yoyo I'm back with updates to the fastest inference engine with minimal quality loss for Qwen3.8-27B. The last few weeks I've been optimizing decode speed and I don't think it can be pushed further, until a newer/better drafter is invented. So I focused on prefill, which I this morning was around 1.300 per second at 4k and now is just below 2.000. The main improvement came from a custom kernel, which matches the quality of fp32 with 0.99997 similarity at int8. Try all of the improvem”