Devs Tuning Local LLMs for Speed
AI developers are experiencing significant challenges in optimizing large language models (LLMs) like Qwen for speed and performance. They are experimenting with different quantization methods, hardware configurations, and speculative decoding techniques to achieve desired throughput (tokens/second) and balance power consumption, often encountering unexpected performance variations and seeking community solutions.
SOURCES (60)
“worth trying if you generate a lot of code/structured stuff, that's where it beats MTP (JSON hit 152 for us). for prose MTP's already better. just costs a couple GB VRAM for the draft model, which you've got room for on a 6000”
“which quant of k2.6? i run q8_k_xl with 2400mhz 8 channel at 6tk/sec with a few 3090s :-(”
I already do. I'm very happy with 5tk/sec if the quality is great.
“PrismaSCOUT "Smaller and better than the prior 5.5 bpp artifact" Oh wow. Before I moved back to llama.cpp + Q6_K quant for better quality I used to run rdtang's 5.5bpp PrismaQuant in vLLM. I didn't know such improvement existed, I'll be happy to go back to vLLM now as it is so much faster. I'm glad I read your post!”
“Mx-llama.cpp, its creator is on the gfx906 discord server, so i guess its optimized for mi50's, but i might be wrong. +llaminar, i think its not a fork.”
“Initially they were struggling to find hardware, using old ones, trying to work efficiently - now they don’t have those limitations. I think it was a factor”
“You can be still happy for the advancement, right? I can't really run even Gemma 31B or Qwen 27B in good quality and speed, but at the same time I know that many of you can and I'm happy for you all.”
“A good distil is very cheap only in relative terms. It's a lot cheaper than training from scratch, but it's still a very expensive project.”
“https://reddit.com/link/oybubtz/video/z2obj8dp01eh1/player Another Demo Guys! (K=8)”
“Started testing a private qwen 35B moe capacity LLM runtime on s26 ultra, early testing shows that active model footprint can fit within the device’s memory limits.( not sharing the methods or architecture used) and results suggest roughly 90 input processing t/s achievable after optimisation and output generation is around 8 tokens/s on this mobile. Point is i learned ai ml based on my interest and no formal PhD , I have compute and resources to test. Anyone willing to join or collab to test on”
“so part of what inspired my benchmark post was that it does seem like folks here are generally converging on "unless you are able to operate at very large scales with a lot of system RAM and VRAM, the best model for code work is generally Qwen3.6-27B." what I'm hoping is that we have some good places to start when it comes to harnesses and server settings for that model depending on what resources you have on your system. e.g. my personal machine has a pair of 5060Tis. what I'v”
“What sort of benefits would the hypervisor provide by keeping it as a VMDK? Benefits like faster access or less resource intensive?”
“Is it actually comparable to the qwen3.6 27b? Can it be used for GPU poor? Or do I stick to the 35b moe? submitted by /u/Adventurous-Gold6413 [link] [comments]”
“Trying to target the best value/bang for my buck while also having the possibility of being easy to scale/upgrade in the future. Currently I'm running this in my NAS CPU: Intel Core i9-12900K Motherboard: MSI PRO Z690-A WIFI ATX LGA1700 Motherboard Memory: 32 GB (2 x 16 GB) DDR5-6000 GPU: 3090, 3060, 22gb vram 2080ti I'm looking to upgrade to the following below and I was wondering if anyone has any advice/suggestions/changes that they would make. Motherboard: ASRock Rack ROMED8-2T/BCM (”
“In their blog post they talk about activating 16 of the 896 experts efficiently. At 2.8TB total size that would make each expert then about 3B only? As: 2800/896=3.125, but base should be reduced as there are also the routing layers and similar. So a pass through the model would only process about that 3B*16+Little=about 50-60B maybe? Or am I missing/misunderstanding something? submitted by /u/Luvirin_Weby [link] [comments]”
“so ive been messing around with qwen3.6 35b a3b (MXFP4 gguf) on my 3060 12gb, doing the usual cpu/gpu offload thing where half the expert layers sit in ram and get pulled over pcie whenever needed. and like everyone whos done this knows the gpu just sits there idle waiting for experts to show up, pcie bandwidth is the actual bottleneck not compute idea was pretty simple, use the models own MTP head (the thing thats normally used for speculative decoding) to draft the next token WHILE current tok”
“We ran the same Qwen3.6-27B locally three ways on one RTX 6000: baseline, MTP, DFlash. The tasks were: quicksort, write a Steam library in JSON, solve a logic puzzle and write a sci-fi story. Outputs: Baseline: 44 tok/s · 1.00x MTP: 65 tok/s · 1.45x · 71% accepted DFlash: 98 tok/s · 2.20x · 30% accepted DFlash drafts 15 tokens in a row, so it flies through repetitive or structured stuff where long runs actually stick, like JSON (152 tok/s, 3.4x). On creative text most of the guesses are wrong, s”
“I just ran my benchmarks on it, and it went blow for blow with qwen3.6 35b. Which is a bit better than 27b in my benchmark( I know most people find the opposite)”
“Sharing a config that's been working really well for running this MoE model on modest hardware. Key trick: force the expert FFN weights onto CPU/system RAM while keeping attention + KV cache on GPU. Since only ~3B params get activated per token (A3B), the CPU-side matmuls stay cheap enough that throughput holds up. Load settings Context Length: 262144 (max supported; KV cache stays on GPU so it's affordable) GPU Offload: 40/40 layers MoE weights forced to CPU: 40 layers ← the important o”
“Stack: i7 12700K | RTX 3090 TI | 96GB RAM Qwen 3.6 27B Q3/Q5 KXL UD I've been pushing Qwen 3.6 27B above 200K ctx all week, and it handles it like a champ. I'm impressed. Today I hit the ceiling at 262K and it's still functioning well and coherent. I'm planning on trying out Yarn to see if I can push it higher. How high have you guys pushed it and how has the quality held up? NOTE: If you're wondering why the context is so high, it's mcp tool and memory/summary bloat (40-”
8 boxes are 1TB, should run fp8 the tps hit might be worth it.
“https://preview.redd.it/nrxkobib5hdh1.png?width=1147&format=png&auto=webp&s=a36a282635bcbb23c0c2bdffa855689eab5e76f9 I'm running Unsloth's Qwen3.6:35B UD Q4_K_M with a 100k context on a single NVIDIA P40 (24GB) using TheTom's TurboQuant fork of llama.cpp . I know this is only burst speed, but I thought it was impressive enough to share with the community. Current setup: TurboQuant enabled (stopped short of dropping K to Turbo2 to preserve agent quality) Reasoning disabled”
“I've got 64GB of regular RAM, and I ran Qwen3 Next 80B at UD-Q4_K_XL, enjoying both nice performance and a lot better internal knowledge than Qwen3.5 35B A3B, which itself is better than Qwen3.6 35B A3B. With DDR4, I get 8.5 tok/s on average, However, Qwen3.5 122B A10B with the UD-Q2-K_XL quants make it possible to fit a ~120B model into the same 64GB RAM. Contrary to popular belief, UD-Q2_K_XL weights, despite being Q2 at their core, are still pretty good, even with MTP enabled. I get way h”
“I've been using DS4 flash as a local "big brain" for when my qwen 3.6 35b a3b gets stuck on something, and for planning. I'm blown away by how capable it is at a 2 bit quant Has anyone used the GGUF at a similar 2bit quant and compared performance? I've got a Nimo Strix Halo box, 128GB, kyuz0 stack with rocm submitted by /u/dbinnunE3 [link] [comments]”
“Interesting :O Could you run the benchmark on 4 DGX Spark with C=4 and C=8? I’ve only found benchmarks using 1 or 4 concurrency :\\”
“I don't get how this work tbh, isn't the 20GB mod lacking of correct BAR values? are you using P2P with the limited bar amount? https://www.reddit.com/r/LocalLLaMA/comments/1r8vi2t/chinese_modded_20gb_3080_rebar_bios/?show=original”
“https://preview.redd.it/un1do2xhrfdh1.png?width=1553&format=png&auto=webp&s=521a90ec16dcff96aec8da5370fe94df358fd68b 17+ hours, from my very first prompt with it through Codex, and it's taking 2+hrs on every subsequent prompt. Now yes, this is a large project that involves deployment and tests, but it was seemingly quite a bit faster with 5.5 extra high turbo with negligible difference in quality. This is just an early observation, I'm even using SerenaMCP and CocoIndex, but”
“Made a thing for a problem I kept hitting. I fine-tune the same base model a bunch of different ways and my disk fills up with near-identical multi-GB checkpoints. Since weights barely move from the base, storing all weights for every model is inefficient deltatensors diffs your fine-tune against the base and only stores the diff, compressed. Works on any trained model, full fine-tune, FSDP, whatever. Before I get the question: It's not like LoRA (except in terms of the diffing idea) since i”
“Gemma 3 12B is way better and even smaller.. why you guys hyping on this thing with all the fail toolcalls?”
“Apple M5 Max 128 GB: CPU/default: 1,06 tok/s, RSS 21,8 GB Metal cu --ram 96: 1,11 → 1,83 tok/s after cache warm up Metal cu --ram 110: până la 2,06 tok/s,”
“You literally replied to me that you get 22 t/s without mtp in some other post, how come it's 38 now lol”
“Sounds incredible, I actually run all these services and more and I see maximum of 4.5GB memory usage (without swap)”
“Benchmark Results Benchmark configuration: threads = 96 type_k = bf16 type_v = bf16 Llama-3.1-8B-Instruct Q8_0 Prompt Size GGML_CPU_Q8_0 t/s ZenDNN_Q8_0 t/s Gain 256 472.28 730.87 54.75% 512 450.86 832.48 84.64% 768 446.81 864.52 93.49% 1024 439.58 800.15 82.03% 2048 405.07 778.34 92.15% tg128 33.08 33.14 0.18% Mixtral-8x7B Q8_0 Prompt Size GGML_CPU_Q8_0 t/s ZenDNN_Q8_0 t/s Gain 256 156.09 297.67 90.70% 512 156.63 389.44 148.64% 768 156.76 417.38 166.25% 1024 154.70 438.73 183.60% 2048 150.11 47”
“I recently learned that the 27b model is considered better than the 35b so I gave it a try and found out something odd… My CPU is maxed out while my GPU chills along at around 20% load… Since the model is split between VRAM and RAM I see that a lot but usually it isn’t that drastically. Only qwen 3.6 27b seems to do that to a point where almost everything is lifted by my cpu… Has anyone has some explanation or idea how to fix that? submitted by /u/Br0lynator [link] [comments]”
“GLM5.2 at quants that fit in 384gb of VRAM is pretty lobotomized, at least in sentdex’s case it was a better idea to run DSv4 Flash at higher precision. Maybe Hy3 changes things a bit”
Prefill is cca 50% faster on 8 and gen speed cca 20% faster.
“I tried both and also ran some mmllu, math amd tool calling benchmarks, this got same or slightly better results than nvidia NVFP4 and it is 10% faster. I used it a lot for coding in opencode and it did great. Maybe I will try FP8 quant just to see if I can see a difference, even if it means I will not be running also a second multimedia model in parallel.”
“I have done hours of coding and reviews with my setup (2x 3090 with Qwen 3.6 27b int4 autoround) and I'm still not confident I can draw any conclusions other than it seems to mostly work good so far.”
“Most "run a model in your browser" demos still lean on a server or WebGPU. This one doesn't: it's a pure WebAssembly inference engine for LiquidAI's LFM 2.5-350M , quantized to 4-bit (Q4_0) , running entirely on-device in the browser — no native code, no backend, no network at inference time. What's in it: Hand-written WASM SIMD kernels (gemm/gemv, GQA attention, conv, RoPE) — no linear-algebra lib. Batched prefill so prompts aren't processed token-by-token. 4-bit w”
“The question is what a one bit 70 B model be better than a higher bit lower B model of the same speed?”
“It's not Q1 - it wasn't quantized to 1 bit. It was trained from scratch with 1 bit parameters.”
“With Bonsai 8b at 1bit being 1gb~ size while being functional and the 27b being dropped also at 1bit while only being 5~gb, is the future for 1bit models looking bright? What will their primary use be? submitted by /u/AnimalPuzzleheaded71 [link] [comments]”
“I can how run 60 tk/s with two slot now, quality seems good, tool call is very stable. I haven't done any coding yet. 2 slot each have 100k KV cache allocated and it took around 21GB of VRAM submitted by /u/Top_Outlandishness78 [link] [comments]”
“If it's dense, does it fit at a reasonable quant on my GPU? If it's MoE, does it fit on my PC? The second quesntion is answered yes for both qwen 3.6 35B and Gemma 4 26B The first one... well I'm on 16GB vram and I don't have a heavy use to justify buying a second GPU.”
“You're telling me they managed to cram Qwen 3.6 27B into 5.7GB with only a 5% drop in capability? Gonna test this on my 8GB laptop 3070.”
“Its at the top of the post body. EDIT : "near fp16 precision" I intended performance in terms of benchmarks/output. Obviously 1,0,-1 cannot be fp16. Bad word choice :)”
“Brought DFLASH over to turboquant, significant speed up across Gemma4 and Qwen3.6 models. submitted by /u/giveen [link] [comments]”
“EDIT : "near fp16 precision" I intended performance in terms of benchmarks/output. Obviously 1,0,-1 cannot be fp16. Bad word choice :) EDIT 2 : Now that more people have tested it reported in, consensus (and my own stuff on more doc/retrieval tasks) is this lands better than Q2 but clearly worse than Q4_K_XL. Hallucinates more, tool-calling loops, etc (using Pi harness). The real story is memory footprint at this quality, which is still nice. Title overstated it - got excited lol. Leav”
“Hey, so I spent nearly an hour rebooting, updating bios, bmc, and oh man yeah can't get it past 272GB/s on triad under ubuntu. Though with High Performance, SMEE disabled, ABPDIS 1 / DfPstates 0, was able to get Passmark threaded memory throuput in windows to go from approx 324 to a bit over 420GB/s, but that might be some sort of cache effect going on there. Sorry to not be of help, but man I'm nearly certain a combo of software enviroment / bios / settings got like 320GB/s on stream tr”
“I binned Qwen3-VL-30B-A3B for timing out at 21 minutes on my DGX Spark, then re-ran it on a 5090 and found the real failure: under GBNF grammar decoding it locks onto one valid JSON item and repeats it until the context runs out. Swept quant, temp, flash attention, context size and repeat penalty with no luck - crank the penalty and you get schema-valid output with zero items, which scores perfectly if you only check validity. The dense 8B looped on 15 of 37 pages too, and gemma-4-31b has the sa”