Speculative decoding in llama.cpp breaks on Qwen 3.5 models
Enthusiasts and hobbyists are struggling to achieve optimal performance when running large language models (LLMs) on their custom hardware setups. They face challenges like slow token generation speeds, compatibility issues with specific models and tools (like llama.cpp), and difficulty scaling performance across multiple GPUs.
SOURCES (60)
“I found out 60% of GPUs arent even connected because of power issues and they have a life-span of 3 years. Jensen needs this to work before all those GPUs become obsolete and enterprise goes with competitors.”
“Weird to center on Ollama and LM Studio. Seems like anyone with more than one local inference capable machine will already themselves moved on to other software.”
“Prerequisites [x] I am running the latest code. Mention the version if possible as well. [x] I carefully followed the README.md. [x] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed). [x] I reviewed the Discussions, and have a new and useful enhancement to share. Feature Description Since you can plug a dgpu in pretty much any strix halo box, via m2 to oculink adapter, pcie slot where available, tb dock etc, it would b”
“Howdy, I was wondering if anyone knew of any turnkey low-power draw solutions to host inference with 10-20GB of VRAM? I have an 4x3090 AI GPU cluster that I'm running big models on, but I am hosting my memory system LLMs (embeddings, rankings, etc) on my gaming 4090 right now. I'm out of PCIE lanes on my AI machine and really don't want to build a whole other system (cpu, ram, mobo, gpu, etc) just to run about 15-20GB of small models. I want to keep things local so a VPS is off the t”
“I find this really surprising. I'm on an 8-channel DDR4 server and haven't seen any DGX systems outperforming mine yet. What generation speed are they seeing with GLM flash?”
is pipeline parallel not the better choice with the low pcie bandwith?
“You can get great prefill speeds on EPYC servers by placing the context on a GPU.”
“All my systems are now dual CPU, so have plenty of lanes, and having moved completely to data center cards, I now have p2p with stock drivers everywhere. The thing with the P40s is, they're running with dual Broadwell, which while they're quite good when offloading layers to RAM, they don't have enough memory bandwidth to handle larger models. Skylake/cascade lake boards with lots of slots are still very expensive. I've looked at both the PEX88096 and the PEX8796 (previous versio”
“IIRC, rockchip started working last year or this year on upstreaming support for their SoCs into mainline Linux kernel. If/when that is done, support should be much better, though you'd still need the device try for each SoC.”
“Hi, I'm running a local server with three AMD MI50. Tensor parallelism is not an option since it's very slow with PCIe 3.0 and those cards are not on the same NUMA node. In order to balance the load, I wanted to do something like round-robin. Every graphic card is running the same model with the same settings and llama-server has to manage requests so request 1 goes to card 1, request 2 to card 2 and so on. It's possible to run one llama-server instance on each gpu, however I don'”
“If you are in doubt, maybe you should read the memory breakdown lines more carefully. 10953 for the model + 4800 context + 260 compute. Seems like that fit in 16384, doesn't it? FYI: The memory breakdown only appears after a succesful load and never after an OOM (should be proof enough already, even for ppl who can't do the math).”
“kinda...but not really. until you get above 200gb/s the interconnect bottleneck kills tensor pairing so you are stuck on pipeline parallel anyway. for PP latency matters more than bandwidth so I'm not sure that its really viable. I tried it with my strix halo and 4x3090 node. I did it over 10gb ethernet, but the results would have been the same or worse if I used usb4”
“Amazing work by our quant hero! Still, does this only apply to CPUs with avx2, but not avx512?”
“The 1.87x prefill gain from RAM-resident over mmap is the real sleeper — that's massive for any batch workload. Good sanity check on the PLE table too, I've hit similar cliffs where forcing stuff to GPU backfires on Apple Silicon. tbh the scaling ratio decay from 2.80x to 1.45x at 245K is what I'd expect, memory bandwidth bound once context fills up.”
“Thanks man, its hyper-specific in its execution rn but I really would like to get it close to something like ninfer for us AMD users.”
“GLM 4.5 Air FP8 would not fit in 2 40gb version of the A100, it has to be the 80GB versions. OP is talking about VLLM and Sglang so no CPU offloading either”
Is tensor parallelism doable with PCIE x16 + x4 combination?
Yeah, makes sense, wanted to do quants myself for a long time but was busy, now that we have langfuse integration working might finally get good agentic data from logged traces for…
“You might want to look into making your own quants with a custom calibration set with Russian. NVFP4 is pretty decent for running with vLLM, just keep more sensitive parts of the model in FP8 and you won't see much performance loss, keeping KV cache at FP16 is more important.”
“Hello everyone, we've been running glm 4.5 air(fp8) since basically its release, and I think it's gotten a bit too dated. Problem is, idk with what do I replace it? We've tried qwen 3.5 122b, but I was having way too many problems with it: - malformed tool calls - SGLang straight up not working - SO related bugs - I think to this date MTP is vLLM is bugged on qwen models ( https://github.com/vllm-project/vllm/issues/35800#issuecomment-5442938245 it seems that it's solved now, but”
“My setup is not that optimized honestly, I just run Ollama with the open-webui container on top. Simple but it works for what I need. Your speed jump from 22 to 27 tok/s is making me think I should try llama.cpp directly too, especially since I also have 16GB and been wanting to test those bigger quants. Do you have any guide you followed for the initial config or you just went with the docs?”
“I’ve been lurking here for a while and experimenting with a 2× V100 setup with 64 GB of total VRAM. I’m running Q8 at full context, and for my use case, HPC research, the results have been surprisingly good. With the DeepSeek harness and llama.cpp compiled from source and MTP 2, I’m currently getting around real 20 tok/s generation, which is quite usable. However, my workflow involves designing, running, and evaluating multiple independent experiments. I’ve seen people running llama.cpp as a ser”
“As the heading goes, at some point it stopped being about improvements and just whether I can. So check out my abomination. GLM-5.3-Flash at IQ3_XXS gets about 20t/s generation in Unsloth Studio. Now if only I can make my second 2x48GB DDR5 ram kit play nice, but computer just refuses to be stable with two different 2x48GB kits. Anyway, love this forum, plenty of ideas and fun. Time to go work some more overtime to pay for this crap. submitted by /u/RedAdo2020 [link] [comments]”
“hoping that maybe this speeds up getting these features added, on a system with 8gb vram and 16gb ram i geniunely feel like Direct IO Weight Streaming is the only practical way to run ai locally if you want to multitask and TurboQuant would just be an added bonus to get our KV Caches smaller”
“I've got a similar setup with R9700 and 5060 Ti. I always try to target at least 200k context for coding, since it often requires that and more in practice. So, I have llama.cpp setup using Vulkan to pool the cards VRAM to get bigger quants at longer context. I'm starting to play around with just running two separate instances of Qwen3.8, one on each card, and have them work as pair programmers. For the 5060 Ti the recent IQ3_XS quant here https://www.reddit.com/r/LocalLLaMA/comments/1w1”
“https://preview.redd.it/rgyg3xyehdmh1.png?width=1600&format=png&auto=webp&s=fe93e29e77fdf98e0054050e8126b30230a2e262 Some time ago, I've seen graph that will show optimal power for inference on RTX 3090. At the time it was around 220W. So I did one not very scientific performance test. Prompt: "write 400 words". And results are quite self explanatory. I kind of expected that these numbers can move depending on inference engine, but seeing that gives me new perspective.”
“I grabbed the latest server-cuda13 image today so I could try out Qwen 3.8 Flash Next and poof, OOMs everywhere. Turns out they are compiling with -DGGML_CUDA_NCCL=ON again. You'll know immediately because each GPU will have two llama-server processes each, instead of one. The extra process, I'm guessing, is the NCCL router. NCCL is fairly awful in my experience if you are using PCIe instead NVLink. Luckily there is an easy way to disable it without recompiling. Add the following env var”
Can you not just use --speculative-config'{"method":"mtp","num_speculative_tokens":3,"max_model_len":524288}' ? I don't have my server running right now, but I'm pretty sure that worked for me before with Qwen3.8-27b
“Im pretty sure that sizing is just because they dont bother quantizing the engram to the same extent since that can just be pushed onto SSD.”
“Me neither :-D I'm already overwhelmed with model release cycles and am quite happy that my stack is totally usable and optimized for my needs. That took enough time and nerves, now I'd like to ACTUALLY use my local AI for something useful ;)”
“You don't. As I don't. We have alike hardware and it's not going to work. Calling it flash is silly of them”
“What if it is the future? ephemera backends generated on the fly optimized for your model/quant/MTP/hardware combinaison? Isn't that sexy?”
What are you using your MR for and what kind of cpp are you getting?
“Comment: llama.cpp publishes CUDA builds for WoA as part of their releases for some time now: https://github.com/ggml org/llama.cpp/releases/tag/b10655 I think publishing similar builds via conda forge would be a nice feature addition :)”
“You probably have too long of cables/not well enough made ones. They might say gen 5 or gen 4 but they cannot support that past certain lengths especially without good shielding.”
“Are you sure your PCIe slots are currently configured? When my 5060 TI x3 was had 1 card running at x1 I was getting 20t/s. Moving all to x4 gets me average 35t/s with tensor split and MTP. Yes, it will occasionally burst to +40, +50, +60 etc.”
“The conservative sensor approach is smart, nothing worse than a monitor that confidently shows you garbage numbers. HotSpot on 50-series is a headache though, fingers crossed someone cracks it cleanly.”
someone found why HIP/rocm is slower than vulkan on RDNA3+ cards, and fixed it https://www.reddit.com/r/ROCm/comments/1vo9kxi/llamacpp_boosts_for_rdna3/ I wonder if something similar happens on the V620.
“ik_llama needs more contributors because currently it's strong only on CPU, CUDA & Hybrid(CPU+CUDA). Vulkan, ROCm, SYCL, etc., backends need massive work. Month ago I did post a below thread since I was getting AMD cards so good to have Vulkan at least. Experts-Volunteers needed for Vulkan on ik_llama.cpp”
“hey, im doing that for RDNA4. I really need to finish up and put something out. its pretty custom at the moment though tbh. muse glimmer works fairly well and it beats out llama.cpp in almost every scenario. tg is somewhere around 15% higher, pp is around 30% higher (compared to both vulkan and rocm)”
“currently working on an inference engine in rust for RDNA4, planning to do multi-gpu at some point but for now its single. I have worked it down about 93% of the max theoretical bandwidth for tg at this point, and maxed out pp to about the same level. Using HIP graphs, combining kernels, and maxing scheduling. At this point I am working on custom quants that strictly run in 4 bit and 8 bit combinations to avoid conversions, but you dont really have this problem as you are working with tensor par”
“So here's the thing, almost everyone use NVIDIA to run their LLMs, we also do the same, a lot of people we've met use like RTX PRO 6000 or even H100, B300 It seems like everyone eyes is looking at NVIDIA. However we do the math that the raw power alone on AMD GPU MI350X is higher than NVIDIA B200. So what give? Apparently its the software, ROCM is not as mature as CUDA, their software stack is still lacking behind as well. However, few weeks ago we've got a handful of MI350X to exper”
“Thanks, I'll edit the post to mention the gfx1030 & take a look at running this again in a little bit with both suggestions. Also, I have now found a way to modify some of the values of the driver for the V620 within Windows registry and change the wattage, voltage, and frequency of the clocks. May post about that, too.”
“I had read that I should not have mismatched PCIe configurations for tensor parallelism, so I upgraded my motherboard from MSI PRO-X670-P-WIFI to Asus ProArt X870E-Creator WIFI. With the old board I had gen4 x16 and gen4 x4. With the new board I have gen5 x8 for both AMD R9700 cards. Both old and new motherboards have the slots directly connected to the CPU according to the documentation. P2P was working for both setups. I did some benchmarks before and after using guidellm, and with the same se”
“Umm switching the order was night and day for generation... I think it was adding a network round trip. Getting over double the TG speeds now. Before CUDA0,RPC0: Q6_K: 112 pp and 25 tg Q5_K_XL: 124 pp and 25 tg After RPC0,CUDA0: Q6_K: 112 pp and 55 tg Q5_K_XL: 112 pp and 55 tg I re-tested the old config right after this to make sure something else didn't change, but yes just swapping the order is the only variable here making that huge difference. (though my Q6 speeds went up from yesterday”
“These models are available in Vulkan, ROCm, and HIP versions in various versions. It would be great is the container offered an option for these as a base as well.”
“Prerequisites [x] I am running the latest code. Mention the version if possible as well. [x] I carefully followed the README.md. [x] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed). [x] I reviewed the Discussions, and have a new and useful enhancement to share. Feature Description While profiling models with the Vulkan backend using the Radeon GPU Profiler (RGP), I noticed the compute dispatches appear without descri”
“Your current environment Motivation vLLM's current Pipeline Parallelism implementation is positioned in the official docs as a capacity or memory fallback: "Multi node multi GPU using tensor parallel and pipeline parallel inference: if the model is too large for a single node, combine tensor parallelism with pipeline parallelism." This positioning is correct for the original PP use case, but it leaves a sizable performance opportunity on the table for long context disaggregated serving , where P”
“🚀 The feature, motivation and pitch The RTX A6000 (NVIDIA Ampere architecture, SM 8.6) only supports fp8e4b15 (E4M3B15) and fp8e5 (E5M2) FP8 formats with extremely limited native FP8 capability. However, vLLM (along with the default configuration of DeepSeek V4) attempts to use the fp8e4nv (E4M3FN) format by default, which is exclusive to NVIDIA Hopper architecture and unsupported on Ampere based GPUs, leading to inference failures. Alternatives No response Additional context No response Before”