llama.cpp Crashes or Crawls on Recommended Hardware

“wait that already got merged? nice, the neural accelerator support has been a long time coming tbh”

“I've been working on overcoming KV cache size issues, allowing the ability to load a slightly larger model and/or a larger context size. Downfall is a hit to tg speeds. Think of it as a "ram disk" for KV Cache, however, ram speed may be a determining factor on the actual hit to speed as well. submitted by /u/giveen [link] [comments]”

“This is the harness companion to my Qwen3.8-27B benchmark post . That post made the model fast; this one makes it useful : pi (the coding agent) against a local llama-server, tool calling, thinking control — and response times that don't hurt. This is a setup guide , not a benchmark post. Every trap, config, and extension here is what I actually run daily. The benchmark side (the game-build harness, scorer, runtime gates, playtest protocol) lives in neon-ladder — this post links to it where”

“ChatGPT is down r/ChatGPT Claude is down r/ClaudeCode Grok is down r/grok my local llama.cpp works as always submitted by /u/jacek2023 [link] [comments]”

“Hello LocalLLaMA, Ever since I got my M5 MacBook Pro 24GB, I have been a bit obsessed with optimizing llama.cpp so it doesn’t cook my laptop :) This led to 3 PRs so far. I can only test on my M5 and an older M1 Pro 32 GB, so very limited model options, so I would appreciate if anyone would try it on their apple + maybe on Qwen 3.8 Flash Next! 1. https://github.com/ggml-org/llama.cpp/pull/28301 General MoE prefill optimization: skips empty work in underfilled expert tiles. On M1-M4 this should he”

“Analyzing it now, thank you very much! Those are some nice numbers. My setup is currently at ~405tps PP16384 and ~12tps TG using Q6_K + Q4 drafter”

“It's been 3 weeks since Qwen3.8-27B release. This model got 0-day support & in last 3 weeks, some optimizations & fixes happened on llama.cpp side. Meanwhile DFlash2 support landed on llama.cpp last week . Ubuntu 26.04.1 got released. ROCm 10.0 got released & llama.cpp also up with Version 10.0 now (Applicable to only AMD cards) Other optimizations & fixes on llama.cpp side So how much t/s are you getting now with all optimizations & stuff? Please share your extreme optim”

heh

“Name and Version Present on current master in ggml/src/ggml cpu/arch/s390/quants.c . Found while working on a z/OS port of llama.cpp; the file is s390x only so this affects IBM Z builds. Operating systems Other? (z/OS, and any Linux on IBM Z) Which llama.cpp modules do you know to be affected? libggml (CPU backend, s390x) Problem description ggml vec dot q5 1 q8 1() reads v acc in its own initialiser, before it has been assigned: https://github.com/ggml org/llama.cpp/blob/master/ggml/src/ggml cp”

“llama.cpp uses layer-split as its default multi-GPU mode. Each GPU holds complete layers and their KV cache. For every generated token, just one activation vector passes over PCIe at the boundary between the two GPUs. So the x2 slot should have a bigger impact only when the model is loading.”

“I wanted a clean answer to whether Ollama was leaving CPU performance on the table on this little Celeron N5095 board, and after running a few tests and getting some input from this community I wanted to do more so I ran Ollama 0.32.1 and a native Release build of llama.cpp commit 9a286ac through the same CPU test. TL;DR: Ollama’s CPU runner was faster across the board on these matched runs. Model llama.cpp tok/s Ollama tok/s Qwen3 0.6B 6.725 7.809 Qwen3 1.7B 2.852 3.321 Qwen3 4B Instruct 1.484”

“i was enjoying my Qwen 3.8 27b coding but at long context PP drops to painfully low tokens per second and the copilot chat timeout because of the times it takes, so i asked claude to see if i can enabled P2P on my two gpus, he points me to https://github.com/aikitoria/open-gpu-kernel-modules/tree/610.43.02-p2p which work on RTX 3090, RTX 4090, and RTX 5090 ( 5060ti not listed) but after following the guide ( i am already on linux and have nvidia open source driver ) i got the 5060tis to list OK”

“I tested it by monitor actual disk reads. It’s working. Just make sure mmap is on with it”

“Hey Guys, So, Claude and I have been working on a V100 specific fork of Llama.cpp. WyvernTKC/llama.cpp-4xV100: My llama.cpp changes for my 4 x V100 Dell C4140 NVlink V100 32GB (128GB VRAM) 386GB RAM I run a Dell C4140 at home, so the primary aim has to improve the Tensor Parallelism. I've got some good improvements and have got a similar offloading method like FreeToken implemented. If you happen to give it a go, please let me know how it goes. Only tested on my NVlink V100's under Windo”

“I posted a new gfx906 based llama.cpp fork a few days ago. One of the main points of critique was that i did not provide sufficient numbers for the gains to be achieved. -- TL;DR: After switching our Qwen 3.8 27B production setup to DFlash2, several of the old gfx906 optimizations turned out to be neutral or outright regressions. We went back through the existing gfx906 work, isolated the problem areas, reworked the small-Q Flash Attention path and added adaptive native/convert selection. Agains”

“not with the current llama branch. Unsloth uploaded a readme regarding this.”

There was a fix went in to llamacpp. Now it behaves much better.

“some of these links are in my post, I posted what was merged and some things in progress”

“Isn't it? ExLlamav3 + heretic allowed me to abliterate Laguna-S-2.1 against a 2.50bpw exl3 quant.”

“Can you update your thread with below ones? (Pasting below one from my other comment) Open/Ongoing PRs: qwen4exp: direct reads for the lazy PLE table (>2x prefill performance improvement on GB10)- #28136 qwen4exp: follow up fixes- #27941 kv-cache : index (seq,pos) cells to make ngram prev-token lookups O(log n) for qwen4exp decode speedup- #27992 qwen4exp : add NextN/MTP draft head (--spec-type draft-mtp) for Qwen3.8-Flash-Next- #27836 models : appropriately flag noscan ssm_a tensors- #28121”

“Yeah, llama seems to want to use the same storage for everything in a gguf chunk or shard or whatever it's called. Nothing magic there, they just moved the embedding table to its own chunk, so it can stay in host memory without forcing all the weights to do likewise.”

“Llama.cpp and vllm use layer splitting which means layers 1-20 live in GPU0 layers 21-40 live in GPU1 only thing communicated is the results of inference in the first 20 layers to the GPU1. This card uses an internal PLX PCIe connection that is sufficient for the KBs of data transferred. People use PCIe 2.0 to connect multiple GPUs for layer splitting without issue. It is old and slow though, 11 tflops fp32 performance sounds stinky. Until you consider that won't be the bottleneck for infere”

“if you are on Qwen Flash Next make sure to update your build often https://preview.redd.it/zoq3j9gnjvmh1.jpg?width=900&format=pjpg&auto=webp&s=bb21bc405696df94fd5ce38c22f4ca6dc596135f merged already (by ServeurpersoCom ) https://github.com/ggml-org/llama.cpp/pull/27978 https://github.com/ggml-org/llama.cpp/pull/28011 https://github.com/ggml-org/llama.cpp/pull/28023 https://github.com/ggml-org/llama.cpp/pull/28123 merged (by 0cc4m ) https://github.com/ggml-org/llama.cpp/pull/28032 in”

“Can't wait to test! This should significantly boost TPS! Now we just need more llama cpp optimizations to be merged in! submitted by /u/vini542reddit [link] [comments]”

“Sorry, It can't be converted to GGUF - llama.cpp doesn't support my model.”

“Hi, could you share your current llama.cpp configs? I have a 5070 ti to, new to this and can't seem to get such speeds, also is it a dedicated AI setup or AI and Gaming?”

“This might be the problem I got yesterday, I saw high disk read during inference with the Flash Next, a ton of swap space usage, while ~24GB of RAM sitting idle. But I think I tried setting lazy mode to off and it didn't help. Do you need a newer build to run this model or the latest stable would do?”

“With b10726, the default --lazy-mode change keeps the 51B-parameter PLE n-gram embedding table of Qwen 3.8 Flash Next on disk: it is mmap'd and its rows are read on demand during inference, even with --load-mode none. It is no longer loaded into RAM unless --lazy-mode off is passed. This change resulted in 50% pp speed penalty, and 15% token generation speed penalty for me. Make sure to add the flag --lazy-mode off if you have enough RAM like me. submitted by /u/whiteh4cker [link”

“Is it the third implementation of https://github.com/ggml-org/llama.cpp/pull/27210 ? :)”

“This is a custom patch for llama.cp made by me, it's not available in mainline, I edited the main body to clarify :)”

“I've made a modification of llama.cpp MTP for people that want to run models like QWEN 27B on 16GB and similar setup, the focus is reducing the memory cost of MTP allowing more speed for less ctx cost. MTP Mode Maximum Draft (n) Available Context TG (t/s) Standard 2 72,192 39.53 MTP Compact Rollback 5 77,312 46.39 On this example of a (well tuned!) IQ4 running on 16GB you get some +5k ctx and enjoy 17.35% increase on token generation. With MTP the more speculative tokens you generate (n-max)”

“Sad that I only have 12gb of vram but this ik_llama is so fast submitted by /u/Needausernameplzz [link] [comments]”

Any similar skill for CUDA folks?

“You need to wait for the fixes to be merged - then you’ll be able to offload the ngram to the SSD which will make you able to deduct from the sizes mentioned above which needs to be loaded int memory the ngram so you’ll be able to use the Q4 comfortably but possibly even higher quant”

“Summary Please add DFlash2 speculative decoding support to llm scaler vllm , targeting Qwen3.8 27B on a 2x Intel Arc Pro B60 platform. DFlash2 landed upstream in vLLM v0.28.0 ("DFlash2 with local convolution and a candidate selector", vllm project/vllm 52816), which is ahead of the current llm scaler base (vLLM 0.21). Target configuration | Item | Value | | | | | Target model | Qwen/Qwen3.8 27B | | Draft model | z lab/Qwen3.8 27B DFlash2 | | GPU | Intel Arc Pro B60 x 2 ( tensor parallel size 2 )”

“TL;DR - yeah it works, no wild skills or patching required. 37-50 tg/s when generating text, spiking over 60tg/s when writing code and mtp hits hard. Your results can be better if you have a better motherboard than my aging x370. ---- Grabbed the docker image from here: https://hub.docker.com/r/rocm/dev-ubuntu-24.04/tags docker pull rocm/dev-ubuntu-24.04:10.0.0-full Started up the docker image, installed git and cmake on the docker image since they appeared missing. Pulled llama.cpp latest from”

“Full BF16 (330 GiB, no quantization) Decode: 6.43 t/s. Prefill: 16.5 t/s (that prefill is off a 77-token prompt. I haven't run a proper length sweep yet) Setup: stock llama.cpp master (b10680) over RPC on a LAN, 4 devices across 3 boxes: 94 GiB on the RTX PRO, 40 on the Mac's Metal, 30 on the 5090, plus 5090's DDR5 as a 4th RPC device. About 39 GiB still spills to local (RTX PRO's) RAM as the device pool can't hold all 234 GiB of non-PLE weights. Still hunting the bottleneck.”

“What version of Bun is running? 1.3.14, 1.5.0 What platform is your computer? Linux Debian 13 x86 64 What steps can reproduce the bug? Im building it with the following command, tried to build and run it with both versions (with and without baseline arg): I then installed that build and tried to run it with the following commands: Nothing happens and it eventually says "killed". Some Info using hwinfo: Its a disposable test vm. What is the expected behavior? To instantly execute the command and”

“I did some of my own testing by having the harness write its tests based on my specs and previous work, basic token gen and work on real projects with opencode and dsh, I get 59 tks in dsh one shot full software and up to 70 tks in testing on already done code base. Way more context and faser than mtp2, had 120k ctx and 41tks previously. It seems that n-max 4 with spec draft min p 0.7 is the fastest setting on Qwen 3.8 UD Q4 K M, rtx 3090. It also works with 205k context which is nice, n max 8 a”

“What is the issue? System: OS: Kubuntu 24.04 Kernel: 7.0.0 30 generic GPU: NVIDIA GeForce RTX 5060 Ti 16 GB NVIDIA driver: 580.173.02 NVIDIA kernel module: open Ollama version: 0.33.2 CUDA runner used by Ollama: cuda v13 Since upgrading from Ollama 0.32.15 to 0.33.1, I have experienced two GPU crashes with the exact same NVIDIA Xid 62 signature, both attributed to llama server , followed by Xid 154 ( GPU Reset Required ). The first crash occurred on 2026 08 27, several hours after upgrading to O”

“i also make it work at 20 t/s without unslloth studio, mb its help you(but with vision and 130k context size its consume 14 gm vram on my machine): llama-server ^ -m "F:\.lmstudio\models\AtomicChat\Qwen3.8-Flash-Next-GGUF\Qwen3.8-Flash-Next-AD-4.27bpw-Q4_K_M-M64-00001-of-00033.gguf" ^ --flash-attn on ^ --load-mode mmap ^ --fit on ^ --ctx-size 180000 ^ --no-context-shift ^ --parallel 1 ^ --cache-ram 0 ^ --ctx-checkpoints 8 ^ --checkpoint-min-step 1024 ^ --tensor-read-lazy on ^ --no-reas”

The fact that we are allowed to use a project like this for free. 😅

“Can anybode please help me run this model with MTP enabled, I can not figure out a way to actually run it. Any help appreciated. Thanks my .ini: [*] # perf flash-attn = true mmap = true warmup = false parallel = 1 threads = 6 threads-batch = 6 batch-size = 2048 ubatch-size = 1024 # caching #sleep-idle-seconds = -1 cache-prompt = true keep = 3000 # sampling + reasoning temp = 1.0 top-p = 0.95 top-k = 20 min-p = 0.0 presence-penalty = 0.0 repeat-penalty = 1.0 reasoning-preserve = true reasoning-ef”

“TL;DR: Every public low-bit GGUF of this model is secretly ~4.70 bpw. Shim the rows to 256 and it becomes a real 3.07 bpw / 11.77 GiB file that runs 262K context on 16GB. Needs patched llama.cpp — not LM Studio or Ollama. In the AtomicChat HuggingFace repo it says "There is currently no good 16 GB option for this model, from anyone." That was true, I wanted to figure out why, and it's a quantizer problem, not a model problem. k-quants and i-quants need the row width to divide by 25”

“"Cheap. Right. On time. Pick any two." ... never gets old. Optimizing for everything at the same time is rarely wise/successful. Obviously, this applies to LLMs and how to run them via llama.cpp as wel. Afer you finally made the bloody thing *load* at all, what is most important to you ofl: quality context length speed (and a few other things...) The amount of knobs, buttons and sliders makes it hard to keep track of it all. And it is hard to understand how stuff interacts and/or impac”

“hey, I just posted a 'feature request' to increase performance of --n-cpu-ffn https://github.com/ggml-org/llama.cpp/issues/27987”

“I expected the Q4 to fit on my system with tensor-read-lazy on but I kept running out of memory. Turns out load-mode auto wasn't using mmap. But once I turned it on manually, I got it to work! llama-bench -m models/qwen/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf --device Vulkan2/Vulkan1/Vulkan0 --tensor-read-lazy on --load-mode mmap -ngl 49 --split-mode layer -ts 19/10/20 WARNING: radv is not a conformant Vulkan implementation, testing use only. WARNING: radv is not a conformant Vulka”

“not tested by me: https://github.com/ggml-org/llama.cpp/pull/27941 https://github.com/ggml-org/llama.cpp/pull/27879”

For new models, wait for couple of weeks to get better performance.

“Thanks — when you say fixes are coming, do you specifically mean the Qwen3.8-Flash-Next QSA long-context path (sparse KV gather/indexer optimizations), or more general fixes? Is there a PR/branch I can test?”

“I'm currently running Qwen3.8-Flash-Next on a CMP 170HX 64GB + RTX 3090 24GB, with 80GB system RAM. With llama.cpp I've already spent quite a bit of time tuning it: layer split across the two GPUs, PLE on CPU, q8 KV, Flash Attention, detached MTP draft on the 170HX, and some custom MTP/runtime work. My current results are roughly: Short context: ~900 tok/s prefill, ~70–80 tok/s decode with MTP 100K: ~600 tok/s cumulative prefill 140K: ~520 tok/s cumulative prefill 262K: ~355 tok/s cumula”

“Can provide configs if people are interested but did not want to do the wall of text. Below is AI assisted drafting of bullet points of what I have achieved so far: LLAMA.CPP 32 t/s decode / 463 t/s prefill at UD-Q4_K_XL with a 640K serving pool (4×160K lanes, q8 KV) The 51B n-gram/PLE table lives mmap'd on NVMe; measured irreducibly cache-hostile (0% cross-token reuse), it self-regulates to near-zero residency at ~zero decode cost. SSD random reads are the right architecture, not a compromi”

“Guys, if you have low VRAM, you should start with a small quant first to verify that everything works correctly. command line: .\bin\Release\llama-server.exe -m J:\llm\models\Qwen3.8-Flash-Next-UD-IQ1_S-00001-of-00003.gguf --parallel 1 -c 10000 results: 2.08.059.754 I slot print_timing: id 0 | task 72 | n_gen = 100, tg = 21.47 t/s, tg_3s = 21.69 t/s 2.11.085.278 I slot print_timing: id 0 | task 72 | n_gen = 169, tg = 22.00 t/s, tg_3s = 22.81 t/s 2.11.266.578 I slot print_timing: id 0 | task 72 |”

“Prerequisites [x] I am running the latest code. Mention the version if possible as well. [x] I carefully followed the README.md. [x] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed). [x] I reviewed the Discussions, and have a new and useful enhancement to share. Feature Description While working on 26358 I realized that if we carefully implement fa+gqa for multi token parallel verify, we should be able to reuse same k”

“What is the issue? What happened After auto updating from 0.31.1 to 0.31.2, my daily driver model ( gemma4:31b with num ctx=65536 baked via /api/create ) went from 33.8 tok/s to 4.7 tok/s on the same hardware, same free VRAM, same everything. The root cause appears to be that 0.31.2 estimates the model 1.2 GiB larger than 0.31.1 does , which pushes it just over my 24 GB VRAM budget, so the scheduler offloads part of it to CPU: | | v0.31.1 | v0.31.2 | | | | | | ollama ps size | 18.8 GiB | 20.0 Gi”

“What is the issue? Ran ollama run gemma4:e2b in WSL 2 Ubuntu 24.04 Got Relevant log output OS WSL2 GPU Nvidia CPU Intel Ollama version 0.30.4”

“Name and Version build: 7462 (6ce3d8579) with Clang 21.1.5 for Android aarch64 using termux (Samsung M33 5G) ggml model f32.gguf Samsung M33 5G CPU: 8× ARM Cortex A53 @ 2.00 GHz (64 bit, ARMv8) RAM: 8 GB Storage: 128 GB GPU: Mali G52 (integrated) OS: Android (you’re running Termux) ABI: arm64 v8a Other: Supports NEON, DOTPROD, FP16 vector ops Operating systems Other? (Please let us know in description), Linux Which llama.cpp modules do you know to be affected? Other (Please specify in the next s”

“🚀 The feature, motivation and pitch I started hacking on vLLM recently for RamaLama and other reasons to see what the hardware support is like compared to llama.cpp. The one big noticeable gap is Vulkan support. It would solve a bunch of problems around vLLM not running great on commodity hardware: https://github.com/containers/ramalama/pull/1677 Alternatives Use llama.cpp Additional context No response Before submitting a new issue... [x] Make sure you already searched for relevant issues, and”

“Name and Version library 531cb1c233800e6acb021dc56d69595e314db072 (gguf v0.4.0 2819 g531cb1c2) Operating systems Linux Which llama.cpp modules do you know to be affected? No response Problem description & steps to reproduce 1. Compile the program below 2. Run it a thousand times and it will probably have a segmentation fault at least once. I used the gdb debugger. Simple program: Shell script to run the program several times: First Bad Commit No response Relevant log output GDB output from crash”