llama.cpp Crashes or Crawls on Recommended Hardware

“ggml org/Qwen3 ASR 1.7B Q8 0.gguf stopped working in wllama 3.6.0 (was working in 3.5.1) from deepseek 4.1 flash: Found it — and verified it end to end. Here's the root cause of the 360 hang. Root cause: the new persistent CPU threadpool in llama.cpp 3.6.0 starves wllama's fixed pthread pool, and extra threads beyond that pool deadlock wllama compiles its wasm with pthreads and a fixed pre spawned pool : nbThreads = floor(hardwareConcurrency/2) = 8 workers ( wllama 360/src/wllama.ts:484 ), passe”

“sglang/flashinfer don't exist for Windows. Please add a way to run Ultra Agent Mode on Windows. Edit: I tried making it running with WSL, just to see if the LoRA was working at all, but it seems it's not: <details <summary Image 1</summary <img width="714" height="799" alt="Image" src="https://github.com/user attachments/assets/c126f1be 37ac 4788 bf60 4ee27960e0d8" / </details <details <summary Image 2</summary <img width="715" height="684" alt="Image" src="https://github.com/user attachments/as”

“Two things I ran into while moving my Strata setup from IQ2_XS to Unsloth's UD-IQ4_XS (RTX 4090, i9-14900K, 64GB DDR5-5800, Arch). 1. Greedy sampling on older builds Before Strata 0.1.45, a run config without a `sampling` block decoded greedy whenever the request didn't set its own temperature. Greedy is what makes these models loop. 0.1.45 fixed this upstream (#2685). A request with no sampling now gets the model card's values: - Thinking: temp 1.0, top_p 0.95, top_k 20 - Non-thinki”

“I am not complaining, I just want to understand. So I am running llama-server on a headless Linux with Qwen3.6 35B A3B at a Q5_K_XL quant (26GB safetensor) and 132k context and KV cache at q8_0. I have a 4060TI 16GB with an old i7 gen 7 processor and 48GB DDR4 sysRAM. When I run a dense model of like 26B (Gemma or Qwen) at a Q4_K_M quant I push it with -ngl 25 or so to the limit of my available VRAM and then creep at around 2 t/s through molasses. But with the MOE version I get relatively good s”

“Either llama.cpp itself or Unsloth studio (because I have studio around for training anyway). I've tried Atomic, Chatbox, AnythingLLM, and a host of others. Also, I'll often just use pi for chat as well. Not much choice if you're terminal only. One odd thing I have found is that big prompts freak almost all of them, usually without good error feedback or even any at all. Try it some time, paste in what you would normally have in a document as a spec sheet, and watch them explode. Edg”

“Is it possible, as I get 50 tps on llama.cpp, and 35tps on coding at long context, I get sustained 40 tps on strata with iq3_xxs flash next which performance better than q4 27b for me, but since I have 32gb ram, there's slow prefill and occasional freezing. submitted by /u/UNO10100f [link] [comments]”

“There's a GGUF. I assume llama.cpp just uses the existing Q2/IQ2 dequant kernels on whatever arch you're running on. Isn't gonna need Blackwell. [re: nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB · Hugging Face]”

“Prerequisites [x] I am running the latest code. Mention the version if possible as well. [x] I carefully followed the README.md. [x] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed). [x] I reviewed the Discussions, and have a new and useful enhancement to share. Feature Description It'd be awesome if the new GUI can be used on linux. Maybe as an Appimage. Motivation All linux LLM GUIs such as unsloth studdio, bionic a”

“If the model support MTP they will not. Fck llama. [re: 137B A6.8B model from Microsoft?]”

“Strata prompt processing is a lot faster than llama.cpp running the same model. For me: llama.cpp = ~200 pp/sec Strata on one GPU = ~1200 pp/sec Strata on two GPUs = ~1800 pp/sec I have 3080 20G so no NVLink, one card is in a PCIe x4 slot. [re: Does strata on multi-gpu without nvlink worth it?]”

“Your current environment <details <summary The output of <code python collect env.py</code </summary </details 🐛 Describe the bug Description On stock vLLM 0.31.0 (zero local modifications), with a repetitive prompt on a hybrid linear attention model (Qwen3.5 9B: gated delta net (GDN) layers + full attention layers), the first speculative verify forward of width ≥ 8 rows that crosses the junction prompt len + 8 makes the request stop after exactly 10 tokens with finish reason="stop" . Greedy sa”

I'm getting [E] SageAttention is unavailable on CUDA0; it requires patched GGML, CUDA Toolkit 12.0 or newer, and SM80 or newer kernels --- diffusion_engine.cpp:906 - I suppose Ampere is a minimum requirement…

“Starta was run with fp16 kv in this case, as was llama.cpp. So it helps probably for longer ctx [re: qwen3.8-flash-next-iq3_s vs -27B-GGUF:UD-Q4_K_XL]”

“I'm using Qwen 3.8 27B on an R9700 and it works great, occasionally I'll switch to Tiel-coder for rapid simple tasks. One big problem I run into that makes real projects come to a halt is that more code takes up more context, compaction becomes more and more frequent between any edits, until it gets stuck in a compaction loop not doing any useful work. I started using Q4_K_XL and Q8 kv cache, for a context of 200k tokens, which helps from my normal 100-150k, but is there a technique to e”

“Your current environment ============================== System Info ============================== OS : Ubuntu 22.04.5 LTS (x86 64) GCC version : (Ubuntu 11.4.0 1ubuntu1 22.04.3) 11.4.0 Clang version : Could not collect CMake version : version 3.22.1 Libc version : glibc 2.35 ============================== PyTorch Info ============================== PyTorch version : 2.10.0+cu129 Is debug build : False CUDA used to build PyTorch : 12.9 ROCM used to build PyTorch : N/A XPU used to build PyTorch :”

“that's how fast gpt is running at over on r/codex lol [re: OMG! If you have a Mac with 64GB, try Qwen3.8-Flash-Next-oQ4e-mtp with oMLX!]”

“Follow-up to last week's post about streaming MoE experts from disk in a browser tab. That ran on hand-written engines for two models. This week the same disk tier sits under llama.cpp's own WebGPU backend, so it works for the MoE models llama.cpp already runs, with no per-model engine. The video is one real recording in Chrome. The sped-up parts say so on screen, with a real-time clock. Measured in that recording (MacBook Pro M4 Pro, 24 GB): Model (GGUF) File On the GPU Load from disk D”

submitted by /u/Itsallabouthirdbase [link] [comments]

“My project is optimising qwen flash next on Mac Mini m5 64gb through SSD streaming. I recently asked LocalLLama for guidance on how to optimise token output. Today a popular method to do this is a lookahead guesser which fetches experts one layer ahead of the current layer. One popular suggestion was building a prediction model to start loading experts two layers ahead. This netted me +8% on chat and +4.2% on code. The way the various optimisations stack on top of each other this may actually ne”

“I'm now getting 3x the speed than with llama.cpp (Unsloth fork): 10.52.637.962 I slot print_timing: id 0 | task 0 | prompt eval time = 620987.80 ms / 38761 tokens ( 16.02 ms per token, 62.42 tokens per second) 10.52.637.966 I slot print_timing: id 0 | task 0 | eval time = 15443.97 ms / 292 tokens ( 53.07 ms per token, 18.84 tokens per second) 10.52.637.967 I slot print_timing: id 0 | task 0 | total time = 636431.76 ms / 39053 tokens Strata (same Hermes session, loaded with Strata and a turn”

“And llama.cpp use cuda as backend, lol [re: Anyone feel like reverse-engineering CUDA?]”

“I originally installed llama cpp via their install script on their page; it used Vulkan by default. The GPU (NVIDIA) would "disappear" and I'd have to restart the machine. I re-built llama cpp for CUDA, and had to remove that original install, but the llama cpp built with/for CUDA is much more stable. If I also get an AMD GPU and want to have both in my PC, do I also have to build a separate llama cpp "instance" for HIP usage? I'm not planning on using the same instan”

“I've made a few posts recently, just talking about the hardware I've got and the model I use. Pair of P100s, Unsloth Qwen3.6 35b A3B UD Q4 K XL . There's been a ton of talk of model and hardware specific inference engines showing crazy performance numbers on models and hardware that have no business doing what they do. Inspired, I've decided to pull out Mr. Opus and see where I could push things, targeting the P100 and this model specifically, and found some pretty great gains. T”

“Name and Version $ llama server version version: 0.6.0 dev (build 1, commit a05e58c) built with GNU 13.3.0 for Linux x86 64 Operating systems Linux GGML backends CUDA Hardware NVIDIA RTX PRO 2000 Blackwell, 16 GB (sm 120) Models ggml org/embeddinggemma 2 GGUF, embeddinggemma 2 Q8 0.gguf Problem description & steps to reproduce With fa on , EmbeddingGemma 2 is about 10x slower when the input length is not a multiple of 256 tokens. GPU utilization is 0% during those requests. With fa off , the spe”

“I'm trying to get Ornith 1.5 9B running (I can also barely run 35B A3B but I want something that I can run relatively easily) with the latest llama.cpp with this command: llama serve --flash-attn on --cache_type-k q8_0 --cache-type-v q8_0 --hf-repo ornith-ai/Ornith-1.5-9B-GGUF --spec-draft-p-min 0.9 --spec-type draft-mtp And this gives me ~15 tok/s TG and ~200 tok/s PP. However, without MTP, I am getting double these numbers. I have a very high draft acceptance rate of 0.95 with a mean lengt”

“This is about Ollama, but the mechanism is the same for anything built on llama.cpp (LM Studio, Jan, Open WebUI in front of Ollama, your own scripts). A local model keeps a KV cache of the prompt it has already processed, and reuses it on the next request. That reuse only survives on a shared prefix : it holds for exactly as long as the new prompt matches the old one from the very first token. The first token that differs throws away everything after it, however identical the rest is. Which mean”

“I've been tuning llama.cpp on my Strix Halo box (Radeon 8060S, 128 GB) and the numbers are good enough to share. It's a branch called rdna35-perf on top of halo-box/strix-llama.cpp , a llama.cpp fork aimed at Strix Halo: https://github.com/asf0/strix-llama.cpp/tree/rdna35-perf (10 commits, ROCm/HIP only) TL;DR Prompt processing: 21–54% faster than Vulkan, and 8–174% faster than stock ROCm. The lead over ROCm grows with context, because stock ROCm flash attention falls off hard at depth.”

“I saw a PR for Llama to get an expert caching mechanism, but it isn't in yet. It was merged last week. https://github.com/ggml-org/llama.cpp/pull/29887 [re: Please recommend Qwen3.8-Flash-Next llama parameters]”

“Best for agentic coding on 5060 TI 16GB is Qwen3.8 27B GSQ IQ3_XXS with DFlash2 on llama.cpp + omp command to run: .\llama-server.exe ` >> -m ".\Qwen3.8-27B-GSQ-RCO-IQ3_XXS.gguf" ` >> -md ".\Qwen3.8-27B-DFlash2-Q4_K_M.gguf" ` >> --alias Qwen3.8-27B-GSQ-IQ3_XXS-DFlash2-96K ` >> -c 98304 ` >> -ngl 999 ` >> --fit off ` >> --flash-attn on ` >> --cache-type-k q4_0 ` >> --cache-type-v q4_0 ` >> --parallel 1 ` >> --ba”

“We've added native support to llama.cpp, so no shim is needed: https://github.com/ggml-org/llama.cpp/pull/30361 Until it's merged, the build steps and run command are on the model card: https://huggingface.co/h2oai/h2o-lightning-4b-native-gguf [re: H2O-Lightning-4B: Apache-2.0 4B Decision model, official #1 open model on JevBench (above Jev)]”

“My Jetson Orin–optimized engine, little-gemma V1.0, substantially outperforms llama.cpp. Even after exhausting every practical GGUF option, however, it still falls short of the ideal performance level for Gemma E4B on the Jetson Orin Nano Super 8GB. I forked exllamav3 and ported key code from little-gemma V1.0. This delivered the prefill improvement I needed. Although the decoding rate took a hit, overall voice-chat latency is lower. EXL3 on the Jetson Orin Nano isn’t for everyone, but if you ne”

“What is the issue? Description: When running models with large attention head dimensions ($head\ dim = 256$, such as qwen3.8:27b or gemma4:31b) on Apple Silicon GPUs (M2 Max, 64GB RAM), Ollama’s compiled MLX SDPA backend panics during prefill execution due to a hardware threadgroup allocation mismatch.The C++ MLX transform layer requests 960 threads per threadgroup, exceeding the Apple Silicon hardware cap of 896 threads per threadgroup for this kernel path, causing the GPU worker to crash and f”

“Literally made a Git account for this. Specs: Bazzite 44 NVIDIA RTX 4070 Ryzen 7 7800x3D Had to roll back to 0.5 bc 0.51 just wasnt working, something with Vulkan. Ive finally got it to work (albeit without online compatibility), and launching from steam works fine, however id like to be able to skip the gui and just get straight into BB to make it feel more like an actual steam game. I followed "The Casual Gamer PH" 's tutorial on youtube, despite the fact im not on steam deck, but no matter wh”

“@sayamss I have one of these cards and am running it with llama.cpp. Curious to know if you compared Radiance performance against it? Do you have any notes on your Radiance config and compile instructions? [re: Building a 4x R9700 setup for a 10 person startup]”

“Summary of project status metrics Since StillDeadcode has released his new radiance inference engine , the writing is on the wall for the old vllm-radiance builds, people have moved on to newer and better. I can see a few more single digit percent speedups available but probably won't chase them. I thought I'd spend the last 5% of effort and finish off my personal build. This borrows from quite a few people who have done work in the Launch80 Discord and is by no means my sole work. I&#39”

“Probably the lack of overthinking (on high)! I've only managed to get 8tg 40pp so far (plz recommend better engine for mixed VRAM and RAM, unsloth's fork of llama.cpp runs but unoptimized and no MTP). It still feels better than Q-FN for anything requiring precision (like security) because despite being way slower, it doesn't hallucinate nearly as much and just gets things right without doubting itself for 50k tokens per decision [re: Benefits of using bigger models than Qwen 3.8 flas”

“Each page load? But the models can be cached after all. Still browser dependent and clears on cache clear but could be an idea to consider [re: WebGPU + llama.cpp = Client-side LLMs]”

“I love the idea of running local models on consumer hardware. I currently use `llama.cpp`, but I’ve been looking for a "better" alternative for a while now. I want something that utilizes my system resources more efficiently—for instance, by managing my RTX with 4GB VRAM more intelligently—and delivers higher tokens-per-second, all without the burden of heavy dependencies like PyTorch. I’ve done my research, and honestly, I haven't found a solution yet. I came across plenty of opti”

“It's not required. Simple layer splitting works just fine. Plenty of people do it. I've got an R9700 and 5060 Ti that I can run layer splitting across on llama.cpp vulkan when I want. You don't need a 32GB card specifically. You're not totally shit out of luck if you didn't get a 5090. [re: NVIDIA reportedly discontinuing RTX 5090, GB202 GPUs to be reserved for RTX PRO series]”

“Any version of llama cpp actually, but by default it pulls the most recent. CUDA or Vulkan / CPU is your choice. It can also automatically download and run ninfer, vllm and more. The system is interfaced in a way where I can rapidly add any backend if none of them fit your needs. (Some people are asking for rocm, so... soon) It doesn't require it! I do have an optional vision model that can be downloaded in the stack, but everything in that little demo was purely using my extraction and navi”

“Why not to compare with stock llama.cpp ? [re: Qwen Flash Next @ 137 tok/s & 3,497 tok/s Prefill w/ 512k context on a 5090, 192gb ram, Windows Build, comparing Strata and Infernix]”

“Yes, use ninfer-all (I use it on my 5070Ti, I get ~90-100tok/s, a 2-3x improvement over llama.cpp) https://github.com/iamwavecut/ninfer-all [re: Qwen3.8-27B with just 12GB VRAM + 8GB RAM - ~18 tok/sec]”

“That seems really interesting! Looks like llama.cpp does already have support for Youtu-VL and Youtu-LLM... At least with the size it's not a huge deal to just use transformers. Anyone tested this out yet? It seems promising. [re: tencent/Youtu-Parsing-Omni · Hugging Face]”

“it aint thaaaat horrible, i run it via llama-swap and with stable diffusion.cpp from leetjeet. basically the same with llama.cpp [re: Qwen/Qwen-Image-2.1-Turbo · Hugging Face]”

“VS Code with llama-vscode extension using Qwen2.5-Coder 7B or 3B if you're VRAM-limited. You just point it at an instance of llama.cpp, comes with presets too. It's super fast, not terrible, but there def room to improve. But it's free and local, so... We'll see if this model can slot in there. [re: Mellum2.1 - a JetBrains Collection]”

“I tested Mellum2.1-12B-A2.5B (Q8) locally using Pi and llama-server. All five tests were one-shot. Results were pretty mixed: - Pelican SVG, Browser OS, Minecraft: Poor results. - Bouncing Hexagon: Physics were okay, but surprisingly it made it run in the terminal. - Flappy Bird: Completed it, but the visuals were very basic and the game was way too difficult. Pelican SVG Browser OS Minecraft Flappy Bird Bouncing Hexagon Okay, these might not look great, but hear me out. This model is actually p”

“It’s more performative than the OrcaSAQ-2-27B release recently [re: Running the uncensored Qwen3.8-27B (HauhauCS) on a 4090 at 262K context and ~130 tok/s]”

“Had this exact issue with a 5090 on an ASUS Z690, Fedora 44 and a llama.cpp build from a few weeks ago. No overclocking of any of my components. Couldn't tell if it was poor settings on my end or what, but my GPU would disconnect several times an hour and the system would lock out several times a day under heavy inference loads. Switched to Ninfer and haven't had a single lockout even while running two subagents in parallel. Yesterday I built llama.cpp v0.6.0 to test out a different GGUF”

“multi gpu is that last line of my post [re: llama : add a GPU cache for MoE experts kept in host memory by am17an · Pull Request #29887 · ggml-org/llama.cpp]”

“You're shooting yourself in the foot by using ollama. Use llama.cpp. [re: Even good models shouldn't authorize their own tool calls — 192 local runs with Ollama and vLLM]”

tysm! it should be 100% possible to use swift.cpp in unsloth studio as it's basically just llama.cpp optimised to run really well (2-3x faster) on Swift models getting the context window up…

“buddy like I said I could understand bespoke engines being 30% faster or similar, but 5-6x times faster is pure incompetence, llama.cpp is like a power virus level of suck.. the worst part is these other projects aren't even funded, llama.cpp devs should be ashamed really. As a dev I'd be embarrassed personally. [re: llama.cpp on the stage]”

“600MB is tiny, try bigger numbers. Server in verbose mode shows cache hit rate when shutting down, check it; the cache only becomes useful above like 90%. [re: llama : add a GPU cache for MoE experts kept in host memory by am17an · Pull Request #29887 · ggml-org/llama.cpp]”

“What is the issue? run gpt oss:latest in ollama version 0.40.1 Relevant log output OS Linux GPU Nvidia CPU Intel Ollama version 0.40.1”

“Environment GPUs: AMD Instinct MI355X (gfx950, DigitalOcean gpu mi355x1 288gb spot , SR IOV VF) and MI350X (gfx950, gpu mi350x1 288gb spot ). Container: lmsysorg/sglang:v0.5.20 rocm720 mi35x ( sha256:ef84c44150da0f90495953458e7386350ffc50a37b2b9ac6802d9a94c454c6dd ). ROCm 7.2.0, hipcc /opt/rocm 7.2.0/bin/hipcc . aiter 0.1.21.dev48 (git 4ad99832823dde2315b361cbd3b54b1c5c12acd5 , 2026 08 29). Vendored composable kernel af9e1d1f1ae347c22feeb08fd2d42645075e0c5d (2026 08 05). torch 2.x ROCm build fro”

“Your current environment I don't have access to the original system so not everything here is populated, but big ticket things like os, amdgpu kernel version, hip version, etc are populated. <details <summary The output of <code python collect env.py</code </summary </details 🐛 Describe the bug Summary Decode performance regresses sharply for Kimi K2.5 MXFP4 on MI355X / MI350X (gfx950) between vLLM ROCm v0.18.0 and v0.19.1 , and does not recover on v0.22.0 . This appears distinct from 43153 (IN”

“Name and Version $ llama server version version: 8604 (6307ec07d) built with GNU 13.3.0 for Linux x86 64 Operating systems Linux Which llama.cpp modules do you know to be affected? llama server Command line Problem description & steps to reproduce I woke up this morning, compiled the latest Github build, and suddenly all of my browsers on multiple computers which use the llama server webui are coming up completely blank. No errors, nothing. Looking at the llama server log, I see: Something appea”

“What is the issue? Following the docs to build Ollama from source, then trying to pull llama3.1 results in a refusal stating the Ollama version is too old. OS Linux, macOS GPU Nvidia, Apple CPU AMD, Apple Ollama version 4d52e5d dirty”

“Doesn't happen with the other Vulkan backend.”