llama.cpp Crashes or Crawls on Recommended Hardware
Developers are encountering significant challenges and inconsistencies when attempting to run large language models (LLMs) locally using llama.cpp. Issues range from crashes and incorrect tool usage to slow performance and unexpected token limitations, despite following recommended configurations and troubleshooting steps. The complexity of setup and optimization is proving frustrating and time-consuming.
SOURCES (60)
“I have not recently tried it, but I did run llama.cpp on my MSI Titan SLI with 2 980M in 2024. From what I recall it was abysmally slow.”
“As I said in my post, it’s my own personal fork of llama.cpp based on a version of llama.cpp from about four months ago, but my changes won’t have affected PP speed.”
“I am using it strictly for LM link. So I can work on a laptop and have models running on desktop. Once I bother to set up alternate link I will go back to llama. Hope we will resolve the MTP issues. Something is off. It worked better with earlier releases.”
“Repository: https://github.com/fredconex/Arandu Hello Guys, This is Arandu a open source app to easily launch models using llama.cpp, it tries to integrate all in one place with easy multi version management of llama.cpp, HuggingFace models search/download, intuitive arguments with presets and more. I've not been updating this too much recently, mainly fixes and stuff to make it compatible with recent llama.cpp versions, I keep this mainly for my own usage but so far some people have been en”
“openPangu-2.0-Flash - 92B A6B & 512K Context Length Model : https://huggingface.co/openpangu/openPangu-2.0-Flash/blob/main/README_EN.md GGUF for ik_llama folks : https://huggingface.co/ji-farthing/openPangu-2.0-Flash-ik-llama-GGUF submitted by /u/pmttyji [link] [comments]”
Maybe you gonna like: https://celsowm.github.io/llama-cpp-wasm/
“https://preview.redd.it/n5pwopmc1vdh1.png?width=1602&format=png&auto=webp&s=fd95510f4b8245454e72df7231a8c06ae0bed655 did you try it with right settings: llama-cli.exe --model "Ternary-Bonsai-27B-Q2_0.gguf" --mmproj "Ternary-Bonsai-27B-mmproj-BF16.gguf" --temp 0.7 --top-p 0.95 --top-k 20 -p "Write me an HTML page for lorem ipsum generator"”
“Thank you, appreciate the insight and for helping me move past my prior level of ignorance”
“Speculative decoding research is extremely important to scaling out models, and there is a lot of momentum right now in that area, and it's cool to witness. But yes, MTP in it's current form slows down agentic usage for nearly all hardware in real world use cases. My objection isn't to MTP, it's to benchmarks such as OP's ( "6x faster REAL CODING!!!") when back in reality it's actually a net slowdown. Folks new to running local models don't know that and are”
“Hey guys, Last week I posted my DFlash benchmarks here (4.44x at 36K context) https://www.reddit.com/r/LocalLLaMA/comments/1uq0h4o/i_tested_freshly_merged_dflash_in_llamacpp_on/ . One of the comments form u/exact_constraint mention there are also n-gram lookup drafters (ngram-mod, ngram-map-k4v). I find out that they stack on top of DFlash, so I benchmarked the full stack. Short version: on varied single prompts they add nothing, on iterative coding they take DFlash from 3.7x to 6x, and they cos”
“Hey everyone, I've been slamming my llama.cpp instance a lot more heavily recently, and on it I'm running Qwen 3.6 27B from Unsloth on my RX 7900 XTX. And after slamming it aggressively I tend to get what looks like kernel panics ? Machine doesn't respond to SSH anymore and I have to hard reboot it. Have you ever ran into such an issue ? And if so what did you do to fix it? Note that I'm running DDR4 RAM, not some kind of ECC there unfortunately. submitted by /u/Brace”
“You could also look at LM Studio. It's user friendly and wraps llamacpp. Not as much control as direct CLI llamacpp parameters but it's easy to use and probably faster than Ollama. Definitely worth looking at Gemma 4 E4B or maybe E2B. Try the QAT (Quantisation aware training) ones since they've done extra work to prevent the quantization process lobotomising them too badly. Might be worth keeping an eye out for the PrismML Bonsai 1.5bit quantization of qwen 3.6 27b (just announced),”
“5% boost on tg for Bonsai models. This is the 1st PR mentioned on yesterday thread (Other Open PRs section) submitted by /u/pmttyji [link] [comments]”
“Hola Strix and AMD stacker frendios. Read the Lineage and Credits, this uses charlie12345/ROCmFPX, won't work on native llama.cpp yet. 122B total · 10B active · 60.70 GiB · 28.50 tok/s MTP-off · BF16 KLD 0.041366 · Decode 28.505 Decode speed + 36.89% faster Size - 13.47gb smaller submitted by /u/RedParaglider [link] [comments]”
“https://i.redd.it/ycnzi4ho6hdh1.gif I run models locally on my own box and I love llama-server , but I do not love memorizing ~220 flags or babysitting rebuilds. LM Studio / Ollama / Jan are great if you want zero-config, but they hide the real llama.cpp server from you. I wanted the opposite — full per-model control, in a browser, without the terminal gymnastics. So I built LlamaForge . It's a thin control panel over llama.cpp's own server. No inference code of its own, no telemetry, ba”
“I can't get this to work at all using the llama.cpp repo provided. I've tried everything. ``` ❯ ./build/bin/llama-cli \ -m Ternary-Bonsai-27B-Q2_0.gguf \ -n 256 \ --temp 0.7 --top-p 0.95 --top-k 20 \ -ngl 99 .... build : b9591-62061f910 model : Ternary-Bonsai-27B-Q2_0.gguf modalities : text available commands: /exit or Ctrl+C stop or exit /regen regenerate the last response /clear clear the chat history /read <file> add a text file /glob <pattern> add text files using globbin”
“Below Upstream Status sections are from https://github.com/PrismML-Eng/Bonsai-demo Upstream Status for Binary Q1_0 is supported out of the box in upstream llama.cpp across many backends: CPU (generic, NEON, and optimized x86), Metal, CUDA, and Vulkan. Runtime Status llama.cpp (CPU, Metal, CUDA, Vulkan) ✅ Merged upstream, works out of the box MLX (1-bit) ⏳ Pending upstream: mlx#3161 ; until it merges, use PrismML-Eng/mlx (branch prism , built automatically by setup.sh ) Upstream Status for Ternar”
“Some fixes & boost(pp) for SYCL/Intel. Merged PRs: [SYCL] Flash Attention with XMX engine via oneDNN graph API (SDPA) on KV f16 for Xe2 ; Qwen3.6-27b-Q8_0 prefill speed up x1.21 at p=512 and x4.26 at p=80k #25222 sycl: Increase minimum buffer size for USM system allocations #25525 [SYCL] Support OP XIELU #25550 [SYCL] support conv2d_dw kernel type is fp16 #25653 [SYCL] fix UT big error of get_rows Q2_K, Q4_K, Q5_K #25656 submitted by /u/pmttyji [link] [comments]”
“Pretty useless for coding tasks. Cut's thinking process. Never responds with result. command line (from PrismML HF page, llama.cpp built from their llama fork): ./llama-cli -m Ternary-Bonsai-27B-Q2_0.gguf -n 256 --temp 0.7 --top-p 0.95 --top-k 20 -ngl 99 "Tell me a story" https://preview.redd.it/97gckbgnxbdh1.png?width=2214&format=png&auto=webp&s=812f08827b7351b62cceced444084b2a29d8d561 Write simple C# function: https://preview.redd.it/trpo6lrrxbdh1.png?width=1127&f”
“According to their GitHub , the file needed is the -Q2_0.gguf version, which requires compiling their version of llama.cpp . Easiest way to download is to use huggingface-cli if you have it available: hf download prism-ml/Ternary-Bonsai-27B-gguf --include "*-Q2_0.gguf*" If not, just download the file from https://huggingface.co/prism-ml/Ternary-Bonsai-27B-gguf/tree/main and save it somewhere. Now, we have to compile llama.cpp: ``` git clone https://github.com/PrismML-Eng/llama.cpp pris”
“I haven’t explored vLLM in depth yet, but recently we’ve seen a number of interesting performance-related features added to it, while llama.cpp appears to be following a different set of priorities (they clearly care about performance, just with their own focus). I’m curious what people think: should llama.cpp begin supporting additional formats such as safetensors alongside GGUF, or is it better for the project to continue focusing exclusively on its current approach? P.S. This is not a feature”
“The current implementation of multi token prediction (MTP) in llama cpp could trigger BF16 compute selection on GPUs that don't support BF16, causing cuBLAS crashes on older architectures (tested on Kepler). I patched llama.cpp's CUDA backend to add a robust capability check: - BF16 supported → use BF16 - No BF16 but fast FP16 available → fall back to FP16 - Older GPUs (Kepler etc.) → fall back to FP32 This keeps MTP working on older cards without affecting newer GPUs. Tested with: - Qwe”
“I might be missing something (likely a skill issue), but I consistently run into problems when setting up inference engines on machines with CUDA GPUs. Recently, I spent significant time trying to get things working on an RTX Pro 6000 Blackwell. I tried installing several inference frameworks without success, including SGLang, TensorRT, and llama.cpp. I was eventually able to build llama.cpp from source successfully. The model loaded correctly, but every inference request to the llama server res”
“I think Ternary 27B still needs llama.cpp support. Right now their custom fork is the way. and new 1-bit 27B Q1_0 which works too. That's surprising. Will try this tonight/tomorrow 27B Q1_0 runs at around 0.5 t/s but it runs. Probably PR merge needed for speed. (Need to look at that custom fork)”
“I found really interesting to automate performance profiling of a linunx program. Here a video demo while LLaMa.cpp is profiling a LVGL space HMI running on a Qemu computer. https://www.youtube.com/watch?v=at_nI4hn7gI project: https://github.com/leonardosalvatore/llama.cpp.debugger submitted by /u/leonardosalvatore [link] [comments]”
“That's a great milestone! Thanks on behalf of all. I'm glad to see some llama.cpp contributors hanging out here regularly(So we could request them to optimize the code extreme level 🚀 to run 1T models just with 8GB VRAM 😄 Yeah, they can do it 💪 )”
“Well, this is an appreciation post on the spiritbuun llama.cpp fork. Some weeks ago I was testing forks and configurations to find which one was the best for my secondary model on my 3060, and the winning combo turned out to be Spiritbuun's fork + CUDA + mudler's Apex I-Compact quantization for the Qwen3.6-35B-A3B model. See https://www.reddit.com/r/LocalLLaMA/comments/1tq0h1p/qwen3635ba3bapex_128k_ctx_on_rtx_3060_12gb_37_ts/ Then Spiritbuun shipped turbo8, and I switched my key cache to”
“Hello, I downloaded a Llama3.2-3B-Instruct yesterday and tried running the model verification through the llama cli according to the instructions on meta's download page. The download was successful, but I've been facing some issues, and I don't quite know if it's my ignorance showing or is something truely not right with the download. I tried downloading the model twice and both times the orig_params.json is missing and when running a midel verification I get hash mismatch for e”
“Thanks to all contributors for our local inference! submitted by /u/sergeysi [link] [comments]”
“I mean I don't get how. Are you using the same models I am? Do you have much faster models available or something? I have currently optimised IK_llama to the greatest extent I am able to figure out. I'm hitting 25tok/s on tg on occasion. But my PP is still never going higher than 150 at best (usually a fair bit lower).”
“the thing the whole thread is circling: for token generation (tg), the iGPU and the CPU pull from the same system RAM, so they hit the same bandwidth ceiling. tg is memory-bound — each token reads the active weights once — so the bus is the wall, and the iGPU can't get around a wall the CPU is already standing at. the iGPU only helps prefill (pp), which is compute-bound. your 285HX is DDR5-5200 dual channel ≈ 83 GB/s. Qwen3.6-35B-A3B is MoE, so decode only reads the ~3B active params, not al”
“128 MEGAbytes. That "smartly putting parts of model in RAM" is exactly what llama.cpp already does (and always has). It sounds like the only novel thing it does is expert caching, and even that I'm skeptical of. I did just realized that I meant a 0.8B model (mixed it up with a different one), but same idea— the filesize even when quantized is like 5x the size of the RAM.”
“Yeah.. at least the OCD will be strong enough with a couple of handfuls to get the Vulkan speed up to proper levels.. I started at 17'ish tg on vanilla and then worked my way up..”
This is an unraid box, so it's mostly docker containers or a linux vm
“I Recently did a bunch of tests and wrote them all up on here, but the short version is that Vulkan basically doesn't work (or when it does, it's at 1tok/s at best). SYCL works pretty well, seems to run the Qwen3.6 35b models at around 12tok/s. The prefill part is around 20tok/s when it works but can be hit and miss and sometimes when you say "do that again" the llm just freezes up for ages. CPU by itself is the most consistent at actually working, and runs at more like 14tok/s”
“For whom is this tread : Everyone with a 24GB GPU (rtx 3090, 7900xtx, rtx 4090) What this Thread is for : Sharing proven/well working llama-server start configs. Requirements for the configs: - Utilizes the the VRAM as much as possible - Provides at least 200.000 tokens KV Cache State next to your start command, how much System RAM (normal RAM) you have, as this could very well influence caching performance/viability of your command. Also, if you possible, include infos regarding you OS and CPU,”
“Hey I am running a few models on a strix halo box. Especially for the larger models (like Qwen 3.5 122B) they work okayish performance wise if the cache is utilised properly but a full cache miss at 100k context causes roughly 10-20 minute of PP time - which is extremely annoying. I will first show what I have already configured (and it helps!), then describe what is still not working well. I am very interested on your input what I could still fine tune. What I've configured so far (and what”
“If you want to just duplicare layer of LLM model, you could do that with forked version of llama.cpp or vLLM without actual modification. And it already shows disappointing result in the currenr models”
“hello people i wanted to share my experience with big and huge models (usually 100B+ models and 200B+ models and more) my laptop specs is very poor I7-8750H 20G Ram GTX 1050 Mobile 4G Vram but what nearly saved me is my NVMe from samsung i have 512G NVMe from samsung and yes as you expected. i run these huge models while throwing most of the parameters in my NVMe but i strictly use MoE models, Dense models will kill my machine always used mmap. and throwing experts in my CPU with Quantized KV Ca”
“I saw a lot of (complete and abortive) jacobian lens projects for HF and PyTorch, but nothing for GGUFs or llama.cpp. So I set Fable 5 on xhigh to solve this problem (with close human supervision of course 😎). Inspired by Anthropic's paper and code, and of course by my favorite inference engine llama.cpp and GGML. I did a code review and a few test passes on it. It has a native GGUF server (based closely on llama.cpp and git synced to it) for model observation AND j-space swapping/abliterat”
“B9966 If you run -sm tensor in production you might want to grab this fix which removes 29 regex recompilations per tensor per token on the decode thread. Claude tell me in one sentence what this means: The code was rebuilding 29 regex patterns from scratch on every tensor of every token instead of building them once and reusing them, so the fix just caches them — same behavior, way less wasted CPU on the decode thread submitted by /u/Bulky-Priority6824 [link] [comments]”
“TL;DR: We modified llama.cpp so that Qwen3.5-35B-A3B (Q4_K_M, ~21 GB GGUF) runs on a laptop RTX 2050 (4 GB VRAM) at around 1.2 tok/s , exposing an OpenAI-compatible API that's actually usable for agentic coding . so I'm here looking for ideas to improve generation speed. Also is there any opensourced MOE model that loads experts for per input prompt and not per token? Non-Techie Description : Think of the AI model as a huge library containing hundreds of books (called experts ). A normal”
“such a great piece of software, i'm grateful to everyone that contributes since its multiple projects rolled into one. i do have a q tho that probably doesnt need its own thread: i was trying to connect opencode to my local model but it won't. i enabled jinja as per a suggestion in another thread, but it wont connect to localhost/port even with the /v1 at the end. is there an idiot-proof guide to this? i wouldn't mind using something besides opencode as well if theres an alternative.”
“https://github.com/ggml-org/llama.cpp/commit/ccb0c3422394fbbfc28fd91f8c77111b748cfa09 It seems to be time for another llama.cpp rebuild, at least if you are on amds ROCm/HIP. There are no benchmarks included and i am still building, so it would be nice if any of you could report back on the performance changes. submitted by /u/milpster [link] [comments]”
“Thanks, I am really struggling to dial in vLLM, so I might just try llama.cpp for that reason alone.”
“Hi, I've been running vLLM on my MI50 because of tensor-parallel support. It works, but I have some complaints. For one, the quants seem much harder to find than GGUFs. Also, model switching is a pain, especially with the insanely long startup time. I recently discovered that llama.cpp supports tensor-parallel (I've been away for a long while). Is there any reason I should stick with vLLM? submitted by /u/FrozenAptPea [link] [comments]”
“Thaks. How do you use such a script? Is that just a proxy between the actual model server (llama-server vLLM, etc) and your tooling? And it strips stuff out, or does your harness support scripting of some sort that you're using?”
“llama.cpp finally supports DPS4, unsloth released GGUF version, all of this waiting for what? 8 tokens per second? I was expecting much better numbers on the Mac Studio M3 Ultra 512GB. am I doing something wrong or this numbers are normal? it tried both Q8 and Q4. ./llama-server \ -m '..../models/unsloth/DeepSeek-V4-Flash-GGUF/DeepSeek-V4-Flash-UD-Q4_K_XL-00001-of-00005.gguf' \ --host 127.0.0.1 \ --port 8080 \ --jinja \ -ngl all \ -np 1 \ -c 10000 \ -b 2048 \ -ub 512 \ -t 24 \ -fa on \ -”