LocalLLaMA Users Facing Technical Hurdles

“full writeup on r/LocalLLaMA and the repo github.com/aic0d3r/qwen38-strix-halo-harness [re: Story time: Qwen3.8-Flash-Next on my Strix Halo laptop vs Claude Opus 5.5 on the same feature]”

“I've been looking at the <32B model space and keep coming back to an interesting question. A few years ago, projects like Hummingbird+ suggested that cheap custom accelerators (FPGA-based) might become the future of local inference. But today it seems like memory capacity is still the real bottleneck rather than raw TOPS. For someone who wants to run modern 20B-32B models at reasonable quants (Q5/Q6 rather than INT4), the options all seem compromised: Consumer GPUs have great bandwidth bu”

“What do you expect the extra 64gb of RAM to cost for a total of 192gb RAM? The current Strix Halo 128GB version costs $3K . The 495 is almost the same as the 395 except slightly higher spec on the memory, so 8% tps improvement. https://www.bosgame.com/blogs/news/new-product-launch-bosgame-m5-max-with-amd-ryzen-ai-max-pro-495-processor https://www.bosgamepc.com/blogs/coming-soon/new-product-launch--bosgame-m5-max-with-amd-ryzen-ai-max-pro-495-processor submitted by /u/Terminator857 [l”

“People always talk about the successes but I'm sharing my failure for posterity. Spent hours trying to get egpu working with a framework strix halo. TLDR; not worth the pain or I'm unskilled or unlucky. Hardware I tried: ADT-link UT3G - Could not get a consistent connection. On booting Fedora, I'd see "nvidia kernel module missing, falling back to nouveau" (even though nouveau is blacklisted so that line turned out to be boot-time noise, not real nouveau). Running nvidia-sm”

“I'm looking through some Strix Halo devices, and things like TUF 14 can have more storage than a 2230 single SSD ProArt or Z13. It caps at 64GB RAM and is way cheaper. My question is what's practical to run on it - >60"GB" models would fit on 128GB variants but run slower and slower. Context would be software development aids, Grammarly-like writing checked/fixer, some experimentation with Lemonade and other tooling. submitted by /u/riklaunim [link] [commen”

“Hey all, Not sure whether best to post here or on the llama.cpp board, however...: I have two Strix Halo systems that run over a (cheap) set of mellanox Connect - X 3 cards for RDMA (Parallel, as it uses USB4 instead of the LAN ports that are 5GBe each) All other models from Step 3.7 Flash (my daily driver) to Laguna S2.1 works amazingly with this setup, and has no issues, other than being slower that a model on a single platform. Deepseek V4 Flash is different, however... I can load it using my”

“Especially now that deepseek flash got updated, im very tempted to buy. For folks already with dual halos, what advantages do you see over one Secondly how would it work with a Bosgame m5? Is the usb4 networking fast enough? submitted by /u/lawanda123 [link] [comments]”

“Name and Version version 1 (1705d43) build with Clang 23.0.0 for Windows Operating systems Windows GGML backends HIP Hardware AMD Strix Halo 128GB Models Qwen 3.6 35B MoE Q8 Problem description & steps to reproduce I was fighting with 30% cpu usage and decreased GPU utilization on AMD Strix halo machine when running Qwen 3.6 35B MoE Q8 generation on ROCm. Vulkan does not have that problem and needs only 5% cpu to run tg. GLM found this issue in src/llama model.cpp around line 1300: This keeps to”

“I finally had a chance to play with AMD's new Ryzen AI Halo box, here I show my configuration that can get you 10-15% performance improvement on LLM inference. submitted by /u/Intrepid_Rub_3566 [link] [comments]”

“Hello all, currently have my strix halo with windows and 96gb vram memory reserved. Which I thought was the proper way to do it until I watched https://youtu.be/Gz62bniDkpg?is=w-hKNFJxRSaHyMiC his latest video on amd strix halo box were he discusses that you can leave 512mb for vram and everything on "memory". And it's just as fast. That would work better, as easier to run larger models but I swear I head that its faster to run with larger vram memory cause of how amd processes vra”

“This is something that never gets mentioned when people complain that it's slow and new users are told to avoid them. This 48 cent figure is worst case scenario, running multiple models/compiling hitting CPU, GPU, and NPU at the same time for 24 hours a day. I can handle only 50tps on Q8_XL Qwen 3.6 35B when it's silent, sipping power, and is the size of a small router. I know your Nvidia card is significantly faster, but if you even consider using more than just raw GPU memory speed/com”

“Fable in retrospect: what worked, what didn’t work, and what kind of misleading Twitter posts led me to waste a bunch of tokens. A post-mortem. Fable did a fairly awesome job fully optimizing my pruning/Reap stack on my 16×32GB V100 Threadripper board. It found optimizations that allowed me to more effectively and quickly observe, prune, and adaptively optimize Minimax M2.7 and Minimax M3. It reduced model load times and came up with patches and/or configurations that made it so I no longer had”

“Hey everyone, I'm trying to use MiniCPM-V to analyze long react-style videos (where a creator is on-screen commenting on a piece of content, photo, or text shown next to them). Right now, I have the audio side figured out—using WhisperX to get word-level transcripts with precise timestamps. My next step is to inject the actual visual changes into the context alongside the transcript. The problem is that these videos are long and have a ton of repetitive frames (like the creator's face or”

“Overview This PR extends the UE4M3 lookup table optimization introduced in #23961 to the ARM implementation of the NVFP4 dot product. The ARM implementation now uses the existing GGML_CPU_UE4M3_TO_FP32 lookup table for UE4M3 scale decoding. This aligns the ARM implementation with the x86 implementation while reusing the shared lookup table infrastructure. llama-bench Baseline (master) Command: ./build-master/bin/llama-bench -m models/Qwen3.5-4B-NVFP4.gguf -p 512 -n 0 -r 5 model size params backe”