Local Models Break Down in Multi-Tool Agent Workflows
Users are finding that even powerful local large language models (LLMs) exhibit unpredictable behavior and performance issues when used in complex agentic workflows or with numerous tools. Achieving reliable results requires significant configuration tweaking, custom tooling, and often, a deep understanding of model internals, creating a frustrating and time-consuming experience.
SOURCES (60)
“It started as a fork of llama-swap , but I have been building it out for myself since then as a convenient tool for all my local AI needs, and by now it has drifted far enough to be its own thing. The main idea…”
“haha, hopefully not, i think the benchmark is really interesting and just want to get feedback on the benchmark and seeing where people think the comparison could be better”
“How was your experience with it and is it just benchmaxed or is it really that good? submitted by /u/Personal-Try2776 [link] [comments]”
“For a few days I've been working on creating a custom local-only harness for some work related research using Codex / GPT 5.6 Sol and the model feels not only dumber than usual, but straight up counter productive. It keeps adding unnecessary guardrails for the local agents, removes tools that I clearly specified I want them to have and always drifts from the original requirements. I need to ask it to change things multiple times, which ends up on some over-complicated final product. This is”
“We have been working on an open-source, model-neutral agent harness for general purpose agents called TrueForge, and wanted to understand how much the harness itself actually matters. So we ran 14 tasks from DevRev Enterprise-Bench through multiple harness/model combinations, three times each with a blind judge. The result that surprised us most: Claude Managed Agents + Opus 4.8: 11/14 tasks solved | $11.8/run | 10.0M tokens/run TrueForge + Opus 4.8: 11/14 tasks solved | $8.6/run | 3.7M tokens/r”
“spent a huge chunk of time prompt engineering Qwen3.5-4B for an Android app decomposition is the way: use deterministic code to present your model with very clean narrow objectives with simple outcomes and a narrow set of tools. Like a menu, there are levels to it that are multiple models runs for each decision. But the harness drives and creates a small surface in each turn small models are very sensitive to context pollution, compounding the specificness hence, everything is slow, prefill spee”
“Hi, I'm trying to build a workflow for doing research tasks on the internet. At the moment I am using Gemma-4-31B-IT-QAT (120k context) for planning,reviewing and orchestration and Gemma-4-12B-IT-QAT(256k context) for execution. Both model quants by unsloth. I am using opencode and wrote 4 agents for this purpose. It works pretty good, but not stable enough. Sometimes it needs an hour for a task which normally takes 8-10min. The planner prepares batches, the orchestrator invokes the executor”
“I've been building Arcon around Qwen3-4B + LoRA. Instead of just making it a chatbot, I'm experimenting with persistent memory, personality/mood, internal state, tools, and eventually having it process things before replying. I'm curious what people who've built local agents think - how far can you realistically push a small model with good architecture around it? I put the whole thing on GitHub if anyone wants to poke around, roast the architecture, or tell me what I'm doing”
“Everyone keeps talking about fitting larger models into local hardware, but parameter count isn’t the whole story. For long context inference, KV cache can become the real memory bottleneck. Every new token adds key and value states that need to stay available, so a model that fits comfortably in VRAM can suddenly become painful at 100k or 200k context. GQA and MQA help by reducing KV heads, and KV cache quantization helps even more, but the cache still grows with context. This makes me think fu”
curious what kind of custom MCP setup youre running with it
“Interesting angle. On the theme work I do, targeting a lower ES version reliably costs real KB once you factor in the helper/polyfill overhead for things like async iterators or optional chaining transforms. We enforce a hard JS budget in CI, so parse/eval time actually matters as much as transfer size to us — on low-end mobile CPUs the eval cost of a heavier polyfilled bundle can dwarf the network savings from gzip. Curious whether pagegym.com's numbers separate parse time from execution ti”
“I wouldn't rely on Qwen3.6-35B-A3B for anything. When it fails, it fails epically. I'm sure there are smarter newer models in the same computational complexity class.”
“Agreed. The best part of ThinkingCap is that it's not a template trick, when doing simple tasks the model reasons less while with complex tasks it reasons as usual.”
“I was planning the software stack for my inference server, picking what to run and what resources to plan for it, when I remembered that LocalAI was kinda like this inference service orchestrator. So, I went to check back in - been about a year and change since I last looked at this. Well it went away from llama.cpp entirely and to their own vllm.cpp and many other tools...but the Issues tab is full of the same agent account, and I did not dare to check the PRs after seing this. Seeing a project”
“This project is completely non functional for actual inference. I ported it to Windows. Everything compiled cleanly. It runs. And the output is pure hallucinated nonsense, even for the simplest possible request like writing a “Hello World” Python script. Complete garbage tokens, incoherent rambling, nothing usable. I assumed I must have screwed up the port. So I tested the exact same setup on Ubuntu using the provided Docker environment (so the environment is identical to what the author claims”
“All frontier models trained for agentic workflows suffer from this to some degree I am having a hard time with Kimi K3 during some coding sessions”
“There are long tails here and there even with frontier models. For example, I ask Claude to work with some data I stored as CSV. It keeps recognizing row number added by its read tool as a separate column. Then I have to ask it read raw data. This happens constantly so in the end it writes this into its memory.”
“🚀 The feature, motivation and pitch While reviewing https://github.com/vllm project/vllm/pull/54238 pullrequestreview 5062756046, it was pointed out that the implementation follows the traditional per parser approach: We recently introduced a new unified approach in 45802. However, that approach relies on Parser Engine, and MuseGlimmerReasoningParser hasn't been migrated to Parser Engine yet. I think it would make more sense to first refactor MuseGlimmerReasoningParser to use Parser Engine. Onc”
“It has taken two months now we finally know the truth about good structure. Good structure is where the dev unit called you+chatbot ends up when it has nothing. GPT custom code runs very much the same regardless of how it is 'structured' - provided function does not differ. Code on a page is more like a list of routes than a structure or a mesh or a prime nexus. The word 'structure' is used by LLMs to provide impressive sounding structural fluff. Sometimes users of these LLMs con”
“TL;DR: 11 coding benchmark boards are normalized by field-size percentile, de-duplicated by benchmark family, and combined across repository, agentic, live-coding, and function-generation tasks. The current snapshot covers 98 model series and 268 evidence rows. I’m looking for missing leaderboards and better signals for local deployment. https://preview.redd.it/i6slt8ddhfmh1.jpg?width=2038&format=pjpg&auto=webp&s=48479132e7c5b8334e5d3433de825f9e26c164fe Hi r/LocalLLaMA — I’m one of t”
“Most agentic task is closer to 30tps. But I have multiple agents hitting it so it just feels like forever.”
“Hi all, I'm looking to try out a local model for agentic coding (for now; seems like an easy starting place). I've used opencode and cloud-based open models for personal projects and I'm hoping to sell colleagues on local models. At home I only have a 2018 laptop so no possibility to try things out for personal use first. At work we have some reasonably beefy hardware that's underutilized, but it's mostly high RAM. Here are the specs for one workstation: GPU1: Nvidia GeForce”
“This base model is only 43m parameters trained on 3m arXiv abstracts. We plan to continue pre-training and post training. If you create fine-tuning datasets or if you know of any datasets that can help shape the behavior for our goal we appreciate all contributors. The goal is to make a local agent that can autonomously do research. Its just a simple loop to search the web & document its findings as an experiment to see what is possible. If we train a language model on nothing but science, p”
“They do. Yeah. No question about their agentic work but imo they are pretty bad at writing. Outputs are unnecessarily repetitive, verbose and jargon-heavy. I like Gemini for its output for that reason”
“Single B70, sure. But my 4xB70 are still a crashy mess on VLLM/VLLM-XPU or LLM-Scaler. Llama.cpp mostly works, but it can take a bit before new models get all the bugs worked out.”
“I built a benchmark called EchoNet. An agent gets a factual question, then searches a syntheic web I made before answering. Some of that web is seeded with misinformation: one fake page, a fake page ranked first in search results, the same fake claim copied across many pages, a loud fake majority around a real primary source, or a genuine update the model's training predates. When a model reads a new page, it weighs two things: what it already knows and what the text says. Usually, they agre”
“Their last version 7.14 was released just a month ago. llama.cpp PR(waiting for approval) for Version 10.0 https://github.com/ggml-org/llama.cpp/pull/27803 Hope this version comes with more boost & improvements. submitted by /u/pmttyji [link] [comments]”
“Look at what actually exists: the leading harnesses offer a handful of models. Open-weight and non-frontier models sit unused. Attempts to aggregate models discount complexity and fail to accommodate user diversity. The solutions come in two forms, and both are incomplete The first asks you to move to a new app, a new workspace, or ecosystem. But the best developers we know change tools constantly, because the best place to work keeps changing. Any solution that requires relocation is betting ag”
“I didn't realize at first because most of my local models can't make or use subagents to multitask fast and speed up work. Apparently, Tiel-Coder-35B-A3B-UD can make up to 4 subagents. i am using Tiel-Coder-35B-A3B-UD-Q4_K_S with 256k context so maybe the bigger quant can use more subagents submitted by /u/Khaledthe [link] [comments]”
“OP here: this project was born out of the frustration/paranoia that AI providers are throttling their models when their server load is too high. So, I set out to model and study the problem mathematically to understand what was happening, what I found was quite surprising.The idea seems natural: as the data center demand increases momentarily through the day, throttling their models (either using quantized versions, reducing the context window or lowering the tier of the model to a smaller one)”
“I have a very noob question. Does the lookup happen as the model pushes through each and every layer to include whatever additional reasoning or thinking was generated by that layer? this is very likely my misunderstanding and maybe a single token gets generated after going through all layers. but essentially, doesn't the ngram lookup get generated at the gfx card level, and if it needs to reach out to the ram, wouldn't that be slow (unless the ngrams were held in the graphics card vram”
“For me at least it’s unbearably slow, the point of intelligent models is longer context tasks and having to work around the limitations of speed really harshes the vibe”
“Your current environment Various environments depending on the platform and the framework. See the repro commands below for non standard environment settings, used in the measurement. 🐛 Describe the bug <details <summary Evaluating GLM 5.1 FP8 for reasoning over large contexts produces unexpectedly poor level of accuracy. </summary The vLLM server, running GLM 5.1 FP8 produces very few correct answers. E.g. a full sweep over 503 questions of "Long Bench V2" yields only 5 correct results on CUDA”
“I've been doing useful tasks with LLMs since GPT-3; I don't really NEED the very best reasoning models, but the stronger the model the lazier I can be. Since I got my Strix Halo my daily drivers have gone from GPT-OSS-120b, to Qwen 3.5 122b, DeepSeek Flash 0713, Qwen 3.8 27b, and now Qwen 3.8 Flash Next that I just got running this morning. I'll use it in OpenLumara for life management, research projects, taking notes, that sort of thing. Then I use it in Hermes or Pi to implement a”
“I use it for testing my automation pipelines locally before switching cloud in production. Handling private information any anything that speed doesn't matter for me (like creating summaries, data extractions, etc) and most importantly to feel the safety of not being locked to big corporations :)”
“What are you actually using your local models for? And what tasks actually requires the very large very high reasoning models? I am struggeling to come up with use cases for my locally hosted models. Coding assistance is great of course, but other than that I can't come up with much more other than research/asking questions. And I can't really think of what I would do with some of the huge models (If I could run them). But I guess this is like when the internet first became a thing, many”
“Sorry I missed the second part of your question, got busy dealing with all the haters :D This harness + Model-host is about a 100k lines. Building a harness requires being able to manage a growing and changing code-base. You have to understand the basics of a LLM chat, keeping the conversation stable to not invalidate the KV cache, using a few libraries to interface with the server, making sure streaming works, and it doesnt cause performance issues with updates, when chat is being streamed lett”
“If you want to run, eval, or train a model to actually use a computer — clicking through desktop apps, filling in web forms, tapping through a phone app — try CUA-Lite . It democratizes all four pieces: the harness, the sandboxes, the data, and the framework to run, eval, SFT, and RL your own model. Code: https://github.com/cua-lite/cua-lite Homepage: https://cua-lite.github.io Blog: https://cua-lite.github.io/blog Harness: modular, not a black box. The code that turns a model into a computer-us”
“Hi r/LocalLLaMA ! We’re Apodex , the team behind Apodex 1.1 , our new model family built to scale agentic intelligence for complex work. We’re excited to be here and answer your questions directly. Apodex 1.1 is designed around sustained, verifiable progress toward real-world objectives—from reasoning and search to working with files, executing code, recovering from failures, and coordinating multiple agents. Open models Apodex 1.1 Apodex-1.1-mini Apodex-1.1-mini-NVFP4 Apodex-1.1-mini-GPTQ-Int4”
“In my experience, local AI kinda takes forever to do a lot of stuff. Looking at Qwen3.8-27B for example, it's thinking is hella smart, but it also sits there deliberating for ages before it starts doing stuff. (yes I know this is harness-dependent and thinking-level dependent). Then there's also all the waves vaguely with quants and KL-divergence and benchmark scores dropping a few points and pareto thresholds and everything else. I don't actually care about any of that stuff. I care”
“What about kv cache reuse for sub agents? For me the main problem have been that.”
“Sure! Quick question first so I can make sense of your feedback later: what's your setup (RAM + Windows version)? I'll DM you a code as soon as you reply.”
“The one time I tried asking Sol to use subagents for a small project, it took a surprisingly long time, used up the entire usage limit in one go, and basically failed the project.I’m pretty sure that plain Sol, serially, could have finished the task faster, cheaper, and far more accurately. I’m also pretty sure that any competent subagent orchestration could have gotten it done with even very simple subagents quickly and cheaply.(Is it really that hard to set up a handful of subagents that all u”
“OP here: this project was born out of the frustration/paranoia that AI providers are throttling their models when their server load is too high. So, I set out to model and study the problem mathematically to understand what was happening, what I found was quite surprising.The idea seems natural: as the data center demand increases momentarily through the day, throttling their models (either using a quantized version, reducing the context window or lowering the tier of the model to a smaller one”
“You nailed the exact failure mode it's built against. The learning engine starts in shadow mode: it watches the baseline heuristics decide, and a decision only earns credit on a later pass when the outcome is verified against what actually happened to that page (stayed cold vs. got touched again). Confidence is a slow curve over verified-outcome observations only — it needs several hundred in a given pressure context before it takes direct control there, and it falls back to the baseline the”
“Appreciate you sharing your findings. They match mine indeed. I think there is still some space to improve the switching dynamics. Might look into that next.”
“Good question. I think keeping context small is the better solution but it requires a different approach. I am looking into that. You can add multiple model profiles in an ini file and i can swap depending on the task. Long/short reasoning, MTP/dflash etc”
“ive been tossing around the idea of dynamically swapping mtp and dflash at certain points where one beats out the other, like long context. any thoughts?”
“Honestly easiest for you if you're at the learning phase is to use LM Studio instead of llama-server as it gives you a user interface to chance settings and you can download models directly inside of it etc. https://lmstudio.ai/ You can move to llama-server easily after you get the hang of the local stuff with LM studio. I'm studio uses the llama.cpp back end anyways which llama-server uses. Get the LM studio, download models from the left menu bottom option and search for the gemma 4 26”
“How do MoE even know which active parameters to use, how come they're not as lobotomized as the average 3b model when there are 3b active parameters MoE out there”
“I used this as a reference and forked it https://pi.dev/packages/pi-subagents I forked it so I can have my own custom frontmattter in future and I didn’t want some of the built in subagents. I did end up taking the scout subagent tho. I’d say to add things as you need it and not to strictly port over what you had before. Prior to pi I was using my own extension to orchestrate across Claud code and cursor since Claude for worrying code is overkill and I did everything with composer 2.5 and used C”
“Check Existing Issues [x] I have searched for any existing and/or related issues. [x] I have searched for any existing and/or related discussions. [x] I have also searched in the CLOSED issues AND CLOSED discussions and found no related items (your issue might already be addressed on the development branch!). [x] I am using the latest version of Open WebUI. Installation Method Docker Open WebUI Version v0.11 Ollama Version (if applicable) No response Operating System Rocky Linux release 9.7 (Blu”
“Check Existing Issues [x] I have searched all existing open AND closed issues and discussions and found none comparable to my request. Verify Feature Scope [x] I believe this feature request is appropriately scoped for the Issues section as described above. Problem Description My Use Case: When I give the chat a long, multi step task (e.g., "Analyze this data file one by one and generate a report"), the context window often runs out in the middle of processing. The Current Behavior: The chat com”
“🚀 The feature, motivation and pitch We currently have the following MoE "oracles", which select the right MoE kernel for each model model executor/layers/fused moe/oracle We have: fp8 nvfp4 mxfp8 unquantized and will soon have mxfp4 Each of these has the following functions: select XX moe backend called by the quantization integration to get the backend convert to XX moe kernel format called by the quantization integration to shuffle the weights make XX moe quant config called by the quantizati”