Local Models Break Down in Multi-Tool Agent Workflows
Users are finding that even powerful local large language models (LLMs) exhibit unpredictable behavior and performance issues when used in complex agentic workflows or with numerous tools. Achieving reliable results requires significant configuration tweaking, custom tooling, and often, a deep understanding of model internals, creating a frustrating and time-consuming experience.
SOURCES (60)
“Spent the last few months building this because I thought of testing AI suggesting SQL "optimizations" I couldn't verify without setting up test infrastructure myself. argus seeds a local Postgres in Docker with realistic synthetic data, generates optimization alternatives, and actually benchmarks them, so you…”
“Blunt network blocking via Zscaler stops the bleeding, but it usually triggers a game of whack-a-mole with shadow IT because developers desperately want these tools. A more sustainable middle ground is wrapping the AI agents in a local zero-trust MCP bridge. Instead of blocking the LLM completely, you force the traffic through a local harness that redacts sensitive company data on the fly. We open-sourced our gateway for this exact workflow: https://github.com/Maha-Strategies/maha-mcp-bridge”
“This happens when the context is too large or ends up with some turns that have bad, triggering data in them. You are better off using a memory system and solid prompting to the model rely on memory with a smaller session history, than trying to keep large amounts of session data. I use qwen3.6 35B all day and it rarely ever loops now and produces pretty decent results for a small model.”
“I write incredibly restrictive harnesses as a hobby. Check out constrained decoding with vllm.”
Am I cooked?! submitted by /u/StationMedical6974 [link] [comments]
“Just because a benchmark is saturated and even has reduced utility certainly does not necessarily make it useless.”
I've had a terrible time with small models calling tools. I put Claude in charge of my configuration and had it write a test script to test various options to improve tool…
“Thanks for the comment! Agreed. There is rapid progress being made here but right now even for the basic use cases we are offering solid performance starts around 31B parameters which you aren't running at all on a normal laptop at all. This is why we offer cloud based models alongside of local. Hosting your data and interface on your laptop or host of your choice and using it with cloud based models is in my opinion still a lot better than defaulting to ChatGPT where you have no ownership a”
I am building my own PHP platform completely centered around AI development, and I have an Agent integrated into it. I wanted to have a submodel to help the main model (GLM…
I as someone with writing Zig a bunch, can safely say if it does it hasn't even worked for me.I am talking from experience from a pre-ai human mitts writing code perspective…
“Running agents locally, you hit a failure that isn't in any benchmark: the model says "done, wrote the file / sent the request / updated the row", and the tool never actually fired. No exception, no bad JSON, the trace looks clean. bigger models make it worse, not better, they narrate more convincingly. The reason it's hard to catch is that the model is not a reliable witness to its own actions. ask it "Are you sure you called the tool?" and it says yes again. You'”
“From my very limited understanding seems that it's something that is part of the model (kind of fixed size array) but if it can be reduced why it can't be increased? For Qwen 3.6 27B i read: because Qwen3.6 leverages extended context for complex tasks, we advise maintaining a context length of at least 128K tokens to preserve thinking capabilities. If i run a simple task that uses about 1k tokens like "write a python script to check if a number is odd or even" setting a context”
“Outside of Embassy in embedded, tokio is the only realistic choice though, because it is likely that any third party async crate has a dependency on it already. Yes, smol, monio, glommio etc exist, but they are marginalised (and as far as I can tell they don't really help that much with mixed IO / compute workloads).In fact, async/await in Rust falls apart with a mixed IO / compute workload since scheduling is cooperative. As soon as you want preemption (most of the time for what I do), it is no”
“I am currently doing research on world models, specially in tje field of robot learning, and, as probably most of you alredy know, JEPA-like models are mentioned over and over. I read the main recent papers from lecun as well as other research groups, and I personally think the whole approach is very promising and can really go somewhere. But after listening a bunch of the recent Y Lecun conferences his ideas looks even too cool compared to "literally everything else" (as he's diss”
“I've been working on an agentic runtime framework. The earlier benchmark eval is promising. But I can see the limits of the test design and the fragility of the runtime itself. I would like to ask for your reviews of the framework and the eval process itself. https://github.com/Tura-AI/tura https://turaai.net/docs#benchmark-current-test-set-record https://turaai.net/blog#why-i-am-building-tura submitted by /u/yohji1984 [link] [comments]”
“yeah this is the actual bottleneck imo - model itself is smart enough now, it's re-explaining the whole project every session that kills it. feels like it should just remember context across sessions instead of starting cold every time”
“Smaller/quantized Models definitely don't always get it right the first time. They can usually figure it out themselves given proper tests, though. TDD is definitely worth it with LLMs, but you will at least have to review the tests, if not write them yourself. Even the big ones make dumb mistakes occasionally, so skipping the tests isn't feasible unless you're writing a prototype.”
I mean Ornith is apparently better at coding than the normal 35B
“I have trouble with 35B for anything nontrivial regardless of cache or quant (up to Q8)”
“It's frontier models closing the gap. It used to be worthwhile to have Legora, Harvey, or other models that were an interface with frontier models, especially when frontier models had a hard time digesting a few pages of PDFs and hallucinated heavily when parsing large amounts of information. That's no longer the case, so having a third party between the user and the frontier model is no longer useful.”
“I am looking to take advantage of the different machines I have around the house rather than investing in more hardware at the moment (OUCH). I have ollama running in docker alongside OpenWeb UI on a Ubuntu machine with a P40. I can run QWEN 3.6 27B fully on GPU and 35B with a bit of spill over into CPU/RAM, but it performs well enough for my use cases. I am looking to try to take advantage of a few other machines in the house with GPUs by having QWEN 3.6 orchestrate smaller models running on th”
“Legora is going to be a bit behind, in exchange for more integrations. Frontier labs will always have the edge in harness/model rollouts.”
“model thinks forever, anyone know of a way to constrain the thinking? reasoning-budget didn't work for me when I tried it.”
“generation is the computation-heavy part, checking correctness is not and hence: speed. It's actually not quite that. Generation is RAM bandwidth intensive, and you have to do one token at a time. For each token, you need to load all of billions of parameters from memory into the GPU's working state, and do a single computation with each of them. Because the generation of each token depends on the last one, you can't do anything in parallel; each of the weights that you load gets use”
“It's not perfect, but it does help scan through settings and combinations to find best fit parameters for local models. (Paywall Removed on this link) Uses robust design of experiments techniques to provide efficient parameter sweeps. It does take a long time (I ran mine overnight), but it's pretty good at finding settings I hadn't anticipated. Step 1: Morris Elementary Effects - finds which parameters, for the given model on the host hardware, have independent and interacting effect”
“I hope this hasn't been discussed to death ... I am using local models for quite a while with pi and it's been really nice. I have seen unsloth offering fine tuning and training with custom datasets. Is there something one can use without unsloth? I am basically running llama.cpp server with a handful of models and would like to try tuning a model for my specific work areas. Any pointers would be appreciated. submitted by /u/DanielSReichenbach [link] [comments]”
“👋 Training my first RL model last year was super fun, now I've RL-trained a model that RL-trains other models... wild times! The agent gets a task, writes the full training job (environment, reward, dataset, hyperparameters), and submits it to real GPUs. When the model it trained scores higher on a hidden eval, the agent gets rewarded. An RL loop with RL loops inside it! 🤯 What I did: Built a harness where the trainer agent ( Qwen3.6-35B-A3B ) writes a complete prime-rl training job: a ver”
“Does anyone know of any evidence/forum/paper analyzing benchmark result differences between cloud inference platforms (togetherai) and running models locally with vLLM under greedy decoding? submitted by /u/No_Cardiologist7609 [link] [comments]”
“Why would you choose to NOT have reasoning? Wouldn’t it perform better with reasoning enabled?”
“Oh it is working but immediately going to start hammering your local setup with requests, hence the guardrails and patcj”
“The bug was real in that early v0.2.93 release and got shut down server-side the next day. Current versions don't do it. Local runs were always the safer default. [what is this?]( https://redd.it/1lzgxii )”
“I’m seeing the same pattern: GPT-5.6 Luna is outperforming Terra on a 12-task legal-AI benchmark. That said, whether either model is “great for legal AI” probably depends heavily on the specific task, benchmark, and harness. https://preview.redd.it/oeiwjsy272dh1.png?width=1257&format=png&auto=webp&s=351f5b8bb90c53bb75214e45730258e97197a534 At comparable efforts, Luna scored 86.86% at High and 87.40% at XHigh, versus 83.47% for Terra at both High and XHigh. Terra Max fell to 82.38%. B”
“The first panel from the Local AI Summit at AIEWF 00:00 Introduction and Panelist Backgrounds in Local AI 05:05 The Inflection Point and Evolution of Open Models 11:35 Specialized Models, Multi-Model Routing, and Data Sovereignty 19:31 Hardware Optimization and the Nvidia Swarming Story 27:05 User Experience Challenges and Abstracting Local AI Complexity 35:00 Distillation Pipelines, Audience Q&A, and Open Source Advocacy submitted by /u/pmttyji [link] [comments]”
“It's clearly noted in their documentation that there is a fair use policy on high-reasoning models.”
“From my experience running many fine-tuned models locally, the base model almost always gives better results. Fine-tunes often feel like a step backward — slower, less coherent, or just different in ways that aren't improvements. The only exception I've seen is coding-specific fine-tunes, and even then the gap is small.”
“For teams running self-hosted models behind agents: how do you decide a model is ready to ship, and what happens when you bump the model version or quant? The thing I kept hitting is that standard benchmarks measure single-shot tool formatting, but production agents fail on sequences a malformed tool call at step 7, a “done” that wasn’t done, a loop. And the same model behaves differently across quant and serving config, so “passed in testing” and “works in prod” aren’t the same thing. So I buil”
“Let's start a discussion about what can be done to make local models more reliable. I've been using Qwen3.6-27B a lot lately, and have noticed the same thing that many others talk about here, which is the tool-call failures and looping that really gets in the way of being able to use what is otherwise a pretty intelligent mode as a fully local stand-in for some of the frontier models. I found that I have to watch it like a hawk constantly, otherwise it'll start to loop after a while,”
“Good question. I don't think they're trying to solve exactly the same problem. OpenClaw is primarily focused on computer use. EverFern also supports full computer use, but it started from the idea of building a more complete desktop AI workspace that combines computer control with browser automation, document handling, coding workflows, long-term memory, and extensibility in one application. Some of the things I'm experimenting with that differ are: Full computer use : the agent can”
“What do you mean by "working feature tree" in a STEP? A STEP import does not have parametric features. Do you mean list of solid bodies? STEP imports also do not have mates, so did you have to mate all the bodies to check movement?”
I like their intelligence comparisons. Are my main reference nowdays.
“Look into artificialanalysis.ai agentic benchmark. I think it's pretty reliable.”
“Know your tasks Prepare a small benchmark on your own tasks for LLM Create a simple notation system At each release benchmark the new model and give it a notation Don't forget pricing is an important factor”
“Everyone has a different use case; so you would need a bespoke benchmark test. I guess you just need to try yourself.”
“I know about DEEPSWE but it lacks many models :( submitted by /u/9r4n4y [link] [comments]”
“This is exactly why I'm excited about Agent A1 from Shanghai AI Lab. They released a 35B MoE agent model under Apache-2.0 that runs locally on consumer hardware - getting around 95 tok/s on a Mac Studio M4 Max with full offline capability. Speedups like MTP are what make these larger agent models actually usable day-to-day. The 262k context window here would be perfect for long coding sessions without cloud dependency.”
“It's just a command line flag or env var in llama.cpp, and performance is leaps and bounds better than ollama. API endpoint, everything you mentioned. Llama.cpp is more efficient on resources, and just better overall than ollama. You should compile it for your environment and see for yourself. You'll wish you had done it sooner.”
“Claude Fable 5 can already independently do majority of the tasks in peer reviewed developer benchmarks. Arguably it's better than most developers, what do you think the developers will do?”
“Agreed. And there isn't any advantage to using the dgx spark or strix halo. Price/performance isn't any better. One could do something like a threadripper multi GPU setup.. but we are all chasing numbers with large dollar signs.”
“You didn't say "That is why people are using 5090s", you've said "GPUs" and the reality is that most of the GPUs used by people are not much better or are even worse. The 4090 or 3090Ti with it's 1TB are the next best thing, then you have the 3090 or 7900XTX with 936 and 970 and that's it for the "around 1TB or more and 24GB or more cards". The rest is either less VRAM or less VRAM and less bandwidth. Actually a lot of the products congregated at the”
“My goal is to improve as a developer, thus I needed to know if local llms can answer technical questions accurately The conclusion is that without rag they don't do too well, but with rag they are very good. Thinking didn't really help, and took so long I only got the scores for e2b and e4b, the rest are still running, it was like only +1% point for thinking. This is what I did: - Downloaded the markdown docs from the github repos for the listed projects (Node, Langchain.js, typescript,”
“I'm not gonna say he's 100% wrong, but I will say it with 99% confidence. That 1% is in case there's a bug in llama.cpp's MTP code. What he said is vibe nonsense that goes against the concept of what MTP is. MTP isn't an alternative generation engine, it just means doing some compute in parallel on a small model that usually results in a speed boost. By its very design it gives 100% identical output compared to not using MTP. If the main model disagrees with what the fast MTP”
“Yeah, you have named the exact thing. It is a corroboration gap, not a capability one. The model is not judging whether Muscula is good, it is surfacing whatever got written down in a form it can read. So "best product" and "best documented for this specific job" are two different competitions, and most founders are unknowingly playing the first one while the model only scores the second. On whether adoption fixes it, here is the part that surprised me. Adoption only helps to”