LLM Memory and Session Context Break Between Every Call
AI developers are struggling with the fragmented and rapidly evolving landscape of LLM tooling. They face challenges in managing datasets, memory, orchestration, and persistent knowledge across sessions, leading to wasted time, resources, and a sense of disillusionment with trendy but ultimately impractical tools.
SOURCES (60)
“Hey all — sharing my project infiniteMem, a deterministic episodic memory for agents that runs fully local (CPU-only, zero external calls). Verified numbers (frozen mini/mini protocol, same prompts as Mem0): - **LoCoMo QA: 70.49** (Mem0 66.9 / Zep 75.1 / MemMachine 87.5 with rerank) -…”
WOW! Thanks, wise man. This helped me a lot to clarify my ideas I'm in a hurry with other tasks this week, but by this weekend i'll test the ideas and see…
“One of the projects I'm working in is taking the core (engine) of roo without any of the vs code related stuff, rewriting it in rust, and putting it behind an API. Qwen 3.5/3.6 27B Q8 were great and wrote me ~200 pages of structured documentation about every component in roo. Kudos to the cline team for a clean base architecture. Read most of it and I now know how the thing works. Plan is to have it run headless and expose an API, I can connect to from anything (like a phone app) to be able”
“Yes, using an llm to avoid spending the odd 20 minutes every few months maintaining a dataview setup is, as previously mentioned, not a serious application”
“3.8 can run pretty fast with a smaller context length, but it absolutely mulches tokens. You really feel the token limits on that model.”
“There's, quietly, a llama.cpp WebGPU backend that works *great*.Some hacking required, it's unsupported, a side project for one of the lead maintainers and someone in school.Note of caution, llama.cpp isn't what it was, the grunt-level maintainers are left to their own devices. There's one key subsystem where things break regularly and the engineering is poor, and the "lead maintainer" is aggro and isn't really involved after delivering their big refactor that was DOA, other than telling people”
“Following u/ Ok_Negotiation_2587 's advice, I’m making a post to share the workflow I use. Hope this will be usefull! After dealing with context degradation on long coding projects, I ended up finding a workflow that has worked extremely well for me. The basic idea is: one permanent GPT control chat + one temporary Codex chat per work batch + regularly replacing the project state near the beginning of the GPT chat. 1. Start with a detailed master plan My main GPT chat starts with a detailed”
“the missing layer is decision context: why a schema changed, which customer constraint drove it, what was tried and rejected, and how to reproduce the current environment. code alone preserves implementation, not intent. i’d want each meaningful change tied to a short decision note plus the relevant test data and deploy assumptions. otherwise AI just recreates the same dead ends faster.”
“Disclaimer first: I work on the ML team at Wald.ai The gap you'll probably hit with Purview is the one u/SometimesImMean described. Regex and pattern matching gets rough on unformatted values, and it's all or nothing. Card number shows up, whole prompt gets killed, dev quietly goes back to their personal sub. Which is the shadow IT problem you're trying to avoid in the first place. That middle ground is what we built Wald for. A small classifier runs on the endpoint, reads the prompt”
“I think it's worth separating retrieval from generation before buying more hardware. Finding relevant notes can be relatively lightweight: embeddings + a local vector index don't require a huge LLM. The larger model only becomes important when you want it to synthesize, reason over, or rewrite the retrieved material. I'd first test whether local embeddings + retrieval solve most of the problem. Then you can decide how much model you actually need for the generation step instead of si”
“Before Submitting [x] I searched open and closed issues and discussions for an existing report. [x] I checked whether this is already fixed on the dev branch or latest source. [x] I understand that maintainers want a well written issue before any code pull request. [x] I am using the latest available version of Open WebUI for my install method. [x] This is not a security vulnerability. Installation Method Git Clone Open WebUI Version dev at 148391ded04caf1ffa78be8b450d121e9623c06c (also present”
“Curious what's missing from affine and appflowy? Those were the exact two I would have recommended.”
“Agent node destroys item lineage → best practice for enriching original items with LLM output?”
“Disclosure: I am the maintainer of OpenRoutiQ. It is free, MIT-licensed, and currently has no paid tier. The underlying problem is that the best model is rarely universal. Requests differ in complexity, quality requirements, latency tolerance, cost constraints, capabilities, and risk. Hardcoding one model for every request leaves substantial quality or efficiency on the table. OpenRoutiQ evaluates the complete request and selects a model, provider, deployment, and reasoning level from a user-sup”
“Respectfully, none of that matters in the slightest. At all. Assume that any and all data that you send over to a cloud-based LLM will be saved in perpetuity. It doesn't matter if you have "memories" enabled or disabled. All of your chat logs can and will be saved regardless of you telling the LLM that you "opt out". That's just how it goes when you're using someone else's computer. They can tell you that they delete everything but there's absolutely no wa”
“You definitely could do it, we do something like this at Cascadia (pipeline-parallel sharded inference), and different machines can send requests to the sharded model (or a replicated model). We only support Intel hardware right now, but I'm sure you could do something similar with llama.cpp or vLLM.”
“With the help of Chatty (that’s what I call ChatGPT), I built a simple system for managing long-term projects without depending too much on ChatGPT’s built-in memory. The problem was pretty simple: ChatGPT was good at remembering things like how I prefer to work, but project information eventually became outdated. “I like to discuss the architecture before writing code” is useful long-term memory. “Version 1.2 has three bugs and this is the next task” is not. That’s project state, and project st”
“mostly the stuff a model actually trips on, not generic schema linting. it flags operations with no operationId (tool naming falls back to method+path, which agents handle worse), missing summaries — that's literally the text the model reads to pick a tool — server urls with unfilled variables, relative or missing server urls, deprecated operations that would get exposed as tools, and specs that declare auth you haven't configured, where it prints the exact serve flags to run. there'”
“I run a local AI agent setup on an 8GB M2 MacBook Air. Every megabyte of RAM matters. When implementing RAG (Retrieval-Augmented Generation) for my agents, every tutorial told me to spin up a Docker container for ChromaDB, Qdrant, or Postgres with pgvector. On an 8GB Mac, Docker Desktop alone takes 2GB+ RAM, pushing the system into heavy SSD swap. The Architecture: 1. Stored knowledge text and JSON-encoded float32 embeddings in a local SQLite file ( friday_memory.db ) via Go. 2. Built native in-”
“Unlike most chain-of-thought LLMs that do one pass of thinking + tool calls, claude will think mid-response. With a bit of trickery (aka using an xml antml tag to enable thinking manually (without the reasoning traces being hidden from us). It will think, write a paragraph, then think again out of nowhere, write another few paragraphs. think again. I doubt this is default behaviour but whatever this is might actually improve reasoning scores iirc. Also, the thinking tokens remain in the context,”
“Hey folks! Basically the title, but with a little more detail: I’m getting a bit tired of the limits of Claude, ChatGPT, and similar services — especially when I’m having it generate test tasks or doing an extended coding session where I use Claude as a sparring partner. So I’ve been thinking about getting myself a small homelab/server and hosting a local LLM, for example something like this Minisforum AI X1 Pro , and running something like Qwen3 14B in Q4 or Q5. My understanding is that this wo”
“Why is the loading showing while the translation is already there ? I would fix that since speed is key in user experience”
“I also want to point out that fine-tuning experiments showed that it was much better at learning new facts without catastrophic forgetting than regular fine-tuning is.”
“Summary Automatically discovered new candidate tool pathway from developer catalog \ awesome python\ . Description: High performance Python streaming data processing framework & reactive event driven RAG indexing engine for real time news stream filtering and anomaly event pipelines. Modeling Evaluation & Value Modeling Category: \ LLM & RAG Event Extraction\ Estimated Impact Score: \ 8.4/10.0\ Architecture Target: \ src/intraday event monitor.py\ Rationale: Provides Python reactive streaming pi”
“Yeah, the problem is I want a historical snapshot of the research record day by day across multiple components so I can check how my AI is interpreting research data daily over a week to a month to a quarter.”
“Swapping away from a model that processed a large context throws away valuable work that took significant time. In my case that is 2400s to rebuild 700k context from scratch. Persisting it on disk and retrieving it back should take 20s. Saved slot takes significant amount of space on the disk, but if you have space, you can save a lot of time. I tested the following on llama server build 10573 with smaller model: the restored cache was matched and 81% reused. Simple solution llama swap can expos”
“thank you for the info, Im trying to bulk update tags into properties with help of llm”
“I still kind of don't get what it does better than just having the LLM write python scripts that you describe in English, especially if it's already intended to write new functions on the spot. Why have this whole convoluted intermediary layer?”
“haha yeah, at the bottom it kinda is, a step is literally a python function with typed in/out. not claiming i reinvented functions lol. the difference is the layer around them: types get enforced when you connect two (not at runtime, so you can't wire incompatible ones together), the model composes them from a plain english description and can only pick known ones or write a new one instead of freestyling a whole script, and you get the plan preview plus reuse for free.”
“There is no index, nothing to go stale. The graph builds from disk in about 100ms for a mid size repo. caveats: ctx map -o CODEBASE_MAP.md writes a file, and that file goes stale like ctags did. Skip it and query live, and I should change that recommendation in the README. Also it reads disk, so unsaved editor buffers are invisible.”
“A lot of data science work is shifting toward building and evaluating systems on top of LLMs now, not just training models the traditional way, so figured this might be useful even if RAG specifically isn't your main focus yet. There's a hands-on build lab on August 29 that walks through building a complete retrieval-augmented generation system from scratch, using small open models rather than a paid API, so there's no ongoing token cost involved. What it covers: Hybrid retrieval, co”
“Thanks for your feedback, we'll note this for the next versions. Also, we definitely agree in the importance of pattern flow and consistency. We see this also play out differently depending on the language which is why we added 14 more of the bigger languages in the next versions. Regarding nested entities or ambiguous PII, you are correct that it is a clear problem that even the smaller models v1.1 and 1.2 encounter; seen even in early benchmark results on v1.3. However in v1.3 we started l”
“I know I’m probably insanely late to the game but I just tried this for the first time. I’m in shock. A roller coaster of emotion. On one hand, an insane amount of time saved. Projects that would take me days and sometimes weeks are done in minutes! On the other hand, us admins will soon be obsolete. If you haven’t tried this yet, please do. It’s an eye opening experience, I’m still in shock. Tell it to make you fields, flows, objects, permission sets, reports, etc. Heck, you can even tell it to”
“this is actually a pretty interesting approach, glad to see someone finally tackling the masking problem instead of just slapping [REDACTED] on everything and calling it a day. the semantic replacement angle is what always bugged me about traditional PII tools, you lose all that context and suddenly the model's output reads like a government form i juggle as a hobby so i'm always thinking about pattern flow and consistency, and it's the same deal with language models, if you break th”
“Since around 2026 08 22, reasoning models served via :cloud tag (e.g. minimax m3:cloud ) intermittently split JSON output between message.reasoning and message.content in the OpenAI compatible /v1/chat/completions endpoint. The content field alone is never valid JSON, making response format: {"type": "json object"} unusable. The think: false parameter is silently ignored. Environment Ollama version: 0.32.14 Model: minimax m3:cloud (also reproducible with other :cloud reasoning models) Endpoint:”
“Required prerequisites [x] I have searched the Issue Tracker that this hasn't already been reported. (comment there if it has.) Motivation Motivation SunMMIO currently conflates two different concepts: A semantic TileView, determined by the tensor layout and the logical T.Tiles computation. A physical tile carrier, constrained by the vector core’s 4096 bit register capacity. Using the 4096 bit limit directly during semantic TileView planning causes several problems: ZZ layouts cannot use their n”
“Our deterministic verification engine passed 66/66 benchmark cases on canonical structured inputs. In live model evaluation, the end-to-end pipeline currently passed 19/66 cases. We are restructuring the benchmark to isolate failures by their first invalid state and to separately measure deterministic verifier correctness, production contract integrity, and live model generation reliability. The next benchmark version will provide stage-level attribution across transport, parsing, schema validat”
Please preserve consistency and correctness across ChatGPT model updates
“Sure , I’m actually working on a ~50M-class model next, targeting under 12–15 MB deployed. If you want a challenge, let’s do model size, CPU tok/s, retrieval accuracy , basic benchmarks like PIQA/HellaSwag/ARCand retrieval speed. Same hardware and public reproducible tests. Beat me on any of them and I’ll happily give you credit. Deal?”
“Summary As I said, I tried adding opencode and openrouter keys for models which have an effort slider, but when I run /effort in cmd it says no thinking levels are available. Expected Behavior Should show the models thinking levels if it has them. Actual Behavior Says it doesn't have any thinking levels. Steps to reproduce the issue 1. Connect a third party API like openrouter or opencode 2. Select a model from them that has thinking levels 3. Run /effort Command Code Version 1.32.1 Operating Sy”
“Superset is a strong baseline. I wanted the same workflow idea with a native macOS client, a faster terminal experience, and Web/mobile access. Warren puts Project → Workspace → Session → Agent into one flow. Use Codex or Claude Code from macOS, Web/PWA, mobile Web, or CLI. Switch between raw Terminal and structured Agent view. Technical detail: Sessions survive client quits and reconnects. The default runtime uses ghostline + libghostty-vt; macOS uses libghostty; Web uses xterm.js. Early Apache”
“Right now standard CoT is a one-way street. You generate forward, dump a pile of scratchpad tokens into the KV cache, and pray the model doesn’t hallucinate halfway through. On phones/laptops that creates two ugly problems: Errors compound. If each step is only (1-ε) reliable, an N-step chain dies as ~e^{-Nε}. There’s no cheap way to check intermediate steps without another full forward pass. KV cache just grows. Intermediate tokens sit there forever. Blind pruning breaks context, so memory is O”
“I don't know how most of the admins use Yamtrack, for me it's a single user instance. The only thing I could find in the docs is WEB CONCURRENCY but are there more ways to trim down the RAM usage for the application? Scaling down threads or processes? Or could some up with more configurations for this? This is my current consumption, is not very high, but for a single user seems a bit too much. I have set in Docker compose hard limits for both containers yamtrack and redis; Current RAM usage: |”
“That ISBN workaround is smart, dealing with 1:N matching is a pain no matter what you're building. The not overwriting existing data part is a nice touch too, most automations just bulldoze whatever you already filled in”
“I made it in 18 months of lunch breaks and evenings. It's not fast, llama.cpp is just wow and does that job. I wrote this one because I wanted to read the whole forward/backward pass in an afternoon and be able to stop anywhere and print a tensor and dig the thing. Most from-scratch projects stop at a toy model. llama2.c runs a small Llama2, llm.c does GPT2 training. TRiP loads real checkpoints across four architectures, PaliGemma included, so the multimodal path (vision encoder, projection,”
“you can also use a sovereign cloud based local llm subscription that gives you access to top models like glm5 but also has the necessary data privacy requriements. We use that when working with some clients that need local models”
“I was doing some experimentation on using Unsloth to do Continued Pretraining of qwen 3 4B. The goal was to teach the llm how to reason about a new domain, in this case a fictional city. Sharing more details in the write-up: https://www.teachmecoolstuff.com/viewarticle/teaching-a-local-llm-a-new-domain submitted by /u/funJS [link] [comments]”
“trying to port a couple months of old aider sessions into memmy's local memory, but parsing aider transcripts is turning out to be a massive headache. the frustrating part is that memmy’s core already works ridiculously well for structured tools like cursor and claude code. when I query past sessions, it pulls up exact architectural decisions without hallucinating, saving tons of time re-feeding context. being able to start a new CLI agent session and have it instantly know why a module was”
“Tons of options for this kind of thing. I'm currently connecting Claude to Notion to build a shared memory store for multiple agentic uses.”