LLM Memory and Session Context Break Between Every Call
192 posts across 31 communities in the last 30 days; 31% from r/LocalLLaMA.
“I’m building a legal search and analytics engine over ~4 million Slovak court decisions. Dense embeddings + BM25 are already running in production for basic retrieval, but I’m hitting a fundamental representation wall when trying to achieve near-100% precision on complex queries…”
“I've been building open models that treat a decision as a closed-set scoring problem rather than text generation. The input is structured context plus a typed question with a fixed set of options. The output is a probability for each option, taken from the candidate-label…”
“If you're moving a RAG prototype to production, this is the checklist I wish I'd had. - Citations on every answer - Refusing when the context doesn't support one - Access control that can't be bypassed - Ingestion that never blocks a user Enjoy !! https://youtu.be/-GK1e4KZqhw”
AI developers are struggling with the fragmented and rapidly evolving landscape of LLM tooling. They face challenges in managing datasets, memory, orchestration, and persistent knowledge across sessions, leading to wasted time, resources, and a sense of disillusionment with trendy but ultimately impractical tools.
SOURCES (60)
“I have been thinking of something similar, and figure the answer for my problems is to have async LLM stuffor use Jev styled models where possible [re: Self hosted LLM Goals / possible?]”
“What do you think about a public repository where contributors can dump their sessions manually. We can also build browser extensions and other collectors to automate this as well. Why this is useful: Right now frontier models do not allow distilation and open llms have very limited budget, if we really want to see open llms get frontier capabliities, we need to contribute. If we all contribute our sessions we can maybe make a difference. - There need to be some sort of privacy control or data a”
“Fair, and thanks for the specifics. You're right that the GGUF path is the least finished one today. It has no drafter (speculative decoding runs in our SGLang build and Agens Engine; a GGUF drafter is on the list) and no flash-attention path for our sparse attention layers yet, so ~15 tok/s there is about what we'd expect. v1 also ships without an MTP head; speculative decoding is done with the DFlash2 drafter. Multi-GPU tensor split wasn't in the fork's test matrix (single-GPU”
“Choose github/plurnk. Plurnk is the only harness with journaled context that the model curates, resulting in your session getting smarter over time rather than suffering from "context rot." It's almost as lean as Pi while giving the model a proper tooling and training universe like Hermes and Claw. [re: I am overwhelmed by Harnesses]”
“Agreed, that makes sense. It seems to not be too prohibitive in most modern hardware setups, but it is definitely not nothing. [re: [Showoff Saturday] Rebuilding my 10-year-old clinical NLP project with browser-local AI]”
“What problem does this solve? Currently codebase memory integration requires user global files ( /AGENTS.md, global hooks, MCP config). Projects need repo local versions that override per project. This is impossible today, the repo has no project level AGENTS.md, hooks, or MCP config, and no mechanism exists to redirect integration files to a project local location. Proposed solution Support project level integration: detect repo root .pi/agent/AGENTS.md , .mcp/ , and hooks/ with priority over u”
“I asked three models (Sonnet 5.5 Medium, GLM 5.3 flash, and GPT 6 Astra Max) a simple dictionary definition question: "is saying "The width of the Aptera is a travesty" a correct use of the word travesty?" For context, the Aptera is a pre-production three wheeled vehicle that is quite wide. See: https://aptera.us Only the local model answered the question correctly. I strongly suspect that this is because of the slop-injection system that the frontier labs call their system p”
“I’m building a legal search and analytics engine over ~4 million Slovak court decisions. Dense embeddings + BM25 are already running in production for basic retrieval, but I’m hitting a fundamental representation wall when trying to achieve near-100% precision on complex queries and run quantitative jurimetric statistics (e.g., "In what % of custody cases involving alcohol did the father get custody, grouped by year or judge?" ). The Problem: Why Flat JSON / Enums and Pure Embeddings B”
“Architectural Concept: Optimizing Long LLM Context Windows with Hybrid AI-Pruning & Unified File-Cache Storage”
“No extension and no tool is the right instinct. Every abstraction layer you remove is a place where the harness can silently lose your state. The context saving is the obvious win, but the real one is steer: most agent frameworks force you into a request-response shape, and the moment your task needs a mid-run correction you have to kill and restart. Persistent running with steering is what makes a subagent feel like a worker instead of a batch job. Keep it small. The frameworks will add this in”
“A few months ago, I shared AI Badger here, a lightweight CLI and VS Code extension for preparing focused repository context for ChatGPT, Claude, and other AI chats. I just released v0.8.0 with a new feature: Codebase Map. Run badger map in any project directory to generate a standalone, interactive HTML report. You can explore folders, inspect language breakdowns, and see where code is concentrated. The short video shows me navigating the AI Badger repository using the generated map. Everything”
“Going to try these out to see if they have the same looping behaviors that other Bonsai models have had. Still requires their fork of llama.cpp (branch prism) so dont expect it to work with vanilla llama.cpp. Will see if there is some way to support this for AnythingLLM Mobile via LLama.rn fork since if anything this will hopefully beat out regular Qwen 3.5 1.7B or 0.8B. Can test on iOS today with LocallyAI on latest version. Feel like the way these models are made (QAT?) kind of lend to a high”
Transform ChatGPT's File Library into an AI-Managed Knowledge System
Persistent Character and Memory in ChatGPT: A Working RAG Architecture
“Hi MLOps community, We are working on optimizing an offline local inference engine running on client machines without discrete GPUs. To fit within 16 GB RAM client constraints, we are evaluating GGUF quantization tiers (Q4_K_M vs Q5_K_M) paired with dynamic KV-cache caps and a single resident daemon model architecture. When benchmarking local CPU inference engines under strict RAM constraints, what metrics (peak RSS, first-token latency, memory bandwidth pressure) do you prioritize to establish”
“Great writeup. Glad to see someone sharing their workflow. Something I can actually understand and compare against. Can you explain your "Two engines per box, TP=2 each, 8 sequences per engine, 256K max context, about 488K tokens of KV cache per engine"? Ive got 2x B70's and am about to pull my 2x R9700's into the same box, so could somewhat replicate your setup. Is engine just a model running TP=2? And your sequences are what I call streams? Right now im running a Q4 version o”
“What it is Coding agents relearn your project every session: which files matter, how the tests run, what failed last time. MuscleMemory watches agent sessions (Claude Code, Codex, Cursor, or any MCP client) and remembers how a task was done, but only when a test passed after the last code change. Why I built it I kept watching agents rediscover the same project over and over. Most memory tools store conversations or facts. I wanted memory that only keeps what was proven, so a skill is trusted on”
“Hello everyone, I’ve recently been reading papers that use LLMs to understand code logic and infer security-related semantics from source code and comments. I’m curious how people working on white-box security analysis in industry actually use LLMs. Can LLMs help overcome limitations of traditional static analysis, such as complex inter-procedural data flows? How reliable is LLM-based code understanding? Could generating multiple possible interpretations and verifying them work? If so, how would”
“I don't get the downvotes. I like LM Studio, but they are trying to pivot to harnesses instead of inference serving to make it easier to monetize. That said, I switched to Unsloth Desktop/Studio because they update llama.cpp faster, do inference faster, and have a nice fine tuning features. [re: Decision 3.0: open, multimodal decision models from 0.6B to 27B. Ranks #1 on both Jev Decision Index boards]”
“I'm working on an on-premises AI project for a business in Europe and would love to speak with engineers who have deployed similar systems in production. The use case involves: Local LLMs for German and Polish business communication WhatsApp and email automation Internal document search and a private knowledge base Finance-related workflows, with calculations handled by deterministic application logic Data storage and model inference on the client's own infrastructure We're evaluatin”
“I'm the developer of RetroGemini. I work at the State of Geneva, where we needed to keep potentially sensitive retrospective feedback on our own infrastructure rather than put it in external cloud tools. I built an alternative to cloud tools with AI assistance. RetroGemini covers collaborative retrospective boards, grouping and voting, action follow-up, and team health checks. The interface is available in English and French. For self-hosting: Docker with persistent SQLite storage for a simp”
“Lets say I'm writing my own journal/diary and want an LLM to parse everything and index names and keywords so I can ask later things about my life, which local small LLM would be a good fit that I can trust won't go off the rails/hallucinate? are there any models specific for this type of thing, like for a "Google Notebook"? submitted by /u/justletmesignupalre [link] [comments]”
“Ive been burned by enough "no code" platforms that promise the world and then choke on anything beyond a simple SMA crossover. Rule based editor sounds fine on paper but hows it handle multi timeframe conditions or partial exits, that stuff always gets messy fast Will kick the tires this weekend and see if it can actually handle the kind of dumb edge cases I throw at it without needing to crack open a script. The comparison table for different param runs is a smart touch if it works sm”
“project: https://2796gaurav.github.io/shad0w/ If you use a local model for classification, you know the wait: seconds per message, for questions it has answered hundreds of times. shad0w sits in front of any OpenAI-compatible server (Ollama works out of the box: llm="ollama/llama3.1" or `shad0w proxy --upstream http://localhost:11434/v1`), logs the model's answers, and after ~1,000 of them compiles a small table that answers the repeats on CPU in microseconds. It only answers where”
“Imagine your team changes its embedding model, chunking strategy, or reranker. Your overall retrieval metrics improve. But a few important customer queries silently lose the documents they need. Would your existing evaluation suite catch that before release? I've been working on independent RAG retrieval audits focused on exactly this kind of evaluation risk. A real example: RouteMind I independently audited an open-source RAG benchmark and identified 15 scoring discrepancies across”
“I've been building open models that treat a decision as a closed-set scoring problem rather than text generation. The input is structured context plus a typed question with a fixed set of options. The output is a probability for each option, taken from the candidate-label logits, so there's no generation step to parse and no sampling. v2.1 updates the 27B and the 9B. On the Decision Index 0.3 public suite (my own runs with the unchanged official scorer, submitted to the board, which adds”
“adapt and create workers and customs blocks [re: Is AI going to replace Notion creators?]”
“Thanks for publishing hyper evm sync — it has been very useful for understanding HyperEVM replay semantics (recorded precompile results, AbciState seeding, the per block receipt format). The repository currently ships no license file, which leaves it all rights reserved by default: nobody can legally vendor, fork, or build on the code even though it is public. Sibling repos in this org are already permissively licensed ( hyperliquid dex/node is Apache 2.0, hyperliquid dex/hyperliquid python sdk”
“Let's say I'm using some obscure open source software or legacy tool Ui. I will provide documentation or examples it should organize them and store them somewhere and refer them in future instead of hallucinating. How to do this skills, agents, artifacts etc? submitted by /u/HamsterUnfair6313 [link] [comments]”
“Context fragmentation across different chat sessions and model switches is definitely one of the biggest bottlenecks when building with LLMs right now. Having a lightweight local repo-level wiki or markdown notes to preserve architecture decisions is a super clean approach. Have you thought about adding automatic session summarization or token-budget tracking so the imported context stays lean? Really great concept and clean execution! [re: I made Dory to stop re-explaining my projects every tim”
“Use opencode go with mcp to notion [re: Smallest / cheapest LLM available in Notion right now?]”
“I used to have to jump between different sites copy pasting outputs from one into inputs of the next. So I built one place where you chain the steps and save the pipeline for future use in a right click context menu. 246 utilities so far, with branching, looping on lines or values, and conditions. It runs entirely in the browser (Web Worker), no input reaches the server (One exception when fetching a URL through a proxy). Pipelines can be shared as a URL fragment React + TypeScript + Vite on Clo”
“How do you reason about when to choose a Jev/"system 1"-style approach versus a "classic" LLM returning structured data? Purely cost vs performance? "Am I willing to give up 1%/10%/20%/etc. of recall/ precision to save $X? So the thing is all usecases requiring classification that I encountered have massive data. E.g. For customer support via phone call we received 200k calls last month and any kind LLM inference will literally burn through money to get usable results. D”
“I'm working on instruction diagnostics and evals as part of Reporails ( https://github.com/reporails/cli ) implementation so I was thinking giving back to the community some items that I ran trough a few thousands of experimentation iteration and giving measurable boost in compliance. 1. Do, then why, then don't. Naturally you'd led with the ban: "Never force push, rewrites history." Reordered: "Fix mistakes with a new commit. Teammates' history stays intact. Never”
“I am a data scientist who started working in the industry before the LLM revolution. Back then, Jupyter Notebooks were a perfect fit for the classical DS pipeline: EDA -> data prep -> fit -> eval -> tune -> save model artefact and notebook. Lately, I have been thinking a lot about how agentic development and LLMs are changing the way data scientists work. Especially in classical ML applications, where you still need to explore data, run experiments, check different hypotheses and”
“The plain-text record thing is smart, makes it easy to diff and grep later. Curious how you handle merge conflicts when two agents update the same project file around the same time [re: [macOS Beta] Agenfold — a shared project record for coding-agent handoffs]”
“The compose.yaml is hard to follow. It's 293 lines, but only about 150 are actual config. The rest are long comments about past bugs, which bury the setup itself: 4 services (secrets-init, database, php, worker) plus the optional ntfy. Can you simplify it like most docker-compose that we have in the wild ? [re: plMail: a self-hosted, Gmail like webmail client for all your Email accounts]”
“In the dawn of the LLM craze, when wild Llama2's roamed free and 4k tokens was considered a fairly long context for local models, I forked an interactive fiction and MUD library called Tale to experiment with LLM generated content. The idea being a text based adventure with totally dynamic content that never runs out of context. That was many generations ago in LLM space, and I haven't really done much with it for a long time either. But with the advent of agents, I started brooding how”
“Small vertical slices of value are the way. Pick a single document you see as authoritative and up to date, and tell it to work with you to ingest docs as a a "repo as memory". I could give you my structure, but it would just introduce noise and cruft. Work out how you will ingest documents. I did a bunch of upfront set up and ingested dozens of documents, but that isn't necessary. I also built a skill that sets up an ingest queue and another that runs each assertion by me so I can”
“I've been building an open-source Python project called EQ-Layer , and I'm looking for developers willing to test it and point out problems. The idea came from something I kept noticing with AI assistants: when a conversation goes wrong, changing the assistant's tone doesn't necessarily fix the actual misunderstanding. EQ-Layer takes a different approach. Before an LLM generates a response, a separate control layer analyzes user intent, emotional signals, conversational breakdown”
“Conditional memory architectures such as DeepSeek Engram use input n-grams to look up learned embeddings, expanding the capacity of large language models (LLMs) with limited additional computation. Beyond model scaling, this architecture has demonstrated the potential to decouple factual knowledge storage from general-purpose computation, offering a promising route to updating factual knowledge while keeping the Transformer backbone fixed. Realizing this potential is challenging because differen”
Keep the Why: Git-native project memory for coding agents
“Figure 1b from our paper Disclosure: I'm one of the authors (Microsoft). The paper, code, dataset are public and ThinkingBox is on Hugging Face OpenEnv as well. Raw evaluation trajectories are not released Links at the bottom. We wanted to know how much of a single agent success rate survives repetition, and whether "the agent finished the task" means the backend/database actually ended up in the correct state. Setup Thinkingbox-Bench includes 507 policy conditioned business workfl”
“Stop using ollama [re: Ollama or vLLM for a language model server at home: which to run for your case, who can reach it once it is up, and what each sends out by default]”
“https://sleepingrobots.com/dreams/stop-using-ollama/ [re: Ollama or vLLM for a language model server at home: which to run for your case, who can reach it once it is up, and what each sends out by default]”
“Expand the replies to this comment to learn how AI was used in this post/project. [re: Ollama or vLLM for a language model server at home: which to run for your case, who can reach it once it is up, and what each sends out by default]”
“The Mellum folks recommend the 2.0 Instruct for FIM, not this new 2.1 Thinking, so alas, you're not quite there. I'm in your tribe, BTW, code-first. [re: Mellum2.1 - a JetBrains Collection]”
“What it is: a web workspace where you pick an open model per question, your notes and uploads stay in the workspace when you change models, and every reply is labeled with the model that wrote it. You can also send one question to up to three models and read the answers side by side. What I'd love tested: - Change models mid-project and tell me if anything about your context gets lost. - The side-by-side compare. Does the labeling make sense at a glance? - Ask for a widget (a calculator, a c”
“If you're moving a RAG prototype to production, this is the checklist I wish I'd had. - Citations on every answer - Refusing when the context doesn't support one - Access control that can't be bypassed - Ingestion that never blocks a user Enjoy !! https://youtu.be/-GK1e4KZqhw submitted by /u/Winter_Mistake_3185 [link] [comments]”
“Hey everyone, Jovan from UkisAI (Swift Qwen) here! For those who don't know us, UkisAI is a small lab making tiny frontier LLMs, tools and datasets (+doing it open-source!). I'm one of the guys running it aka I train the models and post on Reddit. Our first open-source release is Swift, a series of reasoning-efficient LLMs. It is proof of how penalizing pathological overthinking patterns inside of various LLMs can bring their token usage down -58.3% and speed x1.95 without losing accurac”
“Hi Kumo team, I'm a student at the University of Augsburg, working at the Chair of Data Engineering. I'm experimenting with Relational Deep Learning, and we are particularly interested in tasks defined on databases with many tables that span large hop distances. What I've done so far I have reproduced the RelBench GraphSAGE approach and baseline results. My focus is the rel trial dataset, with tasks from RelBench v1 as well as v2/v3. What I'm trying to do now My next step is to reproduce/achieve”
“Opus5.5, Sol6.1, Fable, and Astra have all proven they can and the scene has exploded this past week. Part of that is from the tools and feedback loops maturing though. Are any open weight models (at all, so including K3, GLM5.3, Qwen3.8-Max, and Mimo-2.6) able to do this? Can the larger models this sub regularly runs (GLM 5.3-Flash, Qwen 3.8-Next-Flash, V4.1-deepseek Flash..) handle a simpler one (GBA and PSP having smaller roms and mature pipelines)? submitted by /u/EmPips [link] &”
“Submission checklist [x] This is a feature request, not a bug report or usage question. [x] I added a clear and descriptive title that summarizes the feature request. [x] I used the GitHub search to find a similar feature request and didn't find it. [x] I checked the LangChain documentation and API reference to see if this feature already exists. [x] This is not related to the langchain community package. Package (Required) [ ] langchain [ ] langchain openai [ ] langchain anthropic [ ] langchain”
“Cohere shipped North 2 on Monday, and the headline is cross-session memory. The agent keeps the conversation state instead of starting fresh. For support that is the part that usually breaks. The customer explains once, the bot closes the chat, and a human asks from the top. Every re-ask is a tax somebody pays in time or trust. For those running support agents here, what still gets lost in your handoff even when the memory is perfect? submitted by /u/arslannasir128 [link] [comm”