LLM Memory and Session Context Break Between Every Call
AI developers are struggling with the fragmented and rapidly evolving landscape of LLM tooling. They face challenges in managing datasets, memory, orchestration, and persistent knowledge across sessions, leading to wasted time, resources, and a sense of disillusionment with trendy but ultimately impractical tools.
SOURCES (60)
“Google drive is the best, especially paired with AI tools like claude. I dont treat PRDs as a single document but an evolution with time stamps. Meaning there will be a trail of PRDs and notes on what we changed and why. I combine meeting…”
“I've designed a homelab stack for running local AI (LLMs, vision OCR, voice transcription, image gen) and would appreciate feedback before I start deploying. The 4-layer architecture (strict separation): Layer 1 — Inference vLLM + llama.cpp (coexist, per-model), Whisper STT, vision VLM for OCR Layer 2 — Tools Hermes Agent orchestrator, LiteLLM routing, Playwright browser, SearXNG search, Crawl4AI scraping, pandoc + yt-dlp — 1 container/tool via Docker Compose Layer 3 — Storage TrueNAS bare m”
“Well, yeah, there's going to be context extension as well! This is just the release of the first functioning prototype. There is more to come! Larger context and instruction-tuning being just two parts of it.”
“Shared some content on data engineering based on my experiences. If you are also new to this learn from the article. submitted by /u/Ok_Donut1905 [link] [comments]”
“I've been looking at vector search for a self-hosted setup, and the RAM requirements are worse than I thought. Weaviate keeps its index in memory with no production-ready disk option right now. Their docs say around 6GB for 1M vectors at 768 dims. They do ship an embedded mode, but the docs mark it experimental, and the instance dies when the client process exits, which rules it out for anything running as a service. Their disk-based index landed in v1.36 as a preview; they say not to run it”
“It sounds like you completely missed the point I was making: that actually implementing such a highly dynamic TUI application normally requires a significant amount of state tracking and it is significantly easier to write such apps using a functional, descriptive UI framework instead of something like ncurses.Take it from someone who, in those 24 years, has written UIs in jquery, react and ncurses (and yeah I've tried out Ink and it's great)."all the downsides that come with that" yeah well, wh”
“Working with multiple Claude Code instances I found issues with requirements traceability, test coverage, and inter-instance communication. I was working with knowledge graphs so built a knowledge graph skill for Claude Code - every user story, use-case, feature, test-case is a node; plus i track defects and enhancements so each instance knows its backlog. It saves tokens by knowing where code is implemented, but perhaps more importantly it found systematic errors in requirements quality and tes”
“coding, office agents and rag. about 5 light agent users. 3 rag apps are plugged on to it and we use opencode. note this is all internal, nothing is user facing. too low concurrency (only 4 due to me not being able to squeeze more cuda stuff in there). we do reach the limit and sometimes it gets very slow. for single user it goes up to 300tps on small requests. this will not handle external users very well. if i remember later when im at the computer i can send config. all in all very happy with”
“The landing page with all 52 models selected https://gitbench.gitkraken.com/ GitBench is a benchmark suite that measures how well LLMs do git things like commit message generation, submodule usage, etc. Currently, we have 52 models tested across their provided effort levels as well as Text vs Structured JSON Output. (shoutout to OpenRouter for making this so easy) This started as a way to try out some of the spec-driven development tools like GSD. But, since I work for GitKraken, it seemed like”
“The dashboard is essentially a visual layer built on top of the framework. I use Markdown files for the knowledge itself: business rules, procedures, templates, project records and the links between them. This keeps the information readable by both humans and AI. Each note also contains YAML metadata with structured fields such as IDs, record types, status, dates, relationships and dependencies. The dashboard reads this metadata to generate views, filters, tables, timelines and connections betwe”
“Can you explain the dashboards? Trying to understand them and how they are built out since it doesn’t look like standard Obsidian.”
“readme says: Database Schema requests responses But i dont see the responses def in schema.sql? anyway thanks for the contribution is a cool tool”
“LLMs are building trash composes so often for me. Its borderline useless for NixOS since the documentation wasn't too detailed when I last tried it. I just wonder what it is with declarative systems and LLMs Tbf composes can be usually just pulled from the docs (if they aren't ass themselves, the ones from popular tools usually are good) And then you can work from there. So yes, I definitely check it against the docs before "vibing"”
“About two months ago, I shared an idea about building a node-based ML playground. Well, I had an interface like ComfyUI in mind while building the rough idea, but also something more controllable and usable by both, the seasoned ML guys and the beginners since the code of each node is editable. I did a node like structure for training and data preprocessing templates, while there are others too. Not sure how much it's usable yet though, I would like some opinions. Someone said that it looks”
“Keep it dead simple, "abort" the current run, add a system generated compaction prompt on top of the current conversation, capture result, start new context window with that captured result. You are a helpful AI assistant tasked with summarizing conversations for continuation. Summarize the conversation history concisely, preserving: 1. What was done and what is currently being worked on 2. All file modifications made (file paths and what changed) 3. All errors encountered and how they”
“Any suggestion for the algorithm for compaction? I'm also building harness, but I made a mistake of being "clever" with the context, and now it runs bad (both slow, and losing intelligence) with local model, so I have to rely on pi for now until I can fix my harness. When I rebuild, I would definitely want to do a better compact. Unrelated: I built the harness with the thought that local model would rarely if ever pass 32k context max. BAM! today, both 26B-A4B and 35B-A3B can fit f”
Available for n8n builds, production workflows, Claude API, Notion, GoHighLevel
Beyond the Token Factory: Building the OpenAI Intelligence Grid
It's a wrapper for llama.cpp and a wrapper for hfdownload, yes.
“> I had a difficult time experimenting with DVC, I'm trying to use it for an e2e ML Pipeline. I will revise the concepts and continue working on this. > Meanwhile, I completed session 5 of Python DSMP, covering the remaining data types and operations (set, dictionaries...), Impressed by CampusX :) today's notes: https://heroofjustice.notion.site/mlops submitted by /u/herooffjustice [link] [comments]”
“ok genuine question if you are actually building with long context models kimi k3 just dropped with 1m token context and im already seeing people getting excited about it. stuffing entire codebases into a single prompt. and yeah this part looks cool i get it but just think. like. what happens when something goes wrong with normal context size when your llm call breaks or hallucinates or returns something weird you can at least look at the trace and understand what happened. the prompt was 2k tok”
“If you have some patience and know what you want you can probably vibe code a fairly sophisticated frontend for Bedrock to connect with using Claude Code. I was building an app to make depo designations easier and got curious how hard it would be to add AI functionality. Turns out not hard at all. Bedrock is great.”
“Real talk: I lost count of how many big GGUF/safetensors downloads died halfway. Just today I ate a handful of 5xx errors from the Hub, and huggingface-cli wanted to start over again . At that point a proper resumable downloader isn't a nice-to-have — it's the only thing that works. So I built hftools . Single static Go binary (mac/win/linux, arm64+x86-64), no Python, MIT. The parts that came straight out of real pain: Resume on by default — multipart Range downloads pick up at the exact”
we already have a linux release at https://www.locality.dev/downloads
“Hey everyone I am seeking some guidance around ML orchestration for my hobby project. I have looked a loot into Flyte since most of my use-cases are around multi-modal datasets (lance format) and CV pipelines. The reason I looked into Flyte is: * Runs natively on Kubernetes * Caching so datasets don't have to be preprocessed/filtered based on the needed columns required * Retry logic if training or evaluation fails I can continue from a checkpoint or model stored in MLFlow * Conditional logi”
“Thanks I'll download the tool and start using it, it's a really great idea Do you plan on a Linux release for home labs?”
“Hey everyone I have been searching for clearer alternatives to standard RAG setups. Vector databases are tools but they often feel too complex, unclear and prone to losing context for personal or local projects. I wanted a system where the AIs basic information was as easy to control as code. To solve this problem I built a React/FastAPI sandbox around the Open Knowledge Format (OKF) specification using the Google Agent Development Kit (ADK) 2.0. Of a complicated backend data pipeline the workfl”
“Hi everyone, I've been working on an open specification called the Learner State Layer (LSL). The idea is to create a standard for representing a learner's current knowledge state, rather than just tracking learning activities like course completions or quiz scores (e.g. xAPI, Caliper). The specification is still in its early stages, and I'm interested in hearing different perspectives on the idea. Does this solve a real gap? Are there existing standards or research I should look int”
“My question was more about backups tbh, maybe it would be a nice side feature for your tool that also backups the whole notion?”
“thanks. yeah the idea is a locally synced version of notion that you don't just read, but can update notion from as well. We currently export the underlying data, so we export tables and databases. I think sorts, appearance and automations are not directly the underlying data itself, so i believe we don't export that. These are interactive features that notion app uses to create an interactive view over your data. Although let me know how you imagine that feature wrt Locality. Once you h”
“I’ve been working on a similar solution! Not as a product though. My build is more of a personal second brain/life context system with tons of relational databases. Very interested in checking this out. Does it handle databases and relations or just page content?”
“Bit of trial and error to be honest It's mostly overnight coding, so I write an Issue on GitLab, queue it up to be processed, it will work on the Issue and open an MR for review, I'll review it and queue it back up to fix the points raised in my review. Same as you'd do within a team of human devs.”
“Over past ~10 months I've been iterating on my memory system so I can make a proper assistant, like Rick's garage from Rick and Morty. I benched my latest iteration and it scored top out of any system I know of (470/500 on LongMemEval-S), while being way more token efficient and cheaper. The entire pipeline, storage and retrieval, runs on Qwen3 235B A22B Instruct 2507 with user chosen model as the answering model. The hardest bit wasn't even coming up with the architecture, it was ma”
“I came across this paper today and thought the idea was interesting: https://arxiv.org/abs/2607.08319 The basic idea is bringing Git-like workflows to a lakehouse—working on branches, validating changes, then merging them atomically across multiple Iceberg/delta tables. My first reaction was that this could make testing pipelines and AI-generated transformations much safer. On the other hand, it also feels like another layer of complexity that many teams may never need. Thoughts ? submitte”
“An upcoming work task is going to require the timely unpacking of avro files and the subsequent data being ingested to a database of some kind (currently undefined). I don't think our hardware expectations are anywhere close to realistic and wanted a quick confidence check.... I've seen a calculation online, and wanted to get a second opinion on the processing required to unpack 4tb of avro into a database digestible format such as csv (this might not be our final way of doing things, bu”
“I can't think of anything worse than sharing skills via Dropbox. The version management and AIBoM problems that generates is extremely painful. There's no way to track which version LLM is being used or match it against the skill, and people will likely load up too many skills.You don't have to expose git repos to end users to use git, or some other database, to provision skills.”
“Exactly why open-source harnesses like Open Bias are absolutely essential. https://github.com/open-bias/open-bias”
“Hey everyone, I'm working as an AI researcher, and we've been running into a major bottleneck with how our devs interface with LLMs (Claude/ChatGPT) during daily workflows. The core issue is that most people default to messy, unstructured brain dumps when typing out an instruction. This inevitably leads to sub-optimal, generic code generations, forcing the dev into a frustrating 4-turn correction loop just to get the AI to follow basic project constraints. It destroys engineering velocit”
“Storing memories as Markdown files is a smart call. I do something similar with project notes that agents can read and write to. The tricky part was actually watching the files. When Claude Code updates a memory file, you want to see it happen, not discover it later. I built markjason.sh for this - a tiny macOS editor with live file sync. It watches .md files and shows edits in real-time as agents write them. ~100MB RAM, opens instantly. Only macOS and just .md/.json/.env, but for the watching-a”
“LLM API's charge by the token/length and typically get very dumb if the context (amount of info in their "memory" gets too long). If you have a super duper long skill file (ie jokingly the length of the book) that gets read constantly you're racking up a bigger bill for worse performance. Hope that makes sense.”
“The basic idea is to use node-local storage as a shared model cache so new inference pods can reuse existing weights. It significantly reduces startup time and cuts down on repeated network transfers, especially when autoscaling. I wrote up the approach, some implementation details on GKE: https://hrittikhere.com/posts/model-caching-kubernetes-gke Curious how others are handling model distribution at scale. Are you using node-local caches, RWX storage, image-based models, or something else?  ”
“Before you rebuild anything, check out superhuman docs. It is very similar to Notion, more powerful and easier to build complex setups. It currently comfortably handles 10,000+ rows. (Depending on the number of complex formulas.) They are in beta with new functionality that will handle millions of rows. It has recently released strong AI that will allow you to build your "system" using natural language.”
“CozoDB, a transactional relational-graph-vector database with embedded Datalog in Rust, went dormant in December 2024. We hard-forked it as MnesticDB (not official CozoDB), under an MPL-2.0 license, to continue Ziyang Hu and the Cozo Project Authors' vision of building a "Hippocampus for AI", or agentic memory.But agentic memory isn't a pile of current facts with a log of past decisions. It has to track change over time, and be auditable. We've recently shipped several features that make this po”
“I've been looking for a cheap way to isolate feature sets in my projects and came up with the ddiff format, it's a simple export that simply comes out of a prompt asking for a diff of the analysis of intents along code changes related to features (which you can describe and request a specific ddiff file about), with a more explicit file for code changes, and on the receiving end the Claude prompt is to analyze the ddiff towards implementing the feature natively. I provide a live chat to”
“Ciao a tutti, Ho lavorato a una pipeline data-centric per la costruzione di dataset SFT e KTO per modelli linguistici di piccole dimensioni, prendendo come riferimento modelli che vanno da un modello ternario da 1,58 miliardi di parametri fino a 12 miliardi (con particolare attenzione all'intervallo 1,5-4 miliardi), utilizzando come caso di studio un assistente per la chiamata di strumenti in italiano ("Liara"). Invece di concentrarsi sull'architettura del modello, l'obiett”
“I'm working on production-scale ML infrastructure on Kubernetes. I've got the basics down – pipelines, tracking, deployment – but I want to know what's actually hard. What's keeping you up at night? Not "which tool to use" – the stuff that's genuinely unsolved. I could imagine something like: * Training on spot instances without losing progress? * Fair GPU scheduling across teams? Something else entirely? submitted by /u/Budget_Sense3306 [link] [co”
“I mean, in this particular case one of the biggest pain points they’re talking about are just straight up build times. An LLM that can produce a patch in 5 minutes and then has to wait 15 minutes for a build… to then put together another 1 minute patch and wait another 15 minutes for a build… that’s not ever going to be able to compete with Python where there’s no build time at all.”
“A single-file, dependency-free live dashboard for your local LLM serving box — GPU utilization, per-model throughput, KV/context fill, and system stats for llama.cpp and vLLM , in one green terminal-styled page. No framework, no build step, no external requests. The frontend is one index.html (opens on file:// ); the backend is one stdlib Python file that reads nvidia-smi and each server's Prometheus /metrics . https://github.com/NHClimber87/llm-serve-dashboard What it shows GPUs — per-card”
“Hi everyone, A year ago I began pre-training language models exclusively on 1800’s London data. Recently I have completed my largest dataset ever, containing 40B tokens or 160GB of 1800-1875 english data from England and the United States. I will soon train a 2B parameter model on it, but for now I’ve trained a 500M parameter evaluation model on a 5B token sample. I have also fine tuned the eval model on 1800’s Q&A pairs (using synthetic questions and answers pulled straight from the dataset”
“It's extremely interesting but I've also found it very hard to visualise. I'll take a stab at it lol.”