Coding Agents Shifted the Bottleneck to Verification

“Agents ignore AGENTS.md mid-session.Then you tell them: hey, this is written in AGENTS.md.- "You're absolutely right!" (proceeds with fixing)But not anymore! This is a tool that would check the code immediately with a classifier (Jev etc.) and give the agent advice in the same turn.It catches some mistakes that are difficult to put into lint rules and excels at the cases when semantic understanding is required - e.g. field names/types in a nested structure that contradict each other (more common”

“Well, yes. The whole point is that there is some skill to agentic coding, it's just not what regular coding is. It's a heavy emphasis on QA, using the agents capabilities to set up testing infrastructure to push it to its limit and make sure actual performance is bulletproof. Ultimately whether it's you writing the code or an agent, you own it”

“A lot of software engnieers I work wth now spend 99% of their time tellng agents how to write code. I am curious to know how much we delegate work to agents as Data People - given so much of our job is not just writng code Please include a score out of 10! 10 = agents do all my stuff, 0 = still doing everything by hand submitted by /u/engineer_of-sorts [link] [comments]”

“Split out of 706 , where it was bundled with cosmetic observability gaps and got buried. This is the structural half. There is no mechanism for a deployed fleet agent to pick up a new CargoShip version. The compose file ghostship init emits pins an exact tag (deliberately — :latest would silently drift the agent under a running fleet, which is worse), so updating an agent today means, per box: I did exactly this by hand today to move one writer from 0.37.0 → 0.38.0. Why it matters more than it l”

“I kept running into the same problem with AI agents and automation: the system says “done” , but that doesn’t always prove the final state is actually correct. So I built SCC Runner around a simple flow: Claim → Authority → Evidence → Verdict The idea is to verify important claims against real system evidence instead of trusting the agent’s own output. I’m currently offering free early access. I’d especially like feedback from people building agents, automations, DevOps workflows, or AI-assisted”

“Taking 5x longer to review is something we experienced too. While there are multiple code review tools, I am not sure they come close to level 4 autonomy. We have been experimenting by using multiple models to review not just the code, but design, architecture and the agents' riskiest code commits. We just launched it for others to try it out too - https://truverif.ai/panel-reviewThe main insight was that a multiple models arguing over a code change produces far better insights than a single mod”

“This is a genuine problem that I hear from senior engineers. I'm looking for a solution.--The quality of ai-generated code is (claude, agy, copilot, codex a little better).It's exhausting to read.I used to love learning from my experienced colleagues and taking pride in what we made. We spent time on elegance and craftsmanship.Now I spend nearly the whole workday slogging through convoluted code riddled with footguns. I ride on hopes and dreams I might understand a changeset.It takes”

“I find contra productive reading and manually testing all the code AI are generating, because I am turning myself into an important funnel and the company is paying for tokens and therefore wants fast and good delivery. I use Claude Code with dbt official skills, dbt mcp, lots of unity testing, some specific skills tailored for my projects, and the most important thing I think: deterministic checks. I run lots of python or bash script checks after specific hooks like pre commit or after calling”

“We spent a week working through a large code base refactoring with around 500 agents. The learnings from that work went directly into the ArchKeel releases up to 0.8.3.We still need to understand which components exist, what they are responsible for, and whether the changes and new code stick to our rules. More agents make building faster and harder to stay on track.So we worked on the checks and the reports together. Recursive contracts, evidence for individual rules, and much better explorati”

“Most multi-agent write ups are about quality - which model to use and why, more agents and generating better answers. I am interested in the failures that happen.If you run more than one agent against shared state or shared tools, what actually went wrong?1. Did two agents do the same work, or different work that conflicted? 2. Were your agents copies of each other (homogenous agents), or specialized with different roles (heterogenous)? 3. What did you add to stop or prevent the failure?Anthropi”

“i’ve handed off a decent amount, but fully autonomous is still kinda fake for most stuff lol. agents are great at first drafts, repetitive research, small code tasks, cleanup etc. but anything customer-facing or important still gets checked. feels less like replacing work and more like having a bunch of very fast interns you still need to supervise.”

“The edges are where it breaks. Everything in the middle of a task runs fine once the input is structured. The cleanup always comes from the first and last bit: deciding what the job actually is, then checking whether the result is right. I've been building prompt systems for the last few weeks, and the thing that moved the needle wasn't a smarter model. It was making the input boring. A fixed list of fields to fill in, and an exact output format at the other end. Every field you define i”

“The agents can brainstorm, research, create and refine a plan before generating code. Maybe on huge projects it would struggle because of context limit but looking back on every "ai can't do x" right before it evolved and did that thing, this doesn't inspire confidence and feels like cope trying to grasp anything ai can't handle as good (yet). Most of the things a developer can do, a random ai user can also do, just with more prompts. I really hope I'm wrong and didn&#3”

“Deploying agent integrations in production software requires explicit permission boundaries and transaction tracing. Scope agent execution tokens to minimal necessary resources with explicit time-to-live expiration. Log every agent action with deterministic correlation IDs to maintain full auditability across service boundaries. Implement circuit breakers that automatically pause agent operations if error rates exceed baseline thresholds. Simulating unexpected API disconnects against staging san”

“I’ve started using patterns like Code generates JSON of input data, so the agent can’t forget to paginate; sets some flags to help the agent classify things Agent reads input, does language-reasoning stuff; writes a JSON file with its findings Code reads agent output and makes all the write-capable API calls Steps 1 and 3 have good unit tests. Before this pattern, trying to codify all their logic in the Agent’s markdown was getting really hairy. Edge cases kept getting missed, etc.”

“lol the "I miss components, logic, and endpoints" comment sums it up pretty well. How are you even debugging it when the agent does something unexpected? Are you able to trace why it made a particular decision, or is it mostly changing prompts/state and running it again until it behaves?”

“Actually that video is of a codebase I have been working on since 2023. Most of the growth was features and docs (examples, etc.). But generally I agree that agents default to bloat and spaghetti.”

“Hey HN, we are Aakash and Viswesh and we are building Canary (https://www.runcanary.ai/) - independent verification for AI code. Claude/Codex calls Canary with the changesets, intended behaviour and team knowledge. Canary then deploys agent swarms to investigate potential failures and test suspected runtime bugs in remote sandboxes.To try it on your repository, paste this into your coding agent: Install the Canary CLI with npm i -g @runcanary/cli, then run canary skills and follow its instruc”

“They are creating either the PRD documents, or the design specs the agents will use, based on the requirements. And some are reviewing the code, which is getting harder and harder.”

“That is exactly what we do, all logic in normal functions and model only picks which one to call. Took me long time to accept this, before I was trying to make agent do everything and debugging was nightmare”

“observability on agent chains is definitely a major pain point once u get past a simple prototype. are u handling the state tracking via a centralized event bus or are u trying to hook into the provider logs directly for that visibility”

“The whole "agents as the primary consumer" thing is mind boggling. Not even your devs rely on agents that much. They're still designing the technical solution and need to understand the thing.”

“I've been driving myself slightly crazy with agentic work. I mean, it's awesome and all, but this one thing: no matter how many test harnesses I put in the development loop, it still breaks shit. While happily claiming everything is green. Made me think: I need an overview I can control, a sanity check for both the agents and me. When I looped in agents to do this, it felt like it did a better job. I'm actively working on getting that as good as it can get, but the original use case”

Thanks for the feedback!

“Most agent tooling I see is observability: traces, evals, alerts. Useful, but by the time an alert fires, the agent has already sent the email, run the query, or moved the money. We're building a runtime authorization layer that sits between agents and their tools instead. Rough shape: Every tool call and API request goes through a proxy (Envoy-based gateway/sidecar), so the agent has no network route to tools except through it Each action is checked against policies-as-code (spend limits, d”

“The useful distinction here is run health versus ordinary LLM tracing. I’d make first setup emit one timeline per run: trigger or schedule, customer/workflow ID, model route, retries, spend, and terminal state (success, failed, stale, looped, escalated). Add a cheap structural check—valid JSON/schema or expected tool call—before any subjective quality score. That makes silent failures visible without pretending the monitor can judge every output. If you want a lightweight way to turn a plain-lan”

“me and my friend who quite meta last Dec and building agent verification engine, with ai building is cheap now but validading weather agent does right job or not determinstically still a gap we are seeing. we strated around april/may and launched on Ph two times and ranked top 3,2 .. and spoke to developers and builders who are deploying agents on prod. i wanna see feedback from community how are you all validating weather or not agent is taking right actions and if it’s not doing its job how ar”

“A coding agent across work apps needs one visible task ledger more than another chat surface. Show the requested goal, files and systems touched, assumptions, pending approvals, and every side effect in a single timeline. Agentix Labs connects to this pattern because cross-app agents become manageable when execution state is explicit. For the beta, test read-only discovery first, require approval for writes, use scoped credentials per integration, and let users replay or revert actions. That wil”

“We’re building agents that maintain operational state from updates people give them. Initially, Capture seemed like the easy part. Give people a text box → send the update to the agent → structure it. But the more autonomous the agents become, the more we’ve realised that bad state often starts before the agent ever reasons about anything. Humans don’t communicate like databases. Someone writes: “JD should have api done next fri. testing monday. let ralph know.” There’s shorthand. Relative dates”

“agents are the most annoying thing to debug and evaluate, hands down. a single autonomous run fans out into tool calls, provider retries, sub-agent handoffs, context compression, and by the end you have no idea which step cost what or where it went sideways. we built Infere because that hurt us. what we shipped: request-level traces that group a whole agent run (sub-requests, tool calls, cache reads, retries) into one object you can scan, per-step cost/latency/token breakdown, and evaluators bui”

“this matches my experience exactly the model writing code was never the hard part its the orchestration and verification layer that eats all the time havent found a single tool that does the full loop you described still stitching together github actions and custom scripts myself”

“This is really where the current moat is as a software dev. The old things we liked but didn't do because they were too complex - extreme static typing, proptests, formal verification - are now attainable and valuable.”

“The failure that made me believe this wasn't in an agent at all, it was in a test harness I wrote myself. For a few weeks it printed 0 passed, 0 failed, exit code 0 . Green. I read it every run and my eye stopped at the 0 failed. It was collecting no cases at all, the reference frames weren't being found, so it compared nothing, nothing failed, and it reported success. Everything I shipped in that period went out unverified and I was more confident than if I'd had no check at all. So”

“The context-loss problem is real — I've had the same experience where you spin up three agents, step away for 20 minutes, and come back to a wall of terminal output with no clear sense of what actually happened or what needs attention next. One thing worth checking out alongside your tool: IdeaboxAI (ideabox.ai) is built around exactly this orchestration layer — giving you a unified view of what your agents are doing, surfacing the decisions they made, and letting you intervene without diggi”

“Wonderful data point!It's been interesting investing into different classic verification & testing methods for gfql (CPU/GPU graph queries on dataframes) and Louie (agentic investigation harness) over the last couple of years with different model & harness generations. The ones below are fewer but more longitudinal efforts compared to the shallower ones Dan worked through, and seem largely consistent:Overall, my takeaway on standard software has been LLMs favor smart, guided fuzzing, natural lan”

“The pattern across this thread is pretty clear: agents are a force multiplier on greenfield and a liability on old, quirky codebases - the "Chesterton's Fence" comment nails why. They see weird legacy code and refactor it, not knowing it was written that way to handle an edge case three years ago. The thing that seems to separate people getting value from people fighting it isn't the tool, it's scope discipline: small, contained tasks with tests around them, and keeping the”

“Motivation The MCP drive surface is built for iterative agent use: describe app → set inputs / invoke → inspect → adjust → invoke again. get invocation(index) exists to fetch a past run's kwargs+result, and invoke returns a history index , so the history is clearly meant to be reviewed. But while dogfooding the agent surface on the shipped fast dash==0.6.9 , two gaps make that history almost unusable for a headless agent: 1. There is no way to enumerate the history. The only reader is get invoca”

“Agents make API calls to invoke the LLM, they make tool calls, read/writing files etc. I want to be able to mine this data in order to engineer out failure modes and plug gaps with documentation, skills, custom hooks etc. However I'm not seeing great tooling in this space and wondered if the tooling is just severely lacking or am I missing something? So far dsh makes the easiest with observability built in via the trajectory tab. That's the sort of thing I'm after, but as a standalon”

“yep, some people are extremely insecure and would rather shit on tools than learn to use them correctly. hallucinations are no longer a problem since agents can verify their own code. the agent has access to static analysis, integration testing, e2e, even metrics and error reporting. it sounds more like OP is copy/pasting code from chatgpt or something. agents would be much less error prone. obviously OP should always verify the output, but if the output is truly bad then the agent was likely ju”

“For me it wasn't complexity, it was the second change. I asked for a fix, the suite stayed green, and it turned out the agent had written a test that asserted the bug instead of catching it. Since then green means nothing to me unless I read the test. The slower one is drift: every session it reinvents a pattern that already exists three files over, and after a few weeks nothing in the repo looks like anything else”

“Yeah, it writes the tests to match whatever it did. Putting the actual acceptance criteria in the ticket helped a lot with that, gives the tests a spec to check instead of the code that was just written. Still get useless ones but way fewer.”

“I've been building with AI agents and kept running into the same frustrating problem. The agent could reason, write code, use tools, and do some genuinely impressive stuff. But it would forget things. I'd have to explain the project again, remind it what happened in the last session, point it to the right files, explain decisions we'd already made, and keep feeding it context that should've already been there. And the more I built, the more I realized that a smarter model doesn&#”

“You nailed it on every point :) The handoff between agents is exactly where things can go wrong. I actually built in a simple verification step after your comment made me realize I should explain it better: each agent outputs to a markdown file, and the next agent reads that file as context — but I also have a "fact-check" prompt that runs before anything goes public. It cross-references claims against a short list of verified sources I maintain. Your point about measuring time saved a”

“I have been saying this a lot. If you ask AI it just tends to agree with you no matter what. As for Agent Guidelines, I am not very experienced with them or agentic coding in general but this might be worth looking into.”

“If you’re building multi-agent workflows (especially with frameworks like LangGraph, CrewAI, or AutoGen), you know the pain. Tracing a single LLM call is easy. Tracing 4 agents passing state back and forth, hitting infinite tool loops, and ballooning your context window is incredibly frustrating. I got tired of jumping between 4 different tabs (traces, raw prompt templates, logs, and cost metrics) just to figure out where a swarm lost the plot. So I built a workspace that unifies everything into”

“If you’re building multi-agent workflows (especially with frameworks like LangGraph, CrewAI, or AutoGen), you know the pain. Tracing a single LLM call is easy. Tracing 4 agents passing state back and forth, hitting infinite tool loops, and ballooning your context window is incredibly frustrating. I got tired of jumping between 4 different tabs (traces, raw prompt templates, logs, and cost metrics) just to figure out where a swarm lost the plot. So I built a workspace that unifies everything into”

“Anybody here who feels AI SRE is a gimmick? I mean, what are we doing with this, honestly? Not gonna name names, but our observability platform ships one, and it takes 10–12 minutes to run an incident analysis and comes back with "hey, these are 2 things that are broken, and these are 5 probable causes." In the last three incidents, the causes it flagged were real issues acting up as bugs, but not related to the current incident at all. So I'm getting a faster RCA, just a wrong one”

“I’ve been pretty annoyed recently with my coding agents doing stupid things that are anti-protocol. I can encode rules like “agents shouldn’t write to the README” pretty easily, but things like “database migrations must be autogenerated and not handwritten” are much harder to enforce. Anyone have any tools they use for this or tips for how to deal with issues like that?”

“Coding agents are doing a lot of the integration work at our agency now. Stripe, Twilio, WorkOS, email, the usual stack. Economically it's been good. The part I didn't expect: verification actually got harder. We're producing code faster than anyone can review it, and the edge cases that bite you aren't in unit tests. Webhook fires twice. Events arrive out of order. Agent wrote correct code but got one state transition wrong. PR looks fine, everything compiles, and you still don&”

“Your coding agent ran for 40 minutes, spawned three subagents, and touched 30 files. The diff shows what changed. It doesn't show which subagent did what, what it tried before the thing that worked, where it retried, or the moment it went off plan. I built rungraph to answer those questions. It reads the session your agent already recorded and draws it as a graph: your prompts down the spine, subagents in their own lanes, tool calls as nodes, and the moments you said no marked on the path. C”

“I was honestly getting tired of agents writing slop and then having to spend more time manually reviewing and fixing it than actually building things. So i built this workflow for myself: The idea is pretty simple. Instead of letting one agent do everything, I split the workflow into different SDLC stages: PM, planning, development, QA, and two separate review stages. I also wanted the freedom to pick different models from different providers at each stage (even combo of the CLI agents i already”

“yeah that second half is the real problem right, agent "fixes" it and you just get a different failure on the next run. does your screenshot + log give the agent enough to actually diff what changed, or is it mostly trial and error?”

“the pre-prod gate only catches the failure modes you thought to script. the class that bit us hardest is the drift kind: webhook handler works, then weeks later the provider's event log and your DB quietly disagree and nothing ever threw. so our real last gate ended up living in prod, not before it. a nightly reconciliation job that diffs the provider's records against our state and alerts on mismatch, an alert for "no events received in N hours" (silence is also a failure), an”

“Coding agents are doing a lot of the integration work at our agency now. Stripe, Twilio, WorkOS, email, the usual stack. Economically it's been good. The part I didn't expect: verification actually got harder. We're producing code faster than anyone can review it, and the edge cases that bite you aren't in unit tests. Webhook fires twice. Events arrive out of order. Agent wrote correct code but got one state transition wrong. PR looks fine, everything compiles, and you still don&”

“Everyone is talking about agent observability as the next big MLOps problem, but I am curious what people are running today. Traditional model monitoring covers latency, drift and accuracy well. None of that tells you much once an agent is making a chain of decisions and calling multiple tools in sequence. Does anyone have an agent observability setup that helps debug unexpected behaviour in production rather than producing logs that look complete, but are not useful during an incident? What doe”

“Sometimes I start several coding agents at the same time, each working on a different task. I just let them go and don’t watch them closely.The problem is that I may change my mind later but forget that one of the agents is still working toward the old goal. A couple of hours later, it submit a patch with all tests passing and reports that the job is done. But when I check the result, I find that it has finished implementing a plan I had already abandoned.Has anyone else who uses multiple coding”

“Howdy! Happy Saturday everyone!As a solo founder, I have always tried to maximize my speed by letting coding agents build as much as possible in parallel. However, as an engineer, I don't trust that AI will always make the right decisions and work with the right context. In the past, I always needed to click through my sessions to glance at the AI's output, try to understand what it was doing, and hopefully steer it or stop it in time.As a result, the maximum number of concurrent sessions I coul”

“yeah, batching the edits makes sense. i was asking about the actual verification_status value though. if checks pass and the agent writes again 30 seconds later, does that flip to stale/unknown immediately or stay passed until another check runs? also, this is probably a fit for awesome-ai-plugins. i checked the current list and don't see Flare. i'm involved with the list.”

submitted by /u/SplashyPantsMr [link] [comments]