Correlating Logs Across Services Takes Too Long
Engineering teams are frustrated by inconsistent, complex, and reactive observability setups. They face challenges in centralizing logs, correlating data across services, and quickly debugging incidents, often resorting to manual checks and workarounds. This leads to slower incident resolution and hinders efficient development cycles.
SOURCES (60)
“I did test both SigNoz and HyperDX, so as I recall: 1. Storage model. Both keep hot data on local disk and move it to S3 as it gets older, so you still have to size EBS. Quickwit works the other way round, the index…”
“Hello everyone, I hope I am posting in the correct subreddit. I am administrating a bunch of virtual Windows servers running our own software. As of now, I don't have a unified way of checking log files on those machines, which means that not only do I have to manually log into the machine in order to look at those log files, but they are also not actively being monitored at the moment. I am looking for a solution that would allow me to check those files from a single point of entry (ideally”
“I've been playing around with OpenShip on my homelab and so far like what I'm seeing in terms of simplicity and lightness. It's pretty early on though so am looking forward to hearing about anyone running in production and what caveats I should be aware of. submitted by /u/SMBTechIndia [link] [comments]”
“You are assuming that whatever mechanism you use to free up resources again will a) work when your system is already out of resources b) is fast enough to not stall your real workloads during peaks In a virtual environment we usually already overprovision resources, artificially increasing the load doesn't make any sense here. If I run a 3 node cluster and every node is at 40% capacity, I don't have 3*60% idle resources. I have 2 nodes that can hold all resources if any one node fails.”
“Thanks, I'll have to look into these. I've looked at grafana, but not deployed it, and the others are now to me!”
Summary PantryPilot has explicit HTTP success and failure contracts, but no application owned way to correlate a user observed response with the corresponding server side request event. Feature 007 should add the…
“This is good, unfortunately our security group doesn't send us incidents through tickets, they have automated weekly reports sent to the "network" managers for the affected subnets. We then have to respond to individual items in those reports. I suppose we could start creating our own tickets for every item on those lists that we respond to.”
“The ask is for key performance indicators, but seriously think about risk indicators as well. They both have their uses. Generally see two broad categories, depending on what the end goal is. Type 1: The Good Stuff - where you have measures that give your team credit for the stuff that works well. Boast about your successes in (ex: service uptime, first call resolution, average time to patch) - which specifics depend on what your team does well. If management is just looking to CYA there will be”
“Almost every codebase is calling a REST, gRPC, or a GraphQL API or using SDKs from an external or event internal provider. It gets harder to keep track of everything when the codebase matures and increases in size and from my experience sometimes it gets hard to respond to changes in time or even become aware that a dependency is deprecated and their API has changed completely. Do developers need better tools that to improve dependency and integration management?”
“We were running a routine performance comparison between two APIs versions of the same model last month. We checked latency, track and usage, and general quality scores before cutting over to the new version. Everything looked comparable and the dashboards were happy But then I got curious and pulled individual responses side by side to compare them. On straightforward factual queries the two versions matched up 94% of the time which was expected. But on borderline queries, the kind that sits ne”
“Your current environment <details <summary Sanitized environment information</summary The complete collect env.py output contains private infrastructure metadata, so only the relevant sanitized fields are included here. A fully sanitized output can be supplied if additional fields are required. </details 🐛 Describe the bug CoreEngineProcManager.monitor engine liveness() can interpret process sentinel readiness as proof that an EngineCore process died even when the process is still alive and its”
“update: went ahead and built it. monitors now have a daily budget on top of the per-run threshold, sums cost across all runs today and alerts once you cross it, even if no single run trips the per-run one. live now. Still doesn't catch a run that never reaches the final ping step at all, that'd need a start/end pair. but the case you flagged is handled.”
“I wrote my own script to collate system info until it can broadcast and then trickle the logs over and wipe the old ones as they go. Depends on where you are aggregating it to. We use Splunk at work and I just use a DB behind an API at home, so there are many ways to get telemetry if you try.”
“nice, sveltekit squad op said in another comment they’re hitting a small agent on each machine that exposes an api for stats, so kind of similar idea to glances but more custom to what they need”
“honestly not that often, real incidents are maybe a couple a month if you have traffic. that's why the display barely matters, a status page is for your customers, your own view just needs to group related checks so a lone blip never reads as a full outage.”
“Expand the replies to this comment to learn how AI was used in this post/project.”
“Summary & Context To support continuous model evaluation, feature attribution, and error diagnosis over rolling horizons, this issue proposes extending the prediction logging schema in src/prediction logger.py ( data/prediction history.csv ) and the intraday monitoring ledger ( data/intraday events.json ). Currently, prediction history.csv logs basic forecast target dates, 5 day predictions, base prices, and backfilled actual outcomes. However, when forecast errors occur, the current schema lack”
“the alerts you can't act on are the actual pain. every tool I've used fires on single endpoint blips and you learn to ignore them fast. the step worth building is the one that decides whether to wake you, and the cheapest signal for that is "did something change recently". a failing group of checks plus a deploy 20 minutes ago is an incident, a lone timeout is noise. that correlation is the gap, the monitoring itself is the commodity.”
“Windows/Linux. The premise is diagnostics by correlating events in the kernel of the operating system and using the correlation to provide next steps, RCA, etc. I need more users and feedback. Happy to provide a license granting 20 agent seats and 1000 AI queries per month if anyone wants to test it. ET Ducky submitted by /u/apparentlyunoriginal [link] [comments]”
“From a gap analysis against nest start debug 's graph output, node ace list:routes / ace repl , and Spring Boot Actuator's /mappings and /beans . pnpm dev boots three processes with tsx watch and prints their logs. There is nothing else: no way to list the routes a contract exposes, no way to dump the resolved graph, no REPL against a booted application. The counter argument is real and unusually strong here: the graph is proven at compile time, so a runtime dump is redundant — that is the whole”
“i've worked in application monitoring and observability for a while now. a few months back a friend of mine, who build his saas entirely with ai tools like lovable, texted me asking why nobody was signing up anymore?! turns out his signup form been silently failing for 3 days. there was a error in his logs the whole time, he just had no idea what "unhandled promise rejection" mean, so he never look. thats the thing nobody tell you about building with ai coding tools, they great at”
“Managing OpenTelemetry collectors at scale has been a pain point for a while now. You either run standalone collectors and lose central visibility, or you build custom tooling to keep configs in sync across environments. This takes a different approach by embedding an OTel Collector directly into the agent, so you get Fleet-based central management for both your existing Beats integrations and native OTel pipelines in the same process. submitted by /u/AvenueJay [link] [comments”
“Preflight Checklist [x] I have read the Contributing Guidelines for this project, if it exists. [x] I agree to follow the Code of Conduct that this project adheres to. [x] I have searched the issue tracker for a feature request that matches the one I want to file, without success. Problem Description As part of our Cloud cost reduction efforts this issue applies a reduction to pods resources based on observed utilization over the last 3 4 months. Proposed Solution Appy manual resource request to”
“Feature request: Stream A, B, C, and Z weighted SPL simultaneously through the existing API ProdMesh RTA already includes an HTTP API and WebSocket stream for serving live level data to other machines. It would be valuable for the RTA to calculate and expose A , B , C , and Z weighted SPL simultaneously through that existing API. Current API The existing API settings expose: HTTP API Configurable port Network interface selection Configurable stream rate Live browser dashboard at / API status at”
“The logging format is in good shape. Nothing consumes it — there is no dashboard, no alert, no log based metric, no uptime check, and no way to follow one request from the Cloud Run access log into the application's own lines. This issue is the plan to close that, plus two concrete defects found on the way. What already exists — and is worth keeping Credit first, because most of the hard parts are done and the plan below builds on them rather than replacing them. | Piece | Where | | | | | Profil”
“The logging format is in good shape. Nothing consumes it — there is no dashboard, no alert, no log based metric, no uptime check, and no way to follow one request from the Cloud Run access log into the application's own lines. This issue is the plan to close that, plus two concrete defects found on the way. What already exists — and is worth keeping Credit first, because most of the hard parts are done and the plan below builds on them rather than replacing them. | Piece | Where | | | | | Profil”
“Hi everyone, I would like to hear how experienced DevOps engineers approach monitoring for large public-facing applications. We have a .NET e-commerce platform with: - ASP.NET Core MVC + Angular - SQL Server - Elasticsearch (~10M products) - RabbitMQ - IIS hosting - Multiple public domains/subdomains - Heavy SEO crawling and unknown bots One thing we learned is that monitoring only CPU, memory, and disk is not enough. We have experienced situations where: - CPU and RAM looked normal, but the app”
“I didn't say they those choices aren't viable; I'm strictly talking about architectural decisions. You can solve the problem with different solutions, be they rsyslog or otherwise. That said, I probably wouldn't go with rsyslog as my default choice anymore since the world is moving on to OpenTelemetry.”
“The ask Paperclip already has a good async agent wake protocol. The hermes gateway adapter creates a run with POST /v1/runs , observes it over GET /v1/runs/{id}/events , polls GET /v1/runs/{id} as a fallback, and calls POST /v1/runs/{id}/stop on timeout. That shape is genuinely runtime agnostic — nothing about "create a run, stream its events, stop it" depends on which agent runtime is on the other end. But it is currently reachable only through an adapter named for one specific runtime, whose h”
“:bullseye: What is your goal? The goal is to design a decoupled observability architecture for an environment running 20+ production scenarios. Instead of triggering instant notifications for every transient API timeout …”
“I’ve been thinking about this while working with observability systems and I’m curious how other people handle it. We have metrics, logs, traces, alerts, deployment information, etc. In theory, there should be enough information to understand what happened during an incident. But in practice, it often seems like the hard part is connecting everything. Something like: latency spike → database saturation → retries → downstream failures The individual signals are there, but figuring out that they’r”
“Context: generation currently makes roughly one provider request per event (jobs occasionally emit observationCount=2, but granularity is per event). On OpenRouter, free tier daily caps and per minute limits count requests — so a multi session workload burns 50/day (no credits) or 1000/day (with credits) fast, and a retry storm can exhaust a whole day's allowance (that's how we hit 3703). Ask: an opt in batching mode — coalesce up to N pending events (per session, within a small time window) int”
“Problem homelable knows whether a node answers, but not what flows between nodes. status checker.py stores status and response time ms , and every check method is a binary reachability probe, including prometheus , which treats /metrics as an up/down endpoint without parsing values. Edges are drawn by hand, so a link carries no operational data. Two open requests approach that gap from different directions: 242 asks for monitoring integration and a plugin system. 367 asks for LLDP link discovery”
“To me what you just mentioned is an archive, not observability. Raw tool inputs and outputs are thousands of rows a day of page bodies and payloads, and it only helps if you already know what to look for. To me the chain is: tool call → parse → run rules → derive signal → notify/display. Logging is just the first link, and you need the rest well before scale is the reason Something that shows you, readably, what your tools actually reached and whether it’s worth flagging isn’t overengineering. I”
“Disclosure: I publish this project. Most alerting tools solve a much bigger problem than I had. For small Node services, I wanted something I could add without a new dashboard or agent. Wotchi runs in-process: capture -> normalize/redact -> fingerprint/group -> threshold/cooldown -> bounded queue -> notifier. It supports Express and NestJS and is still an early beta. I’m looking for honest feedback on whether this boundary is useful, what would make you trust it, and where it stop”
“Finding Everything we record today is an aggregate : BackgroundJobTelemetry , PluginLifecycleTelemetry , WebhookTelemetry , PluginLifecycleSloEvaluator . Those answer "is the platform healthy". They cannot answer "this one request took 4 seconds — who spent it". Under ADR 013 (trusted in process) several foreign plugins share one process and one database connection. That is precisely the setting where the per request question is the one an operator actually asks, and the one we have no instrumen”
this. I use it to make sure my backup script completes every night
“Can you share a bit more about your workflow with LLMs wrt analytics? Like are you building dashboards or using Claude to find patterns/insights etc.”
“You don't need multiple SDKs - you need to clearly understand what you want/need to know. Regarding performance: i have seen projects that had 6 analytics tools installed in parallel, and it was ok from a performance standpoint - it is also a bit challenging to get multiple tools installed, as most wire in similar ways into a webpage. I worked in the highly regulated EU B2C space and for GDRP compliance (it can be extremely complicated to get analytics approved) we would avoid even using pos”
“Hi Observability & DevOps Experts, I'm looking for guidance from teams that have successfully scaled observability across large enterprise environments. We operate a large-scale estate spanning AWS, Azure, and on-premises environments and have been using Datadog for several years. Over time, a significant amount of technical debt has accumulated around our observability implementation. Current challenges include: Datadog Agents managed differently across teams and platforms. Custom log c”
“Problem description v6 dev start prints the logs from all network components in chronological order, which often makes it hard to trace the logs of a single component Desired solution commands to view the logs of a single component, e.g. v6 dev logs server etc I have not really thought about how to implement this (this issue is brain dump before I forget) yet but hopefully devspace has something for this.”
“Hey hi everyone, I am just trying to understand how engineers/SREs who dealt with real production latency incidents investigate it Lets say you have the following - Logs - Recent deployment information - Application health - Database metrics - External dependency health/metrics - Infrastructure metrics You just encountered the incident, you dont know the root cause. You are uncertain about the truth. From here how do real engineers go about reasoning to find the root cause - Do you follow a stan”
“Issue Report Category: Feature request Severity: N/A Area: Other Description Execution logs should include logs for pre installing hardware targets and other error messages that surface., Telemetry <details <summary Click to expand telemetry data</summary </details Reported from Olive Studio”
“Expand the replies to this comment to learn how AI was used in this post/project.”
“Hey I am just curious about the snapshot/debugger MCP you are talking about. Is that something you built yourself? How does that fit together exactly?”
“I run a small SaaS and went through the same "everything is overkill" loop for a couple of boxes. The lesson that stuck: the monitoring tool matters less than the delivery channel, because email is where alerts go to die and Discord gets muted, so route the truly critical stuff to something that physically buzzes your phone. Also add a dead-man check on the box itself - if it stops reporting, that IS the alert, otherwise a full lockup looks like a quiet day. Disk space and "someth”
“Expand the replies to this comment to learn how AI was used in this post/project.”