Correlating Logs Across Services Takes Too Long

“I have several selfhosted services exposed via reverse proxy. I use NPMplus+Crowdsec stack. The command line output shows ok statistics. But I would like to have a more convenient way to monitor like web dashboard. The Crowdsec console free tier has an alert quota each month so not very useful after hitting the quota. I am wondering what you all use to monitor. Are there any selfhosted monitoring options you would recommend? Thanks submitted by /u/meshinfinity [link] [comments]”

“Had a client dashboard where "events processed per day" was the headline number. Turned out a chunk of that number was a retry loop from a bug quietly re-firing the same event. Fixed the bug, the impressive number dropped, and suddenly nobody wanted it on the dashboard anymore.”

“Congrats on getting it into production. One useful line to watch with analytics is whether your own instrumentation becomes more expensive to maintain than the insight it returns. I’d keep the event contract small, make retention and deletion explicit, and expose the raw-event-to-derived-metric path when debugging. For a Claude-assisted build, did you keep the schema and collection layer deliberately boring so AI-generated feature work cannot silently change the meaning of existing metrics?”

“Prometheus + Loki + Grafana. Depends where you host. I host all my apps and services on self-managed VMs. Tools are completely free, but you will need some additional computing resources. In my case, it's not a problem. Setup might feel a little complex if you have no experience with it, but it's not hard to learn. For a single application or a couple of services, it's probably overkill, but if you have multiple projects, several backend microservices, and self-hosted DBs, the setup”

“No, Deliberately skipped, TimescaleDB pays off at a scale this solo lab never reaches Would use it without hesitation on a larger deployment though.”

“Fair callout. I use an assistant to help draft/check some replies, and this one came out way too checklist-y. The practical point was simpler: once cron jobs can touch external systems, I care less about the UI and more about logs, retries, permissions, approvals, and what happens when a run is half-finished.”

Dashboarding for k8s hosted microservices

“Companies keep adding automation tools (Zapier, Make, n8n, HubSpot, Airtable, custom APIs). Each tool can tell you a workflow ran - and showed no technical errors - but that doesn’t guarantee the expected business outcome. Common failures: Lead entered a workflow but never hit the CRM. Email automation ran but wasn’t delivered. Sync completed but updated the wrong records. How are you catching these issues today - native alerts, internal checks, customer complaints, or another tool? I’m working”

“It’ll just show that host as offline and the rest of the dashboard keeps working. Alerts are optional, so it won’t spam you unless you set up an offline-host alert. There’s also a cooldown and maintenance silencing.”

“Project Name: sigwood Repo/Website Link: https://github.com/helixmap/sigwood Description: sigwood is a local command-line threat-hunting workbench for the logs your homelab already produces: Zeek, Pi-hole/dnsmasq, syslog, and CloudTrail. It looks for periodic C2-style callbacks, DGA/tunneling-like DNS, port scans, rare syslog events, unusually long connections, and anomalous CloudTrail behavior. Each finding names the method behind it (FFT, HDBSCAN, drain3, z-scores), or explicit heuristics. Add”

“For deployment to microvms (lambda like) vs container based deployments ? Which experience are the best for developers ? Whats the hardest/limitations for each kind ? submitted by /u/poyrazKK [link] [comments]”

“A workflow-level router is a sensible way to avoid loading everything, but installed count alone is not the cost driver. Measure which skill text reaches each prompt, routing mistakes, subagent retries, verifier success, and total task tokens before/after pruning. I maintain Tura and wrote this analysis: https://turaai.net/blog#token-saving-plugins-are-mostly-stupid-idea”

“Hi everyone, I'm looking for advice from people who have experience troubleshooting production systems. I'm less interested in the exact fix and more interested in how you would investigate a problem like this. Environment - Windows Server + IIS - ASP.NET Core MVC + Web APIs - Angular frontend - SQL Server Web Edition on a dedicated server (8 GB RAM) - Elasticsearch cluster (3 nodes) on separate servers - Separate monitoring/tools server - Around 8 million products in Elasticsearch - Tra”

“I’m trying to build a proper infrastructure inventory and source of truth in NetBox, but the amount of manual work is becoming overwhelming. The environment includes a mix of: Physical network devices Virtual machines and hypervisors VLANs, prefixes and IP addresses Cameras IoT devices Other miscellaneous network-connected equipment I already have NetBox running and have started entering the data manually, but at this scale it feels like it will take forever. The infrastructure is also constantl”

“About two months ago, I posted here when Stratora first reached GA. For anyone who missed it, Stratora is a fully on-premises infrastructure monitoring, alerting, inventory, IPAM, topology, and escalation platform. It installs through a single Windows Server MSI, runs entirely on your network, and Community Edition is free forever for up to 100 actively monitored nodes. Since that initial release, there have been more than 100 confirmed Community Edition deployments. More importantly, the feedba”

“Project Name: StatLite Repo/Website Link: https://github.com/PVRLabs/statlite Description: StatLite is an ultra-lightweight, self-hosted monitoring dashboard built specifically for Spring Boot applications that use Actuator. It solves a common pain point for new Spring Boot developers who want to productionize their apps on cheap VPSes: getting useful production visibility without the complexity and high resource cost of a full Prometheus + Grafana stack (which often uses more RAM than the apps”

“Hello everyone, so I am currently using NewRelic for half of my setup and grafana (self-hosted) + prometheus for the other half, however I'm looking to move out of grafana and probably use a cloud service, however NR is kinda expensive for my needs; I need to monitor CPU usage, RAM and DISK + any syslog error (as extra), any recommendations/ideas? submitted by /u/no1saysitwasez [link] [comments]”

“Plus one for netalertx. It isn’t the „smallest“ or most efficient solution but the most mature one. I looked into watchyourlan but I really like the plugins netalertx is offering so I can hook it with my openwrt router and have some insights in different Vlans. Also, the dev is super responsive on GitHub.”

“I want to host stuff - open source, models, everything The thing that is stopping me is observability and scale. Suppose i want to host docuseal, testyourapp or ghost in one of this vps How I keep track nothing breaks and if I start making my own stuff around - how should I scale? Im proficient on denojs . But im open to exploring other server oriented languages like elixir or gleam submitted by /u/Main_Scientist_7449 [link] [comments]”

Exactly what a bit would say.

“You know the story. You write a consumer, test it locally, deploy to staging, everything green. Then in production, the same message gets processed twice. Orders are duplicated. Notifications go out twice. Everyone blames "a race condition" and moves on. We spent the last few months formally verifying what actually happens when consumers crash with at-least-once delivery semantics. The short version: it doesn't matter which broker you use . The race condition is a property of the d”

“https://github.com/Slyke/mqttctl MqttCtl What The App Does Today Dashboard: live diagnostics, transport security badges, DynSec bootstrap status, counts, and recent audit activity Audit: append-only history for writes, app startup, and login outcomes, limit presets, and JSON export with chained SHA-256 integrity metadata DynSec: clients, groups, roles, ACLs, effective-permissions view, default-role management, and clean-start bootstrap of read-all plus read-write-all MQTT Config: raw broker-conf”

“Infrastructure as code; and let it self document the as-built, who then uploads to a wiki. We use Arista AVD for this. It’s all built in and oss.”

submitted by /u/Berg0 [link] [comments]

“Hey guys. Using Open Telemetry collectors have been a pain in the *ss. Managing configs across fleet of agents. filtering huge set of telemetry data which are necessary or not. There are not even a single product out there which does it opensource. So built this quick vibe coded project. Shoot your views on it. Let's connect if it feels interesting for you. https://magpie001.vercel.app/ submitted by /u/praneeth_parker [link] [comments]”

“I agree with what I think u/More_Law6245 is writing. RAID - risks, assumptions, issues, dependencies. Risks are bad things that might happen for which you may or may not have mitigations to reduce risks. Definitely a risk register which is itself only useful if risk management is baked into everything else. Risks should point to tasks and tasks should point to risks. Assumptions should be identified in documentation dependent on them. Issues are risks that have been realized. They stay in the ri”

“Thanks for the quick response. Do you set up your raid log as a separate project?”

Yes. RAID logs can be effectively managed and kept in Asana.

“I haven't used the Checkpoint Endpoint solution but have used the P81 (which is the checkpoint SASE solution). Admittedly we have moved on from it and are not on Todyl. But to the pointed questions: - I'm sure it works but the "walls" you are hitting are common in the world of SASE. And it is not a set and forget or and just next next next type of thing. - We have Todyl running on accounts small (under 10 users) and large (over 500 users). Full Disclosure: I run and MSP. We lev”

“The deterministic reports bit is the most important part imo. I'd make that super visible in onboarding/docs, because "plain English analytics" immediately makes people worry about hallucinated numbers. Also worth showing the raw query/report definition behind an answer, even read-only, so a founder can sanity check it before trusting it. Full disclosure, I built AIForDatabase in a related space, and the trust layer matters way more than the chat UI.”

“Depends heavily on scale. For homelab/small deploys, Uptime Kuma + a Telegram bot is hard to beat (zero cost, docker one-liner). But once you hit 10+ services across multiple locations, the maintenance tax becomes real — you're spending cycles on the monitor instead of what it's monitoring. For that tier, a managed service with multi-region checks starts making sense if your time is worth anything. What's your alert payload and escalation chain look like? That's usually the make-”

“I am looking for some sort of video system to put on our production lines to monitor a specific issue that occurs. It's an added bonus if it has 'ai' capabilities to detect if the issue is occurring, or possibly even start recording if given conditions from the plc. The idea is to have footage for our engineers/techs to review the footage during or after the issue occurs, as well as give us further alert if it's happening. I thought about some sort of cctv/home security system, b”

“fcp in datadog is useful, but i’ve had way more signal from “rage clicks + js errors + support complaints” than a perfect perf dashboard. axe catches the obvious stuff, the sneaky a11y issues still usually show up when someone can’t actually complete the flow.”

“I reran the test about a minute after it failed. However, that doesn't always help. The approach you suggested might be useful. I'm not sure how uptime can be measured if the tests only run every 30 minutes. One possible approach would be to wait 30 minutes before retrying and sending an alert when there are signs of an intermittent failure. If the issue still hasn't resolved after 30 minutes, then something is probably genuinely wrong.”

“In your head? Most of the time I have to Google where the config files are so I can figure out how it works from scratch”

“As others mentioned, review which metrics are collected and frequency of them. Export the metrics to Elasticsearch into a timeseries index where people can view the data, to offload that part from Zabbix.”

“Hello, I’m having trouble with my self-hosted timescaleDB (96GB RAM 12vCPU and 1.5TB) keeping up with data from my Zabbix server. We have close to 20k hosts on Zabbix, all being polled. With compression set after every 2 days, timescale can’t compress enough, my storage gets filled quickly. I have done as much optimizations as I possibly can, I just can’t stop the DB from being filled so quickly (less than a week). I have 1.5TB worth of storage. Also my query performance is poor as well, despite”

“Sooooo system overview with history of basic stats, aka what a proper VPS provides?”

“Thanks — this is exactly the kind of feedback I was hoping for. For customization: v0.1a currently only has a built-in set of default detectors. Custom rules are not implemented yet. I have thought about eventually making it more like switchable detector/rule profiles, instead of one rigid hardcoded report structure. But that is probably a later direction. For now, I’m trying to validate the earlier assumption first: can this root-cause tree report format be understood and used by both humans an”

“Does anyone know if there are any free open source on-prem IT Ticketing Systems that one could use for a small business to keep track of users requests. Having something that monitors are our hardware isn't important as we have smaller systems that do that, it more just for use to keep track of users issues, ticket notes of the real issue and resolution, and to keep all the tickets so we have a history to back on. I know many years ago I have seen systems like this which where very basic but”

“The thing eating most of our time lately isn't fixing data issues, it's figuring out what broke and who owns it. We're on dbt plus Snowflake, a couple years in, and our monitoring is mostly reactive: job failure alerts, Slack pings when a run takes too long, manual checks on a handful of critical tables. None of that tells you anything about root cause, so every incident turns into someone manually tracing lineage backward through a few hundred models trying to find where it actually”

“I’m seeking a centralized log management platform for my small startup. We require a comprehensive solution that encompasses total observability and monitoring, alerting, logging, and metrics. Could you recommend a suitable platform? I’m particularly interested in a true open-source solution that offers full control and can be hosted on our infrastructure. It should be easy to set up and scale. If you have any alternative suggestions, please let me know. I’m also curious to know which platform y”

“For the first 4 months of running LLMs in production, our observability was mostly vibes. App responded → good. App timed out → bad, we basically had no insight into what was actually happening. Soon we realised setting up proper LLM observability was one of the highest-roi things we did. Here's what I learned setting up this, So, our setup is primarily, a gateway-level tracing with OpenTelemetry export to our existing stack. Every LLM request now has: model called, input tokens, output toke”

“Hey r/selfhosted , still pretty new to this and wanted to share my first real monitoring setup, woudl love feedback. Running Prometheus + Node Exporter for metrics, Grafana for the dashbaord, embedded a few panels (CPU, RAM, uptime) directly on my own site via d-solo mode insted of just screenshotting it. Everything sits behind a Cloudflare Tunnel, no open ports, no port forwarding, nothing exposed on the router. Felt like the safeset way to do this as someone who's still learning the securi”

“We've been evaluating Crossplane for about 8 weeks. Our Terraform setup covers 3 cloud providers, around 40 modules, and state management across teams keeps getting messier. Crossplane looked like a fit so we started digging in. The documentation is a lot. Official docs, provider-specific pages for AWS/GCP/Azure, composition tutorials, XRD references, GitHub issues about breaking changes between versions. Every time I thought I understood compositions I'd find some edge case in a GitHub”

“we took over a 500+ model dbt project from a team that has since moved on. documentation is sparse, tribal knowledge is gone, and we're three people trying to keep it running while also building new capability. we have basic freshness and not-null tests on maybe 30% of models, mostly the ones we've had to touch since taking over. the other 70% has essentially no coverage. no lineage documentation worth trusting. no incident process. everything is manual and reactive. the coverage problem”

“I wrote a post on how to generate RED metrics from your traces at the Collector before they hit your backend and why you'd want to do that instead of letting your backend handle it. I also added some tips on how not to blow up your metric cardinality in the process. submitted by /u/Broad_Technology_531 [link] [comments]”

“Optimizing code starts with measuring it, and a measurement is only useful if it is repeatable: a 2% improvement is invisible under 5% of noise. Yet on an untuned machine the same binary can easily run several percent faster or slower between runs. In this post we take a tiny benchmark and tune the machine step by step, re-measuring after every change, until runs become deterministic. submitted by /u/david-alvarez-rosa [link] [comments]”

“had an incident two weeks ago that took our team about 14 hours across 2 days to root cause. intermittent 502 spikes on our main api gateway, no obvious pattern, metrics all looked normal between spikes. after we resolved it i decided to test something. exported the full incident window from our stack. 5 days of k8s pod logs from the affected namespace, prometheus metrics export covering cpu memory and network, the entire slack incident channel transcript, and jira comments from the postmortem.”

“Third one of these i have run and observability was the cleanest yet. if you ask an AI which tool to use to monitor and debug a saas app in production, who does it actually name? i asked all four big ones the same thing a few ways. chatgpt, claude, gemini, perplexity. each modern tool scanned in its own home category. better stack got named zero times. on all four models. not a low score, just absent everywhere. axiom and highlight only show up on claude and nowhere else. openstatus only exists”

“Optimizing code starts with measuring it, and a measurement is only useful if it is repeatable: a 2% improvement is invisible under 5% of noise. Yet on an untuned machine the same binary can easily run several percent faster or slower between runs. In this post we take a tiny benchmark and tune the machine step by step, re-measuring after every change, until runs become deterministic. Continue reading— Tuning a Server for Benchmarking submitted by /u/david-alvarez-rosa [link] [”

“curious to see if any are using: - watch - time - zstd -19 --long=31 -T0 --zstd=ovlog=9 -v -c -- input.tar > out.tar.zst (massive compression) submitted by /u/tui-cli-master [link] [comments]”

“Hello! I've moved to company that is using DataDog for storing logs, monitoring etc. Its not really that used in my team, so i tasked myself with some edits and showing possibilities. I'm coming from company where i have used Grafana for monitoring and alerting, so i'm used to the system that grafana has for alerting - mainly for grouping etc. Here, we have private location for Monitors, that is in our network and so can access internal resources. But, as it happens, local server mig”

“maybe i'm overthinking this. i run a couple of small civo k3s clusters for my own stuff and somewhere along the way my setup turned into web dashboard in one tab, terminal with kubectl in another, and the browser open more or less permanently because every time my home IP changes i have to go in and fix the firewall rule by hand. which is a lot, my ISP rotates it whenever it feels like it. k9s covers the kubernetes side fine. pods, logs, exec, all good. but it does nothing for the provider l”

“Hello fellow sysadmins, I'd like to ask for a general opinion about two systems (or a combination of those): Icinga2 + InfluxDB + Grafana + Prometheus. Background: I come from a world of PRTG, mostly. So I am kinda used to "integrated" solutions, with custom queries via Powershell and SSH. New company: uses "old" Icinga2 (read: still Debian 11), a sole integrated solution made by external company, basically all-in-one Icinga2+InfluxDB+Grafana, with Grafana-state-screensho”

“I am working on a gateway/control-plane idea for LLM traffic from Kubernetes workloads. The core problem: every app is starting to call OpenAI/Anthropic/Gemini/etc directly, but platform teams still need routing, provider key control, budgets, observability, and policy checks before prompts leave the infrastructure. I am trying to think through the right architecture. Options: central gateway sidecar per workload API gateway plugin Kubernetes operator + CRDs SDK-based approach service mesh exten”

“I was tired of writing the same plumbing code over and over again. Something happens over here and you need it to trigger something else over there. Eg. a user signs up. A payment fails. A support ticket comes in. An error fires. And then a Slack message needs to go out, a row needs to land in the database, an email needs to send... So I built my own pipeline and SDK: ingestlayer.com You build a pipeline, point events at it and it runs on every event. Each pipeline is representable in YAML and a”

“I see a lot of people use observability and governance interchangeably for LLM gateways, and I think that's a mistake. Observability is about debugging: latency, token usage, error rates, tracing individual requests. Governance is about control: who can call which model, rate limits per team, PII filtering, audit logs for compliance, cost allocation. Most gateways are strong on observability. Helicone is basically an observability layer. Portkey does both reasonably well. LiteLLM gives you l”

“Hi r/selfhosted , What are your experiences with highly power-efficient servers, in particular servers with low idle power consumption, when the server is also running regular actions such as (database-backed) monitoring or haproxy health checks or tcp keepalives? In particular with several containers and/or vms, I imagine monitoring to be the largest idle load. How does this affect power consumption during idle for you? How does this affect when and how long your cpu goes to sleep? How does thi”