DevOps Teams Fighting Flaky CI/CD Pipelines
Developers and DevOps engineers are experiencing significant challenges with their CI/CD pipelines, including unexpected behavior, difficulty in debugging, lack of visibility into deployments, and increasing complexity due to wider adoption of development tools. This leads to unreliable releases, difficulty in troubleshooting, and pressure to deliver increasingly complex solutions.
SOURCES (60)
“So this: "I worked on feature X, and made the choice to go with a separate lambda for dry-run requests to keep traffic isolated and prevent dry-runs from consuming throttling bandwidth" Is enough for you to then support that application if the guy is sick?…”
“AI wrote extra shitty code that was failing CI because DNS names were incorrect... but instead of acknowledging it's mistake it decided to do something else... AI thought that CI result was 'old/stale' and started break glass workflow (getting escalated permissions and GitHub API admin accesa) to overwrite all CI checks to push broken code to production... When deployment of broken code happened, and everything broke, it decided to troubleshoot things by deleting data from production”
right, at that point youre just testing against a fiction
“you can still run cf2021 dev edition in production just not legal but for 6-8 months while new app is being built nobody really checking licenses on old cf servers. the dependency problem is real though adobe makes it hard on purpose to get old versions. lucee is good suggestion but if you have cf specific tags and functions in that code it might not work without changes.”
“Wtf is this i would quit so fast. This is the type of shit where you just know that shipping 1 feature takes you a year”
“honestly, i think the CI/CD toggle being easy to miss is kind of the point, if you’re scanning repos at scale you’re not gonna click into every single one, so it makes sense to default to “CI exists” as a baseline signal still, a 56 without CI means you’re probably one solid README rewrite away from a clean score, which tracks with most repos i’ve seen”
“5134f81 ( 360) left main red. The ui job's npm run build fails with eight tsc errors; rust , openspec and gitleaks pass, and build / merge are skipped, so nothing shipped. What fails Six TemplateDetail fixtures omit params . 360 made the field required ( ui/src/api/types.ts:94 ), because the contract publishes it always, as [] when empty. The fixtures at ui/src/lib/preview.test.ts:68 , ui/src/pages/print/FieldForm.test.tsx:7,21 and ui/src/pages/print/PrintForm.test.tsx:8,22,333,981 predate that”
“A Job can burn its entire retry budget failing a Check it had a one line fix for, because nothing in Fleet can run that fix automatically — the Manifest schema for saying so is written down but was never implemented. Job 01M1M6YP7G0012GVQ2PCB6VF2E used all 3 implement attempts and was escalated completed failed . Every check passed every attempt ( build , test , typecheck , bridge build , storybook , bridge test ) except one: format ( cargo fmt all check ), which failed all three times on a diff”
“Mid dev in a small startup. Last week our freelancer devops guy added more server instances for our app and it was a disaster. I had to cover him for the whole week help find and analyze bugs it caused. Yesterday when the situation finally calmed down, I went straight to the CEO office, put my big balls on his table and asked for a raise + moving some of his responsibilities to the in-house dev team. Today the devops guy found that I accidently leaked our env file... our backend language has a b”
“Problem The integration suite uses inconsistent setup and ownership patterns. Shared helpers hide file level AVA hooks, the multi codec harness runs hidden concurrent work, several large files cover unrelated behavior, and filenames carry an increasingly unclear mix of product area and execution requirements. The repository also lacks a clear coverage strategy. Line coverage is useful for pure SDK code, but it does not show whether server capabilities, authentication modes, codec paths, or deter”
“Every retro after something blows up ends the same way - "we need another sign-off," "let's add a doc for that." Fair enough in the moment. But I've started noticing heavier process doesn't always kill the risk, it just relocates it. Instead of one obvious failure everyone can point to, you get slower turnaround, people quietly skipping steps when the deadline's breathing down their neck, or a doc nobody's actually opened since the day it was written, sitt”
“Before Submitting [x] I searched open and closed issues and discussions for an existing report. [x] I checked whether this is already fixed on the dev branch or latest source. [x] I understand that maintainers want a well written issue before any code pull request. [x] I am using the latest available version of Open WebUI for my install method. [x] This is not a security vulnerability. Installation Method Docker Open WebUI Version ghcr.io/open webui/open webui:dev (reports 0.11.3) reproduces the”
“Problem The task/instance list is dominated by scheduled and manually triggered routine runs (e.g. pr sweep watch , pr sweep dispatch ) that have no underlying Jira ticket or GitHub issue — they're plumbing, not work. These fill up the list and bury the actual ticket driven work (jira implement, pr checkup dispatched against a real ticket, etc.), making it hard to see what's actually going on. Related: there's currently no easy way to see, at a glance in the dashboard, which instances are sittin”
“have to say, I was honestly surprised that such a huge global company is such a chaotic mess 😅 I work in system architecture myself, and honestly, even in much smaller companies you don't see chaos like that.”
“nice work catching that script trap. most devs just move things to theme settings and call it done, never think about the regeneration part had similar situation where client kept losing their changes every few weeks. they thought they was going crazy. turns out the build process was pulling from a json file nobody touched in months”
“interesting idea but who's doing the analysis here? if youre using an LLM to check LLM-generated code, arent you just trusting a different black box at that point”
“just tried it with a script claude gave me for resizing images, caught an rm -rf that was buried in a cleanup function I totally missed. the explanation was clear enough that I actually understood what it was doing instead of just getting a scary red flag, that's the part I didn't expect to like so much”
“Both CI workflows have been red since the runner images moved on. Master last went fully green on 2025 08 22 ( Merge pull request 263 ); every run since fails before reaching the test stage on most platforms. Noticed while working on 405, where 17 of 36 checks fail for reasons unrelated to the branch. There are two independent groups. 1. Build workflow ( .github/workflows/build.yaml ) pinned JUCE 8.0.6 vs current runner images These are toolchain/SDK problems, not JUCE API drift. The JUCE submod”
“That is a better statement of it than my post managed, and I am going to steal the phrasing. lol "The execution engine did not throw" is the only question n8n actually answers. "The workflow produced the required outcome" is the one people think they are being told, and nothing in the execution record contains it - because the required outcome is not a property of the run, it is a property of what you expected before the run started. That has to live outside the execution. An”
“The permission system is still kind of a mess once you go beyond clean internal use. I've had guests accidentally edit master databases when they were only supposed to see a filtered view of one page. Been burned too many times to trust it without double-checking every share link now.”
“Just use a Vercel Preview Deployment from a separate branch, not production. If you don't want random people accessing it, enable Deployment Protection in the project settings. Then only you/team members can open the preview URL while you test.”
“yeah, after enough time in the codebase you start treating every weird behavior like a personality trait 😭 at some point fixing one bug feels less like debugging and more like negotiating with yourself from 6 months ago.”
“I’d focus less on vendor count and more on dependency risk. track how critical each integration is, how often it breaks and how painful replacement would be”
“I would build a validation for anything i get via an api. So if anything is not how i expect it, i would log it. But i can only speak for myself. Maybe in todays age with lots of AI made software it would help. But LLMs often don't suggest paid services. Maybe a free tier would help bring it into agents reach.”
“That’s a fair point. Sentry or Elastic would usually catch an exception after the webhook handler breaks, and they may already be enough for many teams. The gap I’m exploring is earlier validation: detecting that a payload no longer matches the expected structure before it causes an exception or silently writes bad data. For example, a field changing from a number to a string, or disappearing while the handler still returns 200. The idea would be more of a webhook contract monitor than a replace”
“DevOps is for technical failures, what about business logic changes? new edge cases etc”
“DevOps team should have it monitored and with reports. Platform team maintains and iterates.”
“From: mwk rider@main Found by running create mode end to end on 1.5.0, then adding content. A post's excerpt was over the schema's 180 character cap, so npm run build failed with InvalidContentEntryDataError . The audit run straight after reported 62 ✅ 0 🔧 0 🛑 — it read the dist/ left by the previous successful build, which no longer corresponded to src/ . The site was unbuildable and the tool said clean. The tool deliberately never runs the build (it never executes the audited project's code)”
“What is wrong 93173480 ( 323) made the timer deploy each main sha once, recorded in $ROOT/main seen . That is right, and it is what 298 asked for — it ended the flap where a failing sha was re switched every 60 seconds, restarting the crew each time. The tail of it is that a failed sha now produces exactly one mail and then nothing . main sits ahead of the running release, silently, until someone merges again — and whoever does inherits a base whose health check fails, without being told. How I”
“yeah the runtime/auth consolidation point people are making here is real, but for me the other axis that matters is blast radius, not just divergence. a todo app with an outdated dependency sitting there is basically a non issue. an app with expired or half configured auth, or a rate limiter thats not actually wired up, is the one that turns into a 2am problem. i keep a short list of just the stuff that can actually cause an outage or a breach (auth config, leaked secrets, CVEs in deps that touc”
“Everyone knows this loop. CI build takes four minutes instead of forty seconds, you scroll up, and step 4 is missing its CACHED . Great. Why? BuildKit won't say. The cache keys are internal and there's no flag to print them. So you guess. Lockfile? Base image? Did someone add a build arg? Is dist/ leaking into the context again? Twenty minutes later you either find it or you give up, because the build passed anyway and you have other things to do. That's the gap. BuildKit tells you t”
“From the human test pass ( 318), note N38. README.md:49 references docs/workbench plan.png . The file is at docs/images/workbench plan.png , so the front page of a now public repository shows a broken image where its one screenshot should be. One line, and the first thing a stranger sees. The badges reported alongside it in the same note are no longer broken: the CI badge returns 200 now that the repository is public, so that half needs nothing.”
“The project did not compile. This is what breaks the Windows build. | | | | | | | Commit | 4f236b6cd73a5322b4a940c33b7543fdee8fbadd | | Pushed by | @Shombith03 | | Run | https://github.com/froglet studio/Cosmic Shore/actions/runs/33577768887 | | Static validation | success | | Unity compile | failure (compile) | Open the run above for the failing output. A compile break prints its error CS lines inline and attaches the full editor log as an artifact. This issue closes itself automatically when b”
“Why is the GitHub link not in the post body? Afraid the automod will catch this presumably breaking several rules? Or just fearing that the automated repo checker complains? Which is it, hmm?”
“Same here, manage patching for a very large org (400k+ endpoints), and we used to patch just once a month, but thanks to the AI enabled vulnerability research, you just cannot do this anymore. Something like Google Chrome releases multiple patches every week, which remediate hundreds of vulnerabilities, many of them exploitable, so since the start of this year I've setup automatic deployment rules for everything to run three times per week. Patches go out to Ring-0 for three days, if nobody”
“I love optimizing and automating things. I have seen many labs (EDU sector) doing updates and deployments manually or wasting a lot of times doing things that can be automated in an weekend. I have previously worked in a lab and loved automating some of their processes. Have you seen "consultants" being hired to automate / optimize IT labs workflow? Maybe this is more of a career question as to if this is possible / hourly rate submitted by /u/imSeanGG [link] [comment”
“Set up release please to automate release pull requests and versioning from conventional commits.\n\nAcceptance criteria:\n Add and configure the release please GitHub Actions workflow.\n Keep unreleased changes in CHANGELOG.md until a release is cut.\n Update package and extension versions as part of the release PR.\n Document the automated release process. Include conventional commits (agent) docs”
“After I first set up a deployment pipeline, I would simply drop DB passwords and API keys into GitHub Secrets and feel completely safe because they are encrypted. I recently went through a security breakdown on GitHub Actions that showed me that it could be a mistake to think that way. The main issue is that an encrypted secret is still a static, long-lived target. Once the workflow finishes, that credential stays active indefinitely. In the breakdown, I saw a few default behaviors attackers loo”
“Sounds pretty interesting dude. The problem I find with stuff like this is that one bad file can wreck the whole batch, how does your tool handle a step failing partway through? Does it stop the whole pipeline or skip the faulty file and keep going?”
“Who knows what they're changing behind the scenes. Hopefully it won't take them long to fix it...”
“Summary The onboarding generated .git/hooks/pre commit (wrapper version 3) resolves the traceability linter through three independent absolute paths , baked at generation time: The hook is a per clone untracked file, but <workspace is whatever workspace ran the generator. Generate it in workspace A and the hook is broken in every other clone of the same repository — and it reports that breakage as a compliance verdict. Evidence Observed in gcd ops scripts today. The hook had been regenerated by”
“The code is in Git. The preview is in a browser tab. The useful output is in whichever session made the last change. When a teammate picks up a small SaaS project, they get a repo and a scavenger hunt. That's the problem we're building Agent Space around. It puts coding-agent sessions, project files, previews, and outputs in one cloud workspace. Claude Code, Codex, and other agents can work around the same project, and teammates use their own accounts. The tradeoff is that it is cloud-ho”
“That matches what we ended up doing too - kept the swap and teardown as two separate steps instead of one. New container comes up under a separate name (not a straight up -d , since that just recreates in place), we poll its healthcheck, then only swap traffic and kill the old one once it's actually green. Old container stays around briefly as the rollback path if anything looks off right after the flip. The canary-style extra check before swapping is the piece we don't have yet though -”
“I once read a report about the tech behind WhatsApp (before Facebook). They said that at their scale, no meaningful test environment could be built. So they rotated new code slowly into production and watched the telemetry. WhatsApp was famously efficient. They ran with 50-ish staff and maybe hundred servers and served 750m users with that.”
“If you want to ship an app but allow the underlying dependencies to update independently of it, good luck. You're into a whole world of pain when something breaks your app underneath you. There's a reason why people deploy using Docker, which is a closure of the dependencies your app supports....”
“dont rebuild it, make it fail loudly instead. one missing config row made every send in one of my environments return an error, and the code itself was fine.”
“Cross referenced from felsenuboot/felcrew mcp 1 (alliance channel) — this is a goodwill find from reviewing your public repo, not a complaint. Commit ea9102ce added a Discord status routing import — import { postStatus, ... } from './discord.mjs' — but cave/discord.mjs was never actually committed (it's not in the tree and not gitignored). A fresh clone of your runner will die on module not found the moment it tries to load that import. Requested action: either commit the missing discord.mjs fil”
“Depends which of the four you want to talk about. Reliability and scalability: sooner than most people think, because they're verifiable. Fuzzers, load tests, CVE repro, SLO regressions, crisp reward signals are where these models compound. I'd guess a lot of that is largely automated within 3-5 years for well-instrumented codebases. Cleanliness: never in a one-shot sense, because there's no ground truth. Clean for a 5-person team shipping weekly is not clean for a regulated enterpri”
“The merge process in .claude/skills/github commit process/SKILL.md is correct on paper and expensive in practice. PR 82 is the worked example — one PR, 10 hours of elapsed wall time, six review rounds, four separate approvals requested from the human, and still unmerged at time of filing. This issue is about the process , not about 82's content. The findings the reviews produced were real and worth having; the cost of collecting them is the problem. What actually went wrong on 82 1. Every fix co”
Debugging using logs is a basic skill, independent of leetcode.
“were on 50+ microservices on gcp, still doing our own deploy tooling. keep hearing gitops is the way and honestly cant tell if thats real or just the current hype cycle. not looking for a sales pitch, more curious what broke for you after the demo phase. drift detection, secrets, rollback under load, whatever. did it actually reduce incidents or just move the pain somewhere else what would you tell yourself before adopting it submitted by /u/sereikis [link] [comments]”
“disclosure: i cofounded appnigma, we build native salesforce integrations, so this is the part of the job that never shows up in anyone's project plan. no link, just the thing i wish someone had explained before our first release cycle. everyone budgets for the build. almost nobody budgets for the fact that salesforce ships three major releases a year and each one can quietly change how your package behaves in a customer's org. the bit that catches teams out is the preview window. sandbo”
“We deal with this pretty regularly when taking over project from clients. The AI-generated ones are usually the trickiest: the code works, but no one knows why certain decision were made, and there are no tests to tell you what you're going to break. Before touching anything, we usually document what the system is actually supposed to do. Not the code itself but the expected behavior. Once everyone agrees on what "normal" looks like, the whole thing gets a lot less scary.”
“Yeah the Drive split is the part that would wreck me. A Gmail per project feels tidy until Google treats the cluster as a farm, and then every doc you parked in that account is gone with it. I'd get anything still open exported tonight, keep one real Google identity, and just use the domain inboxes you already have for project mail. Outlook being fussy is a one-hour IMAP fight. Losing a 20-year-old address that a bunch of logins still point at is the thing you don't get back. Also don”
At this point it almost seems wise to use an AI to audit FOSS for you.
“I agree with all of your post. That’s where I think the buy decision on tools like this makes sense though, you get a team of ~300 eng working every day on improving the system vs an internal team who maintain a less polished version side of desk. Honestly I think until recently it probably wasn’t worth it but AI tools are finally getting to the point where they are genuinely good and the concerted effort/product thinking starts to really pull ahead of internal stuff. Appreciate the thoughts 🙏”