DevOps Teams Fighting Flaky CI/CD Pipelines
Developers and DevOps engineers are experiencing significant challenges with their CI/CD pipelines, including unexpected behavior, difficulty in debugging, lack of visibility into deployments, and increasing complexity due to wider adoption of development tools. This leads to unreliable releases, difficulty in troubleshooting, and pressure to deliver increasingly complex solutions.
SOURCES (60)
“I'm sure there's a formula you can make to fix this permanently.. could do something like using references and doing ='Admin Settings'!$B$1.”
“I do Salesforce releases regularly and got tired of finding out something broke after it hit production. Most of the failures weren't dramatic, no red errors, just things silently not working until a user noticed. So I started keeping a checklist. Sharing the ones that have saved me the most, in case they're useful to anyone here: 1. Confirm scheduled Apex jobs got re-queued after a sandbox refresh. A refresh wipes them. Everything looks fine until the job that was supposed to run… doesn”
“Hey all, I was checking github randomly and I saw this project called OpenShip https://github.com/oblien/openship Looks fairly new, i wanted to see if anyone had experience with it and what you think in general submitted by /u/matteozamuner [link] [comments]”
Our ratio of SREs to devs is so low that effectively we have none.
“I had the same question. Two types of cases could be a cronjob and a server with low request volume.”
“Dev team meetings are usually less about whether they like the idea and more about whether it's technically safe, reliable, and strategically annoying for them. I'd expect questions on rate limits, data ownership, failure modes, and what happens if they change the API tomorrow. Go in with a clean architecture story and a clear view on boundaries, that tends to calm everyone down fast.”
“Up until about 6 years ago, 50% of software development was just Googling what other people found solutions for.”
“That’s less than the fully loaded cost of hiring another engineer.If someone is making the case that it’s helping the team get twice as much work done without hiring more people (which I’m neither agreeing with or disagreeing with) then the bean counters would actually prefer it. Hiring people is messy and expensive. Spending on API costs is a dial that you can turn down later if you need to, without laying anyone off and paying severance.”
“Like probably everyone here, my Playwright HTML reports live and die as CI artifacts. A test fails on Tuesday, by Thursday the report is gone, and good luck remembering whether that exact failure already happened two weeks ago. I've been a web dev for about 18 years, and for the past few months I've been building Piwi Dashboard to fix this for my own team. Piwi Dashboard test run page It's a self-hosted dashboard and reporter (MIT licensed, runs as a single Docker container) where ev”
“Was it the last release? Was it only happening on iOS? Was the API suddenly slower? What happened right before the error? How many users were affected? Did an alert fire? Why am I opening another dashboard again? I’ve been asking myself these questions quite a lot lately. Every project I worked on eventually ended up using multiple tools. One for errors. One for analytics. One for releases. One for alerts. Another one for monitoring. After a while, I realized I was spending more time jumping bet”
“did u notice any patterns in where it got confused, like specific ui elements or just general flow issues. im curious if it flagged things u already suspected or if it was totally new info.”
“Been here a few times in various situations. The best thing to do is contact them and ask what they expect and what to prepare for to make it a productive meeting. Don’t get stuck in your head. Ask. And then be ready for every question they might ask as a backup.”
“March 2025. A personal access token in spotbugs GitHub Actions secrets. Normal setup — thousands of projects do exactly this. That token had write access to reviewdog. reviewdog was a dependency of tj-actions/changed-files — used by 23,000+ repositories. Three hops. Each one looked fine in isolation. SAST findings went to AppSec. SCA findings went to the dependency team. CI/CD findings went to DevOps. Every team closed its own list correctly. Nobody owned the line between the lists. The attacker”
“I started creating healthcheck endpoints on my APIs. I make any dependent APIs or resources, like databases, part of the healthcheck. If any of the resources set as dependencies fail, I have the API return a negative response, like a 500.”
“I'm curious how teams working across multiple Git providers manage their day-to-day work. If your organization uses GitHub + Bitbucket or GitHub + GitLab (or more), how do you: Track pull requests? Find stale branches? Monitor releases? Keep track of work across providers? Is it a pain point, or is it basically a non-issue? submitted by /u/Gaurav_0707 [link] [comments]”
When a version bump breaks your build, GitLab fixes it
“We generate SBOMs for every image as part of the build, which looks good on paper until an auditor actually starts poking at them. Half the time the SBOM reflects what was declared in the Dockerfile, not what's actually sitting in the final image after multi-stage copies and layer squashing. Packages get pulled in transitively that never show up cleanly, and attestations don't always carry over cleanly when an image gets copied to another registry or pushed through extra pipeline stages,”
“We're struggling with software supply chain security. Looking for automated vulnerability detection tools and sbom generation that don't need source code access. How are you securing software dependencies today?”
“Isolated hub per customer, own machine and own db each, how are you pushing out a fix once a load of them are live? say there's a bug in invoicing and you're at 150 hubs. that's 150 deploys and 150 migrations to run, not one. shared db it's a single migration and you're done. curious how you're handling the rollout side, at zero customers you'd never feel it but every release starts scaling with how many hubs you've got. or is there already something running the m”
“Azure Dev Ops still doest allow you to embed Microsoft Office documents. You have to find the SharePoint or OneDrive URL, or just attach it, like a chump. Basically...all of Azure Dev Ops tbh”
“Same for internal tools and smaller dependencies. I see no reason to not simply write your own. Less context switching for your developers, and if they need a new feature, build it instead of wait.”
“I'm wondering, isn't that literally just connecting these though MCP and ask about it everyday at 9? And I see no security concern because this can easily be done with a shit local model.”
GitLab Duo Security Review spots logic flaws scanners miss
“dev team missed another deadline again. The word "again" means it has happened at least one other time. Why, and why have you let it happen again? It's the whole fool me once thing. the scope creep was real. client kept adding just one more thing and the team kept saying yes without checking with me first. Not sure if this was the issue before, but this is a huge red flag for your team. I look at it like this. As a project manager, if my team accepted work without informing me, cre”
“so i keep seeing self-healing code talked about like its almost here. error happens, agent writes a fix, deploys it, no humans. and every time i hear it i think about our own numbers and go hmm our triage suggestions are right maybe 60% of the time, and thats with a decent setup. self healing means acting on that automatically. so 4 times out of 10 youre auto deploying a wrong fix, at night, with nobody watching. and a wrong fix isnt neutral, it usually patches the symptom and breaks something e”
“99% of the time, what I need is a basic hyper-focused tool to solve something specific. These days, I'd rather claude build exactly what I need and review that code, instead of what feels like existential risk of allowing rando third-party code (that, let's be honest, I'll never properly and fully audit every time there's a new version). Of course this isn't viable for full front-end frameworks, auth, etc, but now I basically hold the position that dependencies not written by”
“In enterprise settings, the engineering department is protected by the organization structure itself. Think about how a feature is delivered to users. The infrastructure the code is deployed on is managed by dedicated teams of SRE's and Sysadmins, every customer interaction is filtered through CSR's, Account Managers, and finally Product Owners and Project Managers before hitting the backlog. A disgruntled customer choose to pursue legal actions because bad code exposed their data? The l”
“Yeah, I decided to give VSCode a shot at it, but then found how locked down our VSCode extensions were and realized it's going to be a challenge either way.”
“lol yeah I’ve heard that before. Collectively we need to go back to a PMBOK 3 or earlier mindset and start advocating for classic project management and use cases like this to say if you want actual committed dates and budgets you need some PMM basics, and it starts with scope control. As it stands with you, if it were me I’d make this case and say by changing the scope without going through you they automatically change the schedule with no oversight or commitment. Yay Agile!”
“I need to vent, and maybe get some perspective to see if I'm just being a bad engineer or if this is actually as crazy as it feels. We have a relatively small product. A standard Python/Java backend, a Postgres database, serving maybe 10k daily active users. It was running perfectly fine. Fast, easy to debug, simple deployments. Then our CTO went to some tech conference, came back, and decided we urgently need to be "enterprise scale" and "cloud-native." For the last four”
“Yeah same, whenever someone new starts we just export the list of RDP sessions and import it to the other guy and change credentials. Our team is only 4 though”
“I've started looking at open issues before stars. A project with 5k stars and six months of un answered bug reports is a much bigger red flag than one with 800 stars and active maintainers.”
“Our team just each uses the personal versions, guess that is probobly against their ToS but we dont need any features other than a clean RDP interface with saved creds, I guess they could audit us? Haha”
“I just look at how many stars it has and when the last commit was, if it's been dead for 6 months I skip it. Also check the issues tab, if maintainer never answers or it's just bots reporting same bug over and over, that tells you everything. The slop is real but you develop a nose for it after wasting enough time.”
“wayland permission confirmation dialogs are why half team refuses to migrate. if agent bypasses that without custom polkit rules it might actually be usable.”
“Troubleshooting arcane Windows issues, copilot is a good resource, not a be all and end all, but still a good tool in your box.”
“Every place I've worked has had at least one. Mine right now is a ~40-line bash script someone wrote 'just for the migration weekend' about three years ago. It's still the only thing that reconciles two systems that were supposed to be fully merged by that Q2. Nobody wants to own it, everyone's a little afraid to touch it, and it has exactly zero tests. I'm curious what everyone else is quietly sitting on: the cron job with no owner, the one instance nobody can confidentl”
What is the mitigation for this now? pnmp with a min release age check?
“Does anyone know if there are any open source human language analysis tools for Code Review? As a hobby I am trying to write components to integrate every code analysis tool I can find with Gitlab (I want to be the bitnami of Gitlab CICD Components). I have been using AI for code review in work and one of the unique benefits has been the analysis of code/variable comments. Finding typo's in the comments Recognising a comment appears to be a duplicate from somewhere else Suggesting a comment”
“Good Q! My experience has been that RAID logs mostly capture what was decided, but less good on the reasoning, alternatives not chosen, or learnings. But maybe we're just doing it badly, that's what I'm trying to understand :)”
“We spent the last year shipping as fast as we could and the bill came due on secrets management. Right now they're everywhere: hardcoded in a few GitHub repos, sitting in .env files, baked into Jenkins credentials, and on at least three devs' laptops that I know of. It works until it doesn't, and I'd rather fix it before it becomes an incident instead of after. The goal is runtime injection so nothing sensitive lives in the repo or the CI config at all, but I don't have six m”
“I'm building a tool that catches flaky tests before they hit CI. My target users leave traces on GitHub and I want to find them all. Has anyone tried something similar? Is there even a tool for this? How did you find people? Does it actually work? submitted by /u/Competitive_Resist18 [link] [comments]”
“Are you talking to the right people in the company? I know some devops guys who would shut this down because they are afraid of losing their jobs, but the CIO would love it.”
“Is there any way to slow or stop the turnover? At least that seems like a related issue.”
“Coming from the land of the mono-repo (ex-googler), I decided to embrace the multi-repo, plain-git style. Methought, “hey, submodules, what could possibly go wrong”. Well, how about everything: detached HEADs, impossibly tedious commit management, and just about everything else. "OKAY" I said, a simple Python script should be fine, right? Well, NO. It turns out to be more complex. So I thought "others have got this, right? My time in mono-repo land was long and someone will have solved this" — w”
“Great article. Thanks for the write up! Excited to try these out. I organize a lot of my work with multiple branches off base branches, this sounds amazing for handling a tedious manual workflow of updating branches.”
“Hello, As a developer, I ship a lot of code with Claude Code, and I've used pretty much every security scanner out there. Most of them are great at telling you what's wrong. Then they leave you with a long report, and you're back to copying logs into Claude or Cursor, asking your AI to figure out how to fix everything. I wanted something that fit the way many of us build software today. So I built Merge Risk. It scans GitHub repositories for the security mistakes AI coding agents sti”
“I know this scares many people, but you can control things with QA, evals, ship quality gates and so on... I know we might have more bugs in the short term, but is not looking at the code the right direction we should go for the long term?Several important developers switched to this new idea, the ones who work in production grade stuff (one example is Antirez, who just wrote some tweets that inspired this post)”