DevOps Teams Fighting Flaky CI/CD Pipelines

“isn't that what antigravity cli does? i think your best action would be to generalize your product. include AWS, Azure. off the top of my head i feel like intergrating Github Actions would also be a good idea since it's gonna be one tool for all for monitiring. [re: Who needs research right?🤧]”

“ughhh the sync drama is the worst.. your canary note hack sounds like something a guy who's good with his hands would come up with [re: “Merging Files” Undid Hours of Work]”

“Yeah, that surprised me too. The code was the solved part. The moment you touch someone else's platform, Gmail scopes, a review queue, a hard user cap, the hardest problems stop being things you can fix by writing better code. You wait, or you design around the limit. Good reminder that "it compiles" is maybe half the work on anything that integrates with a big provider. [re: I built a native macOS Gmail client (not a web wrapper) — tech stack + the one constraint that reshaped the”

“Summary There is no first class way to ask "does this test actually catch the bug?" — the check that separates the tests pass from the tests mean something . Every reviewer in the scientific flow increment hand rolled it as a four step ritual inside exec , and the ritual is both slow and unsafe. The ritual, as performed Done four times by one reviewer in a single task review, and again by the controller. It worked — it is how a real defect was found (below) — but it costs four tool calls per que”

“You can waste a ton of time patching everything one day, for it only to reset the next day. It really comes down to your risk appetite and compliance requirements. Automate as much as possible is your answer. Platforms like Action1 can do a huge amount of OS and third party patching. CVSS 7.0+ is what i aim to be patched in <14 days. Zero day is <48 hours. Vuln dashboards can be such a trap too.. just focus on your patching principles and work towards that. Lots of other sysadmin work to d”

“Gitea and more recently Forgejo; I can actually have private repos that are truly private as opposed to being private until a Microsoft exec wants to take a peak [re: Which small self-hosted service has stayed useful after the novelty wore off?]”

“Exactly. We had this problem for ages. We started using a platform that puts vulnerability data together with the wider security posture instead of just giving us another giant list to work through. Cybaverse has been pretty useful for that side of things, mainly because the team can actually see what needs attention rather than just staring at severity scores all day. [re: Anyone else feel like vulnerability management creates more work than it solves?]”

“I know some systems make it easier than others - but I WASTE a ton of time on these. For example - systems with "untrusted" certs. We have a few systems that require self sign certs - and it is annoying that our scanner calls these untrusted, and marks it as a high/critical alert. Its a web portal that a dozen people use, and the cert is pushed to their system. I know many VASs alow me to turn off a specific finding, but not the one I'm using. Luckily, the scope of scanning across”

submitted by /u/dani-but-g [link] [comments]

“yes, that is basically the point. the "less crowded" part comes from stacks where webstack devs are going to have the most friction to expand into. webstack dev itself is mostly automated even with vague prompting so it is seeing pressure from business analyst-type folks vibe coding low scale internal tools and prototyping by themselves. this is the sky actually falling on us-> we used to have an economic moat just by virtue of understanding computer programming languages and basic”

“The repo gets a Bun TypeScript project with typecheck, test, lint and format scripts, using Biome for lint and format. CI runs all four on every PR and on every push to main , and fails on any finding. Each job has a timeout of about 10 minutes, so a hung job cannot hold a runner. A concurrency group per branch or PR cancels the in progress run when a new push arrives, so only the latest commit uses CI time; runs on main are not cancelled. prek hooks run a light version locally: pre commit forma”

“A database function checked "has permission OR is creator OR is assignee" and raised an error if not. Looks fine. The app had 3683 auto-seeded tasks and all of them had no creator and no assignee, both NULL. In SQL false OR NULL OR NULL is NULL, not false. The IF skipped the error. So any logged in user, even a role with every permission off, could clone tasks. What got me is that the same condition in an RLS policy is safe, there NULL means deny. Fix was a coalesce to false around the”

“Before Submitting [x] I searched open and closed issues and discussions for an existing report. [x] I reproduced this bug on the latest release AND on the current dev branch, right before submitting this report. I did not just check an old version or rely on a check from days ago. [x] I understand that maintainers want a well written issue before any code pull request. [x] This is not a security vulnerability. Installation Method Docker Open WebUI Version v0.12.3 (same code on dev at 079fb19) Op”

submitted by /u/ElyeProj [link] [comments]

“Soon! I didn't want to containerize before having a decent working version which I consider I only achieved on the last release [re: Gofer, a multi-account webmail]”

“The vendors break standards because they can. They bet you'll write the wrapper code instead of switching. The new guy who does the OIDC spec correctly wins by default because the switching cost is now zero. [re: How much would you pay for an IdP that doesn't suck?]”

“Why build an internal harness instead of just use Claude Code, Codex, or OpenCode + GitHub + connectors or just CLI access to your various context silos like Jira, Google Drive, etc.? [re: My company's internal AI buildout is a disaster. Is this happening with anyone else?]”

“Everyone keeps mentioning a book but I more just meant a set of principles that makes code easy to read, debug, extend, maintain, etc. Basically writing code that makes the developers who have to work on it after you not hate you. [re: What’s up with brilliant techies being flat broke]”

“Dozzle for a quick glance, terminal/Ansible for actual work. I'm yet to see the benefits of using a management tool. I've tried Portainer ages ago and it was more confusing than helpful. [re: Best Docker management tool?]”

“Do you genuinely think a person who has been living in a cave never using git or an IDE, can in just one week get up to the same level of competency as person who has been using it every day for the last five years? No, and that's not just because it's "a skill issue". [re: Are you behind if you don't use AI?]”

“The site guard found problems on main (commit 08f6a03). Run: https://github.com/a alzoubi 07 11/a alzoubi 07 11.github.io/actions/runs/39060543959 Fix the listed files (or revert the commit). This issue closes automatically when the guard is green again.”

“Filed from a repo wide audit of main at 5c46185 . Every claim below was produced by running the quoted command against a clean clone — not inferred from reading code. Reading order and wave dependencies: docs/refactor backlog.md . Wave 5. Finding The README makes a specific, checkable promise: Verify it yourself in 60 seconds No toolchain needed — just curl . 1. The accepted proof is a real, successful testnet transaction: curl s https://horizon testnet.stellar.org/transactions/<TX HASH | grep E”

“possible, but the code isn't the hard part of a huge company. distribution and understanding the problem deeply are. the real risks of your setup are the things you can't judge breaking quietly: security holes, data loss, a database bill that spikes. you won't see those until they're expensive. practical stuff that buys you time: have claude write tests for the important flows and actually run them before each change goes live. learn to read code even if you never write it, you c”

“yep part of the job but it also keeps you in a job if no one else knows how to do it [re: Is "inheriting someone else's undocumented install" the part of the job worth building a game around?]”

“Nah I have been doing what you claimed was impossible for at least 7 years in our openstack cloud [re: If, Cisco [or insert your vendor here] releases a 9.8 CVE, nobody panics, because it's all 'part of the plan.' But when I say that Fortinet has one, well then everyone loses their minds!" -Joker who works in InfoSec]”

vscode.dev and a repo? [re: Self hosted docker development environment?]

“Solo dev, so there's no team limit to enforce, but the same problem shows up when an agent writes most of the code. What helped me was changing what a review consists of rather than fighting the line count. Copilot's PR review goes first and takes the mechanical stuff (it caught a paid feature in my app that never checked the subscription). Then I read the diff next to the agent's session log, then the tests it wrote, then I use the feature by hand. The session log is the part people”

“This is true but a lot of that history and culture has not been capable of producing software which has better quality than what came before. Around here banks are spending hundreds of millions trying to exit mainframe systems, and they've been failing for the better part of a decade. This despite having hundreds of people working on "modern" software paradigms in "modern" languages.I do think the capabilities of AI depends a lot on how you work. If you're doing very low abstraction, fail fast d”

“Yeah, the repo can be yours on paper and still be a black box in practice. I learned this the hard way when a contractor left and the deploy keys, cron jobs, and one weird migration lived only in their head, not the README. [re: If your developer is the only one who knows your codebase, you don't own your product]”

“Cool, it takes potentially over a year for software companies to publish some changes. Still a short time frame for many companies and software vendors, especially if they're on an older version of the language or libraries they use. [re: Google Workspace (GMail) doesn't understand the new BC time zone?]”

submitted by /u/Dubinko [link] [comments]

“that build log looks like it's choking on something in the dockerfile build context, not an AI issue. quick things to check: is there a node_modules or .git folder in your build context that's not in .dockerignore, that'll blow up transfer time and sometimes hang steps like that silently. [re: Contrack 2.0 – Lightweight contacts manager and people search]”

“Before Submitting [x] I searched open and closed issues and discussions for an existing report. [x] I reproduced this bug on the latest release AND on the current dev branch, right before submitting this report. I did not just check an old version or rely on a check from days ago. [x] I understand that maintainers want a well written issue before any code pull request. [x] This is not a security vulnerability. Installation Method Other Open WebUI Version latest Operating System macos 26 Browser”

“So, in this podcast episode with Patrick, talking about how DevOps is not longer focused on cloud but on "software factories", I'm starting to form an opinion. I guess this shift is happening all over the place and my recommendation is to get familiar with the business decisions around that as deploying a software factory needs somewhat different mandate from stakeholders compared to normal cloud strategy. It goes with its own observability, security and continuous delivery practic”

“1 viable/strict jobs have been broken on Trunk for at least three commits in a row. Please investigate. Alert Details Occurred At : Oct 16, 10:50am PDT State : FIRING Team : pytorch dev infra Priority : P2 Description : Detects when trunk has been broken for too long Reason : [ var=Failure Threshold labels={} type=threshold value=1 ], [ var=Number of Jobs Failing labels={} type=query value=1 ] Runbook : https://hud.pytorch.org Dashboard : https://pytorchci.grafana.net/d/e9a2a2e9 66d8 4ae3 ac6a d”

“1 viable/strict jobs have been broken on Trunk for at least three commits in a row. Please investigate. Alert Details Occurred At : Oct 16, 10:50am PDT State : FIRING Team : pytorch dev infra Priority : P2 Description : Detects when trunk has been broken for too long Reason : [ var=Failure Threshold labels={} type=threshold value=1 ], [ var=Number of Jobs Failing labels={} type=query value=1 ] Runbook : https://hud.pytorch.org Dashboard : https://pytorchci.grafana.net/d/e9a2a2e9 66d8 4ae3 ac6a d”

“There are tried and true methodologies to accomplishing this. Make sure when you do bring on developers that they've had experience working on your tech stack at the target scale. Will save a ton of headaches later. [re: The hardest part of building software isn't the coding]”

“1 viable/strict jobs have been broken on Trunk for at least three commits in a row. Please investigate. Alert Details Occurred At : Oct 16, 7:50am PDT State : FIRING Team : pytorch dev infra Priority : P2 Description : Detects when trunk has been broken for too long Reason : [ var=Failure Threshold labels={} type=threshold value=1 ], [ var=Number of Jobs Failing labels={} type=query value=1 ] Runbook : https://hud.pytorch.org Dashboard : https://pytorchci.grafana.net/d/e9a2a2e9 66d8 4ae3 ac6a db”

“1 viable/strict jobs have been broken on Trunk for at least three commits in a row. Please investigate. Alert Details Occurred At : Oct 16, 7:50am PDT State : FIRING Team : pytorch dev infra Priority : P2 Description : Detects when trunk has been broken for too long Reason : [ var=Failure Threshold labels={} type=threshold value=1 ], [ var=Number of Jobs Failing labels={} type=query value=1 ] Runbook : https://hud.pytorch.org Dashboard : https://pytorchci.grafana.net/d/e9a2a2e9 66d8 4ae3 ac6a db”

“A recent update to this github action broke the copyright check. Here is an example : https://github.com/qualcomm linux/qli ci/actions/runs/38771067585/job/116980529535”

“Our CSPM rollout is blocking developers from shipping because a test bucket with no data now needs a full incident response ceremony. We wanted guardrails, apparently we got a committee in CI. Been looking at Orca for this, mainly because it scans IaC on the PR but only flags new problems the change introduced, so devs aren't gated on stuff that was already there. How are you tuning this without disabling everything? 🙂 submitted by /u/Wide-Tank-1bloyaem60 [link] [comments]”

“Before that, a bit of story. Our dev team was located in Ukraine, but we mostly worked on New Zealand & Australian projects. We needed a couple of hours to download staging DB dumps because the download speed was 200-400 kb/s. So we set up a small proxy on our European server, and via cron these dumps were uploaded to that server, and we were able to download them faster. After the team scaled, we added more protection, like only developers who worked on the project having access to these du”

“For the dependency issue, start documenting everything in a simple confluence page or even just a spreadsheet. Every time someone mentions a dependency you didn't know about, write it down with context. After a few sprints you'll have your own reference guide that's actually useful instead of relying on tribal knowledge that nobody wants to share properly. [re: Project manager turned solution manager. Any tips?]”

“Before Submitting [x] I searched open and closed issues and discussions for an existing report. [x] I reproduced this bug on the latest release AND on the current dev branch, right before submitting this report. I did not just check an old version or rely on a check from days ago. [x] I understand that maintainers want a well written issue before any code pull request. [x] This is not a security vulnerability. Installation Method Other Open WebUI Version Open WebUI version: 0.12.0 (ghcr.io/open”

“Good find. This is a good example of how a change on the server side can expose an existing issue in a client library, even when nothing has changed in the mailbox configuration. For anyone else running into this with node-imap, I'd recommend checking the library's issue tracker for a related fix and updating or patching the library if needed. If the same error occurs with other IMAP clients too, then it's worth checking the mailbox's IMAP settings and authentication configuratio”

“Most clean? How much did you use it? Its a buggy mess. edit: Unless you're being sarcastic about the reliability of the original software. [re: This week I resigned after 28 years]”

“Some of what you are describing are bad products, rather than buggy/insecure products - and the distinction is important. That said, one thing that’s changed since the examples you’ve given is that software has gone from versioned, manually updated, and largely opt-in (with the exception of G+, but this was more bad product than buggy) to continually and automatically updated. That is to say what worked well today may cease to continue working well tomorrow. Additionally companies that may have”

“Supersedes 32475, which was closed by a maintainer. GitHub does not let an issue's author reopen an issue that someone else closed (the API answers 422), so the evidence requested in the close message is filed here instead, and a comment on 32475 links to this report. Before Submitting [x] I searched open and closed issues and discussions for an existing report. Related: 29628 / PR 29697 (chat completions route), 31403 / PR 31405 (Anthropic conversion route), 31326 (provider status and Retry Aft”

“If you were a junior or a hobbyist or someone who never really took an interest in basic standards, and then Claude suddenly sent your code quality through the roof and produced less bugs than you, you are not who we should be comparing Claude to. Respectfully, I have to disagree. We compare other AI to how well it performs compared to the average human. That's how we evaluate autonomous vehicle safety, whether a robot is ready to replace a factory worker, etc. I think it's perfectly fai”

“1 viable/strict jobs have been broken on Trunk for at least three commits in a row. Please investigate. Alert Details Occurred At : Oct 15, 12:10am PDT State : FIRING Team : pytorch dev infra Priority : P2 Description : Detects when trunk has been broken for too long Reason : [ var=Failure Threshold labels={} type=threshold value=1 ], [ var=Number of Jobs Failing labels={} type=query value=1 ] Runbook : https://hud.pytorch.org Dashboard : https://pytorchci.grafana.net/d/e9a2a2e9 66d8 4ae3 ac6a d”

“1 viable/strict jobs have been broken on Trunk for at least three commits in a row. Please investigate. Alert Details Occurred At : Oct 15, 12:10am PDT State : FIRING Team : pytorch dev infra Priority : P2 Description : Detects when trunk has been broken for too long Reason : [ var=Failure Threshold labels={} type=threshold value=1 ], [ var=Number of Jobs Failing labels={} type=query value=1 ] Runbook : https://hud.pytorch.org Dashboard : https://pytorchci.grafana.net/d/e9a2a2e9 66d8 4ae3 ac6a d”

“The site guard found problems on main (commit 3d6734b). Run: https://github.com/a alzoubi 07 11/a alzoubi 07 11.github.io/actions/runs/38865487993 Fix the listed files (or revert the commit). This issue closes automatically when the guard is green again.”

“1 viable/strict jobs have been broken on Trunk for at least three commits in a row. Please investigate. Alert Details Occurred At : Oct 15, 10:35pm PDT State : FIRING Team : pytorch dev infra Priority : P2 Description : Detects when trunk has been broken for too long Reason : [ var=Failure Threshold labels={} type=threshold value=1 ], [ var=Number of Jobs Failing labels={} type=query value=1 ] Runbook : https://hud.pytorch.org Dashboard : https://pytorchci.grafana.net/d/e9a2a2e9 66d8 4ae3 ac6a d”

“1 viable/strict jobs have been broken on Trunk for at least three commits in a row. Please investigate. Alert Details Occurred At : Oct 15, 10:35pm PDT State : FIRING Team : pytorch dev infra Priority : P2 Description : Detects when trunk has been broken for too long Reason : [ var=Failure Threshold labels={} type=threshold value=1 ], [ var=Number of Jobs Failing labels={} type=query value=1 ] Runbook : https://hud.pytorch.org Dashboard : https://pytorchci.grafana.net/d/e9a2a2e9 66d8 4ae3 ac6a d”

“1 viable/strict jobs have been broken on Trunk for at least three commits in a row. Please investigate. Alert Details Occurred At : Oct 15, 7:40pm PDT State : FIRING Team : pytorch dev infra Priority : P2 Description : Detects when trunk has been broken for too long Reason : [ var=Failure Threshold labels={} type=threshold value=1 ], [ var=Number of Jobs Failing labels={} type=query value=1 ] Runbook : https://hud.pytorch.org Dashboard : https://pytorchci.grafana.net/d/e9a2a2e9 66d8 4ae3 ac6a db”

“1 viable/strict jobs have been broken on Trunk for at least three commits in a row. Please investigate. Alert Details Occurred At : Oct 15, 7:10pm PDT State : FIRING Team : pytorch dev infra Priority : P2 Description : Detects when trunk has been broken for too long Reason : [ var=Failure Threshold labels={} type=threshold value=1 ], [ var=Number of Jobs Failing labels={} type=query value=1 ] Runbook : https://hud.pytorch.org Dashboard : https://pytorchci.grafana.net/d/e9a2a2e9 66d8 4ae3 ac6a db”

“1 viable/strict jobs have been broken on Trunk for at least three commits in a row. Please investigate. Alert Details Occurred At : Oct 15, 7:10pm PDT State : FIRING Team : pytorch dev infra Priority : P2 Description : Detects when trunk has been broken for too long Reason : [ var=Failure Threshold labels={} type=threshold value=1 ], [ var=Number of Jobs Failing labels={} type=query value=1 ] Runbook : https://hud.pytorch.org Dashboard : https://pytorchci.grafana.net/d/e9a2a2e9 66d8 4ae3 ac6a db”

“1 viable/strict jobs have been broken on Trunk for at least three commits in a row. Please investigate. Alert Details Occurred At : Oct 15, 6:25pm PDT State : FIRING Team : pytorch dev infra Priority : P2 Description : Detects when trunk has been broken for too long Reason : [ var=Failure Threshold labels={} type=threshold value=1 ], [ var=Number of Jobs Failing labels={} type=query value=1 ] Runbook : https://hud.pytorch.org Dashboard : https://pytorchci.grafana.net/d/e9a2a2e9 66d8 4ae3 ac6a db”

“1 viable/strict jobs have been broken on Trunk for at least three commits in a row. Please investigate. Alert Details Occurred At : Oct 15, 6:25pm PDT State : FIRING Team : pytorch dev infra Priority : P2 Description : Detects when trunk has been broken for too long Reason : [ var=Failure Threshold labels={} type=threshold value=1 ], [ var=Number of Jobs Failing labels={} type=query value=1 ] Runbook : https://hud.pytorch.org Dashboard : https://pytorchci.grafana.net/d/e9a2a2e9 66d8 4ae3 ac6a db”