Developers Struggle with Database Management
Developers are experiencing significant pain related to database versioning, performance, reliability, and tooling. They face challenges with schema management, data loss, slow queries, and a lack of robust, user-friendly tools to simplify database workflows. This often leads to repetitive tasks, data loss, and frustration.
SOURCES (60)
“I work on the insights team at TouchBistro (a Canadian restaurant software company), and I led a project to pipe restaurant data from a sharded postgres operational database into Redshift using Debezium. I found articles on the web which talked about Debezium and CDC more…”
“Every backup thread ends with "an untested backup is not a backup," everyone upvotes, and then (in my experience) nobody has an automated restore test anywhere. Genuinely curious about the state of practice: Do you restore-test on a schedule, or only when something breaks / an audit demands it? If you automated it, what did you build? (ephemeral instance? scripts like pgbackrest_auto? CI job?) If you didn't, what stopped you? (time? nowhere safe to restore to? nobody asked?) For th”
“> The only way you can have schemaless data is by never reading it Aha! I now get why it's the popular choice for data lakes!”
“jesus christ, how do people have opinions like these and hang out here? no-sql is not no-schema you need schema its a lot more flexible, you don't need to think of everything ahead of time, you can make changes without having to migrate data or backfill it nosql is designed for fast retrieval, for large unstructured data which you can create indexes to create views in any which way. no-sql has none of the overheads of an rdbms which is overkill for the vast majority of saas use cases. you do”
“Schema less is used when you don't want the DB to handle the schema. The database does not require every record to conform to one centrally enforced schema before it can be written. The structure still exists somewhere, it's simply that the DB is no longer the sole owner of that structure nor responsible of enforcing it. If you endup putting all of the responsibility of the structure in the DB, then it's a bit weird. I generally use schema less db (mongo), outside of indexes everythi”
“So now you're introducing even more processes, even more latency and complexity, and for what benefit?Eventual consistency on users would make it difficult to do things like "WHERE !user.is_banned" when getting the stories, so you need to keep your users database tightly synchronised. Sure, you could pro-actively delete or mark comments when deleting users, but now you're risking having the ban itself fail, and have also now added a much longer write operation, as you have to mark all those comm”
“I went through the exact same thing — spent way more time trying to get databases to talk to each other and keep the structure “right” than actually using the workspace for anything. What changed things for me was hooking up Claude directly to my Notion. Instead of me manually maintaining relations, views, and cleanup, I just describe in plain language what I want done — “find everything that’s overdue and hasn’t been touched,” “merge these two databases,” “clean out anything I haven’t opened in”
[Hiring] Long-term n8n + Postgres build for a 50-state medical consulting practice. Multi-tenant, self-hosted, runs for years
“however you can create an empty "template" database that has all your settings and duplicate it instead of creating new databases.”
“More than 1 - having indexes/keys for all major SQL lookups in sync with their query structure”
“I've been comparing PostgreSQL schemas for so many years now. I've been using all kinds of frameworks for ORM and code-first schema generation, but getting the actual "diff" between what I have and what I want has always been a pain.For a long time, I've relied on a customized version of migra (https://github.com/djrobstep/migra). It was the best tool of all those I've tried, but it's unfortunately deprecated and no longer maintained.I kept dreaming of building my own comparison tool - one with”
“Hey all, Trying to figure how people actually manage DB credentials for apps at scale. Our current setup works, but kinda fragile: Liquibase runs DDLs using shared creds pulled from Parameter Store. A custom Jenkins shared lib provisions dedicated per app creds at the SQL level and drops them into Secrets Manager. Apps pull from there and connect. The pain - no visibility into what uses what and it's forward only, nothing cleans up when service is decommissioned, stale SQL users and secrets”
Rolling releases and SQL
“I've been working on incremental indexing pipelines lately, basically keeping a vector store in sync as the source data changes, and I keep finding the same bugs never show up until it's been running a while. Biggest one for me is deletes. I tested the "new doc comes in, gets embedded" path a hundred times and it was fine. Never really tested what happens when a doc gets deleted upstream. Turns out if you don't handle that, your index just keeps growing with stuff that shou”
“Hi there! I moved our multi-node task orchestrator (something like Airflow but lighter) from periodic polling to PG notifications to save resources. The implementation was straightforward: When the DAG is triggered, enqueue the run and notify within the same PG transaction. A listener process then receives the notification and claims the run (with a skip lock). Now I wonder if I am missing something or what caveats might arise from using PG notifications? Especially when scaling this kind of arc”
“I am working as Data Migration Analyst and i have gotten into a headache of testing if the migration is completed successfully . 14M master data in source and ~ 60 tables , almost 80 tables in target , 20 staging tables . Not a 1 to 1 migration, having crazy transformation going on . Cant rely on any tools , currently manually writing queries to verify . Is there is anything i can do with AI ? I am not looking for simple answers like ‘use AI to generate queries’ or ‘use Ai to optimize queries ‘”
“When making changes or additions to tables in a data vault layout, how do you handle name changes? I'm trying to decide if version numbers vs. dates would be better for referencing. submitted by /u/SQL_Boss_Babe [link] [comments]”
“I’m trying to process and move millions of expensive rows with limited computer. It’s not a hard problem, add some joins, a few filters and a window function. But because compute is controlled by Eng, I’m working with limited resources and RAM. Anyways, my setup gives me a slow feedback loop, and everytime I think I have come up with a solution and my DAG fails I get so frustrated. How do you folks deal with this? submitted by /u/leaver_believer [link] [comments]”
“TL;DR: trying to size the "surrounding" engineering tax on shipping a table (not the transform logic itself). If you've got 30 seconds — even just answering Q1 with a rough number/ratio is genuinely useful. Full context below for anyone who wants it. For any new table (or schema change on an existing one), there's a set of engineering work that's separate from the transformation logic: Metadata — owners, descriptions, tags, lineage, kept in sync as schema evolves PII/maskin”
“It doesn't have too, it could be run locally with no externally managed database.”
“Hello everyone, would like to know if anyone did the ETL pipelines from denodo to fabric migration? How tough it is or what do I need to know before starting the migration. Like person I should be looking for expertise? Or any migration documentation I can follow. TIA submitted by /u/jkrm1920 [link] [comments]”
I am having an issue starting the Mysql server and some other permission-related issues in Linux (Kubuntu). If you would like to see my logs for debugging, send me a DM and…
“In postgres, I have a separate table that records stage (and other field) changes with each 15 min sync, if there are any at that time. I have other tables, too, that take snapshot data. Like revenue pipeline per month, etc.”
“After connecting my workspace via OAuth, I periodically call Notion's API to pull each connected database (Accounts, Contacts, Opportunities, Activities) and upsert the records into postgres. Relationships between records get resolved through the local database joins rather than extra API calls, so the charts query really fast.”
“how do you even sync that with postgres bc i keep running into errors with my notion databases”
“Hey, new to this. Was looking for a managed DBaaS solution - a little specific to India. Have independently done projects across Hetzner, GCP and AWS across RDS, PG etc. Any recommendations? Found this one called selfhost.dev but not sure how strong the community around it is eventhough it has got some pretty interesting tooling and MCP support. Wanted to understand better if solutions as such would help me submitted by /u/Sad-Line3512 [link] [comments]”
“Im building a project with lots of rows in my Postgres database, roughly 100k to a million. I use my local machine as dev data which I promote to prod on a remote vps. Currently my process to do this is just using DBeaver's export data feature to transfer the data directly between the databases but I find that it's taking a longer than it should. For example, Im currently waiting for 25 mins to transfer 80k rows where each row is fairly simple, one int primary key and a few simple proper”
“Lets say in the bronze layer you get a json and instead of a number you get a string. What about converting the column to string? I dont see the problem, if you do it with everything (universal type widening), then your system has the maximum flexibility and users can change things without annoying the data team! The user can just send the data they need and dont have to worry about waiting for schema changes etc. submitted by /u/Similar-Bug-350 [link] [comments]”
“I came across this on Hacker News and felt like I needed to share it with the dev community. The main point is simple: a lot of teams reach for extra databases, queues, search engines, caches, and services before they actually need them. This page lays out where Postgres is usually enough, and where you may actually need something else: https://postgresisenough.dev/ Postgres is not perfect for everything, but it is good enough for a surprising amount of real-world work. The more I build and main”
“reckon that ai code is fixin to lock up your database the second you get ten users. that is a query optimization issue, not a ux bug.”
“most people back up their Notion and just assume it will work when they need it. they never actually test the restore part. here's the catch. if your Notion has linked databases.. tasks connected to a project, a number that calculates from other rows, that kind of setup.. a normal export doesn't keep those connections. you get the text back, but the structure is gone. everything becomes separate, disconnected pages. so technically you have a backup. but the day you actually need it, you”
Supabase or Railway hit the sweet spot for early stage, honestly worth.
“size limit: depends which path you use. the web app (in-browser) caps at 500MB since it builds the backup in memory. the free CLI has no size limit.. it streams straight to disk instead, so it handles workspaces of any size. whole workspace: yes. restora backup --all backs up every database plus every page/subpage in one shot. the web app has "all databases" / "all pages" buttons that do the same thing, just subject to the 500MB cap there.”
“Well I am not sure how smart claude is in this matter but on some level dataverse is basically SQL and doing SQL shouldn't be that difficult.”
“What's the update here... have you found a workable solution? I've built a similar workflow from Granola into Notion and it works, but like u/Moodln_me says below the trickiest or weakest part of your described workflow is the matching to a project. In my workflow we just matached to client based on the domain of one of the event attendees. this isn't 100% reliable but about 90% of the time it works 100% of the time.”
“I've been wanting to deploy dawarich. I have a single postgres and redis container per "domain" that all my containers share. (My domains and basically internal, external, experimental). So I have 3 of each, which cuts down on the number of separate ones, but means I have to set up my own dbs fairly often, and obviously centralization has it's own issues like if it goes down I lose several services. Plus I can't use depends on in docker compose files.”
“I would probably great an “institution.base” base file that automatically pulls in members that have the property of belongs.contains(this.note.name) - or something along those lines. If you make it for one institution and get it going nicely, you can then make it a template for all future institution notes. I’ve done this in the past with project notes where I want a task to belong to a project, and then the project note has a base that dynamically lists its own tasks. But one note of caution,”
“Everything looks fine inside each tool like transactions go through and reports generate so nothing feels broken on its own but the moment data moves between systems it starts getting complicated since numbers don’t match exactly and small differences start showing up that need to be fixed manually even though it's not major it's enough to slow things down and make it hard to trust what’s there. We even tried tracking it end to end and realized how many times the same data gets touched o”
“Valid and built into the project, I've separated via RLS the user files, legal citations and dockets (where legal citations is in the millions). Thank you for the response, it's been a learning experience.”
“So for a past few days i had been thinking of a way to make browsing through databases much easier. SOOooo i made this. ( I know there are better ones out in the wild... but i wanted to learn and build one myself.) This versions allows you to execute sql commands. see their outputs, and also browse through databases, schemas, tables you have. Stack: • Backend: Python, Flask, Psycopg • Frontend: HTML, CSS, JavaScript So any feedback on how to upgrade the frontend page. (I am a good web developer,”
“Yeah. Imma give this a try again and let it run while I'm sleeping... And set it up to just backup my active database for next time!”
“you need to migrate 1 major release at a time, otherwise you probably won't make it, even with LLMs”
“I did the last update of my Postgres DBs with the following guide: https://helgeklein.com/blog/upgrading-postgresql-in-docker-container/ But I‘m not sure if you can do such a big step in versions in one go. You might need a version step in between.”
“Expand the replies to this comment to learn how AI was used in this post/project.”
“https://preview.redd.it/7hqyd8gbkjbh1.png?width=1032&format=png&auto=webp&s=9d9841807741a3d1271144740c89e1386dfaf4cf Does anyone know a good tool for sketching out schemas? I've already tried the usual recommednations - draw.io , plantUML, etc but they all seem pretty clunky/outdated and more multi-purpose than neccessary? So far this seems to be the only one that: - lets you input your schema, and have it correctly automatically generate the entries - keeps it simple, so you don”