Data Engineers Pick Between dbt, Databricks, and Terraform Every Week

“So, my company has moved on from an on-prem setup with little to no governance or data modeling (each business domain had its own data silos, like uploading sheets to our SQL Server RDBMS) to a Databricks lakehouse. In the first projects, we kept the bronze and silver layers source-aligned, without any integration or specific modeling technique, and we addressed business needs in the gold layer with star schemas. My question is: now that we have many tables referencing the same concept (customer”

“Posted in the Microsoft Fabric subreddit originally because that's the platform I'm using but I'd consider this more of a conceptual data engineering question. Also what resources should I be referencing to learn these patterns? It can't just be the blind leading the blind! How does your team manage your Silver environment if there are intermediary tables between processes? We are currently migrating legacy processes to medallion which is why things are a little contrived. Our So”

“I got into a debate at work today about how Unity Catalog should be organized, and I'm curious what others are doing in production. Which approach do you prefer? Option 1: Catalogs by medallion layer bronze silver gold Then have separate catalogs for curated data products or analytics built from the Gold layer. Option 2: Catalogs by business domain finance operations hr Then each catalog contains schemas like bronze, silver, gold What are the strongest reasons to choose one approach over the”

“Hello all In my job I have a requirement to read the content of iceberg tables. Only issue is my work systems have C#. There is no native library to read the content of Iceberg data table. Is there a way to read Iceberg tables in C#. I am totally new to working with open source type of data formats so hoping I would the required help here. TIA submitted by /u/ConsistentString4627 [link] [comments]”

“Hi guys just wanted to check peoples thoughts on this, currently my team is suggesting using DABS for catalog creation but I feel like just because DABS can create Catalogs, we shouldn't be spamming it for everything and I feel like Terraform should own it. Happy to hear people's thoughs on this and you are all approaching this - Happy to learn!!. Also I read a blog post of a very famous persona in Databricks addressing this, attaching the link below, but teams says it's 2 years old.”

“Hi Everyone - We could use your help shaping the bulk copy API in the mssql-python driver. We are extending it to take take support Apache Arrow directly and need your input. The primary question is whether Arrow ingestion should be exposed as: A dedicated bulkcopy_arrow() method, or An enhancement to the existing bulkcopy() API that accepts Arrow sources in addition to row iterators. Here is the discussion thread to participate directly: API Review: Apache Arrow Support in Bulk Copy · microsoft”

“What SQL or data engineering project do you consider unusually interesting and fun to work on? I recently work on a project that kinda like creating a web application with only SQL as the backend and online dashboard as its front end. It is easy to version the SQL models but hard to handle the dashboard part because it's not a codebase. The only downside to this is that you cannot write directly into the app. You have to write/rewrite the data (or the app "settings") through a spre”

“I built a lightweight WMS system in browser using Google Appscript as basically a front end with a (somewhat) friendly ui for what is essentially under the hood just a spreadsheet workbook. This was a natural evolution of baby steps from a warehouse that was previously operated with hand written bin labels for a e-commerce 3PL company. My system is working well and I can introduce new features quickly, but as the business needs are scaling, I’m finding that the time to make this a big boy data e”

“Hello guys, I am trying to gather some benchmark statistics for various technologies used to perform data integration on prem and on cloud. To start off if I had a delimited file with 150 million+ records, what would the run time be like if the etl process would read and aggregate data to get counts, averages and sums and then write them to a file or load them to a oracle database. I am trying to figure out what data integration technology would you use and what time it would take to load data f”

“I saw this post a few years ago and agreed wholeheartedly with the issues and cost of Fivetran but I also understood that not many data teams have the bandwidth to manage ingestion from scratch on top of, or rather underneath, the data modeling effort. Today it seems like you can get it all with Databricks Lakeflow Connect. I’m a solo engineer at an e-commerce/retail company now (started a month ago) and so far I’m getting all the data I need through connectors that Databricks manages, community”

“I concur, I posted some links to resources in my comment, but the subreddit needs comments with links to be approved by mods. There's an adonet provider to use duckdb from c#, and they could also interact with low level stuff, but it would be more complex to implement.”

“I believe there is no official duckdb client for C# but I would bet someone has an open source one you can use Basically just use duckdb to read the tables and get what you need from duckdb instead.”

“This is helpful. The file system I need to work with is like a remote file system like Azure Blob storage. I am able to see the each individual parquet file there. Can I use the libraries to connect to those file systems?”

“They are parquets with methadata, easiest way to read them is through a data warehouse engine, locally you could try setting up something like duckdb and querying the tables through sql. https://duckdb.net/docs/getting-started.html https://duckdb.org/docs/lts/core_extensions/iceberg/overview”

“Define the semantic layer in a yml. Point a Claude agent to the semantic layer to create queries & build a dashboard (if your BI tool supports it). Now you’ve “delivered value” or done “AI enablement” or done “LLM ops”; change job title on LinkedIn to AI & Data Engineer; rinse and repeat. Blame any bugs on LLMs. Kinda being facetious, but not far off from some real-life scenarios :)”

“Imagine the faces on these guys when they find out that AI is more expensive than humans”

“The example xlsx name used isn’t even that bad tbh it’s shitty but I’ve seen worse Proper struggle is when multiple people are working off the same file named the same thing just saved in multiple locations or god forbid, offline. At least your example name indicates a different version was saved that’s like a step up from the trash I’ve seen”

“In a perfect world that would be great. We did but at this point we had to have them make the decision and come to an agreement. Unfortunately some companies are full of egos.”

“Meeting with all three of them to sync changes so they maybe understand that they affect each other.”

“Replace that with different versions of PowerBI that users keep changing requirements for because all three think that they are the “core” user.”