SQL

Level: Intermediate Data Engineering Tech Stack: dbt · Databricks · Delta Lake · SQL · Jinja · hive_metastore Source Code & Practice Files: View on GitHub The Problem: Raw Data Is Useless In my beginner pipeline project, we learned how to ingest raw CSV data and clean it with Python. But there’s a critical question that project doesn’t answer: Once the data is ingested — then what? In a real company, raw data lands in cloud storage every day. A dozen different analysts and data scientists need to query it — each writing their own transformations, often with subtle differences. One analyst filters out cancelled orders, another doesn’t. One team reports revenue in cents, another in dollars. The result is a classic problem in data teams: everyone has a different number, and nobody knows who is right. ...

If you have spent any time in data engineering recently, you have almost certainly heard of dbt. It is on almost every data job description, it is the centrepiece of the modern data stack, and yet a lot of people still cannot give a clear answer to the simple question: “What does dbt actually do?” This post is that clear answer. The Problem dbt Solves Before dbt, here is how data transformation typically worked: ...