Data Engineering

Level: Intermediate Data Engineering Tech Stack: dbt · Databricks · Delta Lake · SQL · Jinja · hive_metastore Source Code & Practice Files: View on GitHub The Problem: Raw Data Is Useless In my beginner pipeline project, we learned how to ingest raw CSV data and clean it with Python. But there’s a critical question that project doesn’t answer: Once the data is ingested — then what? In a real company, raw data lands in cloud storage every day. A dozen different analysts and data scientists need to query it — each writing their own transformations, often with subtle differences. One analyst filters out cancelled orders, another doesn’t. One team reports revenue in cents, another in dollars. The result is a classic problem in data teams: everyone has a different number, and nobody knows who is right. ...

If you have spent any time in data engineering recently, you have almost certainly heard of dbt. It is on almost every data job description, it is the centrepiece of the modern data stack, and yet a lot of people still cannot give a clear answer to the simple question: “What does dbt actually do?” This post is that clear answer. The Problem dbt Solves Before dbt, here is how data transformation typically worked: ...

Level: Beginner (Foundations) Tech Stack: Python · CSV · Clean Code Design Source Code & Practice Files: View on GitHub Welcome to Data Engineering! If you are just starting out in data engineering, you might be overwhelmed by tools like PySpark, Airflow, Snowflake, or Databricks. But here’s a secret: you don’t need any of those to understand the fundamentals of data engineering. Every enterprise data platform starts with one core concept: Ingestion. Before data can be analyzed, it must be loaded, cleaned, and verified. ...

When dealing with massive datasets in Teradata, standard JDBC connections often become the bottleneck. To move 2 Billion+ records efficiently, you need to bypass the SQL layer entirely and use Teradata Parallel Transporter (TPT). Why TPT? Standard SQL extractors pull data row-by-row through the SQL Parser and GDO (Global Distributed Object) layer — this is the primary bottleneck in any ODBC/JDBC connection. TPT uses the Export Operator, which bypasses this layer entirely and pulls data in blocks directly from the AMPs (Access Module Processors), enabling true massive parallelism. ...

When building data pipelines in Apache Spark, one of the most common questions is: “Should I write this in Spark SQL or use the DataFrame API?” The short answer regarding performance is neither. Both are powered by the Catalyst Optimizer, meaning Spark converts both into the same optimized physical plan under the hood. However, from a Data Engineering perspective, the choice significantly impacts how you build, test, scale, and maintain your code. ...

For over a decade, Apache Hadoop was the backbone of Big Data, allowing companies to store massive datasets on commodity hardware. However, as data volume and variety exploded, the limitations of Hadoop became a major bottleneck for modern data teams. The Problem: Why Hadoop is Fading While Hadoop revolutionized distributed storage, it introduced several technical and operational challenges: The Small File Problem: Hadoop’s NameNode often struggles with millions of small files, leading to performance degradation and memory issues. Storage-Compute Coupling: In a traditional Hadoop cluster, if you need more processing power, you are forced to buy more storage disks as well. This leads to inefficient resource utilization and high costs. Operational Complexity: Managing a cluster—from NameNode health to YARN resource allocation—requires significant manual effort and specialized expertise. Lack of ACID Compliance: Ensuring data integrity during partial failures is difficult, often resulting in “dirty data” that requires manual cleanup. What is a Data Lakehouse? A Data Lakehouse is a hybrid architecture that combines the low-cost, flexible storage of a Data Lake with the performance, structure, and reliability of a Data Warehouse. ...

Level: Intermediate to Advanced Data Engineering Tech Stack: Python · Apache Kafka · Docker · PySpark Structured Streaming · JVM The Problem: Batch is Too Slow In modern e-commerce, waiting 24 hours to analyze sales data is no longer acceptable. Businesses need to know what is selling right now — to manage inventory, detect fraud, and trigger real-time marketing. To solve this, I designed and built a decoupled, event-driven streaming architecture locally. This project serves as a blueprint for how enterprise companies move from static batch processing to real-time data-in-motion. ...

Bulk inserting millions of rows into staging tables sounds simple—until row-level locking and full transaction logging turn it into a major pipeline bottleneck. What is TABLOCK? TABLOCK is a table-level lock hint. While row-level locking is great for concurrency, it is expensive for massive ETL jobs. By using TABLOCK, you tell SQL Server to take a single lock on the entire table. Why does it make Inserts faster? Minimal Logging: When used with a SELECT INTO or an INSERT INTO ... SELECT on a heap (a table without a clustered index), TABLOCK allows for “Minimal Logging,” which significantly reduces I/O. Reduced Lock Overhead: The engine doesn’t have to manage millions of individual row locks. Parallelism: In some configurations, it allows multiple threads to write to the table simultaneously.Since it locks the whole table, other users won’t be able to write to it until your job is done The Command: INSERT INTO TargetTable WITH (TABLOCK) SELECT * FROM SourceTable;