Projects

Level: Intermediate Data Engineering Tech Stack: dbt · Databricks · Delta Lake · SQL · Jinja · hive_metastore Source Code & Practice Files: View on GitHub The Problem: Raw Data Is Useless In my beginner pipeline project, we learned how to ingest raw CSV data and clean it with Python. But there’s a critical question that project doesn’t answer: Once the data is ingested — then what? In a real company, raw data lands in cloud storage every day. A dozen different analysts and data scientists need to query it — each writing their own transformations, often with subtle differences. One analyst filters out cancelled orders, another doesn’t. One team reports revenue in cents, another in dollars. The result is a classic problem in data teams: everyone has a different number, and nobody knows who is right. ...

Level: Beginner (Foundations) Tech Stack: Python · CSV · Clean Code Design Source Code & Practice Files: View on GitHub Welcome to Data Engineering! If you are just starting out in data engineering, you might be overwhelmed by tools like PySpark, Airflow, Snowflake, or Databricks. But here’s a secret: you don’t need any of those to understand the fundamentals of data engineering. Every enterprise data platform starts with one core concept: Ingestion. Before data can be analyzed, it must be loaded, cleaned, and verified. ...

Level: Intermediate to Advanced Data Engineering Tech Stack: Python · Apache Kafka · Docker · PySpark Structured Streaming · JVM The Problem: Batch is Too Slow In modern e-commerce, waiting 24 hours to analyze sales data is no longer acceptable. Businesses need to know what is selling right now — to manage inventory, detect fraud, and trigger real-time marketing. To solve this, I designed and built a decoupled, event-driven streaming architecture locally. This project serves as a blueprint for how enterprise companies move from static batch processing to real-time data-in-motion. ...