databricks-autoloder

This project implements a modern data engineering pipeline using Databricks, PySpark, DBT, and Delta Live Tables. It follows the Medallion Architecture, supports realtime data ingestion with Autoloader, and models data with fact and dimension tables, including Slowly Changing Dimensions (SCD Type 2), all orchestrated in a scalable cloud environment

Built a metadata-driven Azure lakehouse that incrementally ingests Spotify-style SQL data with ADF, processes new Parquet files with Databricks Auto Loader, and publishes SCD-managed Delta facts and dimensions.

Built a Databricks lakehouse pipeline that incrementally processes six e-commerce data domains through Bronze, Silver, and Gold layers, including data-quality enforcement, CDC, SCD Type 2 history, and an analytics-ready dimensional model.