Abdus Sattar In the world of big data, few technologies have had as much impact as Apache Spark. Today, Spark powers some of the largest data platforms on Earth, enabling companies to process petabytes of information at lightning speed. From machine learning and real-time analytics to large-scale ETL pipelines, Spark has become a cornerstone of modern data … Read more
Apache Spark has become one of the most important technologies in modern data engineering. It enables organizations to process massive datasets quickly using distributed computing. Whether you are working with batch processing, streaming data, machine learning, or large-scale analytics, Spark provides a unified platform for handling big data efficiently. This guide walks through the core … Read more
Common Table Expressions (CTEs) and subqueries often produce identical results, but they are not interchangeable in a data engineering context. The choice between them affects readability, debuggability, and in some databases, query performance. Getting this right matters when you are writing transformations that run millions of rows in production. This tutorial explains the practical differences, … Read more
Microsoft Fabric is transforming the modern data platform landscape by bringing together: …all inside a single unified SaaS platform. One of the biggest strengths of Microsoft Fabric is flexibility. Fabric gives organizations multiple ways to store, process, and analyze data depending on: At the center of everything is OneLake — Microsoft Fabric’s unified data lake. … Read more
Azure Data Factory (ADF) is one of the most important cloud ETL and orchestration tools in the modern Azure ecosystem.In real-world enterprise projects, ADF is used for: This guide explains 100 practical ADF scenarios in a tutorial/blog format that is useful for: Section 1 – Core Azure Data Factory (ADF) Scenarios Scenario 1 – Incremental … Read more
Introduction Data modeling is one of the most important steps in designing and building a database. Before storing information in any system, businesses need a clear structure for how that data will be organized, connected, and maintained. This is where data modeling becomes essential. Think of data modeling as creating a blueprint for a database. … Read more
Loading a full table on every pipeline run is expensive and slow. Once your source tables grow beyond a few million rows, full loads become impractical. Incremental loading — processing only new and changed data since the last run — is the pattern that makes production data pipelines scalable. This tutorial covers three incremental load … Read more
We are going to give a quick overview of how stored views and materialized views can be created in Databricks. For this demo, we will create our two gold layer entities. We will start by creating a view in the gold layer against our silver table customers_orders. Our view will simply contain some aggregations for … Read more
We are going to see how to use CDF data to propagate changes to downstream tables. For this demo, we will create a new silver table called customers_orders by joining two streams: The orders table with the CDF data of the customers table. We will start by creating a function to upsert ranked updates into … Read more
Change Data Capture or CDC refers to the process of identifying and capturing changes made to data in the data source and then delivering those changes to the target. Those changes could be obviously new records to be inserted from the source to the target. Updated records in the source that need to be reflected … Read more