Abdus Sattar Azure Data Factory (ADF) is one of the most important cloud ETL and orchestration tools in the modern Azure ecosystem.In real-world enterprise projects, ADF is used for: This guide explains 100 practical ADF scenarios in a tutorial/blog format that is useful for: Section 1 – Core Azure Data Factory (ADF) Scenarios Scenario 1 – Incremental … Read more
Introduction Data modeling is one of the most important steps in designing and building a database. Before storing information in any system, businesses need a clear structure for how that data will be organized, connected, and maintained. This is where data modeling becomes essential. Think of data modeling as creating a blueprint for a database. … Read more
Loading a full table on every pipeline run is expensive and slow. Once your source tables grow beyond a few million rows, full loads become impractical. Incremental loading — processing only new and changed data since the last run — is the pattern that makes production data pipelines scalable. This tutorial covers three incremental load … Read more
We are going to give a quick overview of how stored views and materialized views can be created in Databricks. For this demo, we will create our two gold layer entities. We will start by creating a view in the gold layer against our silver table customers_orders. Our view will simply contain some aggregations for … Read more
We are going to see how to use CDF data to propagate changes to downstream tables. For this demo, we will create a new silver table called customers_orders by joining two streams: The orders table with the CDF data of the customers table. We will start by creating a function to upsert ranked updates into … Read more
Change Data Capture or CDC refers to the process of identifying and capturing changes made to data in the data source and then delivering those changes to the target. Those changes could be obviously new records to be inserted from the source to the target. Updated records in the source that need to be reflected … Read more
Learning Objectives Change Data Feed or CDF is a new feature built into Delta Lake that allows to automatically generate CDC feeds about Delta Lake tables. CDF records row-level changes for all the data written into a delta table. These include the raw data along with metadata indicating whether the specified row was inserted, deleted … Read more
Find the second highest salary from the Employee table. Find duplicate records in a table. Retrieve employees who earn more than their manager. Count employees in each department having more than 5 employees. Find employees who joined in the last 6 months. Get departments with no employees. Write a query to find the median salary. … Read more
Data Pipeline Data pipeline: Mapping data flow from collection to analysis for organizational insights. Source, process, deliver for analytics efficiency. Database vs Schema vs Table Database: Organized collection of related data, like a file cabinet for storing information. Schema: Blueprint defining structure and organization within a database. Table: Grid-like structure within a schema, where data … Read more
Introduction Slowly Changing Dimensions, or SCD is a data management concept that determines how tables handle data which change over time. For example, whether you want to overwrite values in the table or maybe retain their history. This can be determined by the SCD type to implement. There are three types of SCD which are … Read more