A metadata-driven pipeline means you build one reusable pipeline that can process many tables/files based on configuration, instead of creating a separate pipeline for every table.

Simple ADF example

Suppose you need to copy these 100 SQL tables into Azure Data Lake:

Customers
Orders
Products
Payments
Transactions
…

Without metadata-driven design, you might build many separate pipelines.

With metadata-driven design, create a configuration table:

Then your ADF pipeline does:

Lookup Configuration Table
↓
ForEach
↓
Read metadata for current table
↓
Copy Activity
↓
Data Lake

The same Copy Activity dynamically uses values such as:

@item().SourceTable
@item().TargetPath
@item().WatermarkColumn

So when you need to add another table, for example Invoices, you add one row to the configuration table rather than building a completely new pipeline.

Short interview answer

“A metadata-driven pipeline uses configuration data to control pipeline execution. For example, in ADF I can maintain a configuration table containing source table, target path, load type and watermark column. A Lookup activity reads the metadata, and a ForEach activity processes each table dynamically. This allows one reusable pipeline to handle many tables and makes the solution easier to maintain and scale.”

Key words to remember:
Configuration Table → Lookup → ForEach → Parameters → Dynamic Content → Reusable Pipeline.