When working with large datasets in Apache Spark or Azure Databricks, you may encounter a situation where a job looks almost finished, but a few tasks continue running for a long time.

One common reason is Data Skew.

In this article, we’ll understand what data skew is, why it affects Spark performance, and how salting can help solve it.

What is Data Skew?

Spark processes large datasets by dividing them into smaller pieces called partitions.

Ideally, the data should be reasonably balanced across those partitions.

For example:

Partition 1 → 10 million rows
Partition 2 → 11 million rows
Partition 3 → 9 million rows
Partition 4 → 10 million rows

The workload is fairly balanced, so Spark workers can process the partitions in roughly the same amount of time.

Now imagine this:

Partition 1 → 5 million rows
Partition 2 → 5 million rows
Partition 3 → 5 million rows
Partition 4 → 80 million rows

Partition 4 has significantly more work than the others.

The first three tasks may finish quickly, while the task processing Partition 4 continues running.

The entire Spark stage has to wait for that slow task.

This is called Data Skew.

In simple terms:

Data Skew = Uneven data distribution → Uneven workload → Slow Spark job

---

A Realistic Example

Suppose we have a large transaction dataset containing customer transactions by country.

The data distribution looks like this:

UAE       → 80 million rows
Pakistan  → 5 million rows
India     → 5 million rows
UK        → 2 million rows

We also have a small country reference table:

country      region
---------------------
UAE          Middle East
Pakistan     South Asia
India        South Asia
UK           Europe

Now we perform a join:

result = transactions_df.join(
    country_df,
    "country"
)

The SQL equivalent would conceptually be:

SELECT *
FROM transactions t
JOIN countries c
    ON t.country = c.country;

The problem is that the join key is highly uneven.

UAE appears 80 million times, while the other countries appear far less frequently.

During a normal shuffle join, records with the same join key need to be brought together.

This means the UAE key can create a very large workload for a small number of tasks.

That is our skew problem.

---

How Can We Detect Data Skew?

One simple check is to examine the distribution of the join key.

transactions_df \
    .groupBy("country") \
    .count() \
    .orderBy("count", ascending=False) \
    .show()

We might see:

+----------+----------+
| country  | count    |
+----------+----------+
| UAE      | 80000000 |
| Pakistan |  5000000 |
| India    |  5000000 |
| UK       |  2000000 |
+----------+----------+

This immediately tells us that UAE is a heavily skewed key.

In a real production environment, I would also inspect the Spark UI.

I would look for signs such as:

• A few tasks taking much longer than others
• Very uneven task input sizes
• Large shuffle reads/writes
• Most tasks finishing while a small number remain active

---

Solution: Salting

One technique for handling heavily skewed join keys is called salting.

The idea is surprisingly simple.

Instead of allowing all 80 million UAE records to use the same join key:

UAE

we artificially divide them into multiple keys:

UAE_0
UAE_1
UAE_2
UAE_3
...
UAE_9

If the distribution is reasonably even, our 80 million UAE records can now be split roughly like:

UAE_0 → ~8 million
UAE_1 → ~8 million
UAE_2 → ~8 million
...
UAE_9 → ~8 million

Instead of one extremely large group, Spark has several smaller groups that can be processed in parallel.

The artificial number we add is called the salt.

---

Step 1: Add Salt to the Large Dataset

Let’s add a salt value between 0 and 9.

from pyspark.sql.functions import rand, floor

transactions_salted = transactions_df.withColumn(
    "salt",
    floor(rand() * 10)
)

Our data may now look like:

transaction_id    country    salt
----------------------------------
10001             UAE        3
10002             UAE        7
10003             UAE        1
10004             UAE        5
10005             Pakistan   2

We can then create a salted join key:

from pyspark.sql.functions import concat, col, lit

transactions_salted = transactions_salted.withColumn(
    "salted_country",
    concat(
        col("country"),
        lit("_"),
        col("salt")
    )
)

Now we have:

country      salt     salted_country
------------------------------------
UAE          3        UAE_3
UAE          7        UAE_7
UAE          1        UAE_1
Pakistan     2        Pakistan_2

The UAE records are no longer represented by only one join key.

---

But There Is a Problem

Our small country table still contains:

UAE
Pakistan
India
UK

But the large dataset now contains:

UAE_0
UAE_1
UAE_2
...
UAE_9

If we join these directly, they won’t match.

Therefore, we need to prepare the smaller side of the join as well.

---

Step 2: Replicate the Small Table

We create salt values from 0 through 9:

from pyspark.sql.functions import explode, sequence

country_salted = country_df.withColumn(
    "salt",
    explode(sequence(lit(0), lit(9)))
)

The original UAE row:

UAE → Middle East

becomes:

UAE    0    Middle East
UAE    1    Middle East
UAE    2    Middle East
UAE    3    Middle East
...
UAE    9    Middle East

Then we create the same salted key:

country_salted = country_salted.withColumn(
    "salted_country",
    concat(
        col("country"),
        lit("_"),
        col("salt")
    )
)

Now both sides contain compatible keys:

Large dataset        Small dataset

UAE_0                 UAE_0
UAE_1                 UAE_1
UAE_2                 UAE_2
...
UAE_9                 UAE_9

---

Step 3: Perform the Salted Join

Now we can join using the salted key:

result = transactions_salted.join(
    country_salted,
    "salted_country"
)

Conceptually, instead of:

80 million UAE records
          ↓
        UAE
          ↓
Huge workload

we have:

80 million UAE records
          ↓
 ┌────────┼────────┐
 ↓        ↓        ↓
UAE_0    UAE_1   ... UAE_9
 ↓        ↓          ↓
Task     Task        Task

The workload can now be distributed across more tasks.

---

Do We Always Need Salting?

No.

This is an important point.

Salting is useful, but it adds complexity and can increase the size of the smaller dataset because rows must be replicated.

Before implementing salting, investigate the actual cause of the performance problem.

If the second table is very small, a broadcast join may be a much simpler solution.

For example:

from pyspark.sql.functions import broadcast

result = transactions_df.join(
    broadcast(country_df),
    "country"
)

Spark sends the small country_df to the executors.

This can avoid the large shuffle that caused the skew problem in the first place.

So for our example:

transactions_df → 92 million rows
country_df      → 200 rows

I would normally investigate a broadcast join before implementing salting.

---

Broadcast Join vs Salting

A useful rule of thumb is:

Small lookup table
        ↓
Consider Broadcast Join first

Large-to-large join
        +
Highly skewed join key
        ↓
Consider Salting

Salting becomes particularly useful when broadcasting isn’t practical and a small number of keys dominate the dataset.

---

What About AQE?

Modern Spark also provides Adaptive Query Execution (AQE).

AQE can dynamically optimize query execution based on runtime statistics and includes capabilities for handling some skewed joins.

Therefore, in a production environment, I wouldn’t immediately start writing salting logic.

I would first:

1. Identify the slow stage
        ↓
2. Inspect Spark UI
        ↓
3. Check key distribution
        ↓
4. Check shuffle and partition sizes
        ↓
5. Consider AQE
        ↓
6. Consider Broadcast Join
        ↓
7. Repartition if appropriate
        ↓
8. Use Salting when necessary

---

Interview Question

Interviewer: What is Data Skew?

A strong short answer is:

“Data skew occurs when data is unevenly distributed across Spark partitions. Some partitions become much larger than others, causing a few tasks to take significantly longer and slowing down the entire Spark stage.”

Interviewer: How would you solve it?

“First I would confirm the skew by checking key distribution and the Spark UI. Depending on the situation, I could use AQE, a broadcast join when one side is small, repartitioning, or salting for heavily skewed keys.”

Interviewer: Explain salting.

“Salting means adding an artificial value to a heavily skewed join key so that records for that key are distributed across multiple partitions. On the other side of the join, we create matching salted keys. This allows Spark to process the skewed key across multiple tasks instead of creating one large bottleneck.”

---

Final Takeaway

Remember this sequence:

Data Skew
   ↓
Uneven key distribution
   ↓
Uneven partitions
   ↓
Few tasks become very large
   ↓
Spark stage becomes slow

And the salting solution:

UAE → 80M rows
       ↓
     SALTING
       ↓
UAE_0  UAE_1  UAE_2 ... UAE_9
  ↓      ↓      ↓          ↓
Task   Task   Task       Task

The most important lesson is:

Salting doesn’t reduce the amount of data. It redistributes the work so Spark can process a heavily skewed key more evenly in parallel.

For a Senior Data Engineer interview, don’t simply say, “I use salting for skew.”

Explain the decision:

Detect the skew → understand the cause → choose the simplest appropriate optimization → validate the performance improvement.

That demonstrates senior-level problem solving rather than just knowledge of Spark terminology.

This can also become a strong Data Engineering blog/LinkedIn article and fits well with your plan to create educational Data Engineering content.