| Term | Short explanation |
|---|---|
| Databricks | Cloud platform commonly used for large-scale data processing using Spark. |
| Apache Spark | Distributed processing engine that processes large datasets across multiple workers. |
| PySpark | Python API for Apache Spark. |
| DataFrame | Distributed tabular data structure used for Spark transformations. |
| Lazy Evaluation | Spark waits to execute transformations until an action actually requires the result. |
| Transformation | Operation such as filter, select, join or groupBy that creates a new DataFrame. |
| Action | Operation such as count() or collect() that triggers Spark execution. |
| Partition | A chunk of data processed by a Spark task. |
| Shuffle | Movement of data between partitions, commonly caused by joins and aggregations. Expensive shuffles can hurt performance. |
| Data Skew | Uneven data distribution where some partitions contain much more data than others. |
| Broadcast Join | Sends a small table to workers to avoid a large shuffle during a join. |
| Delta Lake | Adds ACID transactions and reliability features to data-lake tables. |
| OPTIMIZE | Delta operation used to compact many small files into larger files. |
| Z-Ordering | Organizes related data to improve data skipping for commonly filtered columns. |