spark

The Databricks-flavored performance question. What interviewers want is a candidate who heads straight for the usual suspects (skew, shuffle, spill) through the Spark UI rather than guessing. Here is the diagnostic order and the fixes that genuinely move the needle.

The number-one cause of mysteriously slow Spark jobs: one partition handles most of the work while the rest idle. The signal is reading the symptom (a few straggler tasks) and reaching for the right fix, salting, broadcast, or AQE.

Anyone can say 'filter and dedup.' The signal is the funnel arranged by cost, the MinHash/LSH banding math, and recognizing that the shuffle across billions of documents is what actually runs up the bill, plus the benchmark decontamination people forget until their eval numbers get challenged.

These four get confused constantly, but they sit at different layers: data processing, distributed training, and general orchestration. The signal is fitting the tool to the workload instead of reaching for the one you know. Here is the answer.