Оптимизация Apache Spark jobs

★ 8.5 · data

spark-optimization is a Claude Code skill that optimizes Apache Spark jobs through partitioning strategies, caching, shuffle optimization, and executor memory tuning. It covers production-level patterns for tackling the most common performance bottlenecks: data skew via salting and broadcast joins, Kryo serialization, columnar formats like Parquet and Delta, and Adaptive Query Execution with coalesce and skew-join handling. The skill includes a ready-to-use SparkSession configuration template and a breakdown of key performance factors — shuffle, memory pressure, GC spills, and partition sizing. Aimed at data engineers debugging slow distributed jobs, scaling pipelines to large datasets, or reducing unnecessary shuffle in wide transformations.