data-pipeline-engineer

★ 6.8 · general

data-pipeline-engineer is a Claude Code skill that specializes in designing and implementing ETL/ELT pipelines, streaming architectures, and data warehouses. It covers the full modern data stack — Spark, Kafka, Flink, Airflow, Dagster, and dbt — with native support for Medallion architecture (Bronze/Silver/Gold layers), incremental models, Delta Lake, and Apache Iceberg. Data quality is enforced via Great Expectations and dbt tests at every layer, while orchestration relies on Airflow DAGs with sensors, retries, and task groups. The skill ships with ready-made reference files: a complete dbt project layout, a production Airflow DAG, a Kafka-to-Delta streaming processor, a Great Expectations suite, and a validate-pipeline.sh script. It's built for data engineers handling batch and streaming workloads, Kimball or Data Vault modeling, and teams looking to avoid common pitfalls like full table refreshes, monolithic DAGs, missing watermarks, or undocumented data lineage.