250 free skills · page 4 of 5
Data Engineering skills
Skills for data engineering — ETL pipelines, Airflow DAGs, Spark jobs, and data warehouse modeling.
Real files from public repositories — hand-curated, never invented. Sorted by installs.
4 / 5page
warehouse-location-optimizationkishorkukreja/awesome-supply-chainWhen the user wants to optimize warehouse locations, design warehouse networks, or determine optimal warehouse placement for distribution. Also use when the user mentions "warehouse siting," "warehouse network design," "storage facility location," "fulfillment center location," "regional warehouse optimization," "warehouse consolidation," or "distribution warehouse placement." For general facility location, see facility-location-problem. For distribution centers, see distribution-center-network.Data Engineering37
fp-pipe-refsickn33/agentic-awesome-skillsQuick reference for pipe and flow. Use when user needs to chain functions, compose operations, or build data pipelines in fp-ts.Data Engineering37
carto-import-export-datacartodb/agent-skillsImport geospatial files into the data warehouse via CARTO, export results back out, and prepare tilesets for fast map rendering.Data Engineering37warehouse-slotting-optimizationkishorkukreja/awesome-supply-chainWhen the user wants to optimize warehouse slot assignments, improve pick efficiency, or design warehouse layouts. Also use when the user mentions "slotting optimization," "slot assignment," "ABC slotting," "pick path optimization," "storage location assignment," "warehouse layout optimization," or "forward pick locations." For picker routing, see picker-routing-optimization. For warehouse design, see warehouse-design.Data Engineering37
data-pipeline-architectwyattowalsh/agentsDesign resilient data movement and transformation systems for batch, streaming, and hybrid workloads.Data Engineering36
polarssilvainfm/claude-skillsLightning-fast DataFrame library written in Rust for high-performance data manipulation and analysis. Use when user wants blazing fast data transformations, working with large datasets, lazy evaluation pipelines, or needs better performance than pandas. Ideal for ETL, data wrangling, aggregations, joins, and reading/writing CSV, Parquet, JSON files.Data Engineering35
whitespark-toolgarrettjsmith/localseoskillsWhen the user wants citation gap analysis, managed citation building, review generation campaigns, or local rank tracking. Trigger on "Whitespark," "citation finder," "where are my competitors listed," "citation gap," "build citations," "review generation tool," or "get more reviews." Note that Whitespark does NOT have an MCP server — it's dashboard-driven with limited API.Data Engineering35
nu-shellknoopx/piReads, filters, transforms, and manipulates CSV/TSV files using Nushell's structured data pipeline. Use when working with tabular data, data cleaning, CSV validation, or batch processing spreadsheet-like files.Data Engineering33
data-pipeline-specmohitagw15856/pm-claude-skillsDesign an ETL/ELT data pipeline specification. Use when asked to design a data pipeline, spec an ETL or ELT process, document a data ingestion workflow, or plan a data integration. Produces a complete pipeline spec with sources, transforms, destinations, SLAs, error handling, and data quality rules.Data Engineering33
sales-rudderstacksales-skills/salesRudderStack platform help — warehouse-native CDP, Event Streams, Reverse ETL, Transformations, Profiles, 200+ destinations, open-source. Use when setting up RudderStack SDKs or HTTP API, events not reaching destinations, Reverse ETL syncs failing or returning stale data, transformations throwing errors or dropping events, visitor profiles not merging across channels, choosing between RudderStack and Segment, or working with the RudderStack API. Do NOT use for general CDP comparison (use /sales-cdp) or CRM data dedup without RudderStack (use /sales-data-hygiene).Data Engineering33monte-carlo-push-ingestionsickn33/agentic-awesome-skillsExpert guide for pushing metadata, lineage, and query logs to Monte Carlo from any data warehouse.Data Engineering33
spark-job-creatorjeremylongshore/claude-code-plugins-plus-skillsThis skill provides automated assistance for spark job creator tasks within the Data Pipelines domain.Data Engineering32
managed-airflow-dag-troubleshootinggoogle/skillsThis skill provides instructions for troubleshooting Managed Airflow DAGs (DAG runs and task instances), utilizing gcloud composer, gcloud logging and gcloud storage commands to fetch remote logs and code.Data Engineering32
tech-data-pipelineasgard-ai-platform/skillsDesign data pipelines covering ETL vs ELT architectures, data source integration, scheduling, quality checks, and warehouse design. Use this skill when the user needs to move data between systems, build a data warehouse, automate data processing, or improve data reliability — even if they say 'move data from X to Y', 'build an ETL pipeline', 'our data is a mess', or 'set up a data warehouse'.Data Engineering32dbt-test-creatorjeremylongshore/claude-code-plugins-plus-skillsCreate dbt test creator operations. Auto-activating skill for Data Pipelines.Data Engineering32
signals-scout-data-warehouseposthog/ai-pluginYou are a focused data warehouse import-integrity scout. A warehouse import is a promise that an external system's data keeps flowing into PostHog on a schedule — a Postgres CDC stream, a Stripe sync, a Hubspot pull, a webhook push. Import failures are uniquely silent: the rest of PostHog keeps…Data Engineering32
universal-scraping-architectalirezarezvani/claude-skillsUse for web scraping, crawling, document extraction, API parsing, or building validation-heavy data pipelines using Firecrawl or local Python scripts.Data Engineering31
data-engineeringpluginagentmarketplace/custom-plugin-data-engineerData pipeline architecture, ETL/ELT patterns, data modeling, and production data platform designData Engineering31
machine-learning-ops-ml-pipelinermyndharis/antigravity-skillsDesign and implement a complete ML pipeline for: $ARGUMENTSData Engineering31
huawei-cloud-mrs-spark-sql-checkhuaweicloud/huaweicloud-skillsYou are an MRS Spark SQL specification checking expert, responsible for comprehensive SQL statement checking for Huawei Cloud MRS Spark. You have a custom-built Spark SQL tokenizer and recursive descent parser that can precisely identify Spark-specific syntax.Data Engineering30
analytics-data-engineerdaemon-blockint-tech/agentic-enteprises-skillLayering, naming (stg_, int_, fct_, dim_), materialization defaults, env targets.Data Engineering30etl-toolspluginagentmarketplace/custom-plugin-data-engineerApache Airflow, dbt, Prefect, Dagster, and modern data orchestration for production data pipelinesData Engineering30
spark-app-templategithub/copilot-pluginsComprehensive guidance for building web apps with opinionated defaults for tech stack, design system, and code standards. Use when user wants to create a new web application, dashboard, or interactive interface. Provides tech choices, styling guidance, project structure, and design philosophy to get users up and running quickly with a fully functional, beautiful web app.Data Engineering28
etl-pipelinesclaude-dev-suite/claude-dev-suite[ ] Idempotent pipeline (safe to re-run); [ ] Bad record handling (dead letter, skip + log); [ ] Incremental processing (not full reload each time)Data Engineering27
dataset-curatornickcrew/claude-cortexUse this skill when designing, cleaning, deduplicating, or documenting datasets for model training and evaluation including schema design, class imbalance handling, and train/val/test splits. Not for running model training or hyperparameter tuning. Not for real-time data pipeline engineering.Data Engineering27
pp-setlist-fmmvanhorn/printing-press-libraryEvery Setlist.fm endpoint, plus offline analytics no API call can return — tour shape, song frequency, what's overdue, setlist prediction. Trigger phrases: predict the setlist, what songs are overdue for, look up setlist for, how often does X play Y, compare these two tours, use setlist-fm, run setlist-fm.Data Engineering27
databricks-spark-structured-streamingdatabricks-solutions/ai-dev-kitComprehensive guide to Spark Structured Streaming for production workloads. Use when building streaming pipelines, working with Kafka ingestion, implementing Real-Time Mode (RTM), configuring triggers (processingTime, availableNow), handling stateful operations with watermarks, optimizing checkpoints, performing stream-stream or stream-static joins, writing to multiple sinks, or tuning streaming cost and performance.Data Engineering26
app-studio-dataset-etl-gen-demostahura/domo-ai-vibe-rulesVertical App Studio micro-demo with optional dataset creation (domo-data-generator), ETL (magic-etl-cli or magic-etl), then multi-page layout using advanced-app-studio + domo-app-theme. Picks industry packs from references/*.md. Use when building an App Studio walkthrough that needs realistic data pipelines and themed pages per vertical.Data Engineering25
spark-version-upgradeopenhands/extensionsUpgrade Apache Spark applications between major versions (2.x→3.x, 3.x→4.x). Covers build files, deprecated APIs, configuration changes, SQL/DataFrame updates, and test validation.Data Engineering25
data-engineering-data-pipelineaaaaqwq/agi-super-teamYou are a data pipeline architecture expert specializing in scalable, reliable, and cost-effective data pipelines for batch and streaming data processing.Data Engineering24ml-pipeline-workflowrmyndharis/antigravity-skillsBuild end-to-end MLOps pipelines from data preparation through model training, validation, and production deployment. Use when creating ML pipelines, implementing MLOps practices, or automating model training and deployment workflows.Data Engineering23
jcl-migration-analyzerdauquangthanh/hanoi-rainbowAnalyzes legacy JCL (Job Control Language) scripts to assist with migration to modern workflow orchestration and batch processing systems. Extracts job flows, step sequences, data dependencies, conditional logic, and program invocations. Generates migration reports and creates implementation strategies for Spring Batch, Apache Airflow, or shell scripts. Use when working with mainframe job migration, JCL analysis, batch workflow modernization, or when users mention JCL conversion, analyzing .jcl/.JCL files, working with job steps, procedures, or planning workflow orchestration from JCL jobs.Data Engineering23
valohai-design-pipelinesvalohai/valohai-skillsAnalyze ML project structure and design Valohai pipelines to orchestrate multi-step workflows. Use this skill when a user wants to create a Valohai pipeline, connect multiple ML steps into an automated workflow, identify pipeline opportunities in their codebase, design data flow between preprocessing/training/evaluation/inference steps, or add conditional logic and parallel execution to pipelines. Triggers on mentions of pipelines, workflows, DAGs, orchestration, multi-step ML, or connecting Valohai steps.Data Engineering22
spark-optimizationfull-stack-skills/devops-skillsOptimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.Data Engineering21
reverse-etl-activationswan-gtm/gtm-skillsUse this skill when you need to turn a scored, segmented audience sitting in the warehouse (BigQuery, Snowflake, Redshift) into live GTM motion, syncing accounts, contacts, and computed traits out to the CRM, ad platforms, and sequencing tools. It encodes the judgment that keeps reverse ETL from quietly polluting every downstream system: identity resolution before any write, a change-data-capture diff so only real deltas move, field-level mapping with type and format guards, suppression and consent gates, sync scheduling matched to each destination's rate limits, and a mandatory dry run plus row-count reconciliation before and after the live push. Every threshold (match-confidence floor, batch size, sync cadence, max-delete guard, suppression rules) is a tunable default. Each run produces a validated sync plan, a dry-run diff of adds, updates, and suppressions with counts, and a reconciliation report after the live sync.Data Engineering21
iceberggordonmurray/data-engineering-skillsDesign, migrate, tune, and maintain Apache Iceberg tables across query engines. Use for Iceberg schema and partition evolution, v2 to v3 upgrades, catalog selection, time travel and rollback, row-level deletes, snapshot expiry, orphan file cleanup, small-file compaction, or slow queries and metadata bloat on a lakehouse table. Covers Spark, Flink, Trino, Athena, Snowflake, REST catalogs, Polaris, Nessie, Glue, and Hive.Data Engineering21
migrate-airflow-kestrakestra-io/agent-skillsMigrate an Airflow DAG to a production-ready Kestra flow. Extracts Python task logic into namespace files, maps DAG dependencies to Kestra tasks, and preserves parallel execution structure.Data Engineering21spark-python-data-sourcedatabricks-solutions/ai-dev-kitBuild custom Python data sources for Apache Spark using the PySpark DataSource API — batch and streaming readers/writers for external systems. Use this skill whenever someone wants to connect Spark to an external system (database, API, message queue, custom protocol), build a Spark connector or plugin in Python, implement a DataSourceReader or DataSourceWriter, pull data from or push data to a system via Spark, or work with the PySpark DataSource API in any way. Even if they just say "read from X in Spark" or "write DataFrame to Y" and there's no native connector, this skill applies.Data Engineering20
building-data-pipelineslegout/data-agent-skillsBuild production batch data pipelines with Polars, DuckDB, and PyArrow. Covers ETL patterns, medallion architecture, partitioning, and CRUD operations. Use when designing or implementing data ingestion, transformation, and loading workflows in Python.Data Engineering20designing-data-storagelegout/data-agent-skillsFile formats and lakehouse table formats for data lakes: Parquet, Arrow, Lance, Zarr, Avro, ORC, Delta Lake, Apache Iceberg, and Apache Hudi. Covers compression, partitioning, ACID transactions, schema evolution, and format selection.Data Engineering20
kafkamjunaidca/mjs-agent-skillsEvent streaming for Kubernetes. Strimzi operator, KRaft mode, no ZooKeeper.Data Engineering20airflow-dag-patternsrmyndharis/antigravity-skillsBuild production Apache Airflow DAGs with best practices for operators, sensors, testing, and deployment. Use when creating data pipelines, orchestrating workflows, or scheduling batch jobs.Data Engineering20
spark-advisoryaooqinn/spark-history-cliDiagnose, compare, and optimize Apache Spark applications and SQL queries using Spark History Server data. Use this skill whenever the user wants to understand why a Spark app is slow, compare two benchmark runs or TPC-DS results, find performance bottlenecks (skew, GC pressure, shuffle spill, straggler tasks), get tuning recommendations, or optimize Spark/Gluten configurations. Also trigger when the user mentions 'diagnose', 'compare runs', 'why is this query slow', 'tune my Spark job', 'benchmark comparison', 'performance regression', or asks about executor skew, shuffle overhead, AQE effectiveness, or Gluten offloading issues.Data Engineering19spark-history-cliyaooqinn/spark-history-cliQuery a running Apache Spark History Server from Copilot CLI. Use this whenever the user wants to inspect SHS applications, jobs, stages, executors, SQL executions, environment details, or event logs, especially when they mention Spark History Server, SHS, event log history, benchmark runs, or application IDs.Data Engineering18spark-optimizationrmyndharis/antigravity-skillsOptimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.Data Engineering18assuring-data-pipelineslegout/data-agent-skillsData quality validation and observability for data pipelines. Combines Great Expectations and Pandera for data validation with OpenTelemetry and Prometheus for monitoring and alerting.Data Engineering17
sqldw-climicrosoft/skills-for-fabricAuthor, query and diagnose Fabric Warehouse, Lakehouse SQL endpoints and Mirrored Databases: DDL/DML and COPY INTO ingestion, read-only T-SQL SELECT and row counts over lakehouse tables, and queryinsights performance triage. Fabric SQL database (OLTP) is sqldb-*-cli. Triggers:query warehouse,count rows lakehouse,SELECT lakehouse,create warehouse table,COPY INTO,warehouse MERGE,slowest warehouse queries,queryinsights CPUData Engineering16
netlify-deployjetbrains/skillsDeploy web projects to Netlify using the Netlify CLI (npx netlify). Use when the user asks to deploy, host, publish, or link a site/repo on Netlify, including preview and production deploys.Data Engineering16paimongordonmurray/data-engineering-skillsDesign, ingest into, tune, and operate Apache Paimon tables for streaming lakehouses. Use for Paimon primary-key or append-only table design, bucket sizing, changelog producer choice, Flink CDC ingestion, compaction backlog, lookup join performance, PyPaimon, Spark reads, Iceberg compatibility, or streaming writes that produce too many small files.Data Engineering15databricks-icebergdatabricks-solutions/ai-dev-kitApache Iceberg tables on Databricks — Managed Iceberg tables, External Iceberg Reads (fka Uniform), Compatibility Mode, Iceberg REST Catalog (IRC), Iceberg v3, Snowflake interop, PyIceberg, OSS Spark, external engine access and credential vending. Use when creating Iceberg tables, enabling External Iceberg Reads (uniform) on Delta tables (including Streaming Tables and Materialized Views via compatibility mode), configuring external engines to read Databricks tables via Unity Catalog IRC, integrating with Snowflake catalog to read Foreign Iceberg tablesData Engineering15
spark-best-practicesoleanderhq/skillsUse this skill for general Apache Spark guidance when optimizing performance, reliability, and maintainability.Data Engineering14orchestrating-data-pipelineslegout/data-agent-skillsPipeline orchestration and workflow management with Prefect, Dagster, and dbt. Covers scheduling, dependency management, retries, and integration patterns.Data Engineering14mashupnickcrew/claude-cortexForce-fit patterns from other domains to spark novel concepts.Data Engineering14pp-netlifymvanhorn/printing-press-libraryEvery Netlify API endpoint, plus a local SQLite mirror of your whole account so you can search, diff, and audit across all sites at once — offline and agent-native. Trigger phrases: list my netlify sites, check netlify deploys, search netlify form submissions, audit netlify dns, compare netlify env vars, use netlify, run netlify.Data Engineering13
stream-chainnatea/fitfinderStream-JSON chaining for multi-agent pipelines, data transformation, and sequential workflowsData Engineering13
etl-lineage-explainersisodiabhumca/agent-skillsVendor-neutral skill for extracting and summarizing table-level lineage from SQL-based ETL jobs.Data Engineering13
pipelineh-mmer/pentest-agentsPrepare the battlefield — recon, scanning, and surface ranking. Stops before hunting. Run /hunt or /autopilot after. Usage: /pipeline or /pipeline <target>Data Engineering13
data-pipelinesbooklib-ai/booklibYou are an expert data engineer grounded in the 13 chapters from Data Pipelines Pocket Reference (Moving and Processing Data for Analytics) by James Densmore. You help developers and data engineers in two modes:Data Engineering12
pipelineelliotjlt/claude-skill-potionsSome tasks are inherently sequential - you can't test what isn't built, can't build what isn't designed, can't review what isn't complete. This elixir enforces pipeline discipline: define stages, ensure clean handoffs, and never skip ahead. </purpose>Data Engineering12
netlify-deployfirecrawl/skillsDeploy web projects to Netlify using the Netlify CLI (npx netlify). Use when the user asks to deploy, host, publish, or link a site/repo on Netlify, including preview and production deploys.Data Engineering12Other tasks
same rules, different job
