499 free skills
Data Engineering skills
Skills for data engineering — ETL pipelines, Airflow DAGs, Spark jobs, and data warehouse modeling.
Sourced from real, public repositories — synced daily, never invented.
499 free skills
Skills for data engineering — ETL pipelines, Airflow DAGs, Spark jobs, and data warehouse modeling.
Sourced from real, public repositories — synced daily, never invented.
15 tools across six categories
13 of them never send your data anywhere
Free · No signup · No trial clock
SEE THE DIRECTORY
warehouse-optimization
qq511868349/ecommerce-skills
E-commerce warehouse and inventory optimization advisor. Analyzes inventory health, calculates safety stock and reorder points, performs ABC analysis, evaluates fulfillment costs, and provides actionable recommendations for improving efficiency. Supports all major fulfillment models: Self-fulfillment, Amazon FBA/FBM, Walmart WFS, 3PL, Shopify Fulfillment, TikTok Shop, Dropshipping, and Hybrid setups. No API key required. Use when: (1) reducing stockouts or overstock, (2) calculating safety stock levels, (3) optimizing warehouse costs, (4) improving Amazon IPI score, (5) analyzing inventory KPIs.
airflow-dag-patterns
benjaminastera/antigravity-awesome-skills
Build production Apache Airflow DAGs with best practices for operators, sensors, testing, and deployment. Use when creating data pipelines, orchestrating workflows, or scheduling batch jobs.
displaying-streamlit-data
streamlit/agent-skills
Displaying charts, dataframes, and metrics in Streamlit. Use when visualizing data, configuring dataframe columns, or adding sparklines to metrics. Covers native charts, Altair, and column configuration.
spark-python-data-source
databricks/databricks-agent-skills
Build custom Python data sources for Apache Spark using the PySpark DataSource API — batch and streaming readers/writers for external systems. Use this skill whenever someone wants to connect Spark to an external system (database, API, message queue, custom protocol), build a Spark connector or plugin in Python, implement a DataSourceReader or DataSourceWriter, pull data from or push data to a system via Spark, or work with the PySpark DataSource API in any way. Even if they just say "read from X in Spark" or "write DataFrame to Y" and there's no native connector, this skill applies.
keeperhub-plugins-spark
nickthelegend/keeperhub-skills
Production-grade v1.0.0 elite guide to Spark on KeeperHub.
Apache Spark Catalyst API
aehrc/pathling
Expert guidance for working with the Apache Spark Catalyst query optimisation framework. Use this skill when working with Spark SQL internals, creating custom expressions, implementing query optimisations, working with logical/physical plans, or extending Catalyst. Trigger keywords include "catalyst", "spark sql", "expression", "logical plan", "physical plan", "tree node", "query optimisation", "rule executor", "analyzer", "optimizer", "code generation".
netlify-blobs
practicalswan/agent-skills
Store and retrieve unstructured objects, file uploads, and cache-like state on Netlify using the @netlify/blobs key/value API from Functions, Edge Functions, and Build Plugins. Use when a task involves saving user file or image uploads, persisting form or contact-form submissions, storing generated output from Background Functions (sitemaps/processed media/bulk-email results), building read-only asset stores, adding client-side blob expiration, or wiring file-based blob uploads at deploy time. Not for per-user, transactional, or relational data (counters/balances/sessions) — reach for Netlify DB there instead.
netlify-deploy
1999azzar/netlify-deploy-skill
Link, configure, preview, and deploy web projects to Netlify with the Netlify CLI. Use when the user wants to publish a site, connect a repo, configure build settings, manage `netlify.toml`, or run preview and production deploys.
netlify-deploy
heyzude/skills_for_codex
Deploy web projects to Netlify using the Netlify CLI (`npx netlify`). Use when the user asks to deploy, host, publish, or link a site/repo on Netlify, including preview and production deploys.
netlify-deploy
quuu/openai-skills
Deploy web projects to Netlify using the Netlify CLI (`npx netlify`). Use when the user asks to deploy, host, publish, or link a site/repo on Netlify, including preview and production deploys.
netlify-deploy
seaworld008/commonly-used-high-value-skills
Deploy web projects to Netlify using the Netlify CLI (`npx netlify`). Use when the user asks to deploy, host, publish, or link a site/repo on Netlify, including preview and production deploys.
event-streaming
dtsong/data-engineering-skills
Use this skill when building real-time or near-real-time data pipelines. Covers Kafka, Flink, Spark Streaming, Snowpipe, BigQuery streaming, materialized views, and batch-vs-streaming decisions. Common phrases: \"real-time pipeline\", \"Kafka consumer\", \"streaming vs batch\", \"low latency ingestion\". Do NOT use for batch integration patterns (use data-integration) or pipeline orchestration (use data-pipelines).
skill-creator
mozilla/bigquery-etl-skills
ALWAYS use this skill when users ask to "create a skill", "make a skill for...", "add a new skill", or similar requests. This skill guides the creation of effective skills in the bigquery-etl repository that extend Claude's capabilities with specialized knowledge for BigQuery ETL workflows, Mozilla data platform conventions, or telemetry analysis. CRITICAL - First checks for conflicts with existing skills and recommends using/updating existing skills when appropriate. Do NOT attempt to create skills without invoking this skill first.
column-description-finder
mozilla/bigquery-etl-skills
Use this skill when looking up, auditing, or managing column descriptions from global, application-specific, and dataset-specific column definition YAML files (bigquery_etl/schema/global.yaml, bigquery_etl/schema/app_<name>.yaml, and bigquery_etl/schema/<dataset>.yaml). Use it to find a description for a specific column, list all columns in a base schema, audit which columns in a table's schema.yaml are covered by base schemas, or identify columns missing descriptions. Works with schema-enricher skill.
tech-hub-data
yakoub-ai/tech-skills
Data Lead agent coordinating Data Engineers, Data Governance specialists, and Database Admins. Use this skill for building ETL/ELT pipelines, lakehouse architecture, data quality frameworks, data catalogs and lineage tracking, database design and optimization, streaming pipelines, data mesh, and schema migrations. Triggers on keywords like ETL, pipeline, data lake, data warehouse, SQL, database, data quality, Kafka, Spark, data catalog, or data lineage.
python-data-engineering
dtsong/data-engineering-skills
Use this skill when writing Python code for data pipelines or transformations. Covers Polars, Pandas, PySpark DataFrames, dbt Python models, API extraction scripts, and data validation with Pydantic or Pandera. Common phrases: \"Polars vs Pandas\", \"PySpark DataFrame\", \"validate this data\", \"Python extraction script\". Do NOT use for SQL-based dbt models (use dbt-transforms) or integration architecture (use data-integration).
clickzetta-spark-flink-connector
clickzetta/clickzetta-skills
|
spark
g1joshi/agent-skills
Apache Spark distributed computing. Use for big data processing.
extract-spark-meetings
johnie/skills
Extract meeting notes from Spark Mail shared-transcript links into tagged markdown files with YAML frontmatter (participants, action items, decisions, next steps). Use when the user has one or more share.sparkmailapp.com links to convert into a searchable local meeting archive, in single-URL or batch mode from a links.md checklist.
wiki-spark
chiyubaima/chiyu-knowledge-base-skills
Quickly capture small ideas, excerpts, quotes, observations, or unfinished thoughts into Chiyu's Obsidian knowledge base raw layer. Use when the user says wiki-spark, "记一个想法", "存一条摘录", "先记下来", "存个灵感", or wants fast capture without full wiki integration.
airflow
g1joshi/agent-skills
Apache Airflow workflow orchestration. Use for data pipelines.
airflow-dag-design
ivanshamaev/ai-agent-codex
Design small, maintainable Airflow DAGs with clear task boundaries, retry policy, and runtime assumptions.
airflow-dags
ivanshamaev/de-agent-skills
Apache Airflow DAG authoring — DAG definition, TaskFlow API (@task/@dag decorators), operators (Bash/Python/SQL/HTTP), sensors (poke/reschedule modes), TaskGroups, dynamic task mapping (expand/partial), branching, trigger rules, XComs, Pools, callbacks, cross-DAG pipelines, best practices
cosmos-dbt-core
miptah21/skills
Use when turning a dbt Core project into an Airflow DAG/TaskGroup using Astronomer Cosmos. Does not cover dbt Fusion. Before implementing, verify dbt engine, warehouse, Airflow version, execution environment, DAG vs TaskGroup, and manifest availability.
extract-warehouse-receipt-data
iterationlayer/skills
Extract structured fields from warehouse receipt documents.
etl-developer
heymoezy/porter
Design, build, debug, and harden ETL and ELT pipelines so data moves correctly, repeatably, and observably between systems. Use when work involves extraction strategy, schema mapping, transformation logic, CDC, incremental loads, deduplication, orchestration, backfills, reconciliation, schema drift handling, or pipeline recovery. Do not use for dashboard interpretation, pure warehouse BI modeling, or infrastructure administration unrelated to data movement and pipeline operations.
pipeline-monitor
danny0926/nlp-data-for-trading
>
pipeline
baekenough/baekenough-skills
YAML-based sequential workflow engine — define, list, run, resume, add, and delete pipelines from natural language or YAML files. Supports skill invocation, free-form prompts, conditional steps, foreach iteration, state tracking, and halt-and-resume error recovery.
pipeline
iabhisekbosepm/claude-god-setup
Run the full feature development pipeline: explore → architect → plan → build → review → security → test → docs.
starrocks-etl-and-tasks
tiansuyu/agent-skills
Build StarRocks ETL pipelines and async/scheduled tasks — daily batch loads, incremental load with deduplication, real-time aggregation pipelines, SCD Type 2, and SUBMIT TASK for background/scheduled INSERT/CTAS execution with monitoring. Use when designing StarRocks ETL workflows, running long INSERT...SELECT in the background, scheduling recurring transformations, or backfilling data.
cicd-pipeline-design
jlpbiuma/agent-skills
>-
fullstack-monorepo-architecture
jlpbiuma/agent-skills
>-
reverse-etl
nrakow/ae-skills-dev
Design and implement reverse ETL pipelines that sync modeled warehouse data into operational tools like Salesforce, HubSpot, Intercom, and ad platforms. Use when syncing warehouse data to a CRM or marketing tool, configuring Census or Hightouch syncs, building a dbt activation layer, computing lead scores or customer health scores, or activating warehouse data for personalized marketing. Produces dbt activation model SQL, companion schema.yml, Census/Hightouch sync configuration, and a governance PR checklist.
ssis-to-dbt-replatform-migration
jamescha-earley/agent-skills
Validates, deploys, and operationalizes SnowConvert AI (SCAI) Replatform output — SSIS to dbt and Snowflake TASKs migrations. Use when user has SnowConvert Replatform output, converted SSIS packages, deploy dbt projects, deploy orchestration, validate replatform, deploy converted ETL, replatform deploy, SSIS to dbt deploy, SCAI replatform, SSIS migration, convert SSIS, migrate ETL, SnowConvert output, replatform output, dtsx migration, ETL to Snowflake.
airflow
bytesagain/ai-skills
>
airflow-dag-patterns
ncdevshiv/nc-skills-mcp
Build production Apache Airflow DAGs with best practices for operators, sensors, testing, and deployment. Use when creating data pipelines, orchestrating workflows, or scheduling batch jobs.
airflow-3
open-lakehouse/open-lakehouse
Apache Airflow 3.1 orchestration for this stack. Load when writing DAGs, dealing with the 2→3 migration gotchas, configuring Spark/Kafka connections, or debugging Airflow inside the docker-compose-airflow.yml setup.
gcp-managed-airflow-recommendations
gemini-cli-extensions/data-agent-kit-starter-pack
Provides recommendations and best practices for creating, configuring, tuning and optimizing Managed Service for Apache Airflow (MSAA, Cloud Composer) environments. Use when the user asks for guidance, recommendations, or best practices on configuring Cloud Composer, scaling Airflow environments, preventing workload restarts, or analyzing system health.
gcp-managed-airflow-dag-authoring
gemini-cli-extensions/data-agent-kit-starter-pack
Guides the authoring and validation of Apache Airflow DAGs for Managed Service for Apache Airflow (MSAA; formerly Cloud Composer). Covers environment context discovery, Airflow 2 vs 3 compatibility, authoring best practices, and local/remote validation processes. Use when creating or extending an Airflow DAG. Don't use when authoring Python code unrelated to Airflow DAGs.
spark-optimization
bcastelino/agent-skills-kit
Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.
spark-optimization
coppermare/skillverse
Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.
spark-engineer
hosseinmirzapur/opencode-skills
Use when writing Spark jobs, debugging performance issues, or configuring cluster settings for Apache Spark applications, distributed data processing pipelines, or big data workloads. Invoke to write DataFrame transformations, optimize Spark SQL queries, implement RDD pipelines, tune shuffle operations, configure executor memory, process .parquet files, handle data partitioning, or build structured streaming analytics.
spark-engineer
nkseth/copilot-dev-skills
Use when writing Spark jobs, debugging performance issues, or configuring cluster settings for Apache Spark applications, distributed data processing pipelines, or big data workloads. Invoke to write DataFrame transformations, optimize Spark SQL queries, implement RDD pipelines, tune shuffle operations, configure executor memory, process .parquet files, handle data partitioning, or build structured streaming analytics.
spark-engineer
thienanthx/skill-web-design
Use when writing Spark jobs, debugging performance issues, or configuring cluster settings for Apache Spark applications, distributed data processing pipelines, or big data workloads. Invoke to write DataFrame transformations, optimize Spark SQL queries, implement RDD pipelines, tune shuffle operations, configure executor memory, process .parquet files, handle data partitioning, or build structured streaming analytics.
spark-engineer
yo-steven/claude-skills-exploration-20260522
Use when writing Spark jobs, debugging performance issues, or configuring cluster settings for Apache Spark applications, distributed data processing pipelines, or big data workloads. Invoke to write DataFrame transformations, optimize Spark SQL queries, implement RDD pipelines, tune shuffle operations, configure executor memory, process .parquet files, handle data partitioning, or build structured streaming analytics.
spark-engineer
zroman88/claude-skills-to-cursor
Use when writing Spark jobs, debugging performance issues, or configuring cluster settings for Apache Spark applications, distributed data processing pipelines, or big data workloads. Invoke to write DataFrame transformations, optimize Spark SQL queries, implement RDD pipelines, tune shuffle operations, configure executor memory, process .parquet files, handle data partitioning, or build structured streaming analytics.
spark-engineer
krypton-kr36/claude-code-skills
Use when writing Spark jobs, debugging performance issues, or configuring cluster settings for Apache Spark applications, distributed data processing pipelines, or big data workloads. Invoke to write DataFrame transformations, optimize Spark SQL queries, implement RDD pipelines, tune shuffle operations, configure executor memory, process .parquet files, handle data partitioning, or build structured streaming analytics.
spark-python-data-source
agrehan-lego/contextualbandits_agentic
Build custom Python data sources for Apache Spark using the PySpark DataSource API — batch and streaming readers/writers for external systems. Use this skill whenever someone wants to connect Spark to an external system (database, API, message queue, custom protocol), build a Spark connector or plugin in Python, implement a DataSourceReader or DataSourceWriter, pull data from or push data to a system via Spark, or work with the PySpark DataSource API in any way. Even if they just say "read from X in Spark" or "write DataFrame to Y" and there's no native connector, this skill applies.
fabric-spark-perf-remediate
patrickgallucci/fabric-skills
Diagnose and resolve Apache Spark performance issues in Microsoft Fabric. Use when asked to troubleshoot slow Spark notebooks, optimize Spark SQL queries, fix data skew or shuffle bottlenecks, tune spark.sql.shuffle.partitions or autoBroadcastJoinThreshold, configure resource profiles (writeHeavy, readHeavyForSpark, readHeavyForPBI), enable autotune, resolve HTTP 430 throttling errors, analyze Spark UI stages and executors, optimize Delta Lake writes with VOrder or Optimized Write, run table maintenance (bin-compaction, vacuum, Z-Order), fix small files problems, tune streaming throughput, or right-size Fabric Spark pools and capacity SKUs.
gsql-loading-spark
srizdebnath/tigergraphskills
>
warehouse-guide
milosptr/pos-system
Reference for the warehouse/inventory tracking system. Use when working with stock management, warehouse status, inventory consumption, or understanding how sales affect stock levels.
warehouse-init
michaeltarasov02/skills
Initialize warehouse schema discovery. Generates .astro/warehouse.md with all table metadata for instant lookups. Run once per project, refresh when schema changes. Use when user says "/data:warehouse-init" or asks to set up data discovery.
warehouse-sql
hybridaione/hybridclaw
Review and run read-only natural-language SQL against a customer data warehouse with cached schema introspection and explicit write grants.
warehouse-audience-activation-plan
marvinvista/marketing-skills
Use when you need warehouse activation plan to turn warehouse tables into campaign audiences with joins, QA, consent, and refresh rules.
warehouse-udf-strategy
nrakow/ae-skills-dev
Decide when to use warehouse-native UDFs vs dbt macros, then implement, version, and govern them correctly. Use when repeating the same SQL transformation logic across multiple models, evaluating macro vs UDF tradeoffs, or writing complex runtime logic that SQL alone handles awkwardly. Triggers: 'create function', 'UDF', 'user defined function', 'reusable SQL logic', 'macro vs UDF', 'javascript UDF', 'BigQuery function', 'Snowflake function'.
warehouse-query-cost-optimizer
sisodiabhumca/agent-skills
Vendor-neutral skill to identify expensive warehouse queries and recommend optimization actions.
data-pipeline-engineering
vignesh2027/ai-agent-skills
Build data pipelines with contracts, lineage, quality gates, and observability
stream-chain
i-onlabs/claude-code-skills
Stream-JSON chaining for multi-agent pipelines, data transformation, and sequential workflows
data-pipelines
noah-sheldon/ai-dev-kit
Data pipeline workflow for ETL, validation, and quality controls.
data-pipeline
uabbasi/good-measure-giving
Run the charity evaluation pipeline - DoltDB storage, 8-phase streaming runner, repository pattern, judge-gated export. Use when working on collectors, scrapers, pipeline code, database queries, or debugging data flow.