499 free skills
Data Engineering skills
Skills for data engineering — ETL pipelines, Airflow DAGs, Spark jobs, and data warehouse modeling.
Sourced from real, public repositories — synced daily, never invented.
499 free skills
Skills for data engineering — ETL pipelines, Airflow DAGs, Spark jobs, and data warehouse modeling.
Sourced from real, public repositories — synced daily, never invented.
15 tools across six categories
13 of them never send your data anywhere
Free · No signup · No trial clock
SEE THE DIRECTORY
data-pipeline
dora-rs/dora-skills
Data collection and management for dora-rs. Use when user asks about data recording, replay, dataset creation, or training data collection.
ric-data-pipeline
lichong-a/ric-skills
RIC data pipeline skill for Kafka, NATS, Redis streams, TimescaleDB/PostgreSQL, Elasticsearch indexing, event schemas, ingestion, projection, retries, idempotency, backfills, observability, and non-destructive migration design under ric namespace rules.
spark-diagnostics-cli
microsoft/skills-for-fabric
>
synthetic-data-generation
databricks-solutions/ai-dev-kit
Generate realistic synthetic data using Faker and Spark, with non-linear distributions, integrity constraints, and save to Databricks. Use when creating test data, demo datasets, or synthetic tables.
etl
badhope/skill
ETL pipeline development for data extraction, transformation, and loading. Keywords: etl, pipeline, data pipeline, airflow, spark, 数据管道
data-engineer
glincker/claude-code-marketplace
Data engineering agent for ETL pipelines, data warehousing, and analytics
sflow
snomiao/sflow
Use sflow for TypeScript stream processing pipelines. Trigger this skill when the user needs to process arrays, async iterables, or streams with operations like map, filter, reduce, chunk, merge, debounce, throttle, or any functional pipeline over data. sflow wraps WebStreams API with a chainable, async-first API that handles concurrency, backpressure, and lazy evaluation automatically. Use it instead of hand-rolled for-loops, Promise.all chains, or RxJS when building data pipelines in TypeScript/JavaScript.
spark
quinteroac/spark-marketing-guru
Personalized brainstorming agent that generates sellable utility ideas (CLI, TUI, GUI, agent tools) tailored to the user's stack and niche, with persistent memory of past suggestions. Triggered by: /spark.
spark
cdpath/skills
Collaborative brainstorming that walks a raw idea to a committed spec, then stops. Guides context-gathering (one question per turn), multiple approaches with trade-offs, section-by-section approval, and a hard gate that blocks all coding until the user greenlights the design. Use when the user wants to think through a design before implementation or produce a spec document.
stream-chain
aegntic/cldcde
Stream-JSON chaining for multi-agent pipelines, data transformation, and sequential workflows
keeper-stress-analysis
clickhouse/clickhouse
Analyze ClickHouse Keeper stress-test results from play.clickhouse.com / keeper_stress_tests data warehouse. Use whenever the user asks about Keeper performance, validates Keeper PRs against stress dashboards, investigates regressions or improvements in Keeper nightlies, asks about specific date windows / SHAs / PR-sets in Keeper stress tests, wants per-PR or window-vs-window comparisons, asks "did this PR break Keeper", asks "what changed in Keeper between dates", or wants a summary report of Keeper stress runs. Triggers on terms like "keeper stress", "keeper PR", "keeper p99", "keeper memory", "keeper rps", "keeper nightly", "keeper-stress-tests", "keeper validation", "keeper regression", or any question referencing the keeper-stress Grafana dashboard. ALWAYS prefer this skill over re-deriving the workflow from scratch — it captures hard-learned lessons about cgroup-vs-Keeper memory, bench-harness confounds, noise floors, and per-PR attribution limits.
databricks-synthetic-data-generation
databricks-solutions/ai-dev-kit
Generate realistic synthetic data using Faker and Spark, with non-linear distributions, integrity constraints, and save to Databricks. Use when creating test data, demo datasets, or synthetic tables.
databricks-data-engineering
databricks-solutions/databricks-exec-code-mcp
Production data engineering pipelines following medallion architecture (Bronze/Silver/Gold layers) with data ingestion, transformation, quality checks, Delta Lake optimization, and orchestration. Use when building ETL pipelines, medallion architecture, data lakes, or data transformation workflows.
netlify-deploy
guilhermemarketing/esc-skills
Deploy web projects to Netlify using the Netlify CLI (`npx netlify`). Use when the user asks to deploy, host, publish, or link a site/repo on Netlify, including preview and production deploys.
add-admin-panel
cowardor/tars
Добавить защищённую админ-панель к статичному HTML сайту. Netlify Identity + Netlify Blobs. Владелец может менять тексты, цены, часы, контакты через браузер без знания кода.
duckdb-etl
chrishuffman5/domain-expert
DuckDB as an in-process data transformation and ETL engine. Covers file ingestion (Parquet/CSV/JSON/Excel), SQL-based transformations, Spark replacement for small-to-medium pipelines, dbt-duckdb integration, data lake querying, and pipeline orchestration patterns. Routes to the primary DuckDB agent for core engine expertise. WHEN: \"DuckDB ETL\", \"DuckDB transformation\", \"DuckDB pipeline\", \"DuckDB ingestion\", \"DuckDB CSV to Parquet\", \"DuckDB data loading\", \"dbt-duckdb\", \"DuckDB Spark replacement\", \"DuckDB data lake\", \"DuckDB file conversion\", \"DuckDB batch processing\", \"DuckDB COPY\", \"DuckDB export\".
investigate-airflow-failure
recidiviz/pulse-data
Diagnose a failed Airflow task running on Cloud Composer by pulling task logs from GCS, identifying the first real error in the traceback, and inspecting the task's source and recent git history. Use when the user wants to investigate an Airflow task failure (e.g. "why did X task fail", "investigate the failing generate_airflow_dag_run_history task", "dig into the airflow failure in prod"). Can be invoked directly or as a specialist from the investigate-pd-incident skill.
etl-pipelines
ffsshhttiikk/opencode-agents-skills
ETL pipeline design and implementation
etl
ffsshhttiikk/opencode-agents-skills
Extract Transform Load processes
bolt-pipeliner
ormenesse/boltdbt_pipeliner
Create and adapt Bolt Pipeliner ETL projects using Spark, Pandas, or Polars. Use whenever a user asks to create an ETL pipeline, data lake pipeline, medallion architecture, Airflow pipeline, data-quality workflow, or Bolt Pipeliner project.
airflow
terminalskills/skills
|
apache-spark
terminalskills/skills
>-
package-airflow-blue
getcolors/airflow
Creates and operates a single-node Apache Airflow server with Blue, OpenTofu and Ansible — Airflow under Docker with LocalExecutor, a host Postgres backed up to R2 with WAL-G, TLS and authentication through Caddy, and DAGs pushed from GitHub Actions over rsync. Use when initializing an airflow project, generating colors.yml, selecting a compute, DNS, SMTP or state provider, building or dry-running configuration, provisioning or destroying the server, or restoring the metadata database.
package-airflow-green
getcolors/airflow
Creates and operates a single-node Apache Airflow server with Green, OpenTofu and Ansible — Airflow under Docker with LocalExecutor, a host Postgres backed up to R2 with WAL-G, TLS and authentication through Caddy, and DAGs pushed from GitHub Actions over rsync. Use when initializing an airflow project, generating colors.yml, selecting a compute, DNS, SMTP or state provider, building or dry-running configuration, provisioning or destroying the server, or restoring the metadata database.
package-airflow-red
getcolors/airflow
Creates and operates a single-node Apache Airflow server with Red, OpenTofu and Ansible — Airflow under Docker with LocalExecutor, a host Postgres backed up to R2 with WAL-G, TLS and authentication through Caddy, and DAGs pushed from GitHub Actions over rsync. Use when initializing an airflow project, generating colors.yml, selecting a compute, DNS, SMTP or state provider, building or dry-running configuration, provisioning or destroying the server, or restoring the metadata database.
spark-engineer
neg0diay/opencode-skills
Use when writing Spark jobs, debugging performance issues, or configuring cluster settings for Apache Spark applications, distributed data processing pipelines, or big data workloads. Invoke to write DataFrame transformations, optimize Spark SQL queries, implement RDD pipelines, tune shuffle operations, configure executor memory, process .parquet files, handle data partitioning, or build structured streaming analytics.
spark-engineer
neonetz/opencode-skills
Use when writing Spark jobs, debugging performance issues, or configuring cluster settings for Apache Spark applications, distributed data processing pipelines, or big data workloads. Invoke to write DataFrame transformations, optimize Spark SQL queries, implement RDD pipelines, tune shuffle operations, configure executor memory, process .parquet files, handle data partitioning, or build structured streaming analytics.
ETL Pipeline
aaaaqwq/agi-super-team
Design and automate Extract, Transform, Load data pipelines for data integration and analytics
data-pipeline
sports-data-hq/hockey-skills
Builds automated sports analytics pipelines -- daily data pulls, scheduled prediction generation, model versioning, prediction tracking, and drift alerting. Use when user asks how to automate their model, how to run predictions every morning, how to set up a cron job for sports data, how to track model performance over time, or how to version their model. Do not use for one-time data pulls -- use game-lookup or puckapi-tool directly. Do not use for building the model itself -- see model-building. Do not use for backtesting historical performance -- see backtesting.
airflow-translations
apache/airflow
>
bigdata-analysis
oak-b/bigdata-analysis-skill
Use when analyzing data with Hive/Impala tables, writing SQL for data exploration, or building/deploying Spark ETL jobs on HDFS/YARN. ALWAYS trigger this skill — even if the user does not use these exact words — for any of the following: writing or reviewing a Spark Scala job, migrating SQL from Hive/Impala to Spark, creating or altering Hive tables, inserting data into partitioned tables, joining large tables in Spark SQL, using Spark UDFs, verifying table schema before coding, GROUP BY with text fields, OOM on large tables, INSERT column mismatch or silent data shifts, broadcast join stall or task explosion, DataFrame API being slow, cache() not materializing, metadata not visible after Spark write, date window off-by-one, control character regex not matching, Scala string interpolation bugs in Spark SQL, or any time the user says their Spark job is slow, wrong, or behaving unexpectedly.
nu
knoopx/pi
Reads, filters, transforms, and manipulates structured data using Nushell's pipeline commands. Use when working with CSV/TSV files, parsing command output, transforming tabular data, system administration tasks, or building data pipelines.
netlify-deploy
szoloth/skills
Deploy web projects to Netlify using the Netlify CLI (`npx netlify`). Use when the user asks to deploy, host, publish, or link a site/repo on Netlify, including preview and production deploys.
bigquery-etl-core
mozilla/bigquery-etl-skills
The core skill for working within the bigquery-etl repository. Use this skill when understanding project structure, conventions, and common patterns. Works with model-requirements, query-writer, metadata-manager, sql-test-generator, and bigconfig-generator skills.
segment-cli
sderosiaux/segment-cli
Manage your Segment.com workspace from the CLI. Use when the user asks about Segment sources, destinations, tracking plans, transformations, event delivery, violations, or wants to inspect their analytics pipeline. Triggers on "segment sources", "check my tracking plans", "event volume", "schema violations", "reverse ETL", "tap into events".
query-writer
mozilla/bigquery-etl-skills
Use this skill when writing or updating SQL queries (query.sql) or Python ETL scripts (query.py) following Mozilla BigQuery ETL conventions. ALWAYS checks for and updates existing tests when modifying queries. Coordinates downstream updates to schemas and tests. Works with bigquery-etl-core, metadata-manager, and sql-test-generator skills.
pyspark_etl
ivanshamaev/de-agent-skills
Use when designing, implementing, reviewing, or optimizing production PySpark ETL/DataFrame pipelines at GB-TB+ scale, including schemas, joins, partitioning, window functions, writes, UDF avoidance, and Spark performance diagnostics.
spark-savings
migokg/plugin-store
Spark Savings — earn Sky Savings Rate (SSR) on USDS/DAI. Trigger phrases: spark savings, deposit to spark, earn savings rate, sUSDS APY, sDAI rate, sky savings, MakerDAO savings, DSR rate, deposit USDS, stake USDS for yield, withdraw sUSDS, spark savings rate, 存入Spark储蓄, 查询储蓄利率, Spark储蓄年化, 存USDS赚利息
oleander-spark-submit
oleanderhq/skills
How to submit, monitor, and configure Spark jobs on oleander using the CLI & TypeScript SDK.
hit-spark
summerengine/summer-engine-agent
Use when authoring a hit-spark visual effect — a one-shot burst of additive billboard particles oriented to a surface normal, fired on impact. Trigger on "hit spark", "impact spark", "bullet hit", "sword clash", "metal-on-metal", "ricochet", "impact effect", "spawn sparks at hit point".
tiktok-spark-ads
appeeky/ua-skills
When the user needs to publish TikTok organic posts and use Spark Ads codes to run them as paid ads. Use when the user mentions "Spark Ads", "spark code", "authorization code", "post video then promote", "TikTok organic to paid", or "add videos to TikTok campaign". Follows tiktok-creative-strategy and tiktok-campaign-setup.
airflow-adapter
astronomer/agents
Airflow adapter pattern for v2/v3 API compatibility. Use when working with adapters, version detection, or adding new API methods that need to work across Airflow 2.x and 3.x.
airflow-to-zenml-migration
zenml-io/skills
>-
airflow-principal-engineer
modra40/claude-codex-skills-directory
|
optimize-data-model-compute
masthead-data/for-agents
Optimize BigQuery compute costs by assigning data models (Dataform, dbt, Airflow) to slot reservations or on-demand compute based on Masthead recommendations.
dj-pro
aiunlocked1412/claude-skill-unlock
AI DJ มืออาชีพ — setlist, mixing plan, BPM matching, crowd reading, club/wedding/corporate gig
clickzetta-studio-task-manager
clickzetta/clickzetta-skills
|
ETL Design
dasien/retrowarden
Design extract-transform-load patterns, data pipeline orchestration, and incremental loading strategies
pipeline
baekenough/oh-my-customcode
Invoke and resume YAML-defined pipelines by name — /pipeline auto-dev runs the full release pipeline
data-pipeline
wellux/claude-code-deprecated
>
airflow-dag-patterns
xfstudio/skills
Build production Apache Airflow DAGs with best practices for operators, sensors, testing, and deployment. Use when creating data pipelines, orchestrating workflows, or scheduling batch jobs.
airflow-best-practices
baekenough/oh-my-customcode
Apache Airflow best practices for DAG authoring, testing, and production deployment
gcp-managed-airflow-migrations
gemini-cli-extensions/data-agent-kit-starter-pack
Provides guidance for migrating Apache Airflow DAGs in Managed Service for Apache Airflow (MSAA; formerly Cloud Composer). Covers migration to Airflow 2.11.1 (MSAA Gen 2 and 3) and Airflow 3 (MSAA Gen 3), including environment inspection, GCS download/upload and scanning patterns for breaking changes.
spark-best-practices
baekenough/oh-my-customcode
Apache Spark 4.0.2 best practices for PySpark and Scala distributed data processing
apache-iceberg-expert
wbunker/skills-repo
Apache Iceberg table format expertise covering architecture, metadata layers, catalog management, schema/partition evolution, row-level operations, streaming, engine integrations, performance tuning, and governance. Use when working with Iceberg tables, designing lakehouse architectures, choosing between COW and MOR strategies, configuring catalogs (REST, Glue, Hive, Nessie), optimizing query performance, managing table maintenance, or comparing Iceberg with Delta Lake and Hudi.
spark-joy
cystema/skills
Review code through KonMari (Marie Kondo) principles and report whether it sparks joy. Use when the user wants a diagnostic review of whether code is clean, intuitive, has high glance value, is testable, and isn't shallow; when they ask you to "Marie Kondo," "spark-joy," "tidy-review," or quality-assess a codebase, file, module, or PR; or when they want a felt-sense review that goes beyond a lint pass. Trigger this whenever someone asks whether code "feels right," wants findings on what fails to earn its place, or wants a review oriented around joy, clarity, and keeping only what belongs — even if they don't say "KonMari." Produces a report with ranked findings, not edits.
data-engineering-data-pipeline
lza6/claude-code-cli-config
You are a data pipeline architecture expert specializing in scalable, reliable, and cost-effective data pipelines for batch and streaming data processing.
stream-chain
cornjebus/amair
Stream-JSON chaining for multi-agent pipelines, data transformation, and sequential workflows
pipeliner
a-canary/arc-skills
Build/run pipeliner modules (npm pi-pipeliner) — defineModule/defineChain + Thompson sampling. Use when wiring scheduled jobs, multi-step LLM pipelines, or anything cron-driven.
data-pipelines
kid-sid/codex-spellbook
Use when building or debugging data pipelines with Airflow or Prefect, writing dbt models or tests, designing incremental loads, implementing idempotent ETL/ELT jobs, validating data quality, or orchestrating multi-step data workflows.