Real file from a public repo · synced Aug 26, 2026
spark-python-data-source
Build custom Python data sources for Apache Spark using the PySpark DataSource API — batch and streaming readers/writers for external systems. Use this skill whenever someone wants to connect Spark to an external system (database, API, message queue, custom protocol), build a Spark connector or plugin in Python, implement a DataSourceReader or DataSourceWriter, pull data from or push data to a system via Spark, or work with the PySpark DataSource API in any way. Even if they just say "read from X in Spark" or "write DataFrame to Y" and there's no native connector, this skill applies.
Works with
Install in three steps
- 1
Download the ZIP
One file per agent, plus INSTALL.md with the exact paths below.
- 2
Claude Code: copy SKILL.md into your skills folder
.claude/skills/spark-python-data-source-1g910j/SKILL.md # this project ~/.claude/skills/spark-python-data-source-1g910j/SKILL.md # every project
- 3
Other agents: use the matching file
skill.mdc goes in Cursor’s .cursor/rules/. AGENTS.md works as GitHub Copilot instructions, a legacy .cursorrules, or a Codex / Gemini CLI instructions file.
More Data Engineering skills
All Data Engineering skills
data-pipelineclaude-office-skills/skillsData pipeline and ETL automation - extract, transform, load workflows for data integration and analyticsData Engineering4.1kETL Pipelineclaude-office-skills/skillsDesign and automate Extract, Transform, Load data pipelines for data integration and analyticsData Engineering4.0k
data-throughput-acceleratoraffaan-m/eccUse when large data ingestion, backfill, export, ETL, warehouse loading, manifest catch-up, or table synchronization needs to become much faster while preserving data correctness.Data Engineering3.8k
