profile-source

---
name: "profile-source"
description: "Convert an upstream system (database table, SaaS endpoint, streaming topic, partner feed) into a Source Profile markdown file under docs/sources-of-record/, enforcing the Stage 1 kill-switch — an accepted profile must name at least one consumer DGQ. Use when the user says \"/profile-source\", \"profile this source\", \"characterise this source\", \"interrogate this upstream\", \"document this source\", or describes an upstream ingest (\"we have a new source\", \"we're about to ingest X\", \"we need to characterise this API\", \"the team just gave us access to their database\") and wants the operational and contractual model captured before any pipeline is built. Routes to /write-adr when the decision is a cross-cutting platform commitment rather than a per-source profile."
license: "Apache-2.0"
---

# /profile-source — Source Profile interview

This skill runs a short, branching interview that converts an upstream source the data team does not own into a **Source Profile** markdown file at `docs/sources-of-record/<system>-<entity>.md` in the consumer's project. It is an **interview, not a template** — every profile that lands has been pressure-tested against the Generation kill-switch (a named consumer DGQ), the schema-authority honesty test, the streaming/batch neutrality rule, and the "is this actually an ADR moment?" routing test.

A Source Profile is *not* a statistical column-stats profile (cardinality, null rates, distributions). It is the operational and contractual model of the source — schema authority, delivery model, late-arrival behaviour, pager, replay window — deliberately human-authored and never inferred from a live connection. The **Generation kill-switch** is the one thing it must carry: an accepted profile names at least one consumer DGQ, or a source nobody consumes does not deserve a profile.

This skill is self-contained: the Source Profile contract it writes against ships beside it at [`references/source-profile.md`](references/source-profile.md), and the linter that enforces that contract ships at [`scripts/lint-source-profile.sh`](scripts/lint-source-profile.sh). Read the contract for the frontmatter fields, enum domains, and body section order before running — do not duplicate that material into the interview; reference it.

**Optional deeper reading** — not required to run this skill (a per-skill install works without them): the full repo worldview (including the `CONTEXT.md` entries "Source Profile" and "Generation kill-switch") in [`CONTEXT.md`](https://github.com/sleeplessv/relentless-data-skills/blob/main/CONTEXT.md) and the lifecycle reference in [`docs/data-engineering-101.md`](https://github.com/sleeplessv/relentless-data-skills/blob/main/docs/data-engineering-101.md).

## Soul of the skill: the Generation kill-switch

The single load-bearing question this skill exists to force is:

> **Which Decision-Grade Question under `docs/requirements/` is going to consume this source?**

If the user cannot point to at least one accepted-or-draft DGQ in `consumed_by`, the source does not deserve a profile. A Source Profile with no named consumer is the structural twin of a DGQ with `action_change: nothing` — documentation for documentation's sake, the failure mode this stage exists to refuse. Surface that conclusion explicitly. Recommend the user do their exploratory poking informally — a scratch note, a Slack thread — and come back once a DGQ exists.

Do not soften the kill-switch. Do not paper over the absence by inventing a plausible-sounding consumer DGQ on the user's behalf. The kill-switch (Turn 1) is the one place in this skill where the user must speak first; a recommendation there would coach them past the very gap the skill exists to surface.

The linter enforces the structural contract — an `accepted` profile with an empty `consumed_by` fails `scripts/lint-source-profile.sh`. This skill is the upstream half: it refuses to start the interview without one named consumer.

## Target shape

- **Length.** 6–10 substantive exchanges per profile is typical. Keep going until the operational contract is honestly named — don't truncate to hit the cap, and don't pad to fill it. Sharp, not exhausting.
- **Output.** One markdown file per source at `docs/sources-of-record/<system>-<entity>.md`. If the user is profiling several entities in one upstream system (multiple tables in a database, multiple endpoints in a SaaS API), write one fully-drilled profile first, the rest queued as `status: draft` so a later run can pick them up.
- **Tone.** Interrogative, not adversarial. The goal is to find the honest contract — including "us, by accident" answers — not to make the user feel bad for inheriting a messy upstream.

## Resumability check (run this first)

Before starting a fresh interview, scan `docs/sources-of-record/` in the consumer's project. If one or more Source Profile files exist, **pause and ask**:

> I see existing Source Profiles in `docs/sources-of-record/`. Do you want to:
>
> 1. Continue an existing **draft** or **needs-follow-up** profile (resumable)
> 2. Start a **new** profile
> 3. **List** all profiles and their statuses

Treat `status: accepted`, `status: rejected`, and `status: superseded-by-<path>` files as immutable from this skill's perspective — do not offer to edit them. If the user picks "continue", load the chosen file and resume from the first unfilled section (commonly the section that corresponds to the next unanswered turn). If they pick "list", print one line per file: `<path> — <status> — <system>/<entity>`.

If `docs/sources-of-record/` does not exist, create it on first write — not before.

## Steps

Carry out the seven turns below in order. **One question per turn**, not a wall of multiple questions, except where the spec explicitly says "combined turn" (Turn 6). If the user volunteers material that answers a later turn early, accept it and skip ahead — do not re-ask. Do not deliver multi-paragraph monologues; keep each prompt tight.

Two cross-cutting rules apply to every turn:

- **Recommend before you ask.** For each turn's question, propose a plausible answer first — derived from the source name itself, the cited DGQs, adjacent profiles/ADRs, or sensible defaults — then ask the user to accept, reject, or refine it. Reacting to a concrete proposal is faster and sharper than generating one cold. If you genuinely have no basis for a recommendation, say so and ask open. Never invent the kill-switch answer (Turn 1) this way — the named consumer DGQ is the one place the user must speak first.
- **Explore before you ask.** If a turn's question can be answered by looking at the project — existing DGQs in `docs/requirements/`, ADRs in `docs/adr/`, adjacent Source Profiles in `docs/sources-of-record/`, glossary entries in `CONTEXT.md`, or the data and code itself (table schemas, dbt models, orchestrator configs, OpenAPI specs in the repo) — look first, then bring what you found into the turn. Do not make the user re-derive what the repo already knows. If a stakeholder term collides with `CONTEXT.md`'s glossary, call it out the moment you notice it.

### Turn 1 — Name the consumer DGQ(s)

Ask, verbatim or close:

> Which DGQ(s) under `docs/requirements/` need this source? Point me at one or more by filename or by decision.

Before asking, scan `docs/requirements/` and list the available DGQ files with their `decision` line — so the user is choosing from a concrete set, not naming a file from memory. If `docs/requirements/` is empty or absent, that is itself the answer: there is no consumer; the source does not deserve a profile yet.

Three branches:

- **One or more named consumer DGQs.** The user points to one or more files with `status: accepted` or `status: draft`. Record the paths; they go into `consumed_by`. Continue to Turn 2.
- **A DGQ exists but it is `rejected` or `needs-follow-up`.** Push back: *"That DGQ was rejected — what's the live decision this source is feeding?"* / *"That DGQ is still waiting on the stakeholder — should we finish Stage 0 before profiling Stage 1?"* Do not start the profile against a non-live DGQ; the consumer link must be a decision someone is actually going to act on.
- **No DGQ exists, or the user cannot name one.** **Trigger the kill-switch.** Do not write a profile. Tell the user:
  > There's no named consumer DGQ for this source. A Source Profile without a downstream decision is documentation nobody is using — exactly the failure mode this stage refuses. I recommend doing any exploratory poking informally for now — a scratch note or a Slack thread — and coming back to `/profile-source` once you have at least one DGQ under `docs/requirements/` that needs this source. If you'd like to draft that DGQ now, run `/gather-requirements`.

  Then exit cleanly. Do not start drafting a profile "just in case." The kill-switch is the differentiator; surface it without apology.

### Turn 2 — Identify the source

Ask:

> What's the source — `system`, `entity`, and who operates it upstream? Is this one source, or several?

Capture three fields:

- **`system`** — kebab-case slug for the source system (`stripe`, `salesforce`, `kafka-orders`, `partner-fulfilment-co`).
- **`entity`** — kebab-case slug for the specific table, topic, endpoint, or feed (`charges`, `account`, `order-events`).
- **`owner`** — the team or person who operates the source upstream. Often not the data team. If the honest answer is "nobody, it's a legacy system," record that — `owner: legacy-unmaintained` or similar — rather than inventing a custodian.

Listen for plural sources hiding inside one ask. "Profile our Stripe integration" often unpacks into several profiles — `stripe-charges`, `stripe-customers`, `stripe-subscriptions`. Each is a separate Source Profile.

**Branching rule.** If more than one source surfaces:

1. Ask the user to pick **one** to drill fully right now.
2. For each of the others, write a stub Source Profile to `docs/sources-of-record/<system>-<entity>.md` with `status: draft`, frontmatter populated with the consumer DGQ(s) from Turn 1 and the `system`/`entity`/`owner` identifying the stub, all other linter-required enums set to safe placeholders (`delivery_mode: pull-batch`, `cadence: tbd`, `sensitivity: internal`, `schema_authority: producer-owned`, `late_arrival_window: unknown`), and a note in `## Open follow-ups` that this profile needs resumption. Body sections present but with TBD bullets, so the linter passes.
3. Tell the user the stubs are saved and this session will drill the chosen source. Do not abandon them silently.

### Turn 3 — Schema and schema authority

Ask:

> Where does the source-of-truth schema live, and who can change it without telling you?

Listen for two distinct facts:

- **Schema location.** An OpenAPI doc URL, a JSON Schema file in a repo, a `CREATE TABLE` statement, a Protobuf in `.proto`, an Avro registry, "the engineer's head and the production database," "this Confluence page Kim updates sometimes." Record honestly.
- **Schema authority.** Map the answer to one of `producer-owned | data-team-owned | shared-contract`:
  - **`producer-owned`** — the upstream team can change it without notice. Most application databases and SaaS APIs land here. The most common honest answer.
  - **`data-team-owned`** — the data team owns the schema definition (e.g., a JSON contract the producer fills in). Rare for sources the team does not control; more common for internal events.
  - **`shared-contract`** — there is a written, dual-signed contract (data contracts, formal schema registry with producer commits) that constrains both sides. Aspirational for most teams; verify before recording.

Push the user toward honesty here. The temptation is to write `shared-contract` because it sounds responsible. The skill's stance is explicit: **"what is *actually* true today, not what you want to be true."** If the producer can drop a column tomorrow without a heads-up, that is `producer-owned` and the body should name the risk in plain language.

If the schema location is "nowhere written down," record that — `schema_authority: producer-owned` and a body note that the schema is implicit. That is a risk worth flagging, not a gap to hide.

### Turn 4 — Delivery model (branch on the user's answer)

Ask:

> How do records reach you — streaming, batched pulls, pushed batches, CDC, webhooks, manual exports?

Map the answer to `delivery_mode`: one of `cdc | pull-batch | push-batch | streaming | webhook | manual-export`. **Do not default.** Streaming and batch are both first-class in this repo; the framing must come from the user's answer, not from a built-in lean.

Branch the follow-up questions on the answer. Ask only the relevant sub-questions; don't ask the streaming questions of a batch source and vice versa.

**Streaming / CDC / webhook:**

- **Ordering guarantee.** Per-key, global, none?
- **Partitioning / natural key.** What partitions the stream, and is that the natural key for the entity?
- **Dedup key.** How are duplicates identified — by message id, by composite key, by content hash, not at all?
- **Replay window.** How far back can the topic / log / endpoint replay if the team loses a window? (This also seeds `late_arrival_window` and the Turn 5 replay question.)

**Pull-batch / push-batch / manual-export:**

- **Pull window.** What's the time window each batch covers — last hour, last 24h, last calendar day in source TZ?
- **Idempotency.** If the pipeline reruns the same window, does it produce the same result? (Honest answer is often "mostly, except for soft-deletes.")
- **Watermark / cursor.** What does the team key off — updated_at, an export timestamp, a sequence id, a filename convention?
- **Late-arrival behaviour.** Do records ever arrive in a window *after* that window has been pulled?

**Both branches:**

- **`late_arrival_window`** — free-text, but specific. `"up to 24h, observed empirically"`, `"unbounded; corrections can arrive months later"`, `"not applicable — append-only stream with watermark"`. If the team has never measured it, `"unknown — never measured"` is the honest answer and goes into `## Open follow-ups`.

Record the answers in the body's `## Delivery model` section as a short bulleted list — one bullet per sub-question, not prose. Streaming profiles and batch profiles end up the same shape: a list of named contracts, not a wall of text.

### Turn 5 — Operational contract

Ask:

> When the source breaks, who gets paged, what's the recovery procedure, how far back can it replay, what's the upstream retention, and what SLA — formal or informal — does the upstream commit to?

Five facts to capture in the body's `## Operational contract` section:

- **Pager.** Who is paged — by team, by rotation, by person. If the honest answer is "us, by accident," record that. *"The data team finds out when the Looker dashboard goes blank"* is a real, recordable operational contract; pretending it's a Slack channel monitored by the upstream team is the failure mode.
- **Recovery procedure.** What happens when the source breaks — a backfill script, a manual re-export, a phone call to the partner, "the producer team usually notices and fixes it within a day." Honest, even if embarrassing.
- **Replay window.** How far back the source can replay if the team loses a window. Kafka topic retention, S3 export retention, "the partner FTP keeps 7 days." This is the operational analogue of the `late_arrival_window` recorded in Turn 4.
- **Upstream retention.** How long the upstream keeps the data at all. Separate from replay window: replay is "can we re-fetch the same data?", retention is "how long does it exist upstream?"
- **SLA.** Formal SLA, informal SLA, or none. *"99.9% uptime in their MSA"*, *"the producer team aims for 1-hour freshness but has never written it down"*, *"no SLA, the source is best-effort"*. The skill records what is true; it does not negotiate a better SLA.

Cadence (the frontmatter field) gets recorded too — `continuous | hourly | daily | weekly | ad-hoc | one-off`. Often derivable from the delivery model answer; ask explicitly if not.

### Turn 6 — Undercurrent sweep (single combined turn)

In **one** turn, ask the user to answer three sub-questions. Combine them — do not break this into three separate exchanges.

> Quick undercurrent sweep:
>
> 1. **Sensitivity** — is the underlying data `public`, `internal`, `confidential`, or `regulated`?
> 2. **Governance** — any PII, residency, or retention obligations worth recording? (e.g., "contains GDPR-scoped emails", "must stay in EU", "30-day retention required by contract")
> 3. **Cost envelope** — rough scale class: small (a handful of rows / a single API call), medium (recurring scheduled pull, single-digit GB), large (continuous stream, terabyte scan, expensive partner API).

Map answers into the corresponding frontmatter field (`sensitivity`) and body notes. Governance and cost envelope live in the body — under `## Source identity` or `## Operational contract` as appropriate — not as separate frontmatter fields. If the user says "don't know" on any, mark that with a short placeholder (`unknown`, `tbd`) and add a line to `## Open follow-ups`. Do not infer values silently.

### Turn 7 — Write the profile

Pick a filename: `<system>-<entity>.md` under `docs/sources-of-record/`, both slugs kebab-case throughout (lowercase letters, digits, hyphens; at least one hyphen separating system and entity).

Write the file shaped exactly as the contract in [`references/source-profile.md`](references/source-profile.md) specifies — the eleven frontmatter fields and the five body sections in order. The earlier turns already gathered the content each field and section needs: the consumer DGQ(s) from Turn 1, `system`/`entity`/`owner` from Turn 2, schema and `schema_authority` from Turn 3, `delivery_mode` and `late_arrival_window` from Turn 4, the operational contract and `cadence` from Turn 5, and `sensitivity` plus the governance/cost notes from Turn 6. Read the contract for the exact field list, enum domains, and section order rather than reproducing them here.

`consumed_by` carries the DGQ path(s) from Turn 1; `related_adrs` stays `[]` at first write — populate it later only if an ADR-moment surfaces and the user agrees to run `/write-adr`. Do not invent ADR paths.

After writing, print a one-line confirmation:

```text
Wrote docs/sources-of-record/<system>-<entity>.md
  status: <status>
```

No body summary echoed back — the user just answered the questions; reading it back is noise. Then surface the next queued profile (if Turn 2 created stubs): *"You have N draft profiles queued from this session — want to drill the next one now?"* If the user declines, exit cleanly.

Run the bundled structural linter `scripts/lint-source-profile.sh` — it sits beside this `SKILL.md`, in the skill folder's own `scripts/` — as a cheap sanity check after writing. Resolve it relative to this skill folder and invoke it from the consumer project root so it picks up `docs/sources-of-record/` (or point it elsewhere with `SOURCES_OF_RECORD_ROOT=<dir>`). It is **best-effort**: if the script is present in this install, run it; if it is absent — a partial install that dropped the skill's `scripts/`, say — skip it silently and rely on the consumer's pre-commit hook. It is not a CI gate — just a fast structural check that frontmatter, enums, body sections, and the kill-switch are intact. If the schema enforced by the linter drifts from this prompt, **the linter wins** — record a fix-up issue and ship the profile that passes the linter.

## ADR-moment detection

At **any** point in the interview, if the decision being recorded is a **cross-cutting platform commitment** rather than a per-source profile, pause and route. Signals that you are looking at an ADR-moment:

- The user says something that applies across sources, not just this one. Two example phrasings to listen for:
  > *"We always CDC OLTP sources — that's just our pattern."*
  > *"All our SaaS sources flow through Fivetran; we don't do direct API integrations any more."*

  These are platform commitments, not per-source facts.
- The decision is about **a default delivery mode** across a class of sources (`"streaming for events, batch for application DBs"`).
- The decision is about **a default schema-authority posture** across producers (`"we treat every SaaS source as producer-owned and version it ourselves"`).
- The decision is about **a default pager / on-call posture** (`"data team is always the pager of last resort"`).
- The decision applies **across many sources, teams, or pipelines** rather than to this one entity.

When you spot one of these, stop the profile interview and say something like:

> This isn't a per-source profile — it's a cross-cutting platform commitment. The right artefact is an ADR, not a Source Profile. I recommend running `/write-adr` separately to capture this decision in `docs/adr/`. After the ADR lands, you can come back and link it from any related Source Profiles via the `related_adrs` frontmatter, and the per-source profile will just *reference* the ADR rather than restate it.

Do **not** inline the ADR scaffold here. One skill, one artefact — the ADR belongs to `/write-adr`. If the user insists on continuing the profile, write it but note in `## Open follow-ups` that the recorded pattern likely should be promoted to an ADR.

## After writing

- Do not summarise the body of the Source Profile back to the user — they just answered the questions; reading it back is noise. The one-line confirmation is sufficient.
- If the conversation surfaced a cross-cutting commitment that the user agreed to record separately, remind them once at the end: *"Run `/write-adr` for `<topic>` when you're ready — it's not part of this profile's scope."*
- If a profile ended `needs-follow-up`, name the specific question the user needs to take to the source owner. Vague follow-ups rot; named ones get answered.
- If a profile ended `rejected` — the source was profiled defensively and the user decided the consumer DGQ doesn't justify it after all — leave it in place with the rejection rationale in `## Open follow-ups`. The rejection is a durable record of the deliberate decision not to build, not a TODO to revisit.