event-driven-kafka-listener-skill

---
name: "event-driven-kafka-listener-skill"
description: "Use when implementing, reviewing, refactoring, or debugging Kafka listener/event-driven consumer behavior in backend microservices, especially when AsyncAPI contracts, EventDispatcher/EventHandler strategies, schema evolution, idempotency, offset/ack semantics, poison-pill handling,…"
license: "MIT"
---

# Event-Driven Kafka Listener Skill

## When to Use This Skill
Use when implementing, reviewing, refactoring, or debugging Kafka listener/event-driven consumer behavior in backend microservices, especially when AsyncAPI contracts, EventDispatcher/EventHandler strategies, schema evolution, idempotency, offset/ack semantics, poison-pill handling, retry/DLQ/replay, SASL_SSL, Kafka ACLs, tracing, structured logging, metrics, partition/rebalance behavior, graceful shutdown, Testcontainers Kafka validation, or production-readiness evidence are in scope.


## Roles

- **Backend Engineers**: Own assigned stage and required checks.

- **Integration Architects**: Own assigned stage and required checks.

- **Code Reviewers**: Own assigned stage and required checks.



## Prerequisites

- Confirm the target topic, AsyncAPI path/content, event names, target stack, and existing microservice conventions.

- Identify the event envelope/header strategy and event type discriminator before writing dispatcher logic.

- Identify idempotency key, persistence boundary, offset/ack mode, retry/DLQ/replay policy, authentication protocol, Kafka ACL/certificate boundary, tracing/logging/metrics expectations, and test framework.

- Identify schema evolution/versioning policy, event coverage matrix requirements, consumer group/rebalance behavior, and quality-gate/coverage commands before claiming production readiness.

- Classify whether the request is a new listener architecture, adding a new event type, changing reliability/security/observability, or debugging an existing listener failure.

- Policy check: SASL_SSL for secure transport

- Policy check: Secure secret management

- Policy check: traceparent/W3C tracing for distributed observability

- Policy check: Structured JSON logging for auditability


## Workflow


### Step 1: Ingest AsyncAPI and listener boundary
- Read AsyncAPI channels, topic names, messages, headers, payload schemas, examples, auth, and compatibility constraints.
- Output gaps for missing event type discriminator, schema evolution policy, idempotency key, offset policy, retry/DLQ/replay policy, auth, ACLs, or schema details before implementation.






### Step 2: Design extensible listener architecture
- Define a thin Kafka listener/consumer adapter, event envelope, EventDispatcher, and EventHandler<T> strategy contract.
- Ensure new event types are added by registering handlers, not by modifying the dispatcher or listener core.






### Step 3: Implement reliability and security controls
- Add idempotency guard, transaction boundary, offset/ack semantics, poison-pill/deserialization handling, retry/backoff, DLQ envelope/replay routing, and secure SASL_SSL/secrets/ACL configuration.
- Separate retryable, non-retryable, deserialization, validation, duplicate, unknown-event, authorization, and replay errors.






### Step 4: Add observability and operations hooks
- Emit structured lifecycle logs for received, processing, success, retry, failed, and DLQ outcomes without sensitive payload leakage.
- Propagate traceparent/W3C context, record metrics for processed/failed/retried/DLQ/duplicates/lag/rebalance/unknown-event outcomes, and document scaling/partition/replay assumptions.






### Step 5: Verify with tests and readiness evidence
- Generate unit tests for dispatcher routing, handler registration, schema compatibility, deserialization safety, idempotency, unknown types, offset decision, and error classification.
- Generate Testcontainers Kafka integration tests for consume/process/persist, retry-to-success, retry-to-DLQ, DLQ replay metadata, poison pill, duplicate events, and rebalance-safe processing.
- Require 90% relevant unit coverage and project-native build/test/quality-gate evidence before any production-ready claim.






Read references/compliance.md before claiming production readiness, secure Kafka transport, broker ACL readiness, idempotent processing, offset safety, retry/DLQ/replay readiness, traceability, or structured logging compliance.









## Recipe Orchestrator Contract
- Contract version: `1.0.0`
- Skill ID: `event-driven-kafka-listener-skill`
- Runtime role: `bounded_skill_step`
- Workflow steps: `5`
- Recipe supplies external context packs: `accepted_from_recipe`
- Recipe supplies external control packs: `accepted_from_recipe`
- Control tags:

  - `domain:api_integration`

  - `type:technical`

  - `compliance:SASL_SSL for secure transport`

  - `compliance:Secure secret management`

  - `compliance:traceparent/W3C tracing for distributed observability`

  - `compliance:Structured JSON logging for auditability`

  - `compliance:Strict adherence to Open/Closed Principle/SOLID`

  - `compliance:Minimum 90% relevant unit coverage for production listener readiness`

  - `constraint:Must use Spring Boot and Spring Kafka stack where applicable`

  - `constraint:Must implement EventDispatcher with EventHandler<T> strategies`

  - `constraint:Registering new handlers must not modify the dispatcher core`

  - `constraint:Must use non-blocking retries with exponential backoff`

  - `constraint:Must enforce idempotency via event_id, distributed lock, or DB unique constraint`

  - `constraint:Offsets must commit only after a durable processing, duplicate-safe skip, or DLQ/quarantine decision`

  - `constraint:Must define DLQ envelope and replay safety model`

  - `constraint:Must maintain AsyncAPI event coverage matrix for multi-event listeners`

  - `constraint:Must output strictly concrete code guidance, avoiding generic messaging advice`

- Contract artifact: `references/recipe-interface.json`
- Usage boundary: `references/recipe-usage.md`


## On Failure
If any mandatory gate fails, stop execution and escalate with evidence.
- Poison-pill messages causing deserialization loops or crashes
- Duplicate event processing due to lack of idempotency
- Blocking retries stalling partition consumption
- Unclassified errors leading to improper DLQ routing
- Loss of partition ordering or backpressure failure
- Event loss due to early offset commit
- Unsafe DLQ replay causing duplicate business side effects
- Unknown event type causing infinite retries or silent drops
- Schema evolution breaking old producer payloads or replayed historical events
- Kafka ACLs, secrets, logs, or DLQ records exposing unauthorized data
- Consumer lag accumulation without metrics alerting


## Deep Rule Pack
This package includes a maintainable rule corpus for high-fidelity agent behavior.

- Read `AGENTS.md` when detailed rule guidance, incorrect/correct examples, and evidence expectations are needed.
- Read `rules/` when a specific atomic rule applies to the current task.
- Run `python scripts/validate_rules.py` after editing rules.
- Run `python scripts/build_agents.py` after changing `rules/` to refresh the compiled guide.
- Use `test-cases.json` as evaluation scenarios for the generated skill.



## Reference Files

- [Recipe Interface Contract](references/recipe-interface.json) — Read references/recipe-interface.json if an orchestrator needs machine-readable skill contract fields.

- [Recipe Usage Boundary](references/recipe-usage.md) — Read references/recipe-usage.md if this skill is invoked as a recipe step with external context or control packs.

- [Tool Contract](references/tools.md) — Read references/tools.md if you need AsyncAPI, Spring Kafka, Kafka client configuration, Maven/Gradle, JUnit, Testcontainers Kafka, Jackson serialization, retry/DLQ, or schema validation commands.

- [Compliance Requirements](references/compliance.md) — Read references/compliance.md if a regulatory, audit, or policy check is triggered.

- [Edge Cases](references/edge-cases.md) — Read references/edge-cases.md if the process stalls or exception conditions appear.

- [Compiled Agent Rules](AGENTS.md) — Read AGENTS.md when detailed rule guidance, examples, and validation expectations are needed.