# /data-pipeline - Background Job & Task Queue Architecture

> BullMQ, Redis worker pools, retry backoff, and batch CSV processing.

Canonical: https://www.agentickit.co/commands/data-pipeline
Last updated: 2026-11-18

Architects asynchronous background processing: Redis task queues (BullMQ), worker processes, exponential retries, and data import pipelines.

**Usage:** `Use when processing long-running operations like email dispatch, PDF generation, or bulk exports.`

## Pipeline
1. Configure Redis connection and BullMQ queue definitions
2. Implement worker process handler with exponential failure retry backoff
3. Add job progress tracking and status check API endpoints
4. Set up dead-letter queue alerts for repeated failures

## Output files
- `src/lib/queue.ts`
- `src/workers/...`

**Claude Code frontmatter:** `argument-hint: [pipeline source and destination]` · `model: inherit`

## Related
- [data-engineer](https://www.agentickit.co/agents/data-engineer)
- [backend-architect](https://www.agentickit.co/agents/backend-architect)
- [data-pipelines](https://www.agentickit.co/skills/data-pipelines)
- [postgres-drizzle](https://www.agentickit.co/skills/postgres-drizzle)
