Background Job & Task Queue Architecture

What it does

/data-pipeline is a Cursor slash command. Type it in chat to run a saved workflow: Background Job & Task Queue Architecture.

Architects asynchronous background processing: Redis task queues (BullMQ), worker processes, exponential retries, and data import pipelines.

BullMQ, Redis worker pools, retry backoff, and batch CSV processing. Unlike a skill, a command is something you invoke on purpose. The agent does not decide to run /data-pipeline for you.

Why it exists

Founders retype the same multi-step prompt until it rots. /data-pipeline exists so the pipeline, all 4 steps of it, is a file in .cursor/commands, versioned with the repo.

It ships in the Engineering Kit. It is wired to the Data Pipeline & ETL Engineer (data-engineer) and Senior Backend Architect (backend-architect) agents. Required skills: Asynchronous Background Jobs & Task Queues; PostgreSQL & Drizzle ORM Schema Standards.

When to use it

• Use when processing long-running operations like email dispatch, PDF generation, or bulk exports.
• Use /data-pipeline when you want that pipeline, not a freeform chat. If you only need one step, use a narrower command or a single agent.
• Start a new chat. Do not run this command in a thread that just wrote marketing copy.

When not to use it

• Do not run /data-pipeline as a substitute for reading the diff. The command produces files; you still gate them.
• Do not chain it into a 40-turn chat. Fresh context is part of the design.
• Do not run it if you have not filled CURSOR.md. The pipeline will invent a stack.

Example workflow

1. Configure Redis connection and BullMQ queue definitions
2. Implement worker process handler with exponential failure retry backoff
3. Add job progress tracking and status check API endpoints
4. Set up dead-letter queue alerts for repeated failures

Example usage

Type this in Cursor chat: /data-pipeline Build Redis Queue worker for async bulk email dispatch and analytics aggregation

The command file tells the session which agents to adopt and which skills to read. You should see phase headers, not a single dump of code.

If a phase fails its gate, stop. Do not add 'just continue'.

Example output

• Expected artifact: src/lib/queue.ts
• Expected artifact: src/workers/...

Best practices

• Keep the prompt specific. /data-pipeline Build Redis Queue worker for async bulk email dispatch and analytics aggregation is the shape: object, constraint, and outcome.
• Let the listed agents work in order: data-engineer → backend-architect.
• Save outputs in the repo. Chat-only answers evaporate.
• Engineering commands should leave tests or an audit note, not only implementation files.

Common mistakes

• Typing /data-pipeline with no object ('do the thing'). The pipeline will guess.
• Re-running the command in the same chat after a failed gate instead of fixing the failing file.
• Editing the command file to skip review so it 'goes faster'.
• Skipping the Asynchronous Background Jobs & Task Queues skill that the command depends on.