68 lines
2.1 KiB
Markdown
68 lines
2.1 KiB
Markdown
|
---
|
||
|
|
id: bullmq-workers
|
||
|
|
title: BullMQ Background Workers
|
||
|
|
type: documentation
|
||
|
|
tags:
|
||
|
|
- bullmq
|
||
|
|
- redis
|
||
|
|
- jobs
|
||
|
|
- queue
|
||
|
|
- infra
|
||
|
|
status: published
|
||
|
|
author: asep
|
||
|
|
created_at: 2026-08-20
|
||
|
|
updated_at: 2026-08-20
|
||
|
|
---
|
||
|
|
|
||
|
|
# BullMQ Background Workers
|
||
|
|
|
||
|
|
BullMQ runs the async indexing/embedding pipeline. Jobs are enqueued by the CLI, the
|
||
|
|
git-sync webhook, or an MCP tool, and drained by a long-running worker connected to the
|
||
|
|
shared Redis instance.
|
||
|
|
|
||
|
|
## Connection requirements
|
||
|
|
|
||
|
|
BullMQ requires an **ioredis** connection with `maxRetriesPerRequest: null`. A finite
|
||
|
|
retry count causes the cryptic `Connection in key mode` error on blocking commands.
|
||
|
|
The shared client in `packages/queue` sets this correctly:
|
||
|
|
|
||
|
|
```ts
|
||
|
|
const opts: RedisOptions = {
|
||
|
|
maxRetriesPerRequest: null,
|
||
|
|
lazyConnect: true,
|
||
|
|
enableOfflineQueue: true,
|
||
|
|
};
|
||
|
|
```
|
||
|
|
|
||
|
|
## Job model
|
||
|
|
|
||
|
|
Three job types flow through the `mcpedia-index` queue (prefix `mcpedia:` on Redis):
|
||
|
|
|
||
|
|
| name | data | action |
|
||
|
|
|------------|-------------------|---------------------------------|
|
||
|
|
| `index-doc`| `{ relPath, reason }` | index one content file |
|
||
|
|
| `index-all`| `{ reason }` | full corpus reindex |
|
||
|
|
| `reindex` | (legacy) | alias of full |
|
||
|
|
|
||
|
|
Job IDs use a `__` separator (`doc__<slug>`, `full__<ts>`) — BullMQ reserves `:` for
|
||
|
|
repeatable jobs, so a literal `:` in a custom jobId is rejected.
|
||
|
|
|
||
|
|
## Worker lifecycle
|
||
|
|
|
||
|
|
The worker is a `Worker` with `concurrency: 4`. Each job calls the shared
|
||
|
|
`indexContentFile` / `runFullIndex` entry points in `@mcpedia/core` — the same code
|
||
|
|
path the CLI uses, so behavior never diverges. On completion it logs; on failure it
|
||
|
|
logs the reason and the job is retried per BullMQ defaults.
|
||
|
|
|
||
|
|
Graceful shutdown: `worker.close()` on `SIGINT`/`SIGTERM`. systemd sends SIGTERM on
|
||
|
|
stop, so the process exits cleanly and in-flight jobs are returned to the queue.
|
||
|
|
|
||
|
|
## Inspecting state
|
||
|
|
|
||
|
|
```bash
|
||
|
|
bun run enqueue --all # enqueue a full reindex
|
||
|
|
curl localhost:4020/trpc/queueStatus # waiting/active/completed/failed
|
||
|
|
```
|
||
|
|
|
||
|
|
A stuck queue (waiting > 0, active = 0) means the worker died — check
|
||
|
|
`systemctl status mcpedia-worker` and the journal.
|