mcpedia/PHASES.md

MCPedia — Phase Status

Legend: ✅ built · 🟡 partial · ⬜ deferred

Phase 1 — MVP (✅ DONE)

Capability	Status	Notes
Monorepo (bun + Turbo)	✅	apps/{web,mcp}, packages/{types,config,db,parser,search,core}, scripts
Content as Markdown	✅	content/{docs,writeups,research,notes}/, Git-tracked
Frontmatter parsing	✅	@mcpedia/parser (gray-matter)
Postgres metadata	✅	@mcpedia/db Drizzle, documents table
Postgres FTS	✅	weighted tsvector (title A / body B), GIN index, ts_rank+ts_headline
Core services	✅	Document / Content / Search — single business-logic layer
Indexer	✅	scripts/indexer.ts walks content/ → upserts
Web UI (Next 16)	✅	home (list), doc view (SSG), search (dynamic). react-markdown render
MCP server (stdio)	✅	4 tools; in-memory smoke test passing
Hybrid/semantic search	⬜	Phase 2

Phase 2 — Semantic + API

• packages/embeddings — EmbeddingProvider interface + OpenRouter provider (via 9router /v1, encoding_format:"float"); chunkText + embedChunks batcher. EMBED_DIM=2048 discovered live.
• Schema document_chunks (id, document_id→documents.id cascade, slug, chunk_index, content, embedding real[]). Stored as real[] because pgvector is not installed on the shared imrnes Postgres (installing needs host-level apt — deferred). Cosine computed in-app; instant for a KB-sized corpus.
• scripts/indexer.ts — chunks + embeds + upserts (per-doc replace).
• @mcpedia/search — semanticSearch (cosine) + hybridSearch (FTS + cosine, RRF fusion). keywordSearch unchanged.
• apps/api — Hono + tRPC v11 (@trpc/server fetch adapter, @hono/node-server on :4020): search, semanticSearch, hybridSearch, getDocument, listDocuments, related.
• MCP server — added semantic_search + hybrid_search tools (6 total).
• Web search — keyword/hybrid toggle (?mode=hybrid), hybrid reaches semantically-related docs keyword misses.

Phase 3 — Async + Scale ✅ DONE

• Redis + BullMQ background indexing / embedding workers — packages/queue (ioredis singleton + BullMQ Queue/Worker, prefix mcpedia: on shared imrnes Redis :6379); apps/worker runs startWorker(). Three job types: index-doc, index-all, reindex. Single indexing entry point indexContentFile/runFullIndex in @mcpedia/core shared by the script, worker, and git hook. Verified end-to-end against live Redis (job enqueue → worker → Postgres write).
• Git synchronization hook (auto-reindex on push) — API webhook POST /hooks/reindex (full) and POST /hooks/index?slug= (single) enqueue BullMQ jobs. Wire a Git provider (GitHub/Gitea) post-receive / webhook to POST /hooks/reindex to auto-reindex on push. scripts/enqueue.ts is a one-shot enqueue helper (bun run enqueue --all / <slug>).
• Document revision system (document_revisions) — packages/db migration 0002_document_revisions.sql. Indexer snapshots a revision only when the body actually changes vs the latest revision (pure metadata edits don't bloat history). listRevisions / getRevision / restoreRevision in @mcpedia/core; exposed as tRPC revisions / getRevision / restoreRevision and the mcpedia://docs/{+slug}/revisions MCP Resource.
• MCP Resources (mcpedia://docs/...) — alongside the 6 tools: mcpedia://docs (list), mcpedia://docs/{+slug} (body from disk), mcpedia://docs/{+slug}/chunks (chunk preview), mcpedia://docs/{+slug}/revisions (history). {+slug} uses RFC 6570 reserved expansion so slugs containing / match.

New/changed commands

bun run index            # full reindex (runFullIndex, writes revisions)
bun run enqueue --all    # enqueue a full reindex job (no worker needed)
bun run enqueue <slug>   # enqueue a single-doc reindex job
bun run worker           # start the BullMQ indexing worker (long-running)
bun run api              # Hono+tRPC API on :4020 (added /hooks/* webhooks)

Verification done (real, against imrnes Redis + Postgres)

• turbo run typecheck green across all 13 packages.
• BullMQ e2e: enqueue index-doc → worker completes → documents + document_chunks + document_revisions rows present.
• Revision dedup proven: editing a body creates a new revision; metadata-only reindex does not; restoreRevision writes history back into the live row.
• MCP smoke test passes (tools + all 4 resources).
• API webhook POST /hooks/reindex enqueues → worker drains queue → queueStatus reflects counts.

Phase 4 — Scale-out (only if needed)

• Dedicated search engine (OpenSearch/Elasticsearch) — YAGNI until FTS is insufficient
• Object storage for assets
• Advanced ranking, distributed workers, observability, multi-tenant

Decisions locked (from initial planning)

• Tooling: bun workspaces + Turborepo (repo already used bun; pnpm rejected to minimize churn).
• DB: imrnes Postgres 100.121.180.82:6432/mcpedia for both dev and deploy; driver prepare:false (PgBouncer). Docker Compose reserved for future prod.
• Phase 1 scope: Core + Web + MCP only. tRPC/Hono API, pgvector, auth, BullMQ deferred (YAGNI).