feat(mcpedia): Phase 1 MVP — monorepo, Core, Web UI, MCP server, Postgres FTS
- bun workspaces + Turborepo monorepo (apps/web, apps/mcp; packages/*, scripts) - @mcpedia/core single business-logic layer (Document/Content/Search services) - @mcpedia/db Drizzle schema: documents + weighted tsvector (GIN) for FTS - @mcpedia/parser frontmatter, @mcpedia/search Postgres FTS (ts_rank+ts_headline) - Next.js 16 Web UI (home/doc SSG, search dynamic) + react-markdown render - MCP server (stdio) with 4 tools + in-memory smoke test - scripts/indexer walks content/ -> upserts into Postgres - 4 seed docs; README + PHASES status
This commit is contained in:
asepharyana committed
commit
ec3a2867d4
60 files changed
+3723
No files matched your search
@@ -0,0 +1,435 @@
|
||||
# MCPedia Implementation Plan
|
||||
|
||||
> **For Hermes:** Use subagent-driven-development to implement task-by-task. Phase 1 tasks are bite-sized and ready to execute; Phases 2–4 are outlines to expand later.
|
||||
|
||||
**Goal:** Turn the fresh `create-next-app` Next.js 16 repo into the MCPedia knowledge base — Git/Markdown content, a Core business-logic layer shared by a Next.js Web UI and an MCP server for AI agents, with content metadata in Postgres (FTS search), and an architecture that can grow to semantic search, a tRPC API, auth, and async indexing without rewrites.
|
||||
|
||||
**Architecture:** Monorepo (bun workspaces + Turborepo). Content lives as Markdown in `content/` (Git-managed, human-readable). `packages/core` holds all business logic (Document/Content/Search services). Web (`apps/web`) and MCP (`apps/mcp`) both consume Core — no DB access outside Core. Postgres (imrnes `:6432`) stores document metadata + a `tsvector` for keyword search. Phase 1 ships keyword search only; pgvector/semantic, tRPC/Hono API, auth, and BullMQ are later phases behind clean seamsLine truncated
|
||||
|
||||
**Tech Stack (resolved forks):**
|
||||
- Package manager / monorepo: **bun workspaces + Turborepo** (honor existing repo; NOT pnpm — user decision).
|
||||
- DB: **imrnes Postgres `100.121.180.82:6432`** for both dev and deploy (PgBouncer transaction pool → set `prepare: false` on the driver).
|
||||
- Phase 1 scope: **Core + Web + MCP + Postgres FTS only**. tRPC/Hono API, pgvector, auth, Redis/BullMQ deferred.
|
||||
|
||||
---
|
||||
|
||||
## Environment & preconditions (read first)
|
||||
|
||||
- Repo is currently a flat Next.js 16 app, **not git-initialized**, bun-managed (`packageManager: bun@1.3.14`, `bun.lock` present).
|
||||
- `app/` has only `layout.tsx`, `page.tsx`, `globals.css`, `favicon.ico`. Tailwind v4 (`@tailwindcss/postcss`).
|
||||
- Next.js 16 is "NOT the Next.js you know" (see `AGENTS.md`): **before writing any `apps/web` code, read `node_modules/next/dist/docs/`** (app router, server components, route handlers) for breaking changes. Treat deprecation notices as authoritative.
|
||||
- For every library API used (Next 16, Drizzle, `@modelcontextprotocol/sdk`, gray-matter) **resolve + query Context7** before coding — do not trust training data.
|
||||
- Secrets: `DATABASE_URL` lives in **BWS project `orangevps`** (never committed). Local dev reads it from a gitignored `.env` at repo root.
|
||||
- Seed DB creds: a dedicated `mcpedia` database/user on imrnes; create via `bws` secret or operator. `pgvector` extension is **NOT** required until Phase 2.
|
||||
|
||||
---
|
||||
|
||||
## Phase 0 — Repo restructure (monorepo scaffold)
|
||||
|
||||
### Task 0.1: Initialize git + baseline commit
|
||||
- **Files:** `.gitignore` (already exists; ensure `.env`, `.next/`, `node_modules/`, `*.tsbuildinfo`, `dist/` ignored).
|
||||
- **Step:** `git init && git add -A && git commit -m "chore: baseline create-next-app scaffold"` (author = asepharyana, no Co-Authored-By).
|
||||
- **Verify:** `git log --oneline -1` shows the commit.
|
||||
|
||||
### Task 0.2: Root workspace package.json + Turbo
|
||||
- **Modify:** `package.json` (root) — strip app deps, add workspaces + turbo scripts.
|
||||
- **Create:** `turbo.json`, `tsconfig.base.json`.
|
||||
- Root `package.json` becomes:
|
||||
```json
|
||||
{
|
||||
"name": "mcpedia",
|
||||
"version": "0.1.0",
|
||||
"private": true,
|
||||
"packageManager": "bun@1.3.14",
|
||||
"workspaces": ["apps/*", "packages/*"],
|
||||
"scripts": {
|
||||
"dev": "turbo run dev",
|
||||
"build": "turbo run build",
|
||||
"lint": "turbo run lint",
|
||||
"typecheck": "turbo run typecheck",
|
||||
"index": "bun run scripts/indexer.ts"
|
||||
},
|
||||
"devDependencies": {
|
||||
"turbo": "^2",
|
||||
"typescript": "^5",
|
||||
"prettier": "^3"
|
||||
}
|
||||
}
|
||||
```
|
||||
- `turbo.json`:
|
||||
```json
|
||||
{
|
||||
"$schema": "https://turbo.build/schema.json",
|
||||
"tasks": {
|
||||
"build": { "dependsOn": ["^build"], "outputs": ["dist/**", ".next/**"] },
|
||||
"lint": {},
|
||||
"typecheck": { "dependsOn": ["^build"] },
|
||||
"dev": { "cache": false, "persistent": true }
|
||||
}
|
||||
}
|
||||
```
|
||||
- **Verify:** `bun install` succeeds; `bunx turbo --version` prints a version.
|
||||
|
||||
### Task 0.3: Move Web app into `apps/web`
|
||||
- **Move:** `app/`, `public/`, `next.config.ts`, `next-env.d.ts`, `postcss.config.mjs`, `eslint.config.mjs`, `tsconfig.json` → `apps/web/` (rename root `app` → `apps/web/app`). Keep Tailwind v4. Add `apps/web/package.json`:
|
||||
```json
|
||||
{
|
||||
"name": "@mcpedia/web",
|
||||
"private": true,
|
||||
"scripts": {
|
||||
"dev": "next dev",
|
||||
"build": "next build",
|
||||
"start": "next start",
|
||||
"lint": "eslint .",
|
||||
"typecheck": "tsc --noEmit"
|
||||
},
|
||||
"dependencies": {
|
||||
"next": "16.3.1",
|
||||
"react": "19.2.8",
|
||||
"react-dom": "19.2.8",
|
||||
"@mcpedia/core": "workspace:*",
|
||||
"@mcpedia/config": "workspace:*"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@tailwindcss/postcss": "^4",
|
||||
"@types/node": "^20",
|
||||
"@types/react": "^19",
|
||||
"@types/react-dom": "^19",
|
||||
"eslint": "^9",
|
||||
"eslint-config-next": "16.3.1",
|
||||
"tailwindcss": "^4",
|
||||
"typescript": "^5"
|
||||
}
|
||||
}
|
||||
```
|
||||
- Add `apps/web/tsconfig.json` extending `../../tsconfig.base.json` with `@/*` → `./**/*` and `@mcpedia/*` workspace aliases.
|
||||
- **Verify:** `cd apps/web && bun install && bun run build` compiles (may warn about empty content — fine).
|
||||
|
||||
### Task 0.4: `.env` (gitignored) for dev
|
||||
- **Create:** `.env` (gitignored) with `DATABASE_URL=postgresql://<user>:<pass>@100.121.180.82:6432/mcpedia` (value from BWS; `prepare:false` enforced in db client). Add `.env.example` (committed, placeholder only).
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — MVP: Core + Web + MCP + Postgres FTS (detailed)
|
||||
|
||||
### Task 1.1: `packages/types`
|
||||
- **Create:** `packages/types/package.json` (`@mcpedia/types`, exports `src/index.ts`), `packages/types/src/index.ts`:
|
||||
```ts
|
||||
export type DocSection = "docs" | "writeups" | "research" | "notes";
|
||||
export type DocType = "documentation" | "writeup" | "research" | "note";
|
||||
export type DocStatus = "published" | "draft";
|
||||
|
||||
export interface DocumentMeta {
|
||||
id: string; // slug
|
||||
slug: string;
|
||||
title: string;
|
||||
type: DocType;
|
||||
section: DocSection;
|
||||
status: DocStatus;
|
||||
author: string;
|
||||
tags: string[];
|
||||
path: string; // relative path under content/
|
||||
createdAt: string;
|
||||
updatedAt: string;
|
||||
}
|
||||
|
||||
export interface Document extends DocumentMeta {
|
||||
body: string; // raw markdown (read from disk or stored)
|
||||
}
|
||||
|
||||
export interface SearchHit {
|
||||
doc: DocumentMeta;
|
||||
rank: number;
|
||||
snippet: string;
|
||||
}
|
||||
```
|
||||
- **Verify:** `bunx tsc --noEmit` in package passes.
|
||||
|
||||
### Task 1.2: `packages/config`
|
||||
- **Create:** `packages/config/package.json` (`@mcpedia/config`), `packages/config/src/index.ts`:
|
||||
```ts
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { dirname, resolve } from "node:path";
|
||||
|
||||
const root = resolve(dirname(fileURLToPath(import.meta.url)), "../../..");
|
||||
export const CONTENT_ROOT = process.env.CONTENT_ROOT
|
||||
?? resolve(root, "content");
|
||||
export const DATABASE_URL = process.env.DATABASE_URL ?? "";
|
||||
```
|
||||
- **Verify:** import in a scratch `bun -e` prints a path.
|
||||
|
||||
### Task 1.3: `packages/db` (Drizzle + Postgres, FTS)
|
||||
- **Create:** `packages/db/package.json` (`@mcpedia/db`, deps `drizzle-orm`, `postgres`; dev `drizzle-kit`), `packages/db/src/schema.ts`:
|
||||
```ts
|
||||
import { sql } from "drizzle-orm";
|
||||
import { text, timestamp, integer, index, customType } from "drizzle-orm/pg-core";
|
||||
|
||||
// tsvector isn't a first-class drizzle type; wrap raw.
|
||||
export const documents = pgTable("documents", {
|
||||
id: text("id").primaryKey(), // slug
|
||||
slug: text("slug").notNull().unique(),
|
||||
title: text("title").notNull(),
|
||||
type: text("type").notNull(),
|
||||
section: text("section").notNull(),
|
||||
status: text("status").notNull().default("published"),
|
||||
author: text("author").notNull().default(""),
|
||||
tags: text("tags").array().notNull().default(sql`'{}'`),
|
||||
path: text("path").notNull(),
|
||||
body: text("body").notNull().default(""),
|
||||
searchVector: customType<{ data: string }>({
|
||||
dataType() { return "tsvector"; },
|
||||
})("search_vector")
|
||||
.generatedAlwaysAs(sql`to_tsvector('simple', coalesce(title,'') || ' ' || coalesce(body,''))`),
|
||||
createdAt: timestamp("created_at", { withTimezone: true }).notNull(),
|
||||
updatedAt: timestamp("updated_at", { withTimezone: true }).notNull(),
|
||||
}, (t) => ({
|
||||
searchIdx: index("documents_search_idx").using("gin", t.searchVector),
|
||||
}));
|
||||
```
|
||||
NOTE: `document_chunks` + `embedding vector` are **Phase 2** — do NOT add now.
|
||||
- **Create:** `packages/db/src/client.ts` — postgres-js with `prepare: false` (PgBouncer safe):
|
||||
```ts
|
||||
import { drizzle } from "drizzle-orm/postgres-js";
|
||||
import postgres from "postgres";
|
||||
import { DATABASE_URL } from "@mcpedia/config";
|
||||
import * as schema from "./schema";
|
||||
|
||||
const client = postgres(DATABASE_URL, { prepare: false, max: 5 });
|
||||
export const db = drizzle(client, { schema });
|
||||
export { schema };
|
||||
```
|
||||
- **Create:** `packages/db/drizzle.config.ts` (dialect `postgresql`, schema `./src/schema.ts`, out `./drizzle`).
|
||||
- **Verify:** `bunx drizzle-kit generate` produces a migration; `bunx drizzle-kit push` applies it to imrnes (operator supplies DATABASE_URL). `bunx drizzle-kit studio` optional.
|
||||
|
||||
### Task 1.4: `packages/parser` (frontmatter)
|
||||
- **Create:** `packages/parser/package.json` (`@mcpedia/parser`, dep `gray-matter`), `packages/parser/src/index.ts`:
|
||||
```ts
|
||||
import matter from "gray-matter";
|
||||
import { readFileSync } from "node:fs";
|
||||
import type { DocSection, DocType, DocStatus, DocumentMeta } from "@mcpedia/types";
|
||||
|
||||
const SECTIONS: DocSection[] = ["docs", "writeups", "research", "notes"];
|
||||
|
||||
export function parseFile(absPath: string, relPath: string): { meta: DocumentMeta; body: string } {
|
||||
const raw = readFileSync(absPath, "utf8");
|
||||
const { data, content } = matter(raw);
|
||||
const section = SECTIONS.find((s) => relPath.startsWith(s + "/")) ?? "docs";
|
||||
const slug = relPath.replace(/\.mdx?$/, "");
|
||||
const meta: DocumentMeta = {
|
||||
id: slug,
|
||||
slug,
|
||||
title: data.title ?? slug,
|
||||
type: (data.type ?? "documentation") as DocType,
|
||||
section,
|
||||
status: (data.status ?? "published") as DocStatus,
|
||||
author: data.author ?? "",
|
||||
tags: Array.isArray(data.tags) ? dataAtags : [],
|
||||
path: relPath,
|
||||
createdAt: data.created_at ?? new Date().toISOString(),
|
||||
updatedAt: data.updated_at ?? data.created_at ?? new Date().toISOString(),
|
||||
};
|
||||
return { meta, body: content };
|
||||
}
|
||||
```
|
||||
(fix the obvious `dataAtags` typo to `data.tags` when implementing.)
|
||||
- **Verify:** unit test parsing a sample frontmatter → asserts `slug`, `section`, `tags`.
|
||||
|
||||
### Task 1.5: `packages/search` (FTS query)
|
||||
- **Create:** `packages/search/package.json` (`@mcpedia/search`, dep `@mcpedia/db`, `@mcpedia/types`), `packages/search/src/index.ts`:
|
||||
```ts
|
||||
import { db } from "@mcpedia/db";
|
||||
import { documents } from "@mcpedia/db/schema";
|
||||
import { sql, and, eq } from "drizzle-orm";
|
||||
import type { SearchHit } from "@mcpedia/types";
|
||||
|
||||
export function toTsQuery(q: string): string {
|
||||
// plainto_tsquery-style: AND the terms, escape punctuation.
|
||||
const terms = q.trim().split(/\s+/).filter(Boolean).map((t) => t.replace(/[^\p{L}\p{N}]/gu, ""));
|
||||
return terms.map((t) => `${t}:*`).join(" & ");
|
||||
}
|
||||
|
||||
export async function keywordSearch(q: string, limit = 20): Promise<SearchHit[]> {
|
||||
const query = toTsQuery(q);
|
||||
if (!query) return [];
|
||||
const rows = await db
|
||||
.select({
|
||||
doc: documents,
|
||||
rank: sql<number>`ts_rank(${documents.searchVector}, to_tsquery('simple', ${query}))`,
|
||||
snippet: sql<string>`ts_headline('simple', ${documents.body}, to_tsquery('simple', ${query}), 'MaxWords=20, MinWords=5')`,
|
||||
})
|
||||
.from(documents)
|
||||
.where(and(eq(documents.status, "published"), sql`${documents.searchVector} @@ to_tsquery('simple', ${query})`))
|
||||
.orderBy(sql`ts_rank(${documents.searchVector}, to_tsquery('simple', ${query})) desc`)
|
||||
.limit(limit);
|
||||
return rows.map((r) => ({ doc: r.doc, rank: r.rank, snippet: r.snippet }));
|
||||
}
|
||||
```
|
||||
- **Verify:** after seeding, `keywordSearch("websocket")` returns hits.
|
||||
|
||||
### Task 1.6: `packages/core` (services)
|
||||
- **Create:** `packages/core/package.json` (`@mcpedia/core`, deps `@mcpedia/db`, `@mcpedia/parser`, `@mcpedia/search`, `@mcpedia/config`, `@mcpedia/types`), `packages/core/src/`:
|
||||
- `content.service.ts` — read markdown from `CONTENT_ROOT` by slug/section; list files via `fast-glob`.
|
||||
- `document.service.ts` — `listDocuments(section?, type?, status?)`, `getDocument(slug)` (DB meta + disk body), `getRelated(slug)` (shared tags, later vector).
|
||||
- `search.service.ts` — wraps `keywordSearch`.
|
||||
- `index.ts` — re-export services + types.
|
||||
```ts
|
||||
// document.service.ts (sketch)
|
||||
import { db } from "@mcpedia/db";
|
||||
import { documents } from "@mcpedia/db/schema";
|
||||
import { eq, and } from "drizzle-orm";
|
||||
import { CONTENT_ROOT } from "@mcpedia/config";
|
||||
import { readFileSync, existsSync } from "node:fs";
|
||||
import { resolve } from "node:path";
|
||||
import type { Document, DocumentMeta } from "@mcpedia/types";
|
||||
|
||||
export async function listDocuments(opts: { section?: string; status?: string } = {}): Promise<DocumentMeta[]> {
|
||||
const where = [eq(documents.status, opts.status ?? "published")];
|
||||
if (opts.section) where.push(eq(documents.section, opts.section));
|
||||
return db.select().from(documents).where(and(...where));
|
||||
}
|
||||
|
||||
export async function getDocument(slug: string): Promise<Document | null> {
|
||||
const [meta] = await db.select().from(documents).where(eq(documents.slug, slug));
|
||||
if (!meta) return null;
|
||||
const abs = resolve(CONTENT_ROOT, meta.path);
|
||||
const body = existsSync(abs) ? readFileSync(abs, "utf8") : meta.body;
|
||||
return { ...meta, body };
|
||||
}
|
||||
```
|
||||
- **Verify:** `bunx tsc --noEmit` passes across packages.
|
||||
|
||||
### Task 1.7: `scripts/indexer.ts` (walk content → upsert)
|
||||
- **Create:** `scripts/indexer.ts` — walk `CONTENT_ROOT` (`**/*.{md,mdx}`), `parseFile`, upsert into `documents` (slug PK; `onConflictDoUpdate` title/tags/body/updatedAt). Logs count.
|
||||
```ts
|
||||
import { db } from "@mcpedia/db";
|
||||
import { documents } from "@mcpedia/db/schema";
|
||||
import { parseFile } from "@mcpedia/parser";
|
||||
import { CONTENT_ROOT } from "@mcpedia/config";
|
||||
import { walk } from "fast-glob";
|
||||
import { resolve } from "node:path";
|
||||
|
||||
const files = await glob("**/*.{md,mdx}", { cwd: CONTENT_ROOT, absolute: true });
|
||||
let n = 0;
|
||||
for (const abs of files) {
|
||||
const rel = resolve(CONTENT_ROOT, abs).replace(CONTENT_ROOT + "/", "");
|
||||
const { meta, body } = parseFile(abs, rel);
|
||||
await db.insert(documents).values({ ...meta, body })
|
||||
.onConflictDoUpdate({ target: documents.slug, set: { title: meta.title, tags: meta.tags, body, updatedAt: new Date().toISOString() } });
|
||||
n++;
|
||||
}
|
||||
console.log(`indexed ${n} documents`);
|
||||
```
|
||||
- **Verify:** `bun run index` prints `indexed N documents`; `psql`/`drizzle-kit studio` shows rows + non-empty `search_vector`.
|
||||
|
||||
### Task 1.8: Seed content
|
||||
- **Create:** `content/docs/websocket/contract.md`, `content/writeups/debugging/websocket-timeout.md`, `content/research/mcp/architecture.md`, `content/notes/typescript/patterns.md` with the spec's frontmatter shape (id/title/type/tags/status/author/created_at/updated_at). Add 2–3 short paragraphs each so FTS has text.
|
||||
- **Verify:** `bun run index` ingests them; `keywordSearch("websocket")` returns ≥2 hits.
|
||||
|
||||
### Task 1.9: `apps/web` — pages (Next 16, read `node_modules/next/dist/docs` first)
|
||||
- **Read** Node_modules Next 16 docs (app router, server components, route handlers) — mandatory.
|
||||
- **Create:**
|
||||
- `apps/web/app/page.tsx` — home: list sections + recent docs (server component calling `listDocuments`).
|
||||
- `apps/web/app/(docs)/[...slug]/page.tsx` — document view: `getDocument(slug)`, render markdown (Phase 1: simple `dangerouslySetInnerHTML` of a remark→HTML, or raw pre with Shiki later; keep minimal but legible).
|
||||
- `apps/web/app/search/page.tsx` — search box → `keywordSearch`, render hits + snippets.
|
||||
- Keep Tailwind v4 styling minimal but clean (no shadcn yet — Phase 2 polish).
|
||||
- **Verify:** `bun run build` (apps/web) passes; `bun run dev` then `curl -s localhost:3000/` and `/search?q=websocket` return rendered HTML.
|
||||
|
||||
### Task 1.10: `apps/mcp` — MCP server
|
||||
- **Create:** `apps/mcp/package.json` (`@mcpedia/mcp`, deps `@modelcontextprotocol/sdk`, `zod`, `@mcpedia/core`), `apps/mcp/src/index.ts`:
|
||||
```ts
|
||||
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
|
||||
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
|
||||
import { z } from "zod";
|
||||
import { listDocuments, getDocument } from "@mcpedia/core";
|
||||
import { keywordSearch } from "@mcpedia/search";
|
||||
|
||||
const server = new McpServer({ name: "mcpedia", version: "0.1.0" });
|
||||
|
||||
server.tool("search_documents", { query: z.string(), limit: z.number().optional() },
|
||||
async ({ query, limit }) => {
|
||||
const hits = await keywordSearch(query, limit ?? 20);
|
||||
return { content: [{ type: "text", text: JSON.stringify(hits, null, 2) }] };
|
||||
});
|
||||
|
||||
server.tool("get_document", { slug: z.string() },
|
||||
async ({ slug }) => {
|
||||
const doc = await getDocument(slug);
|
||||
return doc ? { content: [{ type: "text", text: doc.body }] }
|
||||
: { content: [{ type: "text", text: "not found" }], isError: true };
|
||||
});
|
||||
|
||||
server.tool("list_documents", { section: z.string().optional() },
|
||||
async ({ section }) => {
|
||||
const docs = await listDocuments({ section });
|
||||
return { content: [{ type: "text", text: JSON.stringify(docs, null, 2) }] };
|
||||
});
|
||||
|
||||
// Resource: mcpedia://<section>/<path>
|
||||
server.resource("doc", "mcpedia://", async (uri) => {
|
||||
const slug = uri.path.replace(/^\//, "");
|
||||
const doc = await getDocument(slug);
|
||||
return { contents: [{ uri: uri.href, text: doc?.body ?? "" }] };
|
||||
});
|
||||
|
||||
const transport = new StdioServerTransport();
|
||||
await server.connect(transport);
|
||||
```
|
||||
(Exact SDK method names verified via Context7 before coding.)
|
||||
- **Create:** `apps/mcp/bin` or `bun run` entry; add `"start": "bun run src/index.ts"` to its package.json.
|
||||
- **Verify:** smoke test via `scripts/mcp-smoke.ts` that imports the tool handlers and calls `search_documents({query:"websocket"})` → asserts ≥1 hit. (Running a real stdio client in CI is optional; inspector `@modelcontextprotocol/inspector` for manual check.)
|
||||
|
||||
### Task 1.11: Cross-package verification
|
||||
- **Run:** `bun install` (root) → `bunx turbo run lint` → `bunx turbo run build` → `bunx turbo run typecheck`. All green.
|
||||
- **Run:** `bun run index` then `bun run scripts/mcp-smoke.ts`.
|
||||
- **Verify:** no package imports `pg`/`drizzle` outside `@mcpedia/core` or `@mcpedia/db` (architecture guard). Search the tree: `search_files("from \"pg\"", path="apps")`.
|
||||
|
||||
### Task 1.12: README + phase markers
|
||||
- **Modify:** `README.md` — replace create-next-app text with MCPedia overview, monorepo map, "Phase 1 status", how to dev (`bun install && bun run index && bun run dev`), how to run MCP (`bun run --cwd apps/mcp start`).
|
||||
- **Verify:** `cat README.md` reads correctly.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — Semantic search + tRPC/Hono API + auth (outline, expand later)
|
||||
|
||||
- Add `document_chunks` table + `embedding vector` (pgvector extension on imrnes — operator enables `CREATE EXTENSION vector`).
|
||||
- `packages/search`: `EmbeddingProvider` abstraction (OpenAI/Gemini/Ollama/Local) behind an interface (spec principle 6). Chunking in `packages/parser` (by heading/section).
|
||||
- Hybrid ranking: fuse FTS `ts_rank` + cosine distance (Reciprocal Rank Fusion).
|
||||
- `apps/api`: Hono + tRPC router exposing `documents.*`, `search.*`; Web switches from direct Core calls to tRPC client; MCP may also call API or keep Core (decide: keep Core for colocated simplicity, API for external).
|
||||
- `packages/auth`: Auth.js/OIDC adapter; `permissions` table; gate write/mutations. Phase 1 documents are public-read.
|
||||
- Categories + tags linking tables (`categories`, `document_tags`, `references`).
|
||||
- MDX rendering with Shiki + shadcn/ui polish on Web.
|
||||
|
||||
## Phase 3 — Async indexing + revisions (outline)
|
||||
|
||||
- Redis (imrnes or local) + BullMQ workers: parse/chunk/embed/reindex off the request path.
|
||||
- `document_revisions` table + revision API; Git sync hook (content/ push → reindex).
|
||||
- Indexer becomes a worker triggered by file watch / webhook, not a manual script.
|
||||
|
||||
## Phase 4 — Scale (outline, only if needed)
|
||||
|
||||
- Dedicated search engine (OpenSearch) behind the same `SearchService` seam — YAGNI until FTS+pgvector saturates.
|
||||
- Object storage for assets, advanced ranking, distributed workers, observability, multi-tenant. Docker Compose for service orchestration pointing at imrnes Postgres.
|
||||
|
||||
---
|
||||
|
||||
## Risks / tradeoffs / open questions
|
||||
|
||||
- **Next 16 breaking changes** — biggest unknown; mitigated by reading `node_modules/next/dist/docs` before Web tasks. If app-router differs enough, adjust Task 1.9 accordingly.
|
||||
- **PgBouncer + prepared statements** — handled via `prepare: false`; if imrnes uses session pool mode, can drop it.
|
||||
- **FTS language** — using `'simple'` config (no stemming) so mixed ID/EN queries match literally; revisit if relevance is poor (could add `'english'` or a custom ID dict).
|
||||
- **pgvector on imrnes** — extension may need operator enablement; Phase 2 blocker, not Phase 1.
|
||||
- **MCP resource URI shape** — `mcpedia://<section>/<path>` is a sketch; finalize against MCP SDK spec in Task 1.10 (Context7).
|
||||
- **Monorepo driver** — bun workspaces chosen over pnpm per user; Turborepo runs fine under bun. If `turbo` bin resolution misbehaves under bun, fall back to `bunx turbo`.
|
||||
|
||||
## Verification matrix (Phase 1)
|
||||
|
||||
| Check | Command | Expect |
|
||||
| --- | --- | --- |
|
||||
| Install | `bun install` | clean |
|
||||
| Lint | `bunx turbo run lint` | 0 errors |
|
||||
| Typecheck | `bunx turbo run typecheck` | 0 errors |
|
||||
| Build | `bunx turbo run build` | web + mcp build |
|
||||
| Index | `bun run index` | `indexed N documents` |
|
||||
| Web | `bun run dev` (apps/web) + curl `/` and `/search?q=websocket` | rendered HTML, hits |
|
||||
| MCP | `bun run scripts/mcp-smoke.ts` | ≥1 hit for "websocket" |
|
||||
| Arch guard | search for `from "pg"` outside `packages/db`,`packages/core` | none |
|
||||
Reference in new issue
Block a user