From a9b67385c3b7bd56111fb2ab76b345a68e01032e Mon Sep 17 00:00:00 2001 From: asepharyana Date: Fri, 21 Aug 2026 11:35:27 +0700 Subject: [PATCH] feat: revamp to dynamic database-first architecture and cleanup phase docs - Migrate document fetching and CRUD to be PostgreSQL-authoritative - Remove static section enums and add dynamic listSections query - Support custom sections and metadata across API, MCP, and Web UI - Add /api/sections endpoint and update Header, Sidebar, and Forms - Remove obsolete phase planning docs and modernize README/AGENTS --- .../plans/2026-08-19_1637-mcpedia-roadmap.md | 435 ----------- .../plans/2026-08-19_phase2-semantic-api.md | 161 ---- .hermes/plans/hierarchical-folders.md | 151 ---- .hermes/plans/mcp-http-transport.md | 42 - .hermes/plans/phase11-crud-auth-ui.md | 67 -- .hermes/plans/phase3-deploy.md | 24 - .hermes/plans/phase3-plan.md | 114 --- .hermes/plans/phase4-plan.md | 105 --- .hermes/plans/phase7-all.md | 41 - .hermes/plans/phase9-spec.md | 157 ---- .hermes/plans/ui-ux-polish.md | 80 -- AGENTS.md | 21 + PHASES.md | 727 ------------------ README.md | 396 ++++++---- apps/api/src/router.ts | 23 +- apps/mcp/src/auth.test.ts | 2 + apps/mcp/src/index.ts | 41 +- apps/web/app/[section]/[...slug]/page.tsx | 25 +- apps/web/app/[section]/page.tsx | 26 +- apps/web/app/api/sections/route.ts | 15 + apps/web/app/components/DocForm.tsx | 76 +- apps/web/app/components/Header.tsx | 20 +- apps/web/app/components/Sidebar.tsx | 19 +- apps/web/app/create/page.tsx | 17 +- apps/web/app/page.tsx | 31 +- apps/web/app/search/page.tsx | 16 +- content/docs/mcp/streamable-http.md | 8 +- content/docs/websocket/contract.md | 2 +- packages/config/src/index.ts | 19 +- packages/config/src/sections.ts | 78 +- packages/core/src/document.service.ts | 213 ++--- packages/core/src/index.service.test.ts | 7 +- packages/core/src/index.ts | 2 + packages/parser/src/index.ts | 17 +- packages/parser/src/parse.test.ts | 17 +- packages/search/src/index.ts | 21 +- packages/types/src/index.ts | 19 +- 37 files changed, 720 insertions(+), 2515 deletions(-) delete mode 100644 .hermes/plans/2026-08-19_1637-mcpedia-roadmap.md delete mode 100644 .hermes/plans/2026-08-19_phase2-semantic-api.md delete mode 100644 .hermes/plans/hierarchical-folders.md delete mode 100644 .hermes/plans/mcp-http-transport.md delete mode 100644 .hermes/plans/phase11-crud-auth-ui.md delete mode 100644 .hermes/plans/phase3-deploy.md delete mode 100644 .hermes/plans/phase3-plan.md delete mode 100644 .hermes/plans/phase4-plan.md delete mode 100644 .hermes/plans/phase7-all.md delete mode 100644 .hermes/plans/phase9-spec.md delete mode 100644 .hermes/plans/ui-ux-polish.md delete mode 100644 PHASES.md create mode 100644 apps/web/app/api/sections/route.ts diff --git a/.hermes/plans/2026-08-19_1637-mcpedia-roadmap.md b/.hermes/plans/2026-08-19_1637-mcpedia-roadmap.md deleted file mode 100644 index cdda9e2..0000000 --- a/.hermes/plans/2026-08-19_1637-mcpedia-roadmap.md +++ /dev/null @@ -1,435 +0,0 @@ -# MCPedia Implementation Plan - -> **For Hermes:** Use subagent-driven-development to implement task-by-task. Phase 1 tasks are bite-sized and ready to execute; Phases 2–4 are outlines to expand later. - -**Goal:** Turn the fresh `create-next-app` Next.js 16 repo into the MCPedia knowledge base — Git/Markdown content, a Core business-logic layer shared by a Next.js Web UI and an MCP server for AI agents, with content metadata in Postgres (FTS search), and an architecture that can grow to semantic search, a tRPC API, auth, and async indexing without rewrites. - -**Architecture:** Monorepo (bun workspaces + Turborepo). Content lives as Markdown in `content/` (Git-managed, human-readable). `packages/core` holds all business logic (Document/Content/Search services). Web (`apps/web`) and MCP (`apps/mcp`) both consume Core — no DB access outside Core. Postgres (imrnes `:6432`) stores document metadata + a `tsvector` for keyword search. Phase 1 ships keyword search only; pgvector/semantic, tRPC/Hono API, auth, and BullMQ are later phases behind clean seams already present in Core. - -**Tech Stack (resolved forks):** -- Package manager / monorepo: **bun workspaces + Turborepo** (honor existing repo; NOT pnpm — user decision). -- DB: **imrnes Postgres `100.121.180.82:6432`** for both dev and deploy (PgBouncer transaction pool → set `prepare: false` on the driver). -- Phase 1 scope: **Core + Web + MCP + Postgres FTS only**. tRPC/Hono API, pgvector, auth, Redis/BullMQ deferred. - ---- - -## Environment & preconditions (read first) - -- Repo is currently a flat Next.js 16 app, **not git-initialized**, bun-managed (`packageManager: bun@1.3.14`, `bun.lock` present). -- `app/` has only `layout.tsx`, `page.tsx`, `globals.css`, `favicon.ico`. Tailwind v4 (`@tailwindcss/postcss`). -- Next.js 16 is "NOT the Next.js you know" (see `AGENTS.md`): **before writing any `apps/web` code, read `node_modules/next/dist/docs/`** (app router, server components, route handlers) for breaking changes. Treat deprecation notices as authoritative. -- For every library API used (Next 16, Drizzle, `@modelcontextprotocol/sdk`, gray-matter) **resolve + query Context7** before coding — do not trust training data. -- Secrets: `DATABASE_URL` lives in **BWS project `orangevps`** (never committed). Local dev reads it from a gitignored `.env` at repo root. -- Seed DB creds: a dedicated `mcpedia` database/user on imrnes; create via `bws` secret or operator. `pgvector` extension is **NOT** required until Phase 2. - ---- - -## Phase 0 — Repo restructure (monorepo scaffold) - -### Task 0.1: Initialize git + baseline commit -- **Files:** `.gitignore` (already exists; ensure `.env`, `.next/`, `node_modules/`, `*.tsbuildinfo`, `dist/` ignored). -- **Step:** `git init && git add -A && git commit -m "chore: baseline create-next-app scaffold"` (author = asepharyana, no Co-Authored-By). -- **Verify:** `git log --oneline -1` shows the commit. - -### Task 0.2: Root workspace package.json + Turbo -- **Modify:** `package.json` (root) — strip app deps, add workspaces + turbo scripts. -- **Create:** `turbo.json`, `tsconfig.base.json`. -- Root `package.json` becomes: -```json -{ - "name": "mcpedia", - "version": "0.1.0", - "private": true, - "packageManager": "bun@1.3.14", - "workspaces": ["apps/*", "packages/*"], - "scripts": { - "dev": "turbo run dev", - "build": "turbo run build", - "lint": "turbo run lint", - "typecheck": "turbo run typecheck", - "index": "bun run scripts/indexer.ts" - }, - "devDependencies": { - "turbo": "^2", - "typescript": "^5", - "prettier": "^3" - } -} -``` -- `turbo.json`: -```json -{ - "$schema": "https://turbo.build/schema.json", - "tasks": { - "build": { "dependsOn": ["^build"], "outputs": ["dist/**", ".next/**"] }, - "lint": {}, - "typecheck": { "dependsOn": ["^build"] }, - "dev": { "cache": false, "persistent": true } - } -} -``` -- **Verify:** `bun install` succeeds; `bunx turbo --version` prints a version. - -### Task 0.3: Move Web app into `apps/web` -- **Move:** `app/`, `public/`, `next.config.ts`, `next-env.d.ts`, `postcss.config.mjs`, `eslint.config.mjs`, `tsconfig.json` → `apps/web/` (rename root `app` → `apps/web/app`). Keep Tailwind v4. Add `apps/web/package.json`: -```json -{ - "name": "@mcpedia/web", - "private": true, - "scripts": { - "dev": "next dev", - "build": "next build", - "start": "next start", - "lint": "eslint .", - "typecheck": "tsc --noEmit" - }, - "dependencies": { - "next": "16.3.1", - "react": "19.2.8", - "react-dom": "19.2.8", - "@mcpedia/core": "workspace:*", - "@mcpedia/config": "workspace:*" - }, - "devDependencies": { - "@tailwindcss/postcss": "^4", - "@types/node": "^20", - "@types/react": "^19", - "@types/react-dom": "^19", - "eslint": "^9", - "eslint-config-next": "16.3.1", - "tailwindcss": "^4", - "typescript": "^5" - } -} -``` -- Add `apps/web/tsconfig.json` extending `../../tsconfig.base.json` with `@/*` → `./**/*` and `@mcpedia/*` workspace aliases. -- **Verify:** `cd apps/web && bun install && bun run build` compiles (may warn about empty content — fine). - -### Task 0.4: `.env` (gitignored) for dev -- **Create:** `.env` (gitignored) with `DATABASE_URL=postgresql://:@100.121.180.82:6432/mcpedia` (value from BWS; `prepare:false` enforced in db client). Add `.env.example` (committed, placeholder only). - ---- - -## Phase 1 — MVP: Core + Web + MCP + Postgres FTS (detailed) - -### Task 1.1: `packages/types` -- **Create:** `packages/types/package.json` (`@mcpedia/types`, exports `src/index.ts`), `packages/types/src/index.ts`: -```ts -export type DocSection = "docs" | "writeups" | "research" | "notes"; -export type DocType = "documentation" | "writeup" | "research" | "note"; -export type DocStatus = "published" | "draft"; - -export interface DocumentMeta { - id: string; // slug - slug: string; - title: string; - type: DocType; - section: DocSection; - status: DocStatus; - author: string; - tags: string[]; - path: string; // relative path under content/ - createdAt: string; - updatedAt: string; -} - -export interface Document extends DocumentMeta { - body: string; // raw markdown (read from disk or stored) -} - -export interface SearchHit { - doc: DocumentMeta; - rank: number; - snippet: string; -} -``` -- **Verify:** `bunx tsc --noEmit` in package passes. - -### Task 1.2: `packages/config` -- **Create:** `packages/config/package.json` (`@mcpedia/config`), `packages/config/src/index.ts`: -```ts -import { fileURLToPath } from "node:url"; -import { dirname, resolve } from "node:path"; - -const root = resolve(dirname(fileURLToPath(import.meta.url)), "../../.."); -export const CONTENT_ROOT = process.env.CONTENT_ROOT - ?? resolve(root, "content"); -export const DATABASE_URL = process.env.DATABASE_URL ?? ""; -``` -- **Verify:** import in a scratch `bun -e` prints a path. - -### Task 1.3: `packages/db` (Drizzle + Postgres, FTS) -- **Create:** `packages/db/package.json` (`@mcpedia/db`, deps `drizzle-orm`, `postgres`; dev `drizzle-kit`), `packages/db/src/schema.ts`: -```ts -import { sql } from "drizzle-orm"; -import { text, timestamp, integer, index, customType } from "drizzle-orm/pg-core"; - -// tsvector isn't a first-class drizzle type; wrap raw. -export const documents = pgTable("documents", { - id: text("id").primaryKey(), // slug - slug: text("slug").notNull().unique(), - title: text("title").notNull(), - type: text("type").notNull(), - section: text("section").notNull(), - status: text("status").notNull().default("published"), - author: text("author").notNull().default(""), - tags: text("tags").array().notNull().default(sql`'{}'`), - path: text("path").notNull(), - body: text("body").notNull().default(""), - searchVector: customType<{ data: string }>({ - dataType() { return "tsvector"; }, - })("search_vector") - .generatedAlwaysAs(sql`to_tsvector('simple', coalesce(title,'') || ' ' || coalesce(body,''))`), - createdAt: timestamp("created_at", { withTimezone: true }).notNull(), - updatedAt: timestamp("updated_at", { withTimezone: true }).notNull(), -}, (t) => ({ - searchIdx: index("documents_search_idx").using("gin", t.searchVector), -})); -``` - NOTE: `document_chunks` + `embedding vector` are **Phase 2** — do NOT add now. -- **Create:** `packages/db/src/client.ts` — postgres-js with `prepare: false` (PgBouncer safe): -```ts -import { drizzle } from "drizzle-orm/postgres-js"; -import postgres from "postgres"; -import { DATABASE_URL } from "@mcpedia/config"; -import * as schema from "./schema"; - -const client = postgres(DATABASE_URL, { prepare: false, max: 5 }); -export const db = drizzle(client, { schema }); -export { schema }; -``` -- **Create:** `packages/db/drizzle.config.ts` (dialect `postgresql`, schema `./src/schema.ts`, out `./drizzle`). -- **Verify:** `bunx drizzle-kit generate` produces a migration; `bunx drizzle-kit push` applies it to imrnes (operator supplies DATABASE_URL). `bunx drizzle-kit studio` optional. - -### Task 1.4: `packages/parser` (frontmatter) -- **Create:** `packages/parser/package.json` (`@mcpedia/parser`, dep `gray-matter`), `packages/parser/src/index.ts`: -```ts -import matter from "gray-matter"; -import { readFileSync } from "node:fs"; -import type { DocSection, DocType, DocStatus, DocumentMeta } from "@mcpedia/types"; - -const SECTIONS: DocSection[] = ["docs", "writeups", "research", "notes"]; - -export function parseFile(absPath: string, relPath: string): { meta: DocumentMeta; body: string } { - const raw = readFileSync(absPath, "utf8"); - const { data, content } = matter(raw); - const section = SECTIONS.find((s) => relPath.startsWith(s + "/")) ?? "docs"; - const slug = relPath.replace(/\.mdx?$/, ""); - const meta: DocumentMeta = { - id: slug, - slug, - title: data.title ?? slug, - type: (data.type ?? "documentation") as DocType, - section, - status: (data.status ?? "published") as DocStatus, - author: data.author ?? "", - tags: Array.isArray(data.tags) ? dataAtags : [], - path: relPath, - createdAt: data.created_at ?? new Date().toISOString(), - updatedAt: data.updated_at ?? data.created_at ?? new Date().toISOString(), - }; - return { meta, body: content }; -} -``` - (fix the obvious `dataAtags` typo to `data.tags` when implementing.) -- **Verify:** unit test parsing a sample frontmatter → asserts `slug`, `section`, `tags`. - -### Task 1.5: `packages/search` (FTS query) -- **Create:** `packages/search/package.json` (`@mcpedia/search`, dep `@mcpedia/db`, `@mcpedia/types`), `packages/search/src/index.ts`: -```ts -import { db } from "@mcpedia/db"; -import { documents } from "@mcpedia/db/schema"; -import { sql, and, eq } from "drizzle-orm"; -import type { SearchHit } from "@mcpedia/types"; - -export function toTsQuery(q: string): string { - // plainto_tsquery-style: AND the terms, escape punctuation. - const terms = q.trim().split(/\s+/).filter(Boolean).map((t) => t.replace(/[^\p{L}\p{N}]/gu, "")); - return terms.map((t) => `${t}:*`).join(" & "); -} - -export async function keywordSearch(q: string, limit = 20): Promise { - const query = toTsQuery(q); - if (!query) return []; - const rows = await db - .select({ - doc: documents, - rank: sql`ts_rank(${documents.searchVector}, to_tsquery('simple', ${query}))`, - snippet: sql`ts_headline('simple', ${documents.body}, to_tsquery('simple', ${query}), 'MaxWords=20, MinWords=5')`, - }) - .from(documents) - .where(and(eq(documents.status, "published"), sql`${documents.searchVector} @@ to_tsquery('simple', ${query})`)) - .orderBy(sql`ts_rank(${documents.searchVector}, to_tsquery('simple', ${query})) desc`) - .limit(limit); - return rows.map((r) => ({ doc: r.doc, rank: r.rank, snippet: r.snippet })); -} -``` -- **Verify:** after seeding, `keywordSearch("websocket")` returns hits. - -### Task 1.6: `packages/core` (services) -- **Create:** `packages/core/package.json` (`@mcpedia/core`, deps `@mcpedia/db`, `@mcpedia/parser`, `@mcpedia/search`, `@mcpedia/config`, `@mcpedia/types`), `packages/core/src/`: - - `content.service.ts` — read markdown from `CONTENT_ROOT` by slug/section; list files via `fast-glob`. - - `document.service.ts` — `listDocuments(section?, type?, status?)`, `getDocument(slug)` (DB meta + disk body), `getRelated(slug)` (shared tags, later vector). - - `search.service.ts` — wraps `keywordSearch`. - - `index.ts` — re-export services + types. -```ts -// document.service.ts (sketch) -import { db } from "@mcpedia/db"; -import { documents } from "@mcpedia/db/schema"; -import { eq, and } from "drizzle-orm"; -import { CONTENT_ROOT } from "@mcpedia/config"; -import { readFileSync, existsSync } from "node:fs"; -import { resolve } from "node:path"; -import type { Document, DocumentMeta } from "@mcpedia/types"; - -export async function listDocuments(opts: { section?: string; status?: string } = {}): Promise { - const where = [eq(documents.status, opts.status ?? "published")]; - if (opts.section) where.push(eq(documents.section, opts.section)); - return db.select().from(documents).where(and(...where)); -} - -export async function getDocument(slug: string): Promise { - const [meta] = await db.select().from(documents).where(eq(documents.slug, slug)); - if (!meta) return null; - const abs = resolve(CONTENT_ROOT, meta.path); - const body = existsSync(abs) ? readFileSync(abs, "utf8") : meta.body; - return { ...meta, body }; -} -``` -- **Verify:** `bunx tsc --noEmit` passes across packages. - -### Task 1.7: `scripts/indexer.ts` (walk content → upsert) -- **Create:** `scripts/indexer.ts` — walk `CONTENT_ROOT` (`**/*.{md,mdx}`), `parseFile`, upsert into `documents` (slug PK; `onConflictDoUpdate` title/tags/body/updatedAt). Logs count. -```ts -import { db } from "@mcpedia/db"; -import { documents } from "@mcpedia/db/schema"; -import { parseFile } from "@mcpedia/parser"; -import { CONTENT_ROOT } from "@mcpedia/config"; -import { walk } from "fast-glob"; -import { resolve } from "node:path"; - -const files = await glob("**/*.{md,mdx}", { cwd: CONTENT_ROOT, absolute: true }); -let n = 0; -for (const abs of files) { - const rel = resolve(CONTENT_ROOT, abs).replace(CONTENT_ROOT + "/", ""); - const { meta, body } = parseFile(abs, rel); - await db.insert(documents).values({ ...meta, body }) - .onConflictDoUpdate({ target: documents.slug, set: { title: meta.title, tags: meta.tags, body, updatedAt: new Date().toISOString() } }); - n++; -} -console.log(`indexed ${n} documents`); -``` -- **Verify:** `bun run index` prints `indexed N documents`; `psql`/`drizzle-kit studio` shows rows + non-empty `search_vector`. - -### Task 1.8: Seed content -- **Create:** `content/docs/websocket/contract.md`, `content/writeups/debugging/websocket-timeout.md`, `content/research/mcp/architecture.md`, `content/notes/typescript/patterns.md` with the spec's frontmatter shape (id/title/type/tags/status/author/created_at/updated_at). Add 2–3 short paragraphs each so FTS has text. -- **Verify:** `bun run index` ingests them; `keywordSearch("websocket")` returns ≥2 hits. - -### Task 1.9: `apps/web` — pages (Next 16, read `node_modules/next/dist/docs` first) -- **Read** Node_modules Next 16 docs (app router, server components, route handlers) — mandatory. -- **Create:** - - `apps/web/app/page.tsx` — home: list sections + recent docs (server component calling `listDocuments`). - - `apps/web/app/(docs)/[...slug]/page.tsx` — document view: `getDocument(slug)`, render markdown (Phase 1: simple `dangerouslySetInnerHTML` of a remark→HTML, or raw pre with Shiki later; keep minimal but legible). - - `apps/web/app/search/page.tsx` — search box → `keywordSearch`, render hits + snippets. - - Keep Tailwind v4 styling minimal but clean (no shadcn yet — Phase 2 polish). -- **Verify:** `bun run build` (apps/web) passes; `bun run dev` then `curl -s localhost:3000/` and `/search?q=websocket` return rendered HTML. - -### Task 1.10: `apps/mcp` — MCP server -- **Create:** `apps/mcp/package.json` (`@mcpedia/mcp`, deps `@modelcontextprotocol/sdk`, `zod`, `@mcpedia/core`), `apps/mcp/src/index.ts`: -```ts -import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js"; -import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js"; -import { z } from "zod"; -import { listDocuments, getDocument } from "@mcpedia/core"; -import { keywordSearch } from "@mcpedia/search"; - -const server = new McpServer({ name: "mcpedia", version: "0.1.0" }); - -server.tool("search_documents", { query: z.string(), limit: z.number().optional() }, - async ({ query, limit }) => { - const hits = await keywordSearch(query, limit ?? 20); - return { content: [{ type: "text", text: JSON.stringify(hits, null, 2) }] }; - }); - -server.tool("get_document", { slug: z.string() }, - async ({ slug }) => { - const doc = await getDocument(slug); - return doc ? { content: [{ type: "text", text: doc.body }] } - : { content: [{ type: "text", text: "not found" }], isError: true }; - }); - -server.tool("list_documents", { section: z.string().optional() }, - async ({ section }) => { - const docs = await listDocuments({ section }); - return { content: [{ type: "text", text: JSON.stringify(docs, null, 2) }] }; - }); - -// Resource: mcpedia://
/ -server.resource("doc", "mcpedia://", async (uri) => { - const slug = uri.path.replace(/^\//, ""); - const doc = await getDocument(slug); - return { contents: [{ uri: uri.href, text: doc?.body ?? "" }] }; -}); - -const transport = new StdioServerTransport(); -await server.connect(transport); -``` - (Exact SDK method names verified via Context7 before coding.) -- **Create:** `apps/mcp/bin` or `bun run` entry; add `"start": "bun run src/index.ts"` to its package.json. -- **Verify:** smoke test via `scripts/mcp-smoke.ts` that imports the tool handlers and calls `search_documents({query:"websocket"})` → asserts ≥1 hit. (Running a real stdio client in CI is optional; inspector `@modelcontextprotocol/inspector` for manual check.) - -### Task 1.11: Cross-package verification -- **Run:** `bun install` (root) → `bunx turbo run lint` → `bunx turbo run build` → `bunx turbo run typecheck`. All green. -- **Run:** `bun run index` then `bun run scripts/mcp-smoke.ts`. -- **Verify:** no package imports `pg`/`drizzle` outside `@mcpedia/core` or `@mcpedia/db` (architecture guard). Search the tree: `search_files("from \"pg\"", path="apps")`. - -### Task 1.12: README + phase markers -- **Modify:** `README.md` — replace create-next-app text with MCPedia overview, monorepo map, "Phase 1 status", how to dev (`bun install && bun run index && bun run dev`), how to run MCP (`bun run --cwd apps/mcp start`). -- **Verify:** `cat README.md` reads correctly. - ---- - -## Phase 2 — Semantic search + tRPC/Hono API + auth (outline, expand later) - -- Add `document_chunks` table + `embedding vector` (pgvector extension on imrnes — operator enables `CREATE EXTENSION vector`). -- `packages/search`: `EmbeddingProvider` abstraction (OpenAI/Gemini/Ollama/Local) behind an interface (spec principle 6). Chunking in `packages/parser` (by heading/section). -- Hybrid ranking: fuse FTS `ts_rank` + cosine distance (Reciprocal Rank Fusion). -- `apps/api`: Hono + tRPC router exposing `documents.*`, `search.*`; Web switches from direct Core calls to tRPC client; MCP may also call API or keep Core (decide: keep Core for colocated simplicity, API for external). -- `packages/auth`: Auth.js/OIDC adapter; `permissions` table; gate write/mutations. Phase 1 documents are public-read. -- Categories + tags linking tables (`categories`, `document_tags`, `references`). -- MDX rendering with Shiki + shadcn/ui polish on Web. - -## Phase 3 — Async indexing + revisions (outline) - -- Redis (imrnes or local) + BullMQ workers: parse/chunk/embed/reindex off the request path. -- `document_revisions` table + revision API; Git sync hook (content/ push → reindex). -- Indexer becomes a worker triggered by file watch / webhook, not a manual script. - -## Phase 4 — Scale (outline, only if needed) - -- Dedicated search engine (OpenSearch) behind the same `SearchService` seam — YAGNI until FTS+pgvector saturates. -- Object storage for assets, advanced ranking, distributed workers, observability, multi-tenant. Docker Compose for service orchestration pointing at imrnes Postgres. - ---- - -## Risks / tradeoffs / open questions - -- **Next 16 breaking changes** — biggest unknown; mitigated by reading `node_modules/next/dist/docs` before Web tasks. If app-router differs enough, adjust Task 1.9 accordingly. -- **PgBouncer + prepared statements** — handled via `prepare: false`; if imrnes uses session pool mode, can drop it. -- **FTS language** — using `'simple'` config (no stemming) so mixed ID/EN queries match literally; revisit if relevance is poor (could add `'english'` or a custom ID dict). -- **pgvector on imrnes** — extension may need operator enablement; Phase 2 blocker, not Phase 1. -- **MCP resource URI shape** — `mcpedia://
/` is a sketch; finalize against MCP SDK spec in Task 1.10 (Context7). -- **Monorepo driver** — bun workspaces chosen over pnpm per user; Turborepo runs fine under bun. If `turbo` bin resolution misbehaves under bun, fall back to `bunx turbo`. - -## Verification matrix (Phase 1) - -| Check | Command | Expect | -| --- | --- | --- | -| Install | `bun install` | clean | -| Lint | `bunx turbo run lint` | 0 errors | -| Typecheck | `bunx turbo run typecheck` | 0 errors | -| Build | `bunx turbo run build` | web + mcp build | -| Index | `bun run index` | `indexed N documents` | -| Web | `bun run dev` (apps/web) + curl `/` and `/search?q=websocket` | rendered HTML, hits | -| MCP | `bun run scripts/mcp-smoke.ts` | ≥1 hit for "websocket" | -| Arch guard | search for `from "pg"` outside `packages/db`,`packages/core` | none | diff --git a/.hermes/plans/2026-08-19_phase2-semantic-api.md b/.hermes/plans/2026-08-19_phase2-semantic-api.md deleted file mode 100644 index 3bfa61e..0000000 --- a/.hermes/plans/2026-08-19_phase2-semantic-api.md +++ /dev/null @@ -1,161 +0,0 @@ -# MCPedia Phase 2 — Semantic Search + tRPC/Hono API - -> **For Hermes:** implement task-by-task. Spec-first (user rule 2026-08-19). - -**Goal:** Add semantic + hybrid search (pgvector) and a typed tRPC/Hono API so -MCPedia is queryable by embeddings, not just keyword FTS — and expose the -corpus over a programmatic HTTP API. - -**Architecture:** Content (Markdown) → chunk → embed (OpenRouter) → store -`document_chunks` with `vector(N)` in Postgres → `semanticSearch` (cosine) and -`hybridSearch` (FTS + cosine, reciprocal-rank fusion) in `@mcpedia/search` → -exposed via Core, the MCP server (new tools), and a new `apps/api` (Hono + -tRPC v11). - -**Embedding provider:** OpenRouter (`openrouter/llama-nemotron-embed-vl-1b-v2:free`) -via `9router_ai_llm_api_key` + `9router_ai_llm_base_url` (BWS). Dimension is -discovered at first live call (see Step 1.3) and pinned in schema/migration. - -**Tech stack:** drizzle-orm `vector` column + pgvector extension, HNSW index, -`@trpc/server` v11 (fetch adapter), `hono` + `@hono/node-server`. - ---- - -## Task P2.1 — `packages/embeddings` (provider + abstraction) - -**Files:** `packages/embeddings/package.json`, `src/index.ts`, `src/provider.ts`, -`src/openrouter.ts` - -- `EmbeddingProvider` interface: `embed(texts: string[]): Promise`, `readonly model`, `readonly dimensions`. -- `OpenRouterEmbeddingProvider`: POST `${baseUrl}/embeddings` with `{ model, input }`, - `Authorization: Bearer ${key}`. Returns `data[].embedding`. Validate length === dimensions. -- Read `EMBED_BASE_URL`, `EMBED_API_KEY`, `EMBED_MODEL` from `@mcpedia/config` - (with `.env` fallback). Dimensions discovered live (Step 1.3) → export `EMBED_DIM`. -- Chunk helper `chunkText(text, { size=1000, overlap=150 })` in `src/chunk.ts`. - -**Step 1.3 (discover dim):** live call `embed(["test"])`, read `embedding.length`, -pin `EMBED_DIM`, assert mismatch throws. - -**Verify:** `bun run` a temp script: `embed(["hello world"])` prints a vector of -length N (e.g. 1024). Confirm no key is logged. - ---- - -## Task P2.2 — Schema: `document_chunks` + vector extension - -**Files:** `packages/db/src/schema.ts` (add), `packages/db/drizzle.config.ts` -(unchanged), new migration. - -- `CREATE EXTENSION IF NOT EXISTS vector;` (idempotent; run once via psql). -- `document_chunks` table: - - `id` uuid pk default gen_random_uuid() - - `document_id` text → `documents.id` on delete cascade - - `slug` text (denormalized for convenience) - - `chunk_index` integer - - `content` text - - `embedding` vector(EMBED_DIM) - - `created_at` timestamp default now() - - index `chunk_embedding_idx` using hnsw (`embedding` op `vector_cosine_ops`) -- Generate migration with `drizzle-kit generate`, apply via `psql` (drizzle-kit - push is unreliable here — known). - -**Verify:** `\d document_chunks` shows `embedding vector(N)` + HNSW index; -`select count(*) from document_chunks` = 0. - ---- - -## Task P2.3 — Indexer: chunk + embed + upsert - -**Files:** `scripts/indexer.ts` (extend), `packages/core/src/document.service.ts` -(add `indexChunks`). - -- For each published doc: read body (already on disk), `chunkText`, `embed` in - batches (≤ 16), delete existing chunks for slug, insert new rows. -- Guard: if embedding provider fails, log + skip (don't crash the whole index). -- Add `bun run index:embed` (or extend `bun run index` to also embed). - -**Verify:** after running, `select count(*) from document_chunks` > 0; a sample -row has non-null `embedding`. - ---- - -## Task P2.4 — `packages/search`: semantic + hybrid - -**Files:** `packages/search/src/index.ts` (add `semanticSearch`, `hybridSearch`). - -- `semanticSearch(vec, limit)`: order by `embedding <=> ${vec}` asc, filter published. -- `hybridSearch(q, limit)`: run FTS (`ts_rank`) + semantic (cosine) in parallel; - fuse with reciprocal-rank (RRF: score = 1/(k+rank), k=60); return merged hits. -- Keep `keywordSearch` unchanged (Phase 1). - -**Verify:** unit-ish script: embed a query, `semanticSearch` returns relevant -chunks; `hybridSearch("websocket")` returns ≥ keyword results. - ---- - -## Task P2.5 — `packages/core` expose semantic/hybrid - -**Files:** `packages/core/src/search.service.ts`, `index.ts`. - -- Re-export `semanticSearch`, `hybridSearch` from Core. - ---- - -## Task P2.6 — `apps/api` (Hono + tRPC v11) - -**Files:** `apps/api/package.json`, `tsconfig.json`, `src/index.ts`, -`src/router.ts`, `src/trpc.ts`. - -- `initTRPC.create()` router with procedures: `search`, `semanticSearch`, - `hybridSearch`, `getDocument`, `listDocuments` (mirrors MCP tools). -- Mount `fetchRequestHandler` on a Hono app at `/trpc/*`; serve via - `@hono/node-server` `serve({ fetch: app.fetch, port: 4020 })`. -- `createContext` returns `{ db }`. - -**Verify:** `bun run dev` → `curl -X POST localhost:4020/trpc/search` -with JSON body returns hits. - ---- - -## Task P2.7 — MCP server: semantic + hybrid tools - -**Files:** `apps/mcp/src/index.ts` (add `semantic_search`, `hybrid_search`), -extend `smoke.test.ts`. - -- `semantic_search`: embed query → `semanticSearch`. -- `hybrid_search`: embed query → `hybridSearch`. -- Smoke: assert both return ≥1 hit for "websocket". - ---- - -## Task P2.8 — Web: semantic toggle on search - -**Files:** `apps/web/app/search/page.tsx`. - -- Add `mode=keyword|hybrid` query param; server component calls Core - `hybridSearch` when `mode=hybrid`. Minimal UI toggle (link/buttons). -- Keep keyword as default. - -**Verify:** `bun run build`; `curl '/search?q=websocket&mode=hybrid'` returns hits. - ---- - -## Task P2.9 — Verify all + commit - -- `bunx turbo run build` (web + api + mcp), `bun run apps/mcp smoke`, - live API curl, live web hybrid search. -- Update `README.md` + `PHASES.md` (mark Phase 2 ✅). -- `git add -A` (exclude `.env`), commit as asepharyana (no Co-Authored-By). - ---- - -## Risks / decisions -- **Dimension unknown until live call** → P2.1.3 discovers it; pinned EMBED_DIM=2048. -- **pgvector NOT available on shared imrnes Postgres** (extension not installed; - installing needs host-level apt on a managed/shared DB — deferred). PIVOT: - store `embedding` as `real[]` and compute cosine similarity in the app layer. - Brute-force cosine is instant for a KB-sized corpus (dozens of docs / hundreds - of chunks). pgvector+HNSW is the Phase-4 scale-out path. -- **PgBouncer + real[]**: fine; simple queries, no extension needed. -- **API port 4020** (host 4000s range is 4000–4015; 4020 is free for dev). Deploy later. -- **YAGNI**: no auth/revisions this phase (Phase 3). diff --git a/.hermes/plans/hierarchical-folders.md b/.hermes/plans/hierarchical-folders.md deleted file mode 100644 index eb5506e..0000000 --- a/.hermes/plans/hierarchical-folders.md +++ /dev/null @@ -1,151 +0,0 @@ -# Phase 14 — Hierarchical Folder Structure - -> User: "gk ada bedanya, maksud saya inginnya itu bisa yg bertingkat seperti github yg memiliki folder dalam folder" -> Context: after the full dynamic-custom-fields overhaul (Phase 13), the user -> wants document URLs/content organized in **nested folders** like GitHub — -> `writeups/ctf/defcon-quals-2024/pwn-100/...` with subfolders under subfolders, -> not just one level deep. - -## Problem - -The current URL scheme is `/
/` where `slug` can contain `/` -(e.g. `writeups/ctf/defcon-quals-2024/pwn-100-ret2win-alignment` → -`/writeups/ctf/defcon-quals-2024/pwn-100-ret2win-alignment`). This works for -**files** but there are no **folder-level index pages** — navigating to -`/writeups/ctf/defcon-quals-2024/` returns 404 because Next.js catch-all -`[section]/[...slug]/page.tsx` requires at least one slug segment beyond the -section, and the sidebar only shows flat doc titles (no folder tree). - -GitHub's model: `github.com/org/repo/tree/main/path/to/folder/file` — every -folder has an index page (`/path/to/folder/`) listing its contents. - -## Solution - -### 1. Folder Index Pages - -**Create `apps/web/app/[section]/[...slug]/folder.tsx`** (or a parallel route). -Actually — cleaner approach per Next.js App Router: the catch-all -`[section]/[...slug]/page.tsx` handles both. Add logic: if the slug resolves to -an actual markdown file → doc page (existing behavior). If the slug resolves to -a **directory** (folder of docs) → render a folder index listing all docs whose -`path` starts with that prefix. - -**Mechanism:** -- Call `listDocuments()` to get all docs. -- The incoming URL path is `{section}/{...slug}`. -- Build the "folder prefix" = `${section}/${slug.join("/")}/` (with trailing `/`, - or just `${section}/${slug.join("/")}` if no slug segments). -- Filter docs whose `doc.path` starts with that prefix. -- If exactly one doc matches AND its path === prefix (trimmed .md) → it's a - doc page (existing). If zero or multiple match and they all start with the - prefix → it's a folder index. -- Edge: a folder with exactly one doc whose path matches exactly — still a doc - page. A folder is when there are docs at `prefix/sub/...`. - -**Better heuristic:** A slug path is a "folder" if there exist docs whose `path` -is `prefix/deep/...` (i.e., the slug is a parent of other doc paths, not a -leaf itself). A slug is a "leaf doc" if `path === prefix + ".md"`. - -### 2. Sidebar Tree - -**Update `Sidebar.tsx`:** -- `listDocuments()` already returns all docs with their full `slug` and `path`. -- Build a **tree** from the flat list: split each slug by `/`, create nested - folder nodes. -- Render nested `
    ` with indentation (already done via `marginLeft` based on - depth). -- **Folder nodes** (collapsed/expanded) get a folder icon 📁 and a CSS class. -- Clicking a folder → navigates to the folder index page `/{section}/{path}`. -- **Leaf doc nodes** → link to `/{doc.slug}` (existing behavior). -- Group by the first segment after section too (e.g. `ctf/defcon-quals-2024/` - is a folder, then `pwn-100-...` are children). - -Tree-building algorithm (from flat slugs): -``` -For slug "writeups/ctf/defcon-quals-2024/pwn-100-ret2win-alignment": - parts = ["writeups", "ctf", "defcon-quals-2024", "pwn-100-ret2win-alignment"] - → tree: writeups → ctf → defcon-quals-2024 → pwn-100-ret2win-alignment (leaf) -``` - -### 3. DocForm / Folder Selection - -**Update `DocForm.tsx`:** -- Add a "Parent folder" input (autocomplete or text) that shows existing folders - for the selected section. The slug field already supports `/` but the user - experience is better with folder picker. -- When creating, the slug becomes `{parentFolder}/{slug}` automatically. -- Show existing folder structure as ` setSection(e.target.value as typeof section)} - className={baseInputCls} - disabled={isEdit} - > - {SECTION_OPTIONS.map((s) => ( - - ))} - +
    + setSection(e.target.value)} + placeholder="e.g. docs, writeups, ctf, guides" + className={baseInputCls} + required + /> + + {sectionOptions.map((s) => ( + + ))} + +
    - +
    + setType(e.target.value)} + placeholder="e.g. documentation, writeup, note" + className={baseInputCls} + /> + + {DEFAULT_TYPE_LIST.map((t) => ( + + ))} + +