feat: revamp to dynamic database-first architecture and cleanup phase docs
CI / typecheck + build (turbo) (push) Canceled after 0s
CI / typecheck + build (turbo) (push) Canceled after 0s
- Migrate document fetching and CRUD to be PostgreSQL-authoritative - Remove static section enums and add dynamic listSections query - Support custom sections and metadata across API, MCP, and Web UI - Add /api/sections endpoint and update Header, Sidebar, and Forms - Remove obsolete phase planning docs and modernize README/AGENTS
This commit is contained in:
asepharyana committed
1 parent
a937e8f51b
commit
a9b67385c3
37 files changed
+720
-2515
No files matched your search
@@ -1,435 +0,0 @@
|
||||
# MCPedia Implementation Plan
|
||||
|
||||
> **For Hermes:** Use subagent-driven-development to implement task-by-task. Phase 1 tasks are bite-sized and ready to execute; Phases 2–4 are outlines to expand later.
|
||||
|
||||
**Goal:** Turn the fresh `create-next-app` Next.js 16 repo into the MCPedia knowledge base — Git/Markdown content, a Core business-logic layer shared by a Next.js Web UI and an MCP server for AI agents, with content metadata in Postgres (FTS search), and an architecture that can grow to semantic search, a tRPC API, auth, and async indexing without rewrites.
|
||||
|
||||
**Architecture:** Monorepo (bun workspaces + Turborepo). Content lives as Markdown in `content/` (Git-managed, human-readable). `packages/core` holds all business logic (Document/Content/Search services). Web (`apps/web`) and MCP (`apps/mcp`) both consume Core — no DB access outside Core. Postgres (imrnes `:6432`) stores document metadata + a `tsvector` for keyword search. Phase 1 ships keyword search only; pgvector/semantic, tRPC/Hono API, auth, and BullMQ are later phases behind clean seamsLine truncated
|
||||
|
||||
**Tech Stack (resolved forks):**
|
||||
- Package manager / monorepo: **bun workspaces + Turborepo** (honor existing repo; NOT pnpm — user decision).
|
||||
- DB: **imrnes Postgres `100.121.180.82:6432`** for both dev and deploy (PgBouncer transaction pool → set `prepare: false` on the driver).
|
||||
- Phase 1 scope: **Core + Web + MCP + Postgres FTS only**. tRPC/Hono API, pgvector, auth, Redis/BullMQ deferred.
|
||||
|
||||
---
|
||||
|
||||
## Environment & preconditions (read first)
|
||||
|
||||
- Repo is currently a flat Next.js 16 app, **not git-initialized**, bun-managed (`packageManager: bun@1.3.14`, `bun.lock` present).
|
||||
- `app/` has only `layout.tsx`, `page.tsx`, `globals.css`, `favicon.ico`. Tailwind v4 (`@tailwindcss/postcss`).
|
||||
- Next.js 16 is "NOT the Next.js you know" (see `AGENTS.md`): **before writing any `apps/web` code, read `node_modules/next/dist/docs/`** (app router, server components, route handlers) for breaking changes. Treat deprecation notices as authoritative.
|
||||
- For every library API used (Next 16, Drizzle, `@modelcontextprotocol/sdk`, gray-matter) **resolve + query Context7** before coding — do not trust training data.
|
||||
- Secrets: `DATABASE_URL` lives in **BWS project `orangevps`** (never committed). Local dev reads it from a gitignored `.env` at repo root.
|
||||
- Seed DB creds: a dedicated `mcpedia` database/user on imrnes; create via `bws` secret or operator. `pgvector` extension is **NOT** required until Phase 2.
|
||||
|
||||
---
|
||||
|
||||
## Phase 0 — Repo restructure (monorepo scaffold)
|
||||
|
||||
### Task 0.1: Initialize git + baseline commit
|
||||
- **Files:** `.gitignore` (already exists; ensure `.env`, `.next/`, `node_modules/`, `*.tsbuildinfo`, `dist/` ignored).
|
||||
- **Step:** `git init && git add -A && git commit -m "chore: baseline create-next-app scaffold"` (author = asepharyana, no Co-Authored-By).
|
||||
- **Verify:** `git log --oneline -1` shows the commit.
|
||||
|
||||
### Task 0.2: Root workspace package.json + Turbo
|
||||
- **Modify:** `package.json` (root) — strip app deps, add workspaces + turbo scripts.
|
||||
- **Create:** `turbo.json`, `tsconfig.base.json`.
|
||||
- Root `package.json` becomes:
|
||||
```json
|
||||
{
|
||||
"name": "mcpedia",
|
||||
"version": "0.1.0",
|
||||
"private": true,
|
||||
"packageManager": "bun@1.3.14",
|
||||
"workspaces": ["apps/*", "packages/*"],
|
||||
"scripts": {
|
||||
"dev": "turbo run dev",
|
||||
"build": "turbo run build",
|
||||
"lint": "turbo run lint",
|
||||
"typecheck": "turbo run typecheck",
|
||||
"index": "bun run scripts/indexer.ts"
|
||||
},
|
||||
"devDependencies": {
|
||||
"turbo": "^2",
|
||||
"typescript": "^5",
|
||||
"prettier": "^3"
|
||||
}
|
||||
}
|
||||
```
|
||||
- `turbo.json`:
|
||||
```json
|
||||
{
|
||||
"$schema": "https://turbo.build/schema.json",
|
||||
"tasks": {
|
||||
"build": { "dependsOn": ["^build"], "outputs": ["dist/**", ".next/**"] },
|
||||
"lint": {},
|
||||
"typecheck": { "dependsOn": ["^build"] },
|
||||
"dev": { "cache": false, "persistent": true }
|
||||
}
|
||||
}
|
||||
```
|
||||
- **Verify:** `bun install` succeeds; `bunx turbo --version` prints a version.
|
||||
|
||||
### Task 0.3: Move Web app into `apps/web`
|
||||
- **Move:** `app/`, `public/`, `next.config.ts`, `next-env.d.ts`, `postcss.config.mjs`, `eslint.config.mjs`, `tsconfig.json` → `apps/web/` (rename root `app` → `apps/web/app`). Keep Tailwind v4. Add `apps/web/package.json`:
|
||||
```json
|
||||
{
|
||||
"name": "@mcpedia/web",
|
||||
"private": true,
|
||||
"scripts": {
|
||||
"dev": "next dev",
|
||||
"build": "next build",
|
||||
"start": "next start",
|
||||
"lint": "eslint .",
|
||||
"typecheck": "tsc --noEmit"
|
||||
},
|
||||
"dependencies": {
|
||||
"next": "16.3.1",
|
||||
"react": "19.2.8",
|
||||
"react-dom": "19.2.8",
|
||||
"@mcpedia/core": "workspace:*",
|
||||
"@mcpedia/config": "workspace:*"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@tailwindcss/postcss": "^4",
|
||||
"@types/node": "^20",
|
||||
"@types/react": "^19",
|
||||
"@types/react-dom": "^19",
|
||||
"eslint": "^9",
|
||||
"eslint-config-next": "16.3.1",
|
||||
"tailwindcss": "^4",
|
||||
"typescript": "^5"
|
||||
}
|
||||
}
|
||||
```
|
||||
- Add `apps/web/tsconfig.json` extending `../../tsconfig.base.json` with `@/*` → `./**/*` and `@mcpedia/*` workspace aliases.
|
||||
- **Verify:** `cd apps/web && bun install && bun run build` compiles (may warn about empty content — fine).
|
||||
|
||||
### Task 0.4: `.env` (gitignored) for dev
|
||||
- **Create:** `.env` (gitignored) with `DATABASE_URL=postgresql://<user>:<pass>@100.121.180.82:6432/mcpedia` (value from BWS; `prepare:false` enforced in db client). Add `.env.example` (committed, placeholder only).
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — MVP: Core + Web + MCP + Postgres FTS (detailed)
|
||||
|
||||
### Task 1.1: `packages/types`
|
||||
- **Create:** `packages/types/package.json` (`@mcpedia/types`, exports `src/index.ts`), `packages/types/src/index.ts`:
|
||||
```ts
|
||||
export type DocSection = "docs" | "writeups" | "research" | "notes";
|
||||
export type DocType = "documentation" | "writeup" | "research" | "note";
|
||||
export type DocStatus = "published" | "draft";
|
||||
|
||||
export interface DocumentMeta {
|
||||
id: string; // slug
|
||||
slug: string;
|
||||
title: string;
|
||||
type: DocType;
|
||||
section: DocSection;
|
||||
status: DocStatus;
|
||||
author: string;
|
||||
tags: string[];
|
||||
path: string; // relative path under content/
|
||||
createdAt: string;
|
||||
updatedAt: string;
|
||||
}
|
||||
|
||||
export interface Document extends DocumentMeta {
|
||||
body: string; // raw markdown (read from disk or stored)
|
||||
}
|
||||
|
||||
export interface SearchHit {
|
||||
doc: DocumentMeta;
|
||||
rank: number;
|
||||
snippet: string;
|
||||
}
|
||||
```
|
||||
- **Verify:** `bunx tsc --noEmit` in package passes.
|
||||
|
||||
### Task 1.2: `packages/config`
|
||||
- **Create:** `packages/config/package.json` (`@mcpedia/config`), `packages/config/src/index.ts`:
|
||||
```ts
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { dirname, resolve } from "node:path";
|
||||
|
||||
const root = resolve(dirname(fileURLToPath(import.meta.url)), "../../..");
|
||||
export const CONTENT_ROOT = process.env.CONTENT_ROOT
|
||||
?? resolve(root, "content");
|
||||
export const DATABASE_URL = process.env.DATABASE_URL ?? "";
|
||||
```
|
||||
- **Verify:** import in a scratch `bun -e` prints a path.
|
||||
|
||||
### Task 1.3: `packages/db` (Drizzle + Postgres, FTS)
|
||||
- **Create:** `packages/db/package.json` (`@mcpedia/db`, deps `drizzle-orm`, `postgres`; dev `drizzle-kit`), `packages/db/src/schema.ts`:
|
||||
```ts
|
||||
import { sql } from "drizzle-orm";
|
||||
import { text, timestamp, integer, index, customType } from "drizzle-orm/pg-core";
|
||||
|
||||
// tsvector isn't a first-class drizzle type; wrap raw.
|
||||
export const documents = pgTable("documents", {
|
||||
id: text("id").primaryKey(), // slug
|
||||
slug: text("slug").notNull().unique(),
|
||||
title: text("title").notNull(),
|
||||
type: text("type").notNull(),
|
||||
section: text("section").notNull(),
|
||||
status: text("status").notNull().default("published"),
|
||||
author: text("author").notNull().default(""),
|
||||
tags: text("tags").array().notNull().default(sql`'{}'`),
|
||||
path: text("path").notNull(),
|
||||
body: text("body").notNull().default(""),
|
||||
searchVector: customType<{ data: string }>({
|
||||
dataType() { return "tsvector"; },
|
||||
})("search_vector")
|
||||
.generatedAlwaysAs(sql`to_tsvector('simple', coalesce(title,'') || ' ' || coalesce(body,''))`),
|
||||
createdAt: timestamp("created_at", { withTimezone: true }).notNull(),
|
||||
updatedAt: timestamp("updated_at", { withTimezone: true }).notNull(),
|
||||
}, (t) => ({
|
||||
searchIdx: index("documents_search_idx").using("gin", t.searchVector),
|
||||
}));
|
||||
```
|
||||
NOTE: `document_chunks` + `embedding vector` are **Phase 2** — do NOT add now.
|
||||
- **Create:** `packages/db/src/client.ts` — postgres-js with `prepare: false` (PgBouncer safe):
|
||||
```ts
|
||||
import { drizzle } from "drizzle-orm/postgres-js";
|
||||
import postgres from "postgres";
|
||||
import { DATABASE_URL } from "@mcpedia/config";
|
||||
import * as schema from "./schema";
|
||||
|
||||
const client = postgres(DATABASE_URL, { prepare: false, max: 5 });
|
||||
export const db = drizzle(client, { schema });
|
||||
export { schema };
|
||||
```
|
||||
- **Create:** `packages/db/drizzle.config.ts` (dialect `postgresql`, schema `./src/schema.ts`, out `./drizzle`).
|
||||
- **Verify:** `bunx drizzle-kit generate` produces a migration; `bunx drizzle-kit push` applies it to imrnes (operator supplies DATABASE_URL). `bunx drizzle-kit studio` optional.
|
||||
|
||||
### Task 1.4: `packages/parser` (frontmatter)
|
||||
- **Create:** `packages/parser/package.json` (`@mcpedia/parser`, dep `gray-matter`), `packages/parser/src/index.ts`:
|
||||
```ts
|
||||
import matter from "gray-matter";
|
||||
import { readFileSync } from "node:fs";
|
||||
import type { DocSection, DocType, DocStatus, DocumentMeta } from "@mcpedia/types";
|
||||
|
||||
const SECTIONS: DocSection[] = ["docs", "writeups", "research", "notes"];
|
||||
|
||||
export function parseFile(absPath: string, relPath: string): { meta: DocumentMeta; body: string } {
|
||||
const raw = readFileSync(absPath, "utf8");
|
||||
const { data, content } = matter(raw);
|
||||
const section = SECTIONS.find((s) => relPath.startsWith(s + "/")) ?? "docs";
|
||||
const slug = relPath.replace(/\.mdx?$/, "");
|
||||
const meta: DocumentMeta = {
|
||||
id: slug,
|
||||
slug,
|
||||
title: data.title ?? slug,
|
||||
type: (data.type ?? "documentation") as DocType,
|
||||
section,
|
||||
status: (data.status ?? "published") as DocStatus,
|
||||
author: data.author ?? "",
|
||||
tags: Array.isArray(data.tags) ? dataAtags : [],
|
||||
path: relPath,
|
||||
createdAt: data.created_at ?? new Date().toISOString(),
|
||||
updatedAt: data.updated_at ?? data.created_at ?? new Date().toISOString(),
|
||||
};
|
||||
return { meta, body: content };
|
||||
}
|
||||
```
|
||||
(fix the obvious `dataAtags` typo to `data.tags` when implementing.)
|
||||
- **Verify:** unit test parsing a sample frontmatter → asserts `slug`, `section`, `tags`.
|
||||
|
||||
### Task 1.5: `packages/search` (FTS query)
|
||||
- **Create:** `packages/search/package.json` (`@mcpedia/search`, dep `@mcpedia/db`, `@mcpedia/types`), `packages/search/src/index.ts`:
|
||||
```ts
|
||||
import { db } from "@mcpedia/db";
|
||||
import { documents } from "@mcpedia/db/schema";
|
||||
import { sql, and, eq } from "drizzle-orm";
|
||||
import type { SearchHit } from "@mcpedia/types";
|
||||
|
||||
export function toTsQuery(q: string): string {
|
||||
// plainto_tsquery-style: AND the terms, escape punctuation.
|
||||
const terms = q.trim().split(/\s+/).filter(Boolean).map((t) => t.replace(/[^\p{L}\p{N}]/gu, ""));
|
||||
return terms.map((t) => `${t}:*`).join(" & ");
|
||||
}
|
||||
|
||||
export async function keywordSearch(q: string, limit = 20): Promise<SearchHit[]> {
|
||||
const query = toTsQuery(q);
|
||||
if (!query) return [];
|
||||
const rows = await db
|
||||
.select({
|
||||
doc: documents,
|
||||
rank: sql<number>`ts_rank(${documents.searchVector}, to_tsquery('simple', ${query}))`,
|
||||
snippet: sql<string>`ts_headline('simple', ${documents.body}, to_tsquery('simple', ${query}), 'MaxWords=20, MinWords=5')`,
|
||||
})
|
||||
.from(documents)
|
||||
.where(and(eq(documents.status, "published"), sql`${documents.searchVector} @@ to_tsquery('simple', ${query})`))
|
||||
.orderBy(sql`ts_rank(${documents.searchVector}, to_tsquery('simple', ${query})) desc`)
|
||||
.limit(limit);
|
||||
return rows.map((r) => ({ doc: r.doc, rank: r.rank, snippet: r.snippet }));
|
||||
}
|
||||
```
|
||||
- **Verify:** after seeding, `keywordSearch("websocket")` returns hits.
|
||||
|
||||
### Task 1.6: `packages/core` (services)
|
||||
- **Create:** `packages/core/package.json` (`@mcpedia/core`, deps `@mcpedia/db`, `@mcpedia/parser`, `@mcpedia/search`, `@mcpedia/config`, `@mcpedia/types`), `packages/core/src/`:
|
||||
- `content.service.ts` — read markdown from `CONTENT_ROOT` by slug/section; list files via `fast-glob`.
|
||||
- `document.service.ts` — `listDocuments(section?, type?, status?)`, `getDocument(slug)` (DB meta + disk body), `getRelated(slug)` (shared tags, later vector).
|
||||
- `search.service.ts` — wraps `keywordSearch`.
|
||||
- `index.ts` — re-export services + types.
|
||||
```ts
|
||||
// document.service.ts (sketch)
|
||||
import { db } from "@mcpedia/db";
|
||||
import { documents } from "@mcpedia/db/schema";
|
||||
import { eq, and } from "drizzle-orm";
|
||||
import { CONTENT_ROOT } from "@mcpedia/config";
|
||||
import { readFileSync, existsSync } from "node:fs";
|
||||
import { resolve } from "node:path";
|
||||
import type { Document, DocumentMeta } from "@mcpedia/types";
|
||||
|
||||
export async function listDocuments(opts: { section?: string; status?: string } = {}): Promise<DocumentMeta[]> {
|
||||
const where = [eq(documents.status, opts.status ?? "published")];
|
||||
if (opts.section) where.push(eq(documents.section, opts.section));
|
||||
return db.select().from(documents).where(and(...where));
|
||||
}
|
||||
|
||||
export async function getDocument(slug: string): Promise<Document | null> {
|
||||
const [meta] = await db.select().from(documents).where(eq(documents.slug, slug));
|
||||
if (!meta) return null;
|
||||
const abs = resolve(CONTENT_ROOT, meta.path);
|
||||
const body = existsSync(abs) ? readFileSync(abs, "utf8") : meta.body;
|
||||
return { ...meta, body };
|
||||
}
|
||||
```
|
||||
- **Verify:** `bunx tsc --noEmit` passes across packages.
|
||||
|
||||
### Task 1.7: `scripts/indexer.ts` (walk content → upsert)
|
||||
- **Create:** `scripts/indexer.ts` — walk `CONTENT_ROOT` (`**/*.{md,mdx}`), `parseFile`, upsert into `documents` (slug PK; `onConflictDoUpdate` title/tags/body/updatedAt). Logs count.
|
||||
```ts
|
||||
import { db } from "@mcpedia/db";
|
||||
import { documents } from "@mcpedia/db/schema";
|
||||
import { parseFile } from "@mcpedia/parser";
|
||||
import { CONTENT_ROOT } from "@mcpedia/config";
|
||||
import { walk } from "fast-glob";
|
||||
import { resolve } from "node:path";
|
||||
|
||||
const files = await glob("**/*.{md,mdx}", { cwd: CONTENT_ROOT, absolute: true });
|
||||
let n = 0;
|
||||
for (const abs of files) {
|
||||
const rel = resolve(CONTENT_ROOT, abs).replace(CONTENT_ROOT + "/", "");
|
||||
const { meta, body } = parseFile(abs, rel);
|
||||
await db.insert(documents).values({ ...meta, body })
|
||||
.onConflictDoUpdate({ target: documents.slug, set: { title: meta.title, tags: meta.tags, body, updatedAt: new Date().toISOString() } });
|
||||
n++;
|
||||
}
|
||||
console.log(`indexed ${n} documents`);
|
||||
```
|
||||
- **Verify:** `bun run index` prints `indexed N documents`; `psql`/`drizzle-kit studio` shows rows + non-empty `search_vector`.
|
||||
|
||||
### Task 1.8: Seed content
|
||||
- **Create:** `content/docs/websocket/contract.md`, `content/writeups/debugging/websocket-timeout.md`, `content/research/mcp/architecture.md`, `content/notes/typescript/patterns.md` with the spec's frontmatter shape (id/title/type/tags/status/author/created_at/updated_at). Add 2–3 short paragraphs each so FTS has text.
|
||||
- **Verify:** `bun run index` ingests them; `keywordSearch("websocket")` returns ≥2 hits.
|
||||
|
||||
### Task 1.9: `apps/web` — pages (Next 16, read `node_modules/next/dist/docs` first)
|
||||
- **Read** Node_modules Next 16 docs (app router, server components, route handlers) — mandatory.
|
||||
- **Create:**
|
||||
- `apps/web/app/page.tsx` — home: list sections + recent docs (server component calling `listDocuments`).
|
||||
- `apps/web/app/(docs)/[...slug]/page.tsx` — document view: `getDocument(slug)`, render markdown (Phase 1: simple `dangerouslySetInnerHTML` of a remark→HTML, or raw pre with Shiki later; keep minimal but legible).
|
||||
- `apps/web/app/search/page.tsx` — search box → `keywordSearch`, render hits + snippets.
|
||||
- Keep Tailwind v4 styling minimal but clean (no shadcn yet — Phase 2 polish).
|
||||
- **Verify:** `bun run build` (apps/web) passes; `bun run dev` then `curl -s localhost:3000/` and `/search?q=websocket` return rendered HTML.
|
||||
|
||||
### Task 1.10: `apps/mcp` — MCP server
|
||||
- **Create:** `apps/mcp/package.json` (`@mcpedia/mcp`, deps `@modelcontextprotocol/sdk`, `zod`, `@mcpedia/core`), `apps/mcp/src/index.ts`:
|
||||
```ts
|
||||
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
|
||||
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
|
||||
import { z } from "zod";
|
||||
import { listDocuments, getDocument } from "@mcpedia/core";
|
||||
import { keywordSearch } from "@mcpedia/search";
|
||||
|
||||
const server = new McpServer({ name: "mcpedia", version: "0.1.0" });
|
||||
|
||||
server.tool("search_documents", { query: z.string(), limit: z.number().optional() },
|
||||
async ({ query, limit }) => {
|
||||
const hits = await keywordSearch(query, limit ?? 20);
|
||||
return { content: [{ type: "text", text: JSON.stringify(hits, null, 2) }] };
|
||||
});
|
||||
|
||||
server.tool("get_document", { slug: z.string() },
|
||||
async ({ slug }) => {
|
||||
const doc = await getDocument(slug);
|
||||
return doc ? { content: [{ type: "text", text: doc.body }] }
|
||||
: { content: [{ type: "text", text: "not found" }], isError: true };
|
||||
});
|
||||
|
||||
server.tool("list_documents", { section: z.string().optional() },
|
||||
async ({ section }) => {
|
||||
const docs = await listDocuments({ section });
|
||||
return { content: [{ type: "text", text: JSON.stringify(docs, null, 2) }] };
|
||||
});
|
||||
|
||||
// Resource: mcpedia://<section>/<path>
|
||||
server.resource("doc", "mcpedia://", async (uri) => {
|
||||
const slug = uri.path.replace(/^\//, "");
|
||||
const doc = await getDocument(slug);
|
||||
return { contents: [{ uri: uri.href, text: doc?.body ?? "" }] };
|
||||
});
|
||||
|
||||
const transport = new StdioServerTransport();
|
||||
await server.connect(transport);
|
||||
```
|
||||
(Exact SDK method names verified via Context7 before coding.)
|
||||
- **Create:** `apps/mcp/bin` or `bun run` entry; add `"start": "bun run src/index.ts"` to its package.json.
|
||||
- **Verify:** smoke test via `scripts/mcp-smoke.ts` that imports the tool handlers and calls `search_documents({query:"websocket"})` → asserts ≥1 hit. (Running a real stdio client in CI is optional; inspector `@modelcontextprotocol/inspector` for manual check.)
|
||||
|
||||
### Task 1.11: Cross-package verification
|
||||
- **Run:** `bun install` (root) → `bunx turbo run lint` → `bunx turbo run build` → `bunx turbo run typecheck`. All green.
|
||||
- **Run:** `bun run index` then `bun run scripts/mcp-smoke.ts`.
|
||||
- **Verify:** no package imports `pg`/`drizzle` outside `@mcpedia/core` or `@mcpedia/db` (architecture guard). Search the tree: `search_files("from \"pg\"", path="apps")`.
|
||||
|
||||
### Task 1.12: README + phase markers
|
||||
- **Modify:** `README.md` — replace create-next-app text with MCPedia overview, monorepo map, "Phase 1 status", how to dev (`bun install && bun run index && bun run dev`), how to run MCP (`bun run --cwd apps/mcp start`).
|
||||
- **Verify:** `cat README.md` reads correctly.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — Semantic search + tRPC/Hono API + auth (outline, expand later)
|
||||
|
||||
- Add `document_chunks` table + `embedding vector` (pgvector extension on imrnes — operator enables `CREATE EXTENSION vector`).
|
||||
- `packages/search`: `EmbeddingProvider` abstraction (OpenAI/Gemini/Ollama/Local) behind an interface (spec principle 6). Chunking in `packages/parser` (by heading/section).
|
||||
- Hybrid ranking: fuse FTS `ts_rank` + cosine distance (Reciprocal Rank Fusion).
|
||||
- `apps/api`: Hono + tRPC router exposing `documents.*`, `search.*`; Web switches from direct Core calls to tRPC client; MCP may also call API or keep Core (decide: keep Core for colocated simplicity, API for external).
|
||||
- `packages/auth`: Auth.js/OIDC adapter; `permissions` table; gate write/mutations. Phase 1 documents are public-read.
|
||||
- Categories + tags linking tables (`categories`, `document_tags`, `references`).
|
||||
- MDX rendering with Shiki + shadcn/ui polish on Web.
|
||||
|
||||
## Phase 3 — Async indexing + revisions (outline)
|
||||
|
||||
- Redis (imrnes or local) + BullMQ workers: parse/chunk/embed/reindex off the request path.
|
||||
- `document_revisions` table + revision API; Git sync hook (content/ push → reindex).
|
||||
- Indexer becomes a worker triggered by file watch / webhook, not a manual script.
|
||||
|
||||
## Phase 4 — Scale (outline, only if needed)
|
||||
|
||||
- Dedicated search engine (OpenSearch) behind the same `SearchService` seam — YAGNI until FTS+pgvector saturates.
|
||||
- Object storage for assets, advanced ranking, distributed workers, observability, multi-tenant. Docker Compose for service orchestration pointing at imrnes Postgres.
|
||||
|
||||
---
|
||||
|
||||
## Risks / tradeoffs / open questions
|
||||
|
||||
- **Next 16 breaking changes** — biggest unknown; mitigated by reading `node_modules/next/dist/docs` before Web tasks. If app-router differs enough, adjust Task 1.9 accordingly.
|
||||
- **PgBouncer + prepared statements** — handled via `prepare: false`; if imrnes uses session pool mode, can drop it.
|
||||
- **FTS language** — using `'simple'` config (no stemming) so mixed ID/EN queries match literally; revisit if relevance is poor (could add `'english'` or a custom ID dict).
|
||||
- **pgvector on imrnes** — extension may need operator enablement; Phase 2 blocker, not Phase 1.
|
||||
- **MCP resource URI shape** — `mcpedia://<section>/<path>` is a sketch; finalize against MCP SDK spec in Task 1.10 (Context7).
|
||||
- **Monorepo driver** — bun workspaces chosen over pnpm per user; Turborepo runs fine under bun. If `turbo` bin resolution misbehaves under bun, fall back to `bunx turbo`.
|
||||
|
||||
## Verification matrix (Phase 1)
|
||||
|
||||
| Check | Command | Expect |
|
||||
| --- | --- | --- |
|
||||
| Install | `bun install` | clean |
|
||||
| Lint | `bunx turbo run lint` | 0 errors |
|
||||
| Typecheck | `bunx turbo run typecheck` | 0 errors |
|
||||
| Build | `bunx turbo run build` | web + mcp build |
|
||||
| Index | `bun run index` | `indexed N documents` |
|
||||
| Web | `bun run dev` (apps/web) + curl `/` and `/search?q=websocket` | rendered HTML, hits |
|
||||
| MCP | `bun run scripts/mcp-smoke.ts` | ≥1 hit for "websocket" |
|
||||
| Arch guard | search for `from "pg"` outside `packages/db`,`packages/core` | none |
|
||||
@@ -1,161 +0,0 @@
|
||||
# MCPedia Phase 2 — Semantic Search + tRPC/Hono API
|
||||
|
||||
> **For Hermes:** implement task-by-task. Spec-first (user rule 2026-08-19).
|
||||
|
||||
**Goal:** Add semantic + hybrid search (pgvector) and a typed tRPC/Hono API so
|
||||
MCPedia is queryable by embeddings, not just keyword FTS — and expose the
|
||||
corpus over a programmatic HTTP API.
|
||||
|
||||
**Architecture:** Content (Markdown) → chunk → embed (OpenRouter) → store
|
||||
`document_chunks` with `vector(N)` in Postgres → `semanticSearch` (cosine) and
|
||||
`hybridSearch` (FTS + cosine, reciprocal-rank fusion) in `@mcpedia/search` →
|
||||
exposed via Core, the MCP server (new tools), and a new `apps/api` (Hono +
|
||||
tRPC v11).
|
||||
|
||||
**Embedding provider:** OpenRouter (`openrouter/llama-nemotron-embed-vl-1b-v2:free`)
|
||||
via `9router_ai_llm_api_key` + `9router_ai_llm_base_url` (BWS). Dimension is
|
||||
discovered at first live call (see Step 1.3) and pinned in schema/migration.
|
||||
|
||||
**Tech stack:** drizzle-orm `vector` column + pgvector extension, HNSW index,
|
||||
`@trpc/server` v11 (fetch adapter), `hono` + `@hono/node-server`.
|
||||
|
||||
---
|
||||
|
||||
## Task P2.1 — `packages/embeddings` (provider + abstraction)
|
||||
|
||||
**Files:** `packages/embeddings/package.json`, `src/index.ts`, `src/provider.ts`,
|
||||
`src/openrouter.ts`
|
||||
|
||||
- `EmbeddingProvider` interface: `embed(texts: string[]): Promise<number[][]>`, `readonly model`, `readonly dimensions`.
|
||||
- `OpenRouterEmbeddingProvider`: POST `${baseUrl}/embeddings` with `{ model, input }`,
|
||||
`Authorization: Bearer ${key}`. Returns `data[].embedding`. Validate length === dimensions.
|
||||
- Read `EMBED_BASE_URL`, `EMBED_API_KEY`, `EMBED_MODEL` from `@mcpedia/config`
|
||||
(with `.env` fallback). Dimensions discovered live (Step 1.3) → export `EMBED_DIM`.
|
||||
- Chunk helper `chunkText(text, { size=1000, overlap=150 })` in `src/chunk.ts`.
|
||||
|
||||
**Step 1.3 (discover dim):** live call `embed(["test"])`, read `embedding.length`,
|
||||
pin `EMBED_DIM`, assert mismatch throws.
|
||||
|
||||
**Verify:** `bun run` a temp script: `embed(["hello world"])` prints a vector of
|
||||
length N (e.g. 1024). Confirm no key is logged.
|
||||
|
||||
---
|
||||
|
||||
## Task P2.2 — Schema: `document_chunks` + vector extension
|
||||
|
||||
**Files:** `packages/db/src/schema.ts` (add), `packages/db/drizzle.config.ts`
|
||||
(unchanged), new migration.
|
||||
|
||||
- `CREATE EXTENSION IF NOT EXISTS vector;` (idempotent; run once via psql).
|
||||
- `document_chunks` table:
|
||||
- `id` uuid pk default gen_random_uuid()
|
||||
- `document_id` text → `documents.id` on delete cascade
|
||||
- `slug` text (denormalized for convenience)
|
||||
- `chunk_index` integer
|
||||
- `content` text
|
||||
- `embedding` vector(EMBED_DIM)
|
||||
- `created_at` timestamp default now()
|
||||
- index `chunk_embedding_idx` using hnsw (`embedding` op `vector_cosine_ops`)
|
||||
- Generate migration with `drizzle-kit generate`, apply via `psql` (drizzle-kit
|
||||
push is unreliable here — known).
|
||||
|
||||
**Verify:** `\d document_chunks` shows `embedding vector(N)` + HNSW index;
|
||||
`select count(*) from document_chunks` = 0.
|
||||
|
||||
---
|
||||
|
||||
## Task P2.3 — Indexer: chunk + embed + upsert
|
||||
|
||||
**Files:** `scripts/indexer.ts` (extend), `packages/core/src/document.service.ts`
|
||||
(add `indexChunks`).
|
||||
|
||||
- For each published doc: read body (already on disk), `chunkText`, `embed` in
|
||||
batches (≤ 16), delete existing chunks for slug, insert new rows.
|
||||
- Guard: if embedding provider fails, log + skip (don't crash the whole index).
|
||||
- Add `bun run index:embed` (or extend `bun run index` to also embed).
|
||||
|
||||
**Verify:** after running, `select count(*) from document_chunks` > 0; a sample
|
||||
row has non-null `embedding`.
|
||||
|
||||
---
|
||||
|
||||
## Task P2.4 — `packages/search`: semantic + hybrid
|
||||
|
||||
**Files:** `packages/search/src/index.ts` (add `semanticSearch`, `hybridSearch`).
|
||||
|
||||
- `semanticSearch(vec, limit)`: order by `embedding <=> ${vec}` asc, filter published.
|
||||
- `hybridSearch(q, limit)`: run FTS (`ts_rank`) + semantic (cosine) in parallel;
|
||||
fuse with reciprocal-rank (RRF: score = 1/(k+rank), k=60); return merged hits.
|
||||
- Keep `keywordSearch` unchanged (Phase 1).
|
||||
|
||||
**Verify:** unit-ish script: embed a query, `semanticSearch` returns relevant
|
||||
chunks; `hybridSearch("websocket")` returns ≥ keyword results.
|
||||
|
||||
---
|
||||
|
||||
## Task P2.5 — `packages/core` expose semantic/hybrid
|
||||
|
||||
**Files:** `packages/core/src/search.service.ts`, `index.ts`.
|
||||
|
||||
- Re-export `semanticSearch`, `hybridSearch` from Core.
|
||||
|
||||
---
|
||||
|
||||
## Task P2.6 — `apps/api` (Hono + tRPC v11)
|
||||
|
||||
**Files:** `apps/api/package.json`, `tsconfig.json`, `src/index.ts`,
|
||||
`src/router.ts`, `src/trpc.ts`.
|
||||
|
||||
- `initTRPC.create()` router with procedures: `search`, `semanticSearch`,
|
||||
`hybridSearch`, `getDocument`, `listDocuments` (mirrors MCP tools).
|
||||
- Mount `fetchRequestHandler` on a Hono app at `/trpc/*`; serve via
|
||||
`@hono/node-server` `serve({ fetch: app.fetch, port: 4020 })`.
|
||||
- `createContext` returns `{ db }`.
|
||||
|
||||
**Verify:** `bun run dev` → `curl -X POST localhost:4020/trpc/search`
|
||||
with JSON body returns hits.
|
||||
|
||||
---
|
||||
|
||||
## Task P2.7 — MCP server: semantic + hybrid tools
|
||||
|
||||
**Files:** `apps/mcp/src/index.ts` (add `semantic_search`, `hybrid_search`),
|
||||
extend `smoke.test.ts`.
|
||||
|
||||
- `semantic_search`: embed query → `semanticSearch`.
|
||||
- `hybrid_search`: embed query → `hybridSearch`.
|
||||
- Smoke: assert both return ≥1 hit for "websocket".
|
||||
|
||||
---
|
||||
|
||||
## Task P2.8 — Web: semantic toggle on search
|
||||
|
||||
**Files:** `apps/web/app/search/page.tsx`.
|
||||
|
||||
- Add `mode=keyword|hybrid` query param; server component calls Core
|
||||
`hybridSearch` when `mode=hybrid`. Minimal UI toggle (link/buttons).
|
||||
- Keep keyword as default.
|
||||
|
||||
**Verify:** `bun run build`; `curl '/search?q=websocket&mode=hybrid'` returns hits.
|
||||
|
||||
---
|
||||
|
||||
## Task P2.9 — Verify all + commit
|
||||
|
||||
- `bunx turbo run build` (web + api + mcp), `bun run apps/mcp smoke`,
|
||||
live API curl, live web hybrid search.
|
||||
- Update `README.md` + `PHASES.md` (mark Phase 2 ✅).
|
||||
- `git add -A` (exclude `.env`), commit as asepharyana (no Co-Authored-By).
|
||||
|
||||
---
|
||||
|
||||
## Risks / decisions
|
||||
- **Dimension unknown until live call** → P2.1.3 discovers it; pinned EMBED_DIM=2048.
|
||||
- **pgvector NOT available on shared imrnes Postgres** (extension not installed;
|
||||
installing needs host-level apt on a managed/shared DB — deferred). PIVOT:
|
||||
store `embedding` as `real[]` and compute cosine similarity in the app layer.
|
||||
Brute-force cosine is instant for a KB-sized corpus (dozens of docs / hundreds
|
||||
of chunks). pgvector+HNSW is the Phase-4 scale-out path.
|
||||
- **PgBouncer + real[]**: fine; simple queries, no extension needed.
|
||||
- **API port 4020** (host 4000s range is 4000–4015; 4020 is free for dev). Deploy later.
|
||||
- **YAGNI**: no auth/revisions this phase (Phase 3).
|
||||
@@ -1,151 +0,0 @@
|
||||
# Phase 14 — Hierarchical Folder Structure
|
||||
|
||||
> User: "gk ada bedanya, maksud saya inginnya itu bisa yg bertingkat seperti github yg memiliki folder dalam folder"
|
||||
> Context: after the full dynamic-custom-fields overhaul (Phase 13), the user
|
||||
> wants document URLs/content organized in **nested folders** like GitHub —
|
||||
> `writeups/ctf/defcon-quals-2024/pwn-100/...` with subfolders under subfolders,
|
||||
> not just one level deep.
|
||||
|
||||
## Problem
|
||||
|
||||
The current URL scheme is `/<section>/<slug>` where `slug` can contain `/`
|
||||
(e.g. `writeups/ctf/defcon-quals-2024/pwn-100-ret2win-alignment` →
|
||||
`/writeups/ctf/defcon-quals-2024/pwn-100-ret2win-alignment`). This works for
|
||||
**files** but there are no **folder-level index pages** — navigating to
|
||||
`/writeups/ctf/defcon-quals-2024/` returns 404 because Next.js catch-all
|
||||
`[section]/[...slug]/page.tsx` requires at least one slug segment beyond the
|
||||
section, and the sidebar only shows flat doc titles (no folder tree).
|
||||
|
||||
GitHub's model: `github.com/org/repo/tree/main/path/to/folder/file` — every
|
||||
folder has an index page (`/path/to/folder/`) listing its contents.
|
||||
|
||||
## Solution
|
||||
|
||||
### 1. Folder Index Pages
|
||||
|
||||
**Create `apps/web/app/[section]/[...slug]/folder.tsx`** (or a parallel route).
|
||||
Actually — cleaner approach per Next.js App Router: the catch-all
|
||||
`[section]/[...slug]/page.tsx` handles both. Add logic: if the slug resolves to
|
||||
an actual markdown file → doc page (existing behavior). If the slug resolves to
|
||||
a **directory** (folder of docs) → render a folder index listing all docs whose
|
||||
`path` starts with that prefix.
|
||||
|
||||
**Mechanism:**
|
||||
- Call `listDocuments()` to get all docs.
|
||||
- The incoming URL path is `{section}/{...slug}`.
|
||||
- Build the "folder prefix" = `${section}/${slug.join("/")}/` (with trailing `/`,
|
||||
or just `${section}/${slug.join("/")}` if no slug segments).
|
||||
- Filter docs whose `doc.path` starts with that prefix.
|
||||
- If exactly one doc matches AND its path === prefix (trimmed .md) → it's a
|
||||
doc page (existing). If zero or multiple match and they all start with the
|
||||
prefix → it's a folder index.
|
||||
- Edge: a folder with exactly one doc whose path matches exactly — still a doc
|
||||
page. A folder is when there are docs at `prefix/sub/...`.
|
||||
|
||||
**Better heuristic:** A slug path is a "folder" if there exist docs whose `path`
|
||||
is `prefix/deep/...` (i.e., the slug is a parent of other doc paths, not a
|
||||
leaf itself). A slug is a "leaf doc" if `path === prefix + ".md"`.
|
||||
|
||||
### 2. Sidebar Tree
|
||||
|
||||
**Update `Sidebar.tsx`:**
|
||||
- `listDocuments()` already returns all docs with their full `slug` and `path`.
|
||||
- Build a **tree** from the flat list: split each slug by `/`, create nested
|
||||
folder nodes.
|
||||
- Render nested `<ul>` with indentation (already done via `marginLeft` based on
|
||||
depth).
|
||||
- **Folder nodes** (collapsed/expanded) get a folder icon 📁 and a CSS class.
|
||||
- Clicking a folder → navigates to the folder index page `/{section}/{path}`.
|
||||
- **Leaf doc nodes** → link to `/{doc.slug}` (existing behavior).
|
||||
- Group by the first segment after section too (e.g. `ctf/defcon-quals-2024/`
|
||||
is a folder, then `pwn-100-...` are children).
|
||||
|
||||
Tree-building algorithm (from flat slugs):
|
||||
```
|
||||
For slug "writeups/ctf/defcon-quals-2024/pwn-100-ret2win-alignment":
|
||||
parts = ["writeups", "ctf", "defcon-quals-2024", "pwn-100-ret2win-alignment"]
|
||||
→ tree: writeups → ctf → defcon-quals-2024 → pwn-100-ret2win-alignment (leaf)
|
||||
```
|
||||
|
||||
### 3. DocForm / Folder Selection
|
||||
|
||||
**Update `DocForm.tsx`:**
|
||||
- Add a "Parent folder" input (autocomplete or text) that shows existing folders
|
||||
for the selected section. The slug field already supports `/` but the user
|
||||
experience is better with folder picker.
|
||||
- When creating, the slug becomes `{parentFolder}/{slug}` automatically.
|
||||
- Show existing folder structure as `<select>` or tree picker.
|
||||
|
||||
### 4. Example Hierarchy
|
||||
|
||||
Create a real hierarchical structure to demonstrate:
|
||||
```
|
||||
writeups/
|
||||
ctf/
|
||||
defcon-quals-2024/
|
||||
pwn/
|
||||
pwn-100-ret2win-alignment.md
|
||||
pwn-200-bof-heap.md
|
||||
crypto/
|
||||
crypto-100-xor.md
|
||||
crypto-200-rsa.md
|
||||
template/
|
||||
writeup-template.md
|
||||
_index.md ← folder index (optional intro)
|
||||
hackthebox/
|
||||
machine-name/
|
||||
walkthrough.md
|
||||
```
|
||||
|
||||
For now, reorganize the existing Defcon writeup into proper subfolders + add a
|
||||
folder index page. The existing `content/writeups/ctf/defcon-quals-2024/` is
|
||||
already a folder — just need the folder index route to work.
|
||||
|
||||
### 5. Route Changes
|
||||
|
||||
**Current:** `[section]/[...slug]/page.tsx` — catch-all requires ≥1 slug segment.
|
||||
- `/writeups` → `notFound()` (no index for bare section unless we add one)
|
||||
- `/writeups/ctf` → catch-all gets `slug=["ctf"]` → currently treated as a doc
|
||||
(looks up `writeups/ctf` doc, 404 if none)
|
||||
- `/writeups/ctf/defcon-quals-2024/` → catch-all `slug=["ctf","defcon-quals-2024"]`
|
||||
|
||||
**Plan:**
|
||||
1. Add `/writeups/page.tsx` (section index) — lists top-level folders + root
|
||||
docs in that section. (Currently `/docs/page.tsx` exists but `/writeups/page.tsx`
|
||||
doesn't.)
|
||||
2. In `[section]/[...slug]/page.tsx`: at the top of the page component, check if
|
||||
the slug path is a folder (has child docs). If so, render folder index instead
|
||||
of doc page.
|
||||
|
||||
**Section index pages:** Create `[section]/page.tsx` for all 4 sections, or a
|
||||
generic one. Currently only `/docs/page.tsx` exists. Add a shared
|
||||
`SectionIndex` component.
|
||||
|
||||
### 6. Files to Change
|
||||
|
||||
```
|
||||
new: apps/web/app/[section]/page.tsx # generic section index (folder + doc listing)
|
||||
mod: apps/web/app/[section]/[...slug]/page.tsx # add folder-index detection
|
||||
mod: apps/web/app/components/Sidebar.tsx # tree from flat slugs
|
||||
mod: apps/web/app/components/DocForm.tsx # parent folder picker
|
||||
new: content/writeups/ctf/defcon-quals-2024/_index.md # folder intro (optional)
|
||||
new: content/writeups/ctf/_index.md # CTF section intro
|
||||
mod: apps/web/app/docs/page.tsx # may need generic version
|
||||
```
|
||||
|
||||
### 7. Verification
|
||||
|
||||
- `/writeups` → 200, shows folders: ctf/, template/
|
||||
- `/writeups/ctf` → 200, folder index listing defcon-quals-2024/
|
||||
- `/writeups/ctf/defcon-quals-2024` → 200, folder index listing pwn-100-...
|
||||
- `/writeups/ctf/defcon-quals-2024/pwn-100-ret2win-alignment` → 200, doc page
|
||||
- Sidebar shows nested tree with folder icons
|
||||
- DocForm parent folder picker works
|
||||
- `bun run test` green, `turbo run typecheck` green
|
||||
- Live verification via curl
|
||||
|
||||
## Constraints
|
||||
- User: "pastikan semua dinamis dan rapih untuk banyak situasi jadi tergantung
|
||||
user bukan hardcode" — folder detection must be content-driven, not config.
|
||||
- No breaking existing flat slugs.
|
||||
- Section icons/labels stay the same.
|
||||
@@ -1,42 +0,0 @@
|
||||
# MCP HTTP transport + deploy + review actions
|
||||
|
||||
## Goal
|
||||
Make the MCPedia MCP server reachable over the network (not just stdio subprocess), so
|
||||
remote MCP clients (Claude, a Discord bot, a web client) can call its 6 tools + 4 resources.
|
||||
Serve via Streamable HTTP (MCP 2025-03-26 spec), deploy as a supervised systemd service,
|
||||
expose through Caddy on a dedicated subdomain.
|
||||
|
||||
## Design decisions
|
||||
- **StreamableHTTPServerTransport, stateless mode** (`sessionIdGenerator: undefined`).
|
||||
One McpServer + transport per request. No session map, no shared-transport connect race,
|
||||
no memory leak. Re-registering 6 tools + 4 resources per request is negligible for a KB.
|
||||
- **New entry `apps/mcp/src/http.ts`** served by Node `http` (built-in), NOT mounted on the
|
||||
API app — keeps the MCP app's zod-4 isolation intact (api app is zod 3).
|
||||
- **Port 4021** (next free in the 4000s range; 4020 is the API).
|
||||
- **Subdomain `mcp.asepharyana.my.id`** -> 4021 (Cloudflare `*` wildcard already proxies it;
|
||||
Caddy auto-issues LE cert, no extra DNS work).
|
||||
- **CORS** allow on `/mcp` (remote web clients need it).
|
||||
- MCP tools are read-only (search/get/list/related + read-only resources) => open MCP is
|
||||
low-risk. No auth on MCP itself.
|
||||
|
||||
## Files
|
||||
- `apps/mcp/src/http.ts` (NEW) — Node http server, `/mcp` route, stateless transport.
|
||||
- `apps/mcp/package.json` — add `serve:http` script.
|
||||
- `package.json` (root) — add `mcp:http` script (absolute bun path).
|
||||
- `deploy/mcpedia-mcp.service` (NEW) — systemd unit, MCP_PORT=4021.
|
||||
- `/etc/caddy/Caddyfile` — add `mcp.asepharyana.my.id { import proxy 4021 }`.
|
||||
|
||||
## Security finding (review, flagged not silently built)
|
||||
The tRPC `restoreRevision` mutation is exposed UNauthenticated at
|
||||
`https://wiki.asepharyana.my.id/trpc/restoreRevision` — anyone can revert a live doc.
|
||||
The web UI's restore path calls `@mcpedia/core` directly (server component), so the tRPC
|
||||
mutation is dead surface. Fix: guard the mutation with the existing WEBHOOK_SECRET header,
|
||||
or drop it from the router. Will apply the guard (consistent with /hooks auth) unless user
|
||||
prefers removal.
|
||||
|
||||
## Verification
|
||||
- `bun --cwd apps/mcp run typecheck` green.
|
||||
- Live: `curl -XPOST https://mcp.asepharyana.my.id/mcp` initialize -> 200 + serverInfo;
|
||||
tools/list -> 6 tools; resources/list -> 4 resources.
|
||||
- `systemctl is-active mcpedia-mcp` == active.
|
||||
- Commit + push.
|
||||
@@ -1,67 +0,0 @@
|
||||
# MCPedia Phase 11 — CRUD + Auth + Web UI
|
||||
|
||||
## STATUS: ✅ ALL DONE (committed f8b525d, CI+Deploy success)
|
||||
|
||||
## Plan (spec BEFORE implementation, per user rule)
|
||||
|
||||
### Scope: 4 areas
|
||||
1. **CRUD**: Create/Read/Update/Delete documents from web UI + MCP
|
||||
2. **Authentication**: MCP/API writes (x-webhook-secret), Web CRUD (cookie-based ADMIN_PASSWORD)
|
||||
3. **Web + Agent access**: Web forms + MCP write tools
|
||||
4. **UI/UX**: Edit forms, TOC, dark mode
|
||||
|
||||
### Requirements:
|
||||
1. Source of truth = filesystem (markdown files in content/{section}/{slug}.md)
|
||||
2. DB mirrors disk (documents, document_chunks, document_revisions tables)
|
||||
3. Single indexing path: indexContentFile in @mcpedia/core
|
||||
4. Auth: MCP/API writes use WEBHOOK_SECRET; Web uses ADMIN_PASSWORD cookie
|
||||
5. Slug rules: [a-z0-9][a-z0-9/_-]*, no //, no .. traversal
|
||||
6. No breaking existing features (32 original tests still green)
|
||||
7. UI/UX: edit button on doc pages, ?edit=1 inline form, /create page, login, TOC, dark mode
|
||||
|
||||
### Implementation
|
||||
|
||||
#### Backend
|
||||
- `packages/parser`: added `stringifyFile()` (writes markdown with frontmatter)
|
||||
- `packages/core`: `createDocument`, `updateDocument`, `deleteDocument` (file I/O + DB + revision + chunks)
|
||||
- `apps/api`: tRPC CRUD routers (`requireWriteAuth`), fixed `requireWriteAuth` env-constant bug (now uses `ctx.expectedSecret` injected from `createApp(deps)`)
|
||||
- `apps/mcp`: 3 new write tools (`create_document`, `update_document`, `delete_document`) gated by `x-webhook-secret`
|
||||
|
||||
#### Web UI
|
||||
- `apps/web/app/api/auth/login/route.ts`: POST login → verify ADMIN_PASSWORD, set `mcpedia_admin` cookie
|
||||
- `apps/web/app/api/docs/route.ts`: POST (create)
|
||||
- `apps/web/app/api/docs/[...slug]/route.ts`: PUT (update), DELETE (delete)
|
||||
- `apps/web/app/components/DocForm.tsx`: shared create/edit form
|
||||
- `apps/web/app/create/page.tsx`: create form
|
||||
- `apps/web/app/login/page.tsx`: login form
|
||||
- `apps/web/app/[section]/[...slug]/page.tsx`: `?edit=1` inline edit, TOC, dark mode, Edit button
|
||||
- `apps/web/app/docs/page.tsx`: docs index listing
|
||||
- `apps/web/app/components/TOC.tsx`: auto-generated TOC from h2/h3 headings
|
||||
- `apps/web/app/components/ThemeToggle.tsx`: dark mode toggle (localStorage + system default)
|
||||
|
||||
#### New deps (minimal — only for UX):
|
||||
- `rehype-slug` (heading anchors for TOC links)
|
||||
- `github-slugger` (matching slug algorithm for TOC client-side)
|
||||
|
||||
### Gotchas (learned the hard way)
|
||||
1. tRPC fetch adapter expects input directly as JSON body, NOT JSON-RPC envelope
|
||||
2. requireWriteAuth compared `ctx.webhookSecret !== WEBHOOK_SECRET` (module-level env constant) — untestable. Fixed: `ctx.webhookSecret !== ctx.expectedSecret` (injected per-app via deps).
|
||||
3. Next.js catch-all routes: `[...slug]/edit/` is INVALID (catch-all must be last). Used `?edit=1` query param instead.
|
||||
4. Next.js App Router: PUT/DELETE on `/api/docs/route.ts` doesn't match `/api/docs/{slug}` — need dynamic route `/api/docs/[...slug]/route.ts`.
|
||||
5. `@env.example` should be updated.
|
||||
6. `ADMIN_PASSWORD` must be set in VPS `.env` (deployed separately).
|
||||
|
||||
### Verification
|
||||
- Typecheck: ✅ 4/4 apps green
|
||||
- Tests: ✅ 40 tests green (32 original + 8 new), no DB/Redis
|
||||
- Build: ✅ web compiled
|
||||
- CI: ✅ success → Deploy: ✅ success
|
||||
- Live: all 9 endpoints 200, 13 MCP tools live, CRUD e2e verified (login → create → view → delete via cookie auth), MCP create_document verified via header auth
|
||||
- Test docs cleaned up (404 confirmed)
|
||||
|
||||
### Commits
|
||||
1. `57f9001` feat: Phase 11 — CRUD + auth + web UI
|
||||
2. `1dd16eb` feat(web): Phase 11 UI/UX — TOC, dark mode toggle, /docs index
|
||||
3. `98437cb` fix(web): /api/docs accepts cookie OR header (not both required)
|
||||
4. `8ed8c67` fix(web): split PUT/DELETE into /api/docs/[...slug]/route.ts
|
||||
5. `f8b525d` chore: remove test docs
|
||||
@@ -1,24 +0,0 @@
|
||||
# Phase 3 — Deploy + git-sync wiring (remaining work)
|
||||
|
||||
Status: Phase 3/4 code is DONE and e2e-verified (webhook enqueue -> worker drain, 0 failed).
|
||||
What was missing on the host: API + worker never ran as systemd services, and the GitHub
|
||||
push webhook was never created. Also a real integration bug: `assertWebhookAuth` only
|
||||
accepts a plain `x-webhook-secret` header, which GitHub does NOT send (GitHub delivers
|
||||
`X-Hub-Signature-256` = HMAC-SHA256 of raw body). So a real GitHub webhook would 401.
|
||||
|
||||
## Changes
|
||||
1. `apps/api/src/index.ts` — `assertWebhookAuth` now verifies GitHub `X-Hub-Signature-256`
|
||||
(HMAC-SHA256 of raw body w/ WEBHOOK_SECRET) and still accepts `x-webhook-secret` for manual tests.
|
||||
2. root `package.json` scripts — `api`: `bun --cwd apps/api run dev` -> `bun --cwd apps/api src/index.ts`
|
||||
(the `run dev` form errors in bun 1.3.14; direct-file form verified booting + health). `worker` -> same form for consistency.
|
||||
3. systemd — `cp deploy/*.service /etc/systemd/system`, `daemon-reload`, `enable --now mcpedia-api mcpedia-worker`.
|
||||
4. Caddy — expose `/hooks/*` on `wiki.asepharyana.my.id` -> :4020 (no new DNS). Keep web on :4016.
|
||||
5. GitHub webhook — `gh api repos/asepharyana/mcpedia/hooks` POST:
|
||||
`https://wiki.asepharyana.my.id/hooks/reindex`, content_type json, secret=WEBHOOK_SECRET, events=push.
|
||||
|
||||
## Verification
|
||||
- `systemctl is-active mcpedia-api mcpedia-worker` == active.
|
||||
- `curl /health` on :4020 -> ok.
|
||||
- `curl -X POST https://wiki.asepharyana.my.id/hooks/reindex -H "X-Hub-Signature-256: ..."` (or x-webhook-secret) -> 200 + jobId; worker drains.
|
||||
- `gh api .../hooks` lists the webhook.
|
||||
- `turbo run typecheck` green; commit + push.
|
||||
@@ -1,114 +0,0 @@
|
||||
# MCPedia — Phase 3 "Async + Scale" Implementation Plan
|
||||
|
||||
Status: Phase 1 (MVP) + Phase 2 (Semantic+API) DONE. Phase 3 adds async
|
||||
background work, git-driven reindex, document revision history, and MCP
|
||||
Resources. All logic stays in `@mcpedia/core`; new `packages/queue` wires
|
||||
BullMQ; `apps/worker` runs the worker process; the existing API gets a git-sync
|
||||
webhook + job-status procedures; the MCP server gains Resources.
|
||||
|
||||
## Scope (4 features from PHASES.md)
|
||||
|
||||
1. **Redis + BullMQ background indexing/embedding workers**
|
||||
2. **Git synchronization hook** (auto-reindex on push via webhook)
|
||||
3. **Document revision system** (`document_revisions`)
|
||||
4. **MCP Resources** (`mcpedia://docs/...`) alongside existing tools
|
||||
|
||||
## Architecture decisions (locked)
|
||||
|
||||
- **Redis**: shared imrnes Redis `100.121.180.82:6379`, no auth (verified
|
||||
`+PONG`). `REDIS_URL` env (default `redis://100.121.180.82:6379`), optional
|
||||
`REDIS_PASSWORD`. BullMQ key prefix `mcpedia:` to avoid collisions on the
|
||||
shared instance.
|
||||
- **Queue lib**: `bullmq@6.1.2` + `ioredis@6.0.0` (BullMQ peer dep). Pass an
|
||||
ioredis instance; BullMQ duplicates it for blocking commands.
|
||||
- **Single source of truth preserved**: per-doc indexing logic moves into
|
||||
`@mcpedia/core` as `indexContentFile(relPath, reason?)`. The script, the
|
||||
worker, and the git hook ALL call this. Revisions are snapshotted inside it.
|
||||
- **Revisions**: created only when body actually changes vs the latest revision
|
||||
(avoids bloat on every sync). Stored in `document_revisions`.
|
||||
|
||||
## Files touched
|
||||
|
||||
### packages/config
|
||||
- `src/index.ts`: add `REDIS_URL`, `REDIS_PASSWORD`, `QUEUE_PREFIX`.
|
||||
|
||||
### packages/db
|
||||
- `src/schema.ts`: add `documentRevisions` table
|
||||
(id, documentId→documents.id cascade, slug, revisionNo int, title, body,
|
||||
meta jsonb, reason text, createdAt). Index (document_id, revision_no DESC),
|
||||
(slug).
|
||||
- `drizzle/0002_document_revisions.sql`: migration (applied via psql).
|
||||
- `drizzle/meta/0002_snapshot.json` + `_journal.json` entry (keeps drizzle-kit
|
||||
consistent even though we apply manually).
|
||||
|
||||
### packages/core (new)
|
||||
- `src/index.service.ts`:
|
||||
- `indexContentFile(relPath: string, reason = "index")` — parse → upsert
|
||||
`documents` → `indexChunks` → snapshot revision (if changed).
|
||||
- `runFullIndex(reason?)` — walk content, index each, return counts.
|
||||
- `src/revision.service.ts`:
|
||||
- `createRevision(...)`, `listRevisions(slug, limit)`,
|
||||
`getRevision(id)`, `latestRevisionBody(slug)`, `restoreRevision(id)`.
|
||||
- `src/index.ts`: export both.
|
||||
|
||||
### packages/queue (NEW)
|
||||
- `package.json` (@mcpedia/queue): deps bullmq, ioredis, @mcpedia/core,
|
||||
@mcpedia/db, @mcpedia/config.
|
||||
- `src/client.ts`: ioredis instance factory from config.
|
||||
- `src/queue.ts`:
|
||||
- `INDEX_QUEUE = "mcpedia-index"`.
|
||||
- `enqueueIndexDoc(slug, absPath, reason)`, `enqueueFullIndex(reason)`.
|
||||
- `getQueue()` lazy singleton.
|
||||
- `src/worker.ts`: `startWorker()` — BullMQ Worker with 3 job types:
|
||||
`index-doc` (single), `index-all` (full), `reindex` (full, reason=git-push).
|
||||
Graceful shutdown on SIGINT/SIGTERM. Job progress + error handling.
|
||||
|
||||
### apps/worker (NEW)
|
||||
- `package.json` (@mcpedia/worker): script `start: bun src/index.ts`.
|
||||
- `src/index.ts`: `startWorker()` + heartbeat log.
|
||||
|
||||
### apps/api
|
||||
- `src/index.ts`: add `POST /hooks/reindex` (full) and
|
||||
`POST /hooks/index?slug=` (single) webhook routes → enqueue jobs. Mount
|
||||
AFTER /trpc.
|
||||
- `src/router.ts`: add `jobStatus` (id→state/prev/failedData),
|
||||
`queueStatus` (waiting/active/completed/failed counts),
|
||||
`revisions` (slug→list), `restoreRevision` (id→new slug/doc).
|
||||
- `package.json`: add `@mcpedia/queue` dep, `hooks` reused.
|
||||
|
||||
### apps/mcp
|
||||
- `src/index.ts`: register Resources:
|
||||
- `mcpedia://docs` (list all metas)
|
||||
- `mcpedia://docs/{slug}` (full body from disk)
|
||||
- `mcpedia://docs/{slug}/chunks` (chunk previews)
|
||||
- `mcpedia://docs/{slug}/revisions` (revision list)
|
||||
- `src/smoke.test.ts`: add `listResources` + read `mcpedia://docs` assertion.
|
||||
|
||||
### scripts
|
||||
- `scripts/indexer.ts`: refactor `main()` to call `runFullIndex()`.
|
||||
|
||||
### Root
|
||||
- `package.json`: add `"worker": "bun --cwd apps/worker run start"`,
|
||||
`"reindex": "bun run scripts/worker.ts"`? No — `worker` runs the listener;
|
||||
triggering reindex = `bun run api` webhook or `enqueueFullIndex` helper.
|
||||
Add `"enqueue-index": "bun run scripts/enqueue.ts"` (one-shot enqueue).
|
||||
- `.env.example`: add `REDIS_URL`, `REDIS_PASSWORD`, `QUEUE_PREFIX`.
|
||||
|
||||
### Docs
|
||||
- `PHASES.md`: mark Phase 3 items DONE with notes.
|
||||
- `README.md`: document worker, webhook, revisions, MCP resources.
|
||||
|
||||
## Verification (real, not claimed)
|
||||
|
||||
1. `bun install` picks up new deps.
|
||||
2. `bunx turbo run build` + `typecheck` green across workspace.
|
||||
3. **Real BullMQ e2e against imrnes Redis**: script that enqueues an
|
||||
`index-doc` job, starts a Worker, asserts the job completes and the doc row
|
||||
+ chunks + a revision row appear in Postgres. Verifies Redis+ioredis+bullmq
|
||||
+ db + core all wired correctly.
|
||||
4. `bun --cwd apps/mcp run smoke` passes (incl. new resources).
|
||||
5. `bun run index` (runFullIndex) green; verify `documents`,
|
||||
`document_chunks`, `document_revisions` row counts via psql.
|
||||
6. API webhook: `curl -XPOST localhost:4020/hooks/reindex` enqueues; worker
|
||||
processes; `curl localhost:4020/trpc/queueStatus` reflects counts.
|
||||
7. MCP resource read returns real content.
|
||||
@@ -1,105 +0,0 @@
|
||||
# MCPedia Phase 4 — Operability & Correctness Hardening
|
||||
|
||||
> Reinterpretation: the PHASES.md "Scale-out" items (OpenSearch, object storage,
|
||||
> multi-tenant, distributed workers) are YAGNI at KB scale (4 docs). Phase 4 =
|
||||
> make the Phase 3 async + revision machinery **correct, secure, observable, and
|
||||
> deployable** — not speculative infra. Each task below fixes a real gap found
|
||||
> by reading the code, not a hypothetical need.
|
||||
|
||||
## Tasks
|
||||
|
||||
### T1 — `restoreRevision` must rebuild semantic chunks (CORRECTNESS BUG)
|
||||
**Root cause:** `packages/core/src/revision.service.ts` `restoreRevision` writes
|
||||
the old body back into `documents` but never calls `indexChunks(slug, body)`.
|
||||
So after a restore, keyword search (FTS on `documents.body`) is correct but
|
||||
`document_chunks`/embeddings stay on the *new* body → semantic + hybrid search
|
||||
return stale/ghost chunks.
|
||||
|
||||
**Fix:**
|
||||
- Add `reindexChunks(slug)` to `@mcpedia/core` that re-runs `indexChunks(slug, body)`
|
||||
using the live `documents.body` (the new body after the update).
|
||||
- Call it inside `restoreRevision` after the `documents` update (wrap in try/catch
|
||||
like `indexContentFile` so embed failure doesn't abort the restore).
|
||||
- Add a unit-style assertion to the MCP smoke test or a small script: restore →
|
||||
`document_chunks` count matches re-chunked body.
|
||||
|
||||
**Files:** `packages/core/src/revision.service.ts`, `packages/core/src/index.ts`,
|
||||
`packages/core/src/document.service.ts` (export existing `indexChunks` if needed).
|
||||
|
||||
### T2 — Secure the git-sync webhook (SECURITY)
|
||||
**Root cause:** `apps/api/src/index.ts` `/hooks/reindex` and `/hooks/index` accept
|
||||
any request with no `WEBHOOK_SECRET` check — `.env.example` defines `WEBHOOK_SECRET`
|
||||
but the router never reads it.
|
||||
|
||||
**Fix:**
|
||||
- In `apps/api/src/index.ts`, compare `c.req.header("x-webhook-secret")` (or
|
||||
`?secret=`) against `WEBHOOK_SECRET` (from `@mcpedia/config`). If unset/mismatch →
|
||||
`401`. If `WEBHOOK_SECRET` env is empty, reject at startup with a clear log
|
||||
(fail-fast, don't run an open endpoint).
|
||||
- Add `WEBHOOK_SECRET` to `packages/config/src/index.ts` export.
|
||||
- Document the header in README + verify with curl (401 without secret, 200 with).
|
||||
|
||||
**Files:** `apps/api/src/index.ts`, `packages/config/src/index.ts`, README.
|
||||
|
||||
### T3 — Web UI revisions view (UX)
|
||||
**Root cause:** Web UI (server components) calls `@mcpedia/core` directly; there is
|
||||
no revisions surface even though `revisions`/`restoreRevision` tRPC + MCP resource
|
||||
exist.
|
||||
|
||||
**Fix (server-component only, no client JS):**
|
||||
- On the doc page (`apps/web/app/[section]/[...slug]/page.tsx`), fetch
|
||||
`listRevisions(fullSlug, 10)` and render a "History" panel: revision number,
|
||||
reason, createdAt, body length, and a `/api/revisions/restore` link/POST that
|
||||
calls the tRPC `restoreRevision` mutation via a server action or a form POST to
|
||||
a small route handler. Simplest: a `<form method="post" action="/api/revisions/restore">`
|
||||
with hidden `id` + a route handler in `apps/web` calling `restoreRevision`.
|
||||
Keep it read-mostly; restore is a deliberate action.
|
||||
- Add `apps/web/app/api/revisions/restore/route.ts` (POST) → `restoreRevision(id)`
|
||||
→ `revalidatePath` the doc.
|
||||
|
||||
**Files:** `apps/web/app/[section]/[...slug]/page.tsx`,
|
||||
`apps/web/app/api/revisions/restore/route.ts`.
|
||||
|
||||
### T4 — Paginate `listRevisions` / `revisions` API (PERF)
|
||||
**Root cause:** `revision.service.ts` `listRevisions` does `select length(body)`
|
||||
(fine) but the tRPC `revisions` and MCP resource return *full* revision rows
|
||||
including the body in some callers; list endpoints should never carry bodies.
|
||||
|
||||
**Fix:**
|
||||
- Ensure `listRevisions` summary excludes `body` (it already does — `bodyLength`
|
||||
only). Add `offset` param for paging. Confirm MCP resource uses the summary.
|
||||
- No behavior change for the doc page (uses summary).
|
||||
|
||||
**Files:** `packages/core/src/revision.service.ts` (add `offset`), router unchanged.
|
||||
|
||||
### T5 — Deployable as supervised services (OPS)
|
||||
**Root cause:** `apps/api` and `apps/worker` run only ad-hoc; the host already runs
|
||||
`zeavis` via Nix/systemd. Phase-4 operability = provide a systemd unit (or Nix
|
||||
service) so `mcpedia-api` + `mcpedia-worker` start on boot and restart on failure.
|
||||
|
||||
**Fix (Nix-first, per MEMORY):**
|
||||
- Write `mcpedia-api.service` + `mcpedia-worker.service` systemd unit files under
|
||||
`deploy/` (bun run api / bun run worker, `WorkingDirectory`, `Restart=on-failure`,
|
||||
`EnvironmentFile` pointing at `.env`, `After=network-online.target`).
|
||||
- README section "Run as a service" with `cp deploy/*.service /etc/systemd/system && systemctl daemon-reload && systemctl enable --now mcpedia-api mcpedia-worker`.
|
||||
- **Do NOT** `systemctl` on the host without user confirmation (changing live
|
||||
services). Provide the files + instructions only; user runs enable.
|
||||
|
||||
**Files:** `deploy/mcpedia-api.service`, `deploy/mcpedia-worker.service`, README.
|
||||
|
||||
## Verification (all real, against imrnes Redis + Postgres)
|
||||
1. `turbo run typecheck` + `turbo run build` green.
|
||||
2. T1: script — edit a doc, reindex (new revision + new chunks), restore rev #1,
|
||||
assert `document_chunks` count for that slug now matches re-chunk of rev #1 body
|
||||
and `semanticSearch` on a term unique to rev #1 returns it.
|
||||
3. T2: `curl -XPOST localhost:4020/hooks/reindex` → 401; with
|
||||
`-H "x-webhook-secret: $WEBHOOK_SECRET"` → 200 + jobId.
|
||||
4. T3: `next build` includes the History panel; restore form rebuilds chunks
|
||||
(verified via T1 path through the route handler).
|
||||
5. T4: `revisions` API returns summaries without body; offset paging works.
|
||||
6. T5: `systemd-analyze verify deploy/*.service` passes (off-host safe check);
|
||||
README documents enable steps.
|
||||
|
||||
## Out of scope (YAGNI, keep deferred per PHASES.md)
|
||||
OpenSearch/Elasticsearch, object storage, multi-tenant, distributed workers,
|
||||
pgvector migration. Revisit only when corpus > ~10k docs or query latency bites.
|
||||
@@ -1,41 +0,0 @@
|
||||
# MCPedia — "lanjut semua" workstream
|
||||
|
||||
Three real gaps remain (from review): tiny corpus (4 docs), MCP read-only (no write/auth),
|
||||
no observability. This plan closes all three.
|
||||
|
||||
## 1. Content corpus (grow the KB)
|
||||
Author real, useful docs so search/semantic/revisions have something to operate on.
|
||||
Frontmatter schema (from packages/parser): id,title,type,tags,status,author,created_at,updated_at.
|
||||
Sections: docs|writeups|research|notes. Files under content/<section>/...
|
||||
New docs to add:
|
||||
- content/docs/caddy/reverse-proxy.md (ops reference, tags: caddy, reverse-proxy, tls)
|
||||
- content/docs/bullmq/workers.md (queue/worker reference, tags: bullmq, redis, jobs)
|
||||
- content/docs/mcp/streamable-http.md (MCP transport reference, tags: mcp, protocol, http)
|
||||
- content/notes/postgres/full-text-search.md (PG FTS notes, tags: postgres, fts, tsvector)
|
||||
- content/writeups/infra/cloudflare-525.md (debugging writeup, tags: cloudflare, tls, 525)
|
||||
After adding: `bun run index` to reindex (writes revisions + chunks), verify counts.
|
||||
|
||||
## 2. MCP write-tools + auth
|
||||
Add mutating + admin tools to the MCP server (currently read-only):
|
||||
- `index_document(slug)` -> enqueueIndexDoc (requires MCP auth header)
|
||||
- `reindex_all()` -> enqueueFullIndex (requires MCP auth header)
|
||||
- `queue_status()` -> getQueue counts (read, public)
|
||||
- `restore_revision(id)` -> restoreRevision (requires MCP auth header)
|
||||
Auth: MCP client must send header `x-webhook-secret` (reuse WEBHOOK_SECRET). StreamableHTTP
|
||||
transport: read the Authorization/header in http.ts, pass to server via a factory closure
|
||||
capturing the request; tools check it. Stateless per-request server already created fresh,
|
||||
so threading the header is clean. Guard write-tools with the same requireWriteAuth logic.
|
||||
Verify: unauthenticated call to index_document -> error; authenticated -> enqueues job.
|
||||
|
||||
## 3. Observability
|
||||
- `GET /metrics` on the API (Prometheus text format): queue counts (waiting/active/
|
||||
completed/failed/delayed), uptime, service name. Public (safe to expose).
|
||||
- Caddy: expose /metrics on wiki. domain -> :4020 (add to handle list).
|
||||
- tRPC `queueStatus` already exists; /metrics reuses getQueue.
|
||||
Verify: curl /metrics -> text exposition with mcpedia_queue_* gauges.
|
||||
|
||||
## Verification
|
||||
- typecheck green (turbo run typecheck).
|
||||
- MCP smoke extended: tools/list shows new tools; authenticated index_document enqueues.
|
||||
- /metrics returns 200 text; queue drains.
|
||||
- commit + push.
|
||||
@@ -1,157 +0,0 @@
|
||||
# MCPedia Phase 9 — Test Coverage + Observability Hardening
|
||||
|
||||
**Goal:** Add a real test suite (CI-gated) covering every layer of MCPedia — pure
|
||||
logic, Core services, the API surface, the MCP server + auth gates, and the Web
|
||||
UI — so regressions are caught before deploy. The suite must run green in CI
|
||||
with **no external services** (no Postgres, no Redis) by using in-process fakes.
|
||||
|
||||
## Constraints recap
|
||||
- Tooling: bun workspaces + Turborepo, bun 1.3.14 has a built-in `bun:test` runner.
|
||||
- DB is `imrnes` Postgres at `:6432` (no DB in CI) — tests must NOT touch it.
|
||||
- pgvector is NOT installed (vectors are `real[]`, cosine in-app) — confirmed.
|
||||
- Existing smoke test (`apps/mcp/src/smoke.test.ts`) is a script with `main()`
|
||||
run via `bun run smoke`, NOT a `bun:test` file. It hits the DB → cannot run in CI.
|
||||
- Secrets (`WEBHOOK_SECRET`, `DATABASE_URL`, `EMBED_*`) live in `.env` (gitignored)
|
||||
or BWS for deploy — never in tests or committed config.
|
||||
|
||||
## Decisions
|
||||
|
||||
1. **Runner:** `bun:test` — zero-config, built into the bun 1.3.14 toolchain
|
||||
already used. No extra deps. Add a `test` task to `turbo.json` and `package.json`
|
||||
scripts; add a `Test` step to CI.
|
||||
2. **No live DB in CI.** Tests that would need Postgres/Redis/Embeddings use
|
||||
**in-process fakes** (memory stores + a stub embedder returning fixed vectors).
|
||||
This means Core service tests cannot use the real `@mcpedia/db` singleton —
|
||||
they must accept an injected DB (drizzle-pg mem or a hand-rolled fake). We will
|
||||
**refactor the Core services' DB access behind injectable handles** where cheap,
|
||||
and for the MCP/HTTP auth-layer tests we stub `@mcpedia/queue` + `@mcpedia/core`
|
||||
at the module boundary (the transport/auth logic does not need a real queue).
|
||||
3. **Test boundaries by package:**
|
||||
- `packages/embeddings` — pure: `chunkText`, `cosine`. Real assertions, no I/O.
|
||||
- `packages/search` — pure: `toTsQuery`, `cosine`. SQL-bearing functions
|
||||
(`keywordSearch`/`semanticSearch`/`hybridSearch`) tested via a **fake db**
|
||||
injected into `@mcpedia/db`, OR via the `cosine`/fusion helpers in isolation.
|
||||
- `packages/core` — `snapshotRevision` dedup logic (refactor to accept an inject
|
||||
fn or test the public `indexContentFile`/`restoreRevision` with fakes).
|
||||
Focus: revision-dedup correctness + `restoreRevision` triggers `reindexChunks`.
|
||||
- `apps/api` — Hono app: `/health`, `/metrics` shape, `/hooks/*` auth (401 w/o
|
||||
secret, 200 + enqueue w/ secret using a fake queue), tRPC `restoreRevision`
|
||||
mutation auth gate (401 w/o secret).
|
||||
- `apps/mcp` — auth gates on write tools: `index_document`/`reindex_all`/
|
||||
`restore_revision` error without secret, enqueue with secret (fake queue).
|
||||
Read tools + resources via `InMemoryTransport` (reuse smoke style but without
|
||||
DB).
|
||||
- `apps/web` — render correctness of home (lists sections), doc page
|
||||
(renders title + markdown + history panel when revisions exist), search page
|
||||
(keyword/hybrid toggle, empty state). These need a fake Core.
|
||||
|
||||
## Approach per test (minimal, high-signal)
|
||||
|
||||
### embeddings: `packages/embeddings/src/chunk.test.ts`
|
||||
- `chunkText("hello world")` with short size → single chunk.
|
||||
- `chunkText` long text → multiple chunks, overlap honored, no word splits past boundary.
|
||||
- `chunkText("")` / `" "` → `[]`.
|
||||
|
||||
### search: `packages/search/src/cosine.test.ts` (new tiny file) + refactor
|
||||
- `cosine([1,0],[0,1])` ≈ 0; `cosine([1,1],[1,1])` = 1; `cosine([],[1])` = 0.
|
||||
- `toTsQuery("a b c")` → `"a:* & b:* & c:*"`; empty/garbage → `""`.
|
||||
|
||||
### core: `packages/core/src/index.service.test.ts`
|
||||
The hard part: `indexContentFile`/`restoreRevision`/`snapshotRevision` call `db`
|
||||
directly. Two options:
|
||||
- **Option A (chosen):** extract `snapshotRevision`'s "latest body" + "insert"
|
||||
steps behind the existing `db` but make `indexContentFile` test the revision
|
||||
*decision* by inserting a doc + revision directly via `db` in a test Postgres
|
||||
(too heavy for CI).
|
||||
- **Option B (chosen):** test the **pure decision logic** by refactoring
|
||||
`snapshotRevision` to export a pure helper
|
||||
`shouldCreateRevision(latestBody, body): boolean` — `true` when latest is null
|
||||
or latest.body !== body. Then a unit test asserts the dedup truth table;
|
||||
`indexContentFile` is verified by the existing e2e (manual `bun run index`).
|
||||
This is the CI-safe win.
|
||||
- `restoreRevision` correctness: assert it calls `reindexChunks(slug)` — we can
|
||||
test by spying. Since `reindexChunks` is in the same module, we'll export a
|
||||
seam: `restoreRevision(id, { reindexChunks: spy })` — keep backward compat by
|
||||
defaulting. (Or test the public contract via the API layer instead.)
|
||||
|
||||
### api: `apps/api/src/index.test.ts`
|
||||
- Build the Hono `app` from a testable factory that accepts a fake queue + fake
|
||||
webhook secret. Current `index.ts` throws at import if `WEBHOOK_SECRET` unset —
|
||||
that breaks import in CI. **Refactor:** move the fail-fast check into
|
||||
`listen()`/serve start, so the app is constructable without a secret for
|
||||
testing. Export `createApp(opts?)` returning the Hono instance.
|
||||
- `/health` → 200 `{ok:true}`.
|
||||
- `/metrics` → 200, text/plain, contains `mcpedia_uptime_seconds` +
|
||||
`mcpedia_queue_jobs` for each state (fake queue returns 0/1).
|
||||
- `POST /hooks/reindex` w/o `x-webhook-secret` → 401; with matching secret →
|
||||
200 + `{ok:true, jobId}` (fake queue records the enqueue).
|
||||
- tRPC: build a client against the app, call `restoreRevision` without secret →
|
||||
error; the public `listDocuments` returns from a fake DB.
|
||||
|
||||
### mcp: `apps/mcp/src/auth.test.ts`
|
||||
- `createMcpServer()` (no secret) → `index_document`/`reindex_all`/`restore_revision`
|
||||
throw "unauthorized".
|
||||
- `createMcpServer("secret")` → same tools reach the enqueue call (fake queue).
|
||||
- Read tools still work without secret (server loads, resources list).
|
||||
|
||||
### web: `apps/web/app/search/page.test.tsx` (or a lighter harness)
|
||||
- This is the hardest to test without a browser. **Decision:** keep web tests
|
||||
minimal — assert that `toTsQuery`/render helpers exist; full DOM tests deferred
|
||||
(needs playwright + a running server). We'll instead add a **contract test**
|
||||
that the search page's `dynamic = "force-dynamic"` export exists (static-
|
||||
generation guard, the kind of thing that broke CI before).
|
||||
|
||||
## File layout (new files)
|
||||
```
|
||||
packages/embeddings/src/chunk.test.ts
|
||||
packages/search/src/cosine.test.ts
|
||||
packages/core/src/index.service.test.ts # snapshotRevision + restoreRevision seam
|
||||
apps/api/src/index.test.ts # Hono /health /metrics /hooks + tRPC gate
|
||||
apps/mcp/src/auth.test.ts # write-tool auth gates
|
||||
```
|
||||
|
||||
## turbo.json
|
||||
Add a `test` task (like `typecheck`, no dependsOn, cache false so it always runs):
|
||||
```jsonc
|
||||
"test": { "cache": false }
|
||||
```
|
||||
Each app/pkg gets `"test": "bun test"` in its package.json.
|
||||
|
||||
## CI (`.github/workflows/ci.yml`)
|
||||
After `Build`, add:
|
||||
```yaml
|
||||
- name: Test
|
||||
run: bun run test # -> turbo run test
|
||||
```
|
||||
|
||||
## Source changes required (enablers)
|
||||
1. `apps/api/src/app.ts` (NEW) — extracted `createApp(deps?)` factory returning a
|
||||
`Promise<Hono>`. Pure construction (no process exit, no fail-fast). Accepts
|
||||
injected `ApiDeps` (`{ queue, webhookSecret }`); when omitted, lazily
|
||||
imports the real queue + uses `WEBHOOK_SECRET` (production path). `/dashboard`
|
||||
now renders the HTML from a new `apps/api/src/dashboard.ts` module.
|
||||
2. `apps/api/src/dashboard.ts` (NEW) — the self-contained dashboard HTML, split
|
||||
out of the original `index.ts` so the route is testable + the const is
|
||||
importable. XSS-safe (esc() on all KB-sourced fields; documented in comment).
|
||||
3. `apps/api/src/index.ts` — now a thin re-export of `createApp`/`start` + the
|
||||
`isMain` bootstrap. Systemd unit still runs `bun --cwd apps/api src/index.ts`.
|
||||
4. `packages/core/src/index.service.ts` — exported `shouldCreateRevision` (pure
|
||||
predicate for the revision dedup invariant). `restoreRevision` in
|
||||
`revision.service.ts` now accepts an optional `opts.reindex` seam (defaults
|
||||
to the real `reindexChunks`), making the chunk-rebuild contract testable.
|
||||
5. `apps/mcp/src/smoke.ts` — renamed from `smoke.test.ts` (so `bun test` doesn't
|
||||
treat the integration smoke as a unit run), and fixed stale assertions:
|
||||
expected tool set updated to all 10 tools (Phase 7 additions + write tools),
|
||||
`list_documents(section=docs)` count updated to 4 (post-Phase-7 corpus).
|
||||
|
||||
## Verification
|
||||
- `bun run test` (local) → 32 tests green across 6 packages (embeddings 5,
|
||||
search 8, core 4, parser 5, mcp 6, api 8), **no live DB needed** (mocks
|
||||
stub `@mcpedia/db`, `@mcpedia/queue`, `@mcpedia/core`).
|
||||
- `bun run typecheck` → green (4 apps, no test-only type errors).
|
||||
- `bun --cwd apps/mcp run smoke` → green (integration, needs live DB — runs in
|
||||
CI on the deploy host, not in CI's no-services job).
|
||||
- Live API check: `/health`, `/metrics`, `/dashboard`, `/hooks/*` auth gate
|
||||
all 200/401-verified against a temp-port server.
|
||||
- Commit + push; worker redeploy not needed (code + tests only).
|
||||
|
||||
@@ -1,80 +0,0 @@
|
||||
# Phase 15 — UI/UX Polish Pass
|
||||
|
||||
> User: "perbagus ui ux nya"
|
||||
|
||||
## Audit
|
||||
|
||||
### Current issues
|
||||
1. **Section index** (`[section]/page.tsx`):
|
||||
- `buildFolderTree` uses path segments instead of doc titles for leaf nodes
|
||||
- Folder tree + flat list at bottom is redundant
|
||||
- No folder icons (📁), no doc type indicators, no dates
|
||||
- Tree rendering is very basic (no visual hierarchy)
|
||||
|
||||
2. **Doc page** (`[section]/[...slug]/page.tsx`):
|
||||
- CustomFieldBadges shows key=value but labels all badges with key name as title
|
||||
- No structured metadata card layout
|
||||
- Related docs cards are functional but visually flat
|
||||
|
||||
3. **Folder index** (`FolderIndexPage` in `[...slug]/page.tsx`):
|
||||
- Shows doc slug parts (e.g. "pwn-100-ret2win-alignment") instead of titles
|
||||
- No date, no tags, no description
|
||||
- Subfolders are plain text links, no folder count or doc count
|
||||
|
||||
4. **Homepage** (`page.tsx`):
|
||||
- Folder tree inside cards is cramped (text-xs, no spacing)
|
||||
- Section cards show "View all (N)" but folder tree duplicates that
|
||||
|
||||
5. **Sidebar** (`Sidebar.tsx`):
|
||||
- Tree uses indentation via inline style but no visual depth cues
|
||||
- No hover expand for folders
|
||||
- Active state only on exact match, not parent folders
|
||||
|
||||
## Plan
|
||||
|
||||
### 1. Section index — enhanced tree
|
||||
- Use doc titles for leaf nodes (already have `doc` reference in tree)
|
||||
- Add 📁 for folders, 📄 for docs
|
||||
- Show doc count per folder
|
||||
- Remove flat list (redundant with tree)
|
||||
- Add "View all" link per section
|
||||
- Better visual hierarchy: folder headers with counts, doc titles with dates
|
||||
|
||||
### 2. Doc page — metadata card
|
||||
- Replace inline badge row with a structured metadata card
|
||||
- Show custom fields as labeled badges (key → value with color)
|
||||
- Keep CustomFieldBadges for backward compat but improve layout
|
||||
- Add metadata card with: author, date, tags, and all custom fields
|
||||
|
||||
### 3. Folder index — rich listing
|
||||
- Show doc titles (fetch from listDocuments, match by path)
|
||||
- Show update date per doc
|
||||
- Show tags per doc
|
||||
- Add doc count for subfolders
|
||||
- Better visual separation between subfolders and docs
|
||||
|
||||
### 4. Homepage — cleaner tree
|
||||
- Increase font size for tree items
|
||||
- Show section icon + label in tree header
|
||||
- Better spacing between sections
|
||||
|
||||
### 5. Sidebar — depth cues
|
||||
- Use ml-4 per level instead of inline style
|
||||
- Add section divider lines
|
||||
- Highlight active section + parent folders
|
||||
|
||||
## Files to change
|
||||
```
|
||||
mod: apps/web/app/[section]/page.tsx # enhanced tree, use doc titles
|
||||
mod: apps/web/app/[section]/[...slug]/page.tsx # FolderIndexPage + metadata card
|
||||
mod: apps/web/app/page.tsx # cleaner section tree cards
|
||||
mod: apps/web/app/components/Sidebar.tsx # depth + active parent highlight
|
||||
mod: apps/web/app/globals.css # additional CSS vars if needed
|
||||
```
|
||||
|
||||
## Verification
|
||||
- All endpoints still 200
|
||||
- `bun run test` green
|
||||
- `turbo run typecheck` green
|
||||
- `next build` succeeds
|
||||
- Live visual check of folder index, section index, doc page
|
||||
Reference in new issue
Block a user