feat: revamp to dynamic database-first architecture and cleanup phase docs
CI / typecheck + build (turbo) (push) Canceled after 0s
CI / typecheck + build (turbo) (push) Canceled after 0s
- Migrate document fetching and CRUD to be PostgreSQL-authoritative - Remove static section enums and add dynamic listSections query - Support custom sections and metadata across API, MCP, and Web UI - Add /api/sections endpoint and update Header, Sidebar, and Forms - Remove obsolete phase planning docs and modernize README/AGENTS
This commit is contained in:
File diff suppressed because one or more lines are too long
@@ -1,161 +0,0 @@
|
||||
# MCPedia Phase 2 — Semantic Search + tRPC/Hono API
|
||||
|
||||
> **For Hermes:** implement task-by-task. Spec-first (user rule 2026-08-19).
|
||||
|
||||
**Goal:** Add semantic + hybrid search (pgvector) and a typed tRPC/Hono API so
|
||||
MCPedia is queryable by embeddings, not just keyword FTS — and expose the
|
||||
corpus over a programmatic HTTP API.
|
||||
|
||||
**Architecture:** Content (Markdown) → chunk → embed (OpenRouter) → store
|
||||
`document_chunks` with `vector(N)` in Postgres → `semanticSearch` (cosine) and
|
||||
`hybridSearch` (FTS + cosine, reciprocal-rank fusion) in `@mcpedia/search` →
|
||||
exposed via Core, the MCP server (new tools), and a new `apps/api` (Hono +
|
||||
tRPC v11).
|
||||
|
||||
**Embedding provider:** OpenRouter (`openrouter/llama-nemotron-embed-vl-1b-v2:free`)
|
||||
via `9router_ai_llm_api_key` + `9router_ai_llm_base_url` (BWS). Dimension is
|
||||
discovered at first live call (see Step 1.3) and pinned in schema/migration.
|
||||
|
||||
**Tech stack:** drizzle-orm `vector` column + pgvector extension, HNSW index,
|
||||
`@trpc/server` v11 (fetch adapter), `hono` + `@hono/node-server`.
|
||||
|
||||
---
|
||||
|
||||
## Task P2.1 — `packages/embeddings` (provider + abstraction)
|
||||
|
||||
**Files:** `packages/embeddings/package.json`, `src/index.ts`, `src/provider.ts`,
|
||||
`src/openrouter.ts`
|
||||
|
||||
- `EmbeddingProvider` interface: `embed(texts: string[]): Promise<number[][]>`, `readonly model`, `readonly dimensions`.
|
||||
- `OpenRouterEmbeddingProvider`: POST `${baseUrl}/embeddings` with `{ model, input }`,
|
||||
`Authorization: Bearer ${key}`. Returns `data[].embedding`. Validate length === dimensions.
|
||||
- Read `EMBED_BASE_URL`, `EMBED_API_KEY`, `EMBED_MODEL` from `@mcpedia/config`
|
||||
(with `.env` fallback). Dimensions discovered live (Step 1.3) → export `EMBED_DIM`.
|
||||
- Chunk helper `chunkText(text, { size=1000, overlap=150 })` in `src/chunk.ts`.
|
||||
|
||||
**Step 1.3 (discover dim):** live call `embed(["test"])`, read `embedding.length`,
|
||||
pin `EMBED_DIM`, assert mismatch throws.
|
||||
|
||||
**Verify:** `bun run` a temp script: `embed(["hello world"])` prints a vector of
|
||||
length N (e.g. 1024). Confirm no key is logged.
|
||||
|
||||
---
|
||||
|
||||
## Task P2.2 — Schema: `document_chunks` + vector extension
|
||||
|
||||
**Files:** `packages/db/src/schema.ts` (add), `packages/db/drizzle.config.ts`
|
||||
(unchanged), new migration.
|
||||
|
||||
- `CREATE EXTENSION IF NOT EXISTS vector;` (idempotent; run once via psql).
|
||||
- `document_chunks` table:
|
||||
- `id` uuid pk default gen_random_uuid()
|
||||
- `document_id` text → `documents.id` on delete cascade
|
||||
- `slug` text (denormalized for convenience)
|
||||
- `chunk_index` integer
|
||||
- `content` text
|
||||
- `embedding` vector(EMBED_DIM)
|
||||
- `created_at` timestamp default now()
|
||||
- index `chunk_embedding_idx` using hnsw (`embedding` op `vector_cosine_ops`)
|
||||
- Generate migration with `drizzle-kit generate`, apply via `psql` (drizzle-kit
|
||||
push is unreliable here — known).
|
||||
|
||||
**Verify:** `\d document_chunks` shows `embedding vector(N)` + HNSW index;
|
||||
`select count(*) from document_chunks` = 0.
|
||||
|
||||
---
|
||||
|
||||
## Task P2.3 — Indexer: chunk + embed + upsert
|
||||
|
||||
**Files:** `scripts/indexer.ts` (extend), `packages/core/src/document.service.ts`
|
||||
(add `indexChunks`).
|
||||
|
||||
- For each published doc: read body (already on disk), `chunkText`, `embed` in
|
||||
batches (≤ 16), delete existing chunks for slug, insert new rows.
|
||||
- Guard: if embedding provider fails, log + skip (don't crash the whole index).
|
||||
- Add `bun run index:embed` (or extend `bun run index` to also embed).
|
||||
|
||||
**Verify:** after running, `select count(*) from document_chunks` > 0; a sample
|
||||
row has non-null `embedding`.
|
||||
|
||||
---
|
||||
|
||||
## Task P2.4 — `packages/search`: semantic + hybrid
|
||||
|
||||
**Files:** `packages/search/src/index.ts` (add `semanticSearch`, `hybridSearch`).
|
||||
|
||||
- `semanticSearch(vec, limit)`: order by `embedding <=> ${vec}` asc, filter published.
|
||||
- `hybridSearch(q, limit)`: run FTS (`ts_rank`) + semantic (cosine) in parallel;
|
||||
fuse with reciprocal-rank (RRF: score = 1/(k+rank), k=60); return merged hits.
|
||||
- Keep `keywordSearch` unchanged (Phase 1).
|
||||
|
||||
**Verify:** unit-ish script: embed a query, `semanticSearch` returns relevant
|
||||
chunks; `hybridSearch("websocket")` returns ≥ keyword results.
|
||||
|
||||
---
|
||||
|
||||
## Task P2.5 — `packages/core` expose semantic/hybrid
|
||||
|
||||
**Files:** `packages/core/src/search.service.ts`, `index.ts`.
|
||||
|
||||
- Re-export `semanticSearch`, `hybridSearch` from Core.
|
||||
|
||||
---
|
||||
|
||||
## Task P2.6 — `apps/api` (Hono + tRPC v11)
|
||||
|
||||
**Files:** `apps/api/package.json`, `tsconfig.json`, `src/index.ts`,
|
||||
`src/router.ts`, `src/trpc.ts`.
|
||||
|
||||
- `initTRPC.create()` router with procedures: `search`, `semanticSearch`,
|
||||
`hybridSearch`, `getDocument`, `listDocuments` (mirrors MCP tools).
|
||||
- Mount `fetchRequestHandler` on a Hono app at `/trpc/*`; serve via
|
||||
`@hono/node-server` `serve({ fetch: app.fetch, port: 4020 })`.
|
||||
- `createContext` returns `{ db }`.
|
||||
|
||||
**Verify:** `bun run dev` → `curl -X POST localhost:4020/trpc/search`
|
||||
with JSON body returns hits.
|
||||
|
||||
---
|
||||
|
||||
## Task P2.7 — MCP server: semantic + hybrid tools
|
||||
|
||||
**Files:** `apps/mcp/src/index.ts` (add `semantic_search`, `hybrid_search`),
|
||||
extend `smoke.test.ts`.
|
||||
|
||||
- `semantic_search`: embed query → `semanticSearch`.
|
||||
- `hybrid_search`: embed query → `hybridSearch`.
|
||||
- Smoke: assert both return ≥1 hit for "websocket".
|
||||
|
||||
---
|
||||
|
||||
## Task P2.8 — Web: semantic toggle on search
|
||||
|
||||
**Files:** `apps/web/app/search/page.tsx`.
|
||||
|
||||
- Add `mode=keyword|hybrid` query param; server component calls Core
|
||||
`hybridSearch` when `mode=hybrid`. Minimal UI toggle (link/buttons).
|
||||
- Keep keyword as default.
|
||||
|
||||
**Verify:** `bun run build`; `curl '/search?q=websocket&mode=hybrid'` returns hits.
|
||||
|
||||
---
|
||||
|
||||
## Task P2.9 — Verify all + commit
|
||||
|
||||
- `bunx turbo run build` (web + api + mcp), `bun run apps/mcp smoke`,
|
||||
live API curl, live web hybrid search.
|
||||
- Update `README.md` + `PHASES.md` (mark Phase 2 ✅).
|
||||
- `git add -A` (exclude `.env`), commit as asepharyana (no Co-Authored-By).
|
||||
|
||||
---
|
||||
|
||||
## Risks / decisions
|
||||
- **Dimension unknown until live call** → P2.1.3 discovers it; pinned EMBED_DIM=2048.
|
||||
- **pgvector NOT available on shared imrnes Postgres** (extension not installed;
|
||||
installing needs host-level apt on a managed/shared DB — deferred). PIVOT:
|
||||
store `embedding` as `real[]` and compute cosine similarity in the app layer.
|
||||
Brute-force cosine is instant for a KB-sized corpus (dozens of docs / hundreds
|
||||
of chunks). pgvector+HNSW is the Phase-4 scale-out path.
|
||||
- **PgBouncer + real[]**: fine; simple queries, no extension needed.
|
||||
- **API port 4020** (host 4000s range is 4000–4015; 4020 is free for dev). Deploy later.
|
||||
- **YAGNI**: no auth/revisions this phase (Phase 3).
|
||||
@@ -1,151 +0,0 @@
|
||||
# Phase 14 — Hierarchical Folder Structure
|
||||
|
||||
> User: "gk ada bedanya, maksud saya inginnya itu bisa yg bertingkat seperti github yg memiliki folder dalam folder"
|
||||
> Context: after the full dynamic-custom-fields overhaul (Phase 13), the user
|
||||
> wants document URLs/content organized in **nested folders** like GitHub —
|
||||
> `writeups/ctf/defcon-quals-2024/pwn-100/...` with subfolders under subfolders,
|
||||
> not just one level deep.
|
||||
|
||||
## Problem
|
||||
|
||||
The current URL scheme is `/<section>/<slug>` where `slug` can contain `/`
|
||||
(e.g. `writeups/ctf/defcon-quals-2024/pwn-100-ret2win-alignment` →
|
||||
`/writeups/ctf/defcon-quals-2024/pwn-100-ret2win-alignment`). This works for
|
||||
**files** but there are no **folder-level index pages** — navigating to
|
||||
`/writeups/ctf/defcon-quals-2024/` returns 404 because Next.js catch-all
|
||||
`[section]/[...slug]/page.tsx` requires at least one slug segment beyond the
|
||||
section, and the sidebar only shows flat doc titles (no folder tree).
|
||||
|
||||
GitHub's model: `github.com/org/repo/tree/main/path/to/folder/file` — every
|
||||
folder has an index page (`/path/to/folder/`) listing its contents.
|
||||
|
||||
## Solution
|
||||
|
||||
### 1. Folder Index Pages
|
||||
|
||||
**Create `apps/web/app/[section]/[...slug]/folder.tsx`** (or a parallel route).
|
||||
Actually — cleaner approach per Next.js App Router: the catch-all
|
||||
`[section]/[...slug]/page.tsx` handles both. Add logic: if the slug resolves to
|
||||
an actual markdown file → doc page (existing behavior). If the slug resolves to
|
||||
a **directory** (folder of docs) → render a folder index listing all docs whose
|
||||
`path` starts with that prefix.
|
||||
|
||||
**Mechanism:**
|
||||
- Call `listDocuments()` to get all docs.
|
||||
- The incoming URL path is `{section}/{...slug}`.
|
||||
- Build the "folder prefix" = `${section}/${slug.join("/")}/` (with trailing `/`,
|
||||
or just `${section}/${slug.join("/")}` if no slug segments).
|
||||
- Filter docs whose `doc.path` starts with that prefix.
|
||||
- If exactly one doc matches AND its path === prefix (trimmed .md) → it's a
|
||||
doc page (existing). If zero or multiple match and they all start with the
|
||||
prefix → it's a folder index.
|
||||
- Edge: a folder with exactly one doc whose path matches exactly — still a doc
|
||||
page. A folder is when there are docs at `prefix/sub/...`.
|
||||
|
||||
**Better heuristic:** A slug path is a "folder" if there exist docs whose `path`
|
||||
is `prefix/deep/...` (i.e., the slug is a parent of other doc paths, not a
|
||||
leaf itself). A slug is a "leaf doc" if `path === prefix + ".md"`.
|
||||
|
||||
### 2. Sidebar Tree
|
||||
|
||||
**Update `Sidebar.tsx`:**
|
||||
- `listDocuments()` already returns all docs with their full `slug` and `path`.
|
||||
- Build a **tree** from the flat list: split each slug by `/`, create nested
|
||||
folder nodes.
|
||||
- Render nested `<ul>` with indentation (already done via `marginLeft` based on
|
||||
depth).
|
||||
- **Folder nodes** (collapsed/expanded) get a folder icon 📁 and a CSS class.
|
||||
- Clicking a folder → navigates to the folder index page `/{section}/{path}`.
|
||||
- **Leaf doc nodes** → link to `/{doc.slug}` (existing behavior).
|
||||
- Group by the first segment after section too (e.g. `ctf/defcon-quals-2024/`
|
||||
is a folder, then `pwn-100-...` are children).
|
||||
|
||||
Tree-building algorithm (from flat slugs):
|
||||
```
|
||||
For slug "writeups/ctf/defcon-quals-2024/pwn-100-ret2win-alignment":
|
||||
parts = ["writeups", "ctf", "defcon-quals-2024", "pwn-100-ret2win-alignment"]
|
||||
→ tree: writeups → ctf → defcon-quals-2024 → pwn-100-ret2win-alignment (leaf)
|
||||
```
|
||||
|
||||
### 3. DocForm / Folder Selection
|
||||
|
||||
**Update `DocForm.tsx`:**
|
||||
- Add a "Parent folder" input (autocomplete or text) that shows existing folders
|
||||
for the selected section. The slug field already supports `/` but the user
|
||||
experience is better with folder picker.
|
||||
- When creating, the slug becomes `{parentFolder}/{slug}` automatically.
|
||||
- Show existing folder structure as `<select>` or tree picker.
|
||||
|
||||
### 4. Example Hierarchy
|
||||
|
||||
Create a real hierarchical structure to demonstrate:
|
||||
```
|
||||
writeups/
|
||||
ctf/
|
||||
defcon-quals-2024/
|
||||
pwn/
|
||||
pwn-100-ret2win-alignment.md
|
||||
pwn-200-bof-heap.md
|
||||
crypto/
|
||||
crypto-100-xor.md
|
||||
crypto-200-rsa.md
|
||||
template/
|
||||
writeup-template.md
|
||||
_index.md ← folder index (optional intro)
|
||||
hackthebox/
|
||||
machine-name/
|
||||
walkthrough.md
|
||||
```
|
||||
|
||||
For now, reorganize the existing Defcon writeup into proper subfolders + add a
|
||||
folder index page. The existing `content/writeups/ctf/defcon-quals-2024/` is
|
||||
already a folder — just need the folder index route to work.
|
||||
|
||||
### 5. Route Changes
|
||||
|
||||
**Current:** `[section]/[...slug]/page.tsx` — catch-all requires ≥1 slug segment.
|
||||
- `/writeups` → `notFound()` (no index for bare section unless we add one)
|
||||
- `/writeups/ctf` → catch-all gets `slug=["ctf"]` → currently treated as a doc
|
||||
(looks up `writeups/ctf` doc, 404 if none)
|
||||
- `/writeups/ctf/defcon-quals-2024/` → catch-all `slug=["ctf","defcon-quals-2024"]`
|
||||
|
||||
**Plan:**
|
||||
1. Add `/writeups/page.tsx` (section index) — lists top-level folders + root
|
||||
docs in that section. (Currently `/docs/page.tsx` exists but `/writeups/page.tsx`
|
||||
doesn't.)
|
||||
2. In `[section]/[...slug]/page.tsx`: at the top of the page component, check if
|
||||
the slug path is a folder (has child docs). If so, render folder index instead
|
||||
of doc page.
|
||||
|
||||
**Section index pages:** Create `[section]/page.tsx` for all 4 sections, or a
|
||||
generic one. Currently only `/docs/page.tsx` exists. Add a shared
|
||||
`SectionIndex` component.
|
||||
|
||||
### 6. Files to Change
|
||||
|
||||
```
|
||||
new: apps/web/app/[section]/page.tsx # generic section index (folder + doc listing)
|
||||
mod: apps/web/app/[section]/[...slug]/page.tsx # add folder-index detection
|
||||
mod: apps/web/app/components/Sidebar.tsx # tree from flat slugs
|
||||
mod: apps/web/app/components/DocForm.tsx # parent folder picker
|
||||
new: content/writeups/ctf/defcon-quals-2024/_index.md # folder intro (optional)
|
||||
new: content/writeups/ctf/_index.md # CTF section intro
|
||||
mod: apps/web/app/docs/page.tsx # may need generic version
|
||||
```
|
||||
|
||||
### 7. Verification
|
||||
|
||||
- `/writeups` → 200, shows folders: ctf/, template/
|
||||
- `/writeups/ctf` → 200, folder index listing defcon-quals-2024/
|
||||
- `/writeups/ctf/defcon-quals-2024` → 200, folder index listing pwn-100-...
|
||||
- `/writeups/ctf/defcon-quals-2024/pwn-100-ret2win-alignment` → 200, doc page
|
||||
- Sidebar shows nested tree with folder icons
|
||||
- DocForm parent folder picker works
|
||||
- `bun run test` green, `turbo run typecheck` green
|
||||
- Live verification via curl
|
||||
|
||||
## Constraints
|
||||
- User: "pastikan semua dinamis dan rapih untuk banyak situasi jadi tergantung
|
||||
user bukan hardcode" — folder detection must be content-driven, not config.
|
||||
- No breaking existing flat slugs.
|
||||
- Section icons/labels stay the same.
|
||||
@@ -1,42 +0,0 @@
|
||||
# MCP HTTP transport + deploy + review actions
|
||||
|
||||
## Goal
|
||||
Make the MCPedia MCP server reachable over the network (not just stdio subprocess), so
|
||||
remote MCP clients (Claude, a Discord bot, a web client) can call its 6 tools + 4 resources.
|
||||
Serve via Streamable HTTP (MCP 2025-03-26 spec), deploy as a supervised systemd service,
|
||||
expose through Caddy on a dedicated subdomain.
|
||||
|
||||
## Design decisions
|
||||
- **StreamableHTTPServerTransport, stateless mode** (`sessionIdGenerator: undefined`).
|
||||
One McpServer + transport per request. No session map, no shared-transport connect race,
|
||||
no memory leak. Re-registering 6 tools + 4 resources per request is negligible for a KB.
|
||||
- **New entry `apps/mcp/src/http.ts`** served by Node `http` (built-in), NOT mounted on the
|
||||
API app — keeps the MCP app's zod-4 isolation intact (api app is zod 3).
|
||||
- **Port 4021** (next free in the 4000s range; 4020 is the API).
|
||||
- **Subdomain `mcp.asepharyana.my.id`** -> 4021 (Cloudflare `*` wildcard already proxies it;
|
||||
Caddy auto-issues LE cert, no extra DNS work).
|
||||
- **CORS** allow on `/mcp` (remote web clients need it).
|
||||
- MCP tools are read-only (search/get/list/related + read-only resources) => open MCP is
|
||||
low-risk. No auth on MCP itself.
|
||||
|
||||
## Files
|
||||
- `apps/mcp/src/http.ts` (NEW) — Node http server, `/mcp` route, stateless transport.
|
||||
- `apps/mcp/package.json` — add `serve:http` script.
|
||||
- `package.json` (root) — add `mcp:http` script (absolute bun path).
|
||||
- `deploy/mcpedia-mcp.service` (NEW) — systemd unit, MCP_PORT=4021.
|
||||
- `/etc/caddy/Caddyfile` — add `mcp.asepharyana.my.id { import proxy 4021 }`.
|
||||
|
||||
## Security finding (review, flagged not silently built)
|
||||
The tRPC `restoreRevision` mutation is exposed UNauthenticated at
|
||||
`https://wiki.asepharyana.my.id/trpc/restoreRevision` — anyone can revert a live doc.
|
||||
The web UI's restore path calls `@mcpedia/core` directly (server component), so the tRPC
|
||||
mutation is dead surface. Fix: guard the mutation with the existing WEBHOOK_SECRET header,
|
||||
or drop it from the router. Will apply the guard (consistent with /hooks auth) unless user
|
||||
prefers removal.
|
||||
|
||||
## Verification
|
||||
- `bun --cwd apps/mcp run typecheck` green.
|
||||
- Live: `curl -XPOST https://mcp.asepharyana.my.id/mcp` initialize -> 200 + serverInfo;
|
||||
tools/list -> 6 tools; resources/list -> 4 resources.
|
||||
- `systemctl is-active mcpedia-mcp` == active.
|
||||
- Commit + push.
|
||||
@@ -1,67 +0,0 @@
|
||||
# MCPedia Phase 11 — CRUD + Auth + Web UI
|
||||
|
||||
## STATUS: ✅ ALL DONE (committed f8b525d, CI+Deploy success)
|
||||
|
||||
## Plan (spec BEFORE implementation, per user rule)
|
||||
|
||||
### Scope: 4 areas
|
||||
1. **CRUD**: Create/Read/Update/Delete documents from web UI + MCP
|
||||
2. **Authentication**: MCP/API writes (x-webhook-secret), Web CRUD (cookie-based ADMIN_PASSWORD)
|
||||
3. **Web + Agent access**: Web forms + MCP write tools
|
||||
4. **UI/UX**: Edit forms, TOC, dark mode
|
||||
|
||||
### Requirements:
|
||||
1. Source of truth = filesystem (markdown files in content/{section}/{slug}.md)
|
||||
2. DB mirrors disk (documents, document_chunks, document_revisions tables)
|
||||
3. Single indexing path: indexContentFile in @mcpedia/core
|
||||
4. Auth: MCP/API writes use WEBHOOK_SECRET; Web uses ADMIN_PASSWORD cookie
|
||||
5. Slug rules: [a-z0-9][a-z0-9/_-]*, no //, no .. traversal
|
||||
6. No breaking existing features (32 original tests still green)
|
||||
7. UI/UX: edit button on doc pages, ?edit=1 inline form, /create page, login, TOC, dark mode
|
||||
|
||||
### Implementation
|
||||
|
||||
#### Backend
|
||||
- `packages/parser`: added `stringifyFile()` (writes markdown with frontmatter)
|
||||
- `packages/core`: `createDocument`, `updateDocument`, `deleteDocument` (file I/O + DB + revision + chunks)
|
||||
- `apps/api`: tRPC CRUD routers (`requireWriteAuth`), fixed `requireWriteAuth` env-constant bug (now uses `ctx.expectedSecret` injected from `createApp(deps)`)
|
||||
- `apps/mcp`: 3 new write tools (`create_document`, `update_document`, `delete_document`) gated by `x-webhook-secret`
|
||||
|
||||
#### Web UI
|
||||
- `apps/web/app/api/auth/login/route.ts`: POST login → verify ADMIN_PASSWORD, set `mcpedia_admin` cookie
|
||||
- `apps/web/app/api/docs/route.ts`: POST (create)
|
||||
- `apps/web/app/api/docs/[...slug]/route.ts`: PUT (update), DELETE (delete)
|
||||
- `apps/web/app/components/DocForm.tsx`: shared create/edit form
|
||||
- `apps/web/app/create/page.tsx`: create form
|
||||
- `apps/web/app/login/page.tsx`: login form
|
||||
- `apps/web/app/[section]/[...slug]/page.tsx`: `?edit=1` inline edit, TOC, dark mode, Edit button
|
||||
- `apps/web/app/docs/page.tsx`: docs index listing
|
||||
- `apps/web/app/components/TOC.tsx`: auto-generated TOC from h2/h3 headings
|
||||
- `apps/web/app/components/ThemeToggle.tsx`: dark mode toggle (localStorage + system default)
|
||||
|
||||
#### New deps (minimal — only for UX):
|
||||
- `rehype-slug` (heading anchors for TOC links)
|
||||
- `github-slugger` (matching slug algorithm for TOC client-side)
|
||||
|
||||
### Gotchas (learned the hard way)
|
||||
1. tRPC fetch adapter expects input directly as JSON body, NOT JSON-RPC envelope
|
||||
2. requireWriteAuth compared `ctx.webhookSecret !== WEBHOOK_SECRET` (module-level env constant) — untestable. Fixed: `ctx.webhookSecret !== ctx.expectedSecret` (injected per-app via deps).
|
||||
3. Next.js catch-all routes: `[...slug]/edit/` is INVALID (catch-all must be last). Used `?edit=1` query param instead.
|
||||
4. Next.js App Router: PUT/DELETE on `/api/docs/route.ts` doesn't match `/api/docs/{slug}` — need dynamic route `/api/docs/[...slug]/route.ts`.
|
||||
5. `@env.example` should be updated.
|
||||
6. `ADMIN_PASSWORD` must be set in VPS `.env` (deployed separately).
|
||||
|
||||
### Verification
|
||||
- Typecheck: ✅ 4/4 apps green
|
||||
- Tests: ✅ 40 tests green (32 original + 8 new), no DB/Redis
|
||||
- Build: ✅ web compiled
|
||||
- CI: ✅ success → Deploy: ✅ success
|
||||
- Live: all 9 endpoints 200, 13 MCP tools live, CRUD e2e verified (login → create → view → delete via cookie auth), MCP create_document verified via header auth
|
||||
- Test docs cleaned up (404 confirmed)
|
||||
|
||||
### Commits
|
||||
1. `57f9001` feat: Phase 11 — CRUD + auth + web UI
|
||||
2. `1dd16eb` feat(web): Phase 11 UI/UX — TOC, dark mode toggle, /docs index
|
||||
3. `98437cb` fix(web): /api/docs accepts cookie OR header (not both required)
|
||||
4. `8ed8c67` fix(web): split PUT/DELETE into /api/docs/[...slug]/route.ts
|
||||
5. `f8b525d` chore: remove test docs
|
||||
@@ -1,24 +0,0 @@
|
||||
# Phase 3 — Deploy + git-sync wiring (remaining work)
|
||||
|
||||
Status: Phase 3/4 code is DONE and e2e-verified (webhook enqueue -> worker drain, 0 failed).
|
||||
What was missing on the host: API + worker never ran as systemd services, and the GitHub
|
||||
push webhook was never created. Also a real integration bug: `assertWebhookAuth` only
|
||||
accepts a plain `x-webhook-secret` header, which GitHub does NOT send (GitHub delivers
|
||||
`X-Hub-Signature-256` = HMAC-SHA256 of raw body). So a real GitHub webhook would 401.
|
||||
|
||||
## Changes
|
||||
1. `apps/api/src/index.ts` — `assertWebhookAuth` now verifies GitHub `X-Hub-Signature-256`
|
||||
(HMAC-SHA256 of raw body w/ WEBHOOK_SECRET) and still accepts `x-webhook-secret` for manual tests.
|
||||
2. root `package.json` scripts — `api`: `bun --cwd apps/api run dev` -> `bun --cwd apps/api src/index.ts`
|
||||
(the `run dev` form errors in bun 1.3.14; direct-file form verified booting + health). `worker` -> same form for consistency.
|
||||
3. systemd — `cp deploy/*.service /etc/systemd/system`, `daemon-reload`, `enable --now mcpedia-api mcpedia-worker`.
|
||||
4. Caddy — expose `/hooks/*` on `wiki.asepharyana.my.id` -> :4020 (no new DNS). Keep web on :4016.
|
||||
5. GitHub webhook — `gh api repos/asepharyana/mcpedia/hooks` POST:
|
||||
`https://wiki.asepharyana.my.id/hooks/reindex`, content_type json, secret=WEBHOOK_SECRET, events=push.
|
||||
|
||||
## Verification
|
||||
- `systemctl is-active mcpedia-api mcpedia-worker` == active.
|
||||
- `curl /health` on :4020 -> ok.
|
||||
- `curl -X POST https://wiki.asepharyana.my.id/hooks/reindex -H "X-Hub-Signature-256: ..."` (or x-webhook-secret) -> 200 + jobId; worker drains.
|
||||
- `gh api .../hooks` lists the webhook.
|
||||
- `turbo run typecheck` green; commit + push.
|
||||
@@ -1,114 +0,0 @@
|
||||
# MCPedia — Phase 3 "Async + Scale" Implementation Plan
|
||||
|
||||
Status: Phase 1 (MVP) + Phase 2 (Semantic+API) DONE. Phase 3 adds async
|
||||
background work, git-driven reindex, document revision history, and MCP
|
||||
Resources. All logic stays in `@mcpedia/core`; new `packages/queue` wires
|
||||
BullMQ; `apps/worker` runs the worker process; the existing API gets a git-sync
|
||||
webhook + job-status procedures; the MCP server gains Resources.
|
||||
|
||||
## Scope (4 features from PHASES.md)
|
||||
|
||||
1. **Redis + BullMQ background indexing/embedding workers**
|
||||
2. **Git synchronization hook** (auto-reindex on push via webhook)
|
||||
3. **Document revision system** (`document_revisions`)
|
||||
4. **MCP Resources** (`mcpedia://docs/...`) alongside existing tools
|
||||
|
||||
## Architecture decisions (locked)
|
||||
|
||||
- **Redis**: shared imrnes Redis `100.121.180.82:6379`, no auth (verified
|
||||
`+PONG`). `REDIS_URL` env (default `redis://100.121.180.82:6379`), optional
|
||||
`REDIS_PASSWORD`. BullMQ key prefix `mcpedia:` to avoid collisions on the
|
||||
shared instance.
|
||||
- **Queue lib**: `bullmq@6.1.2` + `ioredis@6.0.0` (BullMQ peer dep). Pass an
|
||||
ioredis instance; BullMQ duplicates it for blocking commands.
|
||||
- **Single source of truth preserved**: per-doc indexing logic moves into
|
||||
`@mcpedia/core` as `indexContentFile(relPath, reason?)`. The script, the
|
||||
worker, and the git hook ALL call this. Revisions are snapshotted inside it.
|
||||
- **Revisions**: created only when body actually changes vs the latest revision
|
||||
(avoids bloat on every sync). Stored in `document_revisions`.
|
||||
|
||||
## Files touched
|
||||
|
||||
### packages/config
|
||||
- `src/index.ts`: add `REDIS_URL`, `REDIS_PASSWORD`, `QUEUE_PREFIX`.
|
||||
|
||||
### packages/db
|
||||
- `src/schema.ts`: add `documentRevisions` table
|
||||
(id, documentId→documents.id cascade, slug, revisionNo int, title, body,
|
||||
meta jsonb, reason text, createdAt). Index (document_id, revision_no DESC),
|
||||
(slug).
|
||||
- `drizzle/0002_document_revisions.sql`: migration (applied via psql).
|
||||
- `drizzle/meta/0002_snapshot.json` + `_journal.json` entry (keeps drizzle-kit
|
||||
consistent even though we apply manually).
|
||||
|
||||
### packages/core (new)
|
||||
- `src/index.service.ts`:
|
||||
- `indexContentFile(relPath: string, reason = "index")` — parse → upsert
|
||||
`documents` → `indexChunks` → snapshot revision (if changed).
|
||||
- `runFullIndex(reason?)` — walk content, index each, return counts.
|
||||
- `src/revision.service.ts`:
|
||||
- `createRevision(...)`, `listRevisions(slug, limit)`,
|
||||
`getRevision(id)`, `latestRevisionBody(slug)`, `restoreRevision(id)`.
|
||||
- `src/index.ts`: export both.
|
||||
|
||||
### packages/queue (NEW)
|
||||
- `package.json` (@mcpedia/queue): deps bullmq, ioredis, @mcpedia/core,
|
||||
@mcpedia/db, @mcpedia/config.
|
||||
- `src/client.ts`: ioredis instance factory from config.
|
||||
- `src/queue.ts`:
|
||||
- `INDEX_QUEUE = "mcpedia-index"`.
|
||||
- `enqueueIndexDoc(slug, absPath, reason)`, `enqueueFullIndex(reason)`.
|
||||
- `getQueue()` lazy singleton.
|
||||
- `src/worker.ts`: `startWorker()` — BullMQ Worker with 3 job types:
|
||||
`index-doc` (single), `index-all` (full), `reindex` (full, reason=git-push).
|
||||
Graceful shutdown on SIGINT/SIGTERM. Job progress + error handling.
|
||||
|
||||
### apps/worker (NEW)
|
||||
- `package.json` (@mcpedia/worker): script `start: bun src/index.ts`.
|
||||
- `src/index.ts`: `startWorker()` + heartbeat log.
|
||||
|
||||
### apps/api
|
||||
- `src/index.ts`: add `POST /hooks/reindex` (full) and
|
||||
`POST /hooks/index?slug=` (single) webhook routes → enqueue jobs. Mount
|
||||
AFTER /trpc.
|
||||
- `src/router.ts`: add `jobStatus` (id→state/prev/failedData),
|
||||
`queueStatus` (waiting/active/completed/failed counts),
|
||||
`revisions` (slug→list), `restoreRevision` (id→new slug/doc).
|
||||
- `package.json`: add `@mcpedia/queue` dep, `hooks` reused.
|
||||
|
||||
### apps/mcp
|
||||
- `src/index.ts`: register Resources:
|
||||
- `mcpedia://docs` (list all metas)
|
||||
- `mcpedia://docs/{slug}` (full body from disk)
|
||||
- `mcpedia://docs/{slug}/chunks` (chunk previews)
|
||||
- `mcpedia://docs/{slug}/revisions` (revision list)
|
||||
- `src/smoke.test.ts`: add `listResources` + read `mcpedia://docs` assertion.
|
||||
|
||||
### scripts
|
||||
- `scripts/indexer.ts`: refactor `main()` to call `runFullIndex()`.
|
||||
|
||||
### Root
|
||||
- `package.json`: add `"worker": "bun --cwd apps/worker run start"`,
|
||||
`"reindex": "bun run scripts/worker.ts"`? No — `worker` runs the listener;
|
||||
triggering reindex = `bun run api` webhook or `enqueueFullIndex` helper.
|
||||
Add `"enqueue-index": "bun run scripts/enqueue.ts"` (one-shot enqueue).
|
||||
- `.env.example`: add `REDIS_URL`, `REDIS_PASSWORD`, `QUEUE_PREFIX`.
|
||||
|
||||
### Docs
|
||||
- `PHASES.md`: mark Phase 3 items DONE with notes.
|
||||
- `README.md`: document worker, webhook, revisions, MCP resources.
|
||||
|
||||
## Verification (real, not claimed)
|
||||
|
||||
1. `bun install` picks up new deps.
|
||||
2. `bunx turbo run build` + `typecheck` green across workspace.
|
||||
3. **Real BullMQ e2e against imrnes Redis**: script that enqueues an
|
||||
`index-doc` job, starts a Worker, asserts the job completes and the doc row
|
||||
+ chunks + a revision row appear in Postgres. Verifies Redis+ioredis+bullmq
|
||||
+ db + core all wired correctly.
|
||||
4. `bun --cwd apps/mcp run smoke` passes (incl. new resources).
|
||||
5. `bun run index` (runFullIndex) green; verify `documents`,
|
||||
`document_chunks`, `document_revisions` row counts via psql.
|
||||
6. API webhook: `curl -XPOST localhost:4020/hooks/reindex` enqueues; worker
|
||||
processes; `curl localhost:4020/trpc/queueStatus` reflects counts.
|
||||
7. MCP resource read returns real content.
|
||||
@@ -1,105 +0,0 @@
|
||||
# MCPedia Phase 4 — Operability & Correctness Hardening
|
||||
|
||||
> Reinterpretation: the PHASES.md "Scale-out" items (OpenSearch, object storage,
|
||||
> multi-tenant, distributed workers) are YAGNI at KB scale (4 docs). Phase 4 =
|
||||
> make the Phase 3 async + revision machinery **correct, secure, observable, and
|
||||
> deployable** — not speculative infra. Each task below fixes a real gap found
|
||||
> by reading the code, not a hypothetical need.
|
||||
|
||||
## Tasks
|
||||
|
||||
### T1 — `restoreRevision` must rebuild semantic chunks (CORRECTNESS BUG)
|
||||
**Root cause:** `packages/core/src/revision.service.ts` `restoreRevision` writes
|
||||
the old body back into `documents` but never calls `indexChunks(slug, body)`.
|
||||
So after a restore, keyword search (FTS on `documents.body`) is correct but
|
||||
`document_chunks`/embeddings stay on the *new* body → semantic + hybrid search
|
||||
return stale/ghost chunks.
|
||||
|
||||
**Fix:**
|
||||
- Add `reindexChunks(slug)` to `@mcpedia/core` that re-runs `indexChunks(slug, body)`
|
||||
using the live `documents.body` (the new body after the update).
|
||||
- Call it inside `restoreRevision` after the `documents` update (wrap in try/catch
|
||||
like `indexContentFile` so embed failure doesn't abort the restore).
|
||||
- Add a unit-style assertion to the MCP smoke test or a small script: restore →
|
||||
`document_chunks` count matches re-chunked body.
|
||||
|
||||
**Files:** `packages/core/src/revision.service.ts`, `packages/core/src/index.ts`,
|
||||
`packages/core/src/document.service.ts` (export existing `indexChunks` if needed).
|
||||
|
||||
### T2 — Secure the git-sync webhook (SECURITY)
|
||||
**Root cause:** `apps/api/src/index.ts` `/hooks/reindex` and `/hooks/index` accept
|
||||
any request with no `WEBHOOK_SECRET` check — `.env.example` defines `WEBHOOK_SECRET`
|
||||
but the router never reads it.
|
||||
|
||||
**Fix:**
|
||||
- In `apps/api/src/index.ts`, compare `c.req.header("x-webhook-secret")` (or
|
||||
`?secret=`) against `WEBHOOK_SECRET` (from `@mcpedia/config`). If unset/mismatch →
|
||||
`401`. If `WEBHOOK_SECRET` env is empty, reject at startup with a clear log
|
||||
(fail-fast, don't run an open endpoint).
|
||||
- Add `WEBHOOK_SECRET` to `packages/config/src/index.ts` export.
|
||||
- Document the header in README + verify with curl (401 without secret, 200 with).
|
||||
|
||||
**Files:** `apps/api/src/index.ts`, `packages/config/src/index.ts`, README.
|
||||
|
||||
### T3 — Web UI revisions view (UX)
|
||||
**Root cause:** Web UI (server components) calls `@mcpedia/core` directly; there is
|
||||
no revisions surface even though `revisions`/`restoreRevision` tRPC + MCP resource
|
||||
exist.
|
||||
|
||||
**Fix (server-component only, no client JS):**
|
||||
- On the doc page (`apps/web/app/[section]/[...slug]/page.tsx`), fetch
|
||||
`listRevisions(fullSlug, 10)` and render a "History" panel: revision number,
|
||||
reason, createdAt, body length, and a `/api/revisions/restore` link/POST that
|
||||
calls the tRPC `restoreRevision` mutation via a server action or a form POST to
|
||||
a small route handler. Simplest: a `<form method="post" action="/api/revisions/restore">`
|
||||
with hidden `id` + a route handler in `apps/web` calling `restoreRevision`.
|
||||
Keep it read-mostly; restore is a deliberate action.
|
||||
- Add `apps/web/app/api/revisions/restore/route.ts` (POST) → `restoreRevision(id)`
|
||||
→ `revalidatePath` the doc.
|
||||
|
||||
**Files:** `apps/web/app/[section]/[...slug]/page.tsx`,
|
||||
`apps/web/app/api/revisions/restore/route.ts`.
|
||||
|
||||
### T4 — Paginate `listRevisions` / `revisions` API (PERF)
|
||||
**Root cause:** `revision.service.ts` `listRevisions` does `select length(body)`
|
||||
(fine) but the tRPC `revisions` and MCP resource return *full* revision rows
|
||||
including the body in some callers; list endpoints should never carry bodies.
|
||||
|
||||
**Fix:**
|
||||
- Ensure `listRevisions` summary excludes `body` (it already does — `bodyLength`
|
||||
only). Add `offset` param for paging. Confirm MCP resource uses the summary.
|
||||
- No behavior change for the doc page (uses summary).
|
||||
|
||||
**Files:** `packages/core/src/revision.service.ts` (add `offset`), router unchanged.
|
||||
|
||||
### T5 — Deployable as supervised services (OPS)
|
||||
**Root cause:** `apps/api` and `apps/worker` run only ad-hoc; the host already runs
|
||||
`zeavis` via Nix/systemd. Phase-4 operability = provide a systemd unit (or Nix
|
||||
service) so `mcpedia-api` + `mcpedia-worker` start on boot and restart on failure.
|
||||
|
||||
**Fix (Nix-first, per MEMORY):**
|
||||
- Write `mcpedia-api.service` + `mcpedia-worker.service` systemd unit files under
|
||||
`deploy/` (bun run api / bun run worker, `WorkingDirectory`, `Restart=on-failure`,
|
||||
`EnvironmentFile` pointing at `.env`, `After=network-online.target`).
|
||||
- README section "Run as a service" with `cp deploy/*.service /etc/systemd/system && systemctl daemon-reload && systemctl enable --now mcpedia-api mcpedia-worker`.
|
||||
- **Do NOT** `systemctl` on the host without user confirmation (changing live
|
||||
services). Provide the files + instructions only; user runs enable.
|
||||
|
||||
**Files:** `deploy/mcpedia-api.service`, `deploy/mcpedia-worker.service`, README.
|
||||
|
||||
## Verification (all real, against imrnes Redis + Postgres)
|
||||
1. `turbo run typecheck` + `turbo run build` green.
|
||||
2. T1: script — edit a doc, reindex (new revision + new chunks), restore rev #1,
|
||||
assert `document_chunks` count for that slug now matches re-chunk of rev #1 body
|
||||
and `semanticSearch` on a term unique to rev #1 returns it.
|
||||
3. T2: `curl -XPOST localhost:4020/hooks/reindex` → 401; with
|
||||
`-H "x-webhook-secret: $WEBHOOK_SECRET"` → 200 + jobId.
|
||||
4. T3: `next build` includes the History panel; restore form rebuilds chunks
|
||||
(verified via T1 path through the route handler).
|
||||
5. T4: `revisions` API returns summaries without body; offset paging works.
|
||||
6. T5: `systemd-analyze verify deploy/*.service` passes (off-host safe check);
|
||||
README documents enable steps.
|
||||
|
||||
## Out of scope (YAGNI, keep deferred per PHASES.md)
|
||||
OpenSearch/Elasticsearch, object storage, multi-tenant, distributed workers,
|
||||
pgvector migration. Revisit only when corpus > ~10k docs or query latency bites.
|
||||
@@ -1,41 +0,0 @@
|
||||
# MCPedia — "lanjut semua" workstream
|
||||
|
||||
Three real gaps remain (from review): tiny corpus (4 docs), MCP read-only (no write/auth),
|
||||
no observability. This plan closes all three.
|
||||
|
||||
## 1. Content corpus (grow the KB)
|
||||
Author real, useful docs so search/semantic/revisions have something to operate on.
|
||||
Frontmatter schema (from packages/parser): id,title,type,tags,status,author,created_at,updated_at.
|
||||
Sections: docs|writeups|research|notes. Files under content/<section>/...
|
||||
New docs to add:
|
||||
- content/docs/caddy/reverse-proxy.md (ops reference, tags: caddy, reverse-proxy, tls)
|
||||
- content/docs/bullmq/workers.md (queue/worker reference, tags: bullmq, redis, jobs)
|
||||
- content/docs/mcp/streamable-http.md (MCP transport reference, tags: mcp, protocol, http)
|
||||
- content/notes/postgres/full-text-search.md (PG FTS notes, tags: postgres, fts, tsvector)
|
||||
- content/writeups/infra/cloudflare-525.md (debugging writeup, tags: cloudflare, tls, 525)
|
||||
After adding: `bun run index` to reindex (writes revisions + chunks), verify counts.
|
||||
|
||||
## 2. MCP write-tools + auth
|
||||
Add mutating + admin tools to the MCP server (currently read-only):
|
||||
- `index_document(slug)` -> enqueueIndexDoc (requires MCP auth header)
|
||||
- `reindex_all()` -> enqueueFullIndex (requires MCP auth header)
|
||||
- `queue_status()` -> getQueue counts (read, public)
|
||||
- `restore_revision(id)` -> restoreRevision (requires MCP auth header)
|
||||
Auth: MCP client must send header `x-webhook-secret` (reuse WEBHOOK_SECRET). StreamableHTTP
|
||||
transport: read the Authorization/header in http.ts, pass to server via a factory closure
|
||||
capturing the request; tools check it. Stateless per-request server already created fresh,
|
||||
so threading the header is clean. Guard write-tools with the same requireWriteAuth logic.
|
||||
Verify: unauthenticated call to index_document -> error; authenticated -> enqueues job.
|
||||
|
||||
## 3. Observability
|
||||
- `GET /metrics` on the API (Prometheus text format): queue counts (waiting/active/
|
||||
completed/failed/delayed), uptime, service name. Public (safe to expose).
|
||||
- Caddy: expose /metrics on wiki. domain -> :4020 (add to handle list).
|
||||
- tRPC `queueStatus` already exists; /metrics reuses getQueue.
|
||||
Verify: curl /metrics -> text exposition with mcpedia_queue_* gauges.
|
||||
|
||||
## Verification
|
||||
- typecheck green (turbo run typecheck).
|
||||
- MCP smoke extended: tools/list shows new tools; authenticated index_document enqueues.
|
||||
- /metrics returns 200 text; queue drains.
|
||||
- commit + push.
|
||||
@@ -1,157 +0,0 @@
|
||||
# MCPedia Phase 9 — Test Coverage + Observability Hardening
|
||||
|
||||
**Goal:** Add a real test suite (CI-gated) covering every layer of MCPedia — pure
|
||||
logic, Core services, the API surface, the MCP server + auth gates, and the Web
|
||||
UI — so regressions are caught before deploy. The suite must run green in CI
|
||||
with **no external services** (no Postgres, no Redis) by using in-process fakes.
|
||||
|
||||
## Constraints recap
|
||||
- Tooling: bun workspaces + Turborepo, bun 1.3.14 has a built-in `bun:test` runner.
|
||||
- DB is `imrnes` Postgres at `:6432` (no DB in CI) — tests must NOT touch it.
|
||||
- pgvector is NOT installed (vectors are `real[]`, cosine in-app) — confirmed.
|
||||
- Existing smoke test (`apps/mcp/src/smoke.test.ts`) is a script with `main()`
|
||||
run via `bun run smoke`, NOT a `bun:test` file. It hits the DB → cannot run in CI.
|
||||
- Secrets (`WEBHOOK_SECRET`, `DATABASE_URL`, `EMBED_*`) live in `.env` (gitignored)
|
||||
or BWS for deploy — never in tests or committed config.
|
||||
|
||||
## Decisions
|
||||
|
||||
1. **Runner:** `bun:test` — zero-config, built into the bun 1.3.14 toolchain
|
||||
already used. No extra deps. Add a `test` task to `turbo.json` and `package.json`
|
||||
scripts; add a `Test` step to CI.
|
||||
2. **No live DB in CI.** Tests that would need Postgres/Redis/Embeddings use
|
||||
**in-process fakes** (memory stores + a stub embedder returning fixed vectors).
|
||||
This means Core service tests cannot use the real `@mcpedia/db` singleton —
|
||||
they must accept an injected DB (drizzle-pg mem or a hand-rolled fake). We will
|
||||
**refactor the Core services' DB access behind injectable handles** where cheap,
|
||||
and for the MCP/HTTP auth-layer tests we stub `@mcpedia/queue` + `@mcpedia/core`
|
||||
at the module boundary (the transport/auth logic does not need a real queue).
|
||||
3. **Test boundaries by package:**
|
||||
- `packages/embeddings` — pure: `chunkText`, `cosine`. Real assertions, no I/O.
|
||||
- `packages/search` — pure: `toTsQuery`, `cosine`. SQL-bearing functions
|
||||
(`keywordSearch`/`semanticSearch`/`hybridSearch`) tested via a **fake db**
|
||||
injected into `@mcpedia/db`, OR via the `cosine`/fusion helpers in isolation.
|
||||
- `packages/core` — `snapshotRevision` dedup logic (refactor to accept an inject
|
||||
fn or test the public `indexContentFile`/`restoreRevision` with fakes).
|
||||
Focus: revision-dedup correctness + `restoreRevision` triggers `reindexChunks`.
|
||||
- `apps/api` — Hono app: `/health`, `/metrics` shape, `/hooks/*` auth (401 w/o
|
||||
secret, 200 + enqueue w/ secret using a fake queue), tRPC `restoreRevision`
|
||||
mutation auth gate (401 w/o secret).
|
||||
- `apps/mcp` — auth gates on write tools: `index_document`/`reindex_all`/
|
||||
`restore_revision` error without secret, enqueue with secret (fake queue).
|
||||
Read tools + resources via `InMemoryTransport` (reuse smoke style but without
|
||||
DB).
|
||||
- `apps/web` — render correctness of home (lists sections), doc page
|
||||
(renders title + markdown + history panel when revisions exist), search page
|
||||
(keyword/hybrid toggle, empty state). These need a fake Core.
|
||||
|
||||
## Approach per test (minimal, high-signal)
|
||||
|
||||
### embeddings: `packages/embeddings/src/chunk.test.ts`
|
||||
- `chunkText("hello world")` with short size → single chunk.
|
||||
- `chunkText` long text → multiple chunks, overlap honored, no word splits past boundary.
|
||||
- `chunkText("")` / `" "` → `[]`.
|
||||
|
||||
### search: `packages/search/src/cosine.test.ts` (new tiny file) + refactor
|
||||
- `cosine([1,0],[0,1])` ≈ 0; `cosine([1,1],[1,1])` = 1; `cosine([],[1])` = 0.
|
||||
- `toTsQuery("a b c")` → `"a:* & b:* & c:*"`; empty/garbage → `""`.
|
||||
|
||||
### core: `packages/core/src/index.service.test.ts`
|
||||
The hard part: `indexContentFile`/`restoreRevision`/`snapshotRevision` call `db`
|
||||
directly. Two options:
|
||||
- **Option A (chosen):** extract `snapshotRevision`'s "latest body" + "insert"
|
||||
steps behind the existing `db` but make `indexContentFile` test the revision
|
||||
*decision* by inserting a doc + revision directly via `db` in a test Postgres
|
||||
(too heavy for CI).
|
||||
- **Option B (chosen):** test the **pure decision logic** by refactoring
|
||||
`snapshotRevision` to export a pure helper
|
||||
`shouldCreateRevision(latestBody, body): boolean` — `true` when latest is null
|
||||
or latest.body !== body. Then a unit test asserts the dedup truth table;
|
||||
`indexContentFile` is verified by the existing e2e (manual `bun run index`).
|
||||
This is the CI-safe win.
|
||||
- `restoreRevision` correctness: assert it calls `reindexChunks(slug)` — we can
|
||||
test by spying. Since `reindexChunks` is in the same module, we'll export a
|
||||
seam: `restoreRevision(id, { reindexChunks: spy })` — keep backward compat by
|
||||
defaulting. (Or test the public contract via the API layer instead.)
|
||||
|
||||
### api: `apps/api/src/index.test.ts`
|
||||
- Build the Hono `app` from a testable factory that accepts a fake queue + fake
|
||||
webhook secret. Current `index.ts` throws at import if `WEBHOOK_SECRET` unset —
|
||||
that breaks import in CI. **Refactor:** move the fail-fast check into
|
||||
`listen()`/serve start, so the app is constructable without a secret for
|
||||
testing. Export `createApp(opts?)` returning the Hono instance.
|
||||
- `/health` → 200 `{ok:true}`.
|
||||
- `/metrics` → 200, text/plain, contains `mcpedia_uptime_seconds` +
|
||||
`mcpedia_queue_jobs` for each state (fake queue returns 0/1).
|
||||
- `POST /hooks/reindex` w/o `x-webhook-secret` → 401; with matching secret →
|
||||
200 + `{ok:true, jobId}` (fake queue records the enqueue).
|
||||
- tRPC: build a client against the app, call `restoreRevision` without secret →
|
||||
error; the public `listDocuments` returns from a fake DB.
|
||||
|
||||
### mcp: `apps/mcp/src/auth.test.ts`
|
||||
- `createMcpServer()` (no secret) → `index_document`/`reindex_all`/`restore_revision`
|
||||
throw "unauthorized".
|
||||
- `createMcpServer("secret")` → same tools reach the enqueue call (fake queue).
|
||||
- Read tools still work without secret (server loads, resources list).
|
||||
|
||||
### web: `apps/web/app/search/page.test.tsx` (or a lighter harness)
|
||||
- This is the hardest to test without a browser. **Decision:** keep web tests
|
||||
minimal — assert that `toTsQuery`/render helpers exist; full DOM tests deferred
|
||||
(needs playwright + a running server). We'll instead add a **contract test**
|
||||
that the search page's `dynamic = "force-dynamic"` export exists (static-
|
||||
generation guard, the kind of thing that broke CI before).
|
||||
|
||||
## File layout (new files)
|
||||
```
|
||||
packages/embeddings/src/chunk.test.ts
|
||||
packages/search/src/cosine.test.ts
|
||||
packages/core/src/index.service.test.ts # snapshotRevision + restoreRevision seam
|
||||
apps/api/src/index.test.ts # Hono /health /metrics /hooks + tRPC gate
|
||||
apps/mcp/src/auth.test.ts # write-tool auth gates
|
||||
```
|
||||
|
||||
## turbo.json
|
||||
Add a `test` task (like `typecheck`, no dependsOn, cache false so it always runs):
|
||||
```jsonc
|
||||
"test": { "cache": false }
|
||||
```
|
||||
Each app/pkg gets `"test": "bun test"` in its package.json.
|
||||
|
||||
## CI (`.github/workflows/ci.yml`)
|
||||
After `Build`, add:
|
||||
```yaml
|
||||
- name: Test
|
||||
run: bun run test # -> turbo run test
|
||||
```
|
||||
|
||||
## Source changes required (enablers)
|
||||
1. `apps/api/src/app.ts` (NEW) — extracted `createApp(deps?)` factory returning a
|
||||
`Promise<Hono>`. Pure construction (no process exit, no fail-fast). Accepts
|
||||
injected `ApiDeps` (`{ queue, webhookSecret }`); when omitted, lazily
|
||||
imports the real queue + uses `WEBHOOK_SECRET` (production path). `/dashboard`
|
||||
now renders the HTML from a new `apps/api/src/dashboard.ts` module.
|
||||
2. `apps/api/src/dashboard.ts` (NEW) — the self-contained dashboard HTML, split
|
||||
out of the original `index.ts` so the route is testable + the const is
|
||||
importable. XSS-safe (esc() on all KB-sourced fields; documented in comment).
|
||||
3. `apps/api/src/index.ts` — now a thin re-export of `createApp`/`start` + the
|
||||
`isMain` bootstrap. Systemd unit still runs `bun --cwd apps/api src/index.ts`.
|
||||
4. `packages/core/src/index.service.ts` — exported `shouldCreateRevision` (pure
|
||||
predicate for the revision dedup invariant). `restoreRevision` in
|
||||
`revision.service.ts` now accepts an optional `opts.reindex` seam (defaults
|
||||
to the real `reindexChunks`), making the chunk-rebuild contract testable.
|
||||
5. `apps/mcp/src/smoke.ts` — renamed from `smoke.test.ts` (so `bun test` doesn't
|
||||
treat the integration smoke as a unit run), and fixed stale assertions:
|
||||
expected tool set updated to all 10 tools (Phase 7 additions + write tools),
|
||||
`list_documents(section=docs)` count updated to 4 (post-Phase-7 corpus).
|
||||
|
||||
## Verification
|
||||
- `bun run test` (local) → 32 tests green across 6 packages (embeddings 5,
|
||||
search 8, core 4, parser 5, mcp 6, api 8), **no live DB needed** (mocks
|
||||
stub `@mcpedia/db`, `@mcpedia/queue`, `@mcpedia/core`).
|
||||
- `bun run typecheck` → green (4 apps, no test-only type errors).
|
||||
- `bun --cwd apps/mcp run smoke` → green (integration, needs live DB — runs in
|
||||
CI on the deploy host, not in CI's no-services job).
|
||||
- Live API check: `/health`, `/metrics`, `/dashboard`, `/hooks/*` auth gate
|
||||
all 200/401-verified against a temp-port server.
|
||||
- Commit + push; worker redeploy not needed (code + tests only).
|
||||
|
||||
@@ -1,80 +0,0 @@
|
||||
# Phase 15 — UI/UX Polish Pass
|
||||
|
||||
> User: "perbagus ui ux nya"
|
||||
|
||||
## Audit
|
||||
|
||||
### Current issues
|
||||
1. **Section index** (`[section]/page.tsx`):
|
||||
- `buildFolderTree` uses path segments instead of doc titles for leaf nodes
|
||||
- Folder tree + flat list at bottom is redundant
|
||||
- No folder icons (📁), no doc type indicators, no dates
|
||||
- Tree rendering is very basic (no visual hierarchy)
|
||||
|
||||
2. **Doc page** (`[section]/[...slug]/page.tsx`):
|
||||
- CustomFieldBadges shows key=value but labels all badges with key name as title
|
||||
- No structured metadata card layout
|
||||
- Related docs cards are functional but visually flat
|
||||
|
||||
3. **Folder index** (`FolderIndexPage` in `[...slug]/page.tsx`):
|
||||
- Shows doc slug parts (e.g. "pwn-100-ret2win-alignment") instead of titles
|
||||
- No date, no tags, no description
|
||||
- Subfolders are plain text links, no folder count or doc count
|
||||
|
||||
4. **Homepage** (`page.tsx`):
|
||||
- Folder tree inside cards is cramped (text-xs, no spacing)
|
||||
- Section cards show "View all (N)" but folder tree duplicates that
|
||||
|
||||
5. **Sidebar** (`Sidebar.tsx`):
|
||||
- Tree uses indentation via inline style but no visual depth cues
|
||||
- No hover expand for folders
|
||||
- Active state only on exact match, not parent folders
|
||||
|
||||
## Plan
|
||||
|
||||
### 1. Section index — enhanced tree
|
||||
- Use doc titles for leaf nodes (already have `doc` reference in tree)
|
||||
- Add 📁 for folders, 📄 for docs
|
||||
- Show doc count per folder
|
||||
- Remove flat list (redundant with tree)
|
||||
- Add "View all" link per section
|
||||
- Better visual hierarchy: folder headers with counts, doc titles with dates
|
||||
|
||||
### 2. Doc page — metadata card
|
||||
- Replace inline badge row with a structured metadata card
|
||||
- Show custom fields as labeled badges (key → value with color)
|
||||
- Keep CustomFieldBadges for backward compat but improve layout
|
||||
- Add metadata card with: author, date, tags, and all custom fields
|
||||
|
||||
### 3. Folder index — rich listing
|
||||
- Show doc titles (fetch from listDocuments, match by path)
|
||||
- Show update date per doc
|
||||
- Show tags per doc
|
||||
- Add doc count for subfolders
|
||||
- Better visual separation between subfolders and docs
|
||||
|
||||
### 4. Homepage — cleaner tree
|
||||
- Increase font size for tree items
|
||||
- Show section icon + label in tree header
|
||||
- Better spacing between sections
|
||||
|
||||
### 5. Sidebar — depth cues
|
||||
- Use ml-4 per level instead of inline style
|
||||
- Add section divider lines
|
||||
- Highlight active section + parent folders
|
||||
|
||||
## Files to change
|
||||
```
|
||||
mod: apps/web/app/[section]/page.tsx # enhanced tree, use doc titles
|
||||
mod: apps/web/app/[section]/[...slug]/page.tsx # FolderIndexPage + metadata card
|
||||
mod: apps/web/app/page.tsx # cleaner section tree cards
|
||||
mod: apps/web/app/components/Sidebar.tsx # depth + active parent highlight
|
||||
mod: apps/web/app/globals.css # additional CSS vars if needed
|
||||
```
|
||||
|
||||
## Verification
|
||||
- All endpoints still 200
|
||||
- `bun run test` green
|
||||
- `turbo run typecheck` green
|
||||
- `next build` succeeds
|
||||
- Live visual check of folder index, section index, doc page
|
||||
Reference in New Issue
Block a user