feat(mcpedia): Phase 2 — semantic + hybrid search, tRPC/Hono API

- @mcpedia/embeddings: OpenRouter provider (9router /v1, encoding_format float),
  chunkText + embedChunks; EMBED_DIM=2048
- document_chunks table (real[] embedding) — pgvector NOT available on shared
  imrnes Postgres, so cosine is computed in-app (KB-scale fine); pgvector deferred
- indexer: chunk + embed + upsert per document
- @mcpedia/search: semanticSearch (cosine) + hybridSearch (FTS+cosine RRF)
- apps/api: Hono + tRPC v11 (6 procedures), serve on :4020
- MCP: semantic_search + hybrid_search tools (6 total)
- web: keyword/hybrid toggle; .env.example + README + PHASES updated

.hermes/plans/2026-08-19_phase2-semantic-api.md

@@ -0,0 +1,161 @@
+# MCPedia Phase 2 — Semantic Search + tRPC/Hono API
+
+> **For Hermes:** implement task-by-task. Spec-first (user rule 2026-08-19).
+
+**Goal:** Add semantic + hybrid search (pgvector) and a typed tRPC/Hono API so
+MCPedia is queryable by embeddings, not just keyword FTS — and expose the
+corpus over a programmatic HTTP API.
+
+**Architecture:** Content (Markdown) → chunk → embed (OpenRouter) → store
+`document_chunks` with `vector(N)` in Postgres → `semanticSearch` (cosine) and
+`hybridSearch` (FTS + cosine, reciprocal-rank fusion) in `@mcpedia/search` →
+exposed via Core, the MCP server (new tools), and a new `apps/api` (Hono +
+tRPC v11).
+
+**Embedding provider:** OpenRouter (`openrouter/llama-nemotron-embed-vl-1b-v2:free`)
+via `9router_ai_llm_api_key` + `9router_ai_llm_base_url` (BWS). Dimension is
+discovered at first live call (see Step 1.3) and pinned in schema/migration.
+
+**Tech stack:** drizzle-orm `vector` column + pgvector extension, HNSW index,
+`@trpc/server` v11 (fetch adapter), `hono` + `@hono/node-server`.
+
+---
+
+## Task P2.1 — `packages/embeddings` (provider + abstraction)
+
+**Files:** `packages/embeddings/package.json`, `src/index.ts`, `src/provider.ts`,
+`src/openrouter.ts`
+
+- `EmbeddingProvider` interface: `embed(texts: string[]): Promise<number[][]>`, `readonly model`, `readonly dimensions`.
+- `OpenRouterEmbeddingProvider`: POST `${baseUrl}/embeddings` with `{ model, input }`,
+  `Authorization: Bearer ${key}`. Returns `data[].embedding`. Validate length === dimensions.
+- Read `EMBED_BASE_URL`, `EMBED_API_KEY`, `EMBED_MODEL` from `@mcpedia/config`
+  (with `.env` fallback). Dimensions discovered live (Step 1.3) → export `EMBED_DIM`.
+- Chunk helper `chunkText(text, { size=1000, overlap=150 })` in `src/chunk.ts`.
+
+**Step 1.3 (discover dim):** live call `embed(["test"])`, read `embedding.length`,
+pin `EMBED_DIM`, assert mismatch throws.
+
+**Verify:** `bun run` a temp script: `embed(["hello world"])` prints a vector of
+length N (e.g. 1024). Confirm no key is logged.
+
+---
+
+## Task P2.2 — Schema: `document_chunks` + vector extension
+
+**Files:** `packages/db/src/schema.ts` (add), `packages/db/drizzle.config.ts`
+(unchanged), new migration.
+
+- `CREATE EXTENSION IF NOT EXISTS vector;` (idempotent; run once via psql).
+- `document_chunks` table:
+  - `id` uuid pk default gen_random_uuid()
+  - `document_id` text → `documents.id` on delete cascade
+  - `slug` text (denormalized for convenience)
+  - `chunk_index` integer
+  - `content` text
+  - `embedding` vector(EMBED_DIM)
+  - `created_at` timestamp default now()
+  - index `chunk_embedding_idx` using hnsw (`embedding` op `vector_cosine_ops`)
+- Generate migration with `drizzle-kit generate`, apply via `psql` (drizzle-kit
+  push is unreliable here — known).
+
+**Verify:** `\d document_chunks` shows `embedding vector(N)` + HNSW index;
+`select count(*) from document_chunks` = 0.
+
+---
+
+## Task P2.3 — Indexer: chunk + embed + upsert
+
+**Files:** `scripts/indexer.ts` (extend), `packages/core/src/document.service.ts`
+(add `indexChunks`).
+
+- For each published doc: read body (already on disk), `chunkText`, `embed` in
+  batches (≤ 16), delete existing chunks for slug, insert new rows.
+- Guard: if embedding provider fails, log + skip (don't crash the whole index).
+- Add `bun run index:embed` (or extend `bun run index` to also embed).
+
+**Verify:** after running, `select count(*) from document_chunks` > 0; a sample
+row has non-null `embedding`.
+
+---
+
+## Task P2.4 — `packages/search`: semantic + hybrid
+
+**Files:** `packages/search/src/index.ts` (add `semanticSearch`, `hybridSearch`).
+
+- `semanticSearch(vec, limit)`: order by `embedding <=> ${vec}` asc, filter published.
+- `hybridSearch(q, limit)`: run FTS (`ts_rank`) + semantic (cosine) in parallel;
+  fuse with reciprocal-rank (RRF: score = 1/(k+rank), k=60); return merged hits.
+- Keep `keywordSearch` unchanged (Phase 1).
+
+**Verify:** unit-ish script: embed a query, `semanticSearch` returns relevant
+chunks; `hybridSearch("websocket")` returns ≥ keyword results.
+
+---
+
+## Task P2.5 — `packages/core` expose semantic/hybrid
+
+**Files:** `packages/core/src/search.service.ts`, `index.ts`.
+
+- Re-export `semanticSearch`, `hybridSearch` from Core.
+
+---
+
+## Task P2.6 — `apps/api` (Hono + tRPC v11)
+
+**Files:** `apps/api/package.json`, `tsconfig.json`, `src/index.ts`,
+`src/router.ts`, `src/trpc.ts`.
+
+- `initTRPC.create()` router with procedures: `search`, `semanticSearch`,
+  `hybridSearch`, `getDocument`, `listDocuments` (mirrors MCP tools).
+- Mount `fetchRequestHandler` on a Hono app at `/trpc/*`; serve via
+  `@hono/node-server` `serve({ fetch: app.fetch, port: 4020 })`.
+- `createContext` returns `{ db }`.
+
+**Verify:** `bun run dev` → `curl -X POST localhost:4020/trpc/search`
+with JSON body returns hits.
+
+---
+
+## Task P2.7 — MCP server: semantic + hybrid tools
+
+**Files:** `apps/mcp/src/index.ts` (add `semantic_search`, `hybrid_search`),
+extend `smoke.test.ts`.
+
+- `semantic_search`: embed query → `semanticSearch`.
+- `hybrid_search`: embed query → `hybridSearch`.
+- Smoke: assert both return ≥1 hit for "websocket".
+
+---
+
+## Task P2.8 — Web: semantic toggle on search
+
+**Files:** `apps/web/app/search/page.tsx`.
+
+- Add `mode=keyword|hybrid` query param; server component calls Core
+  `hybridSearch` when `mode=hybrid`. Minimal UI toggle (link/buttons).
+- Keep keyword as default.
+
+**Verify:** `bun run build`; `curl '/search?q=websocket&mode=hybrid'` returns hits.
+
+---
+
+## Task P2.9 — Verify all + commit
+
+- `bunx turbo run build` (web + api + mcp), `bun run apps/mcp smoke`,
+  live API curl, live web hybrid search.
+- Update `README.md` + `PHASES.md` (mark Phase 2 ✅).
+- `git add -A` (exclude `.env`), commit as asepharyana (no Co-Authored-By).
+
+---
+
+## Risks / decisions
+- **Dimension unknown until live call** → P2.1.3 discovers it; pinned EMBED_DIM=2048.
+- **pgvector NOT available on shared imrnes Postgres** (extension not installed;
+  installing needs host-level apt on a managed/shared DB — deferred). PIVOT:
+  store `embedding` as `real[]` and compute cosine similarity in the app layer.
+  Brute-force cosine is instant for a KB-sized corpus (dozens of docs / hundreds
+  of chunks). pgvector+HNSW is the Phase-4 scale-out path.
+- **PgBouncer + real[]**: fine; simple queries, no extension needed.
+- **API port 4020** (host 4000s range is 4000–4015; 4020 is free for dev). Deploy later.
+- **YAGNI**: no auth/revisions this phase (Phase 3).

PHASES.md

@@ -19,13 +19,13 @@ Legend: ✅ built · 🟡 partial · ⬜ deferred
 
 ## Phase 2 — Semantic + API
 
-- [ ] `pgvector` + embedding column on `document_chunks`
+- [x] `packages/embeddings` — `EmbeddingProvider` interface + OpenRouter provider (via 9router `/v1`, `encoding_format:"float"`); `chunkText` + `embedChunks` batcher. `EMBED_DIM=2048` discovered live.
-- [ ] Chunking + embedding provider abstraction (`EmbeddingProvider`: OpenAI/Gemini/Ollama/local)
+- [x] Schema `document_chunks` (id, document_id→documents.id cascade, slug, chunk_index, content, `embedding real[]`). Stored as `real[]` because pgvector **is not installed** on the shared imrnes Postgres (installing needs host-level apt — deferred). Cosine computed in-app; instant for a KB-sized corpus.
-- [ ] Hybrid search (FTS score + cosine, reciprocal-rank fusion)
+- [x] `scripts/indexer.ts` — chunks + embeds + upserts (per-doc replace).
-- [ ] tRPC + Hono API (`apps/api`) sharing `@mcpedia/core`
+- [x] `@mcpedia/search` — `semanticSearch` (cosine) + `hybridSearch` (FTS + cosine, RRF fusion). `keywordSearch` unchanged.
-- [ ] Tags / Categories / References as first-class tables
+- [x] `apps/api` — Hono + tRPC v11 (`@trpc/server` fetch adapter, `@hono/node-server` on :4020): `search`, `semanticSearch`, `hybridSearch`, `getDocument`, `listDocuments`, `related`.
-- [ ] Auth (Auth.js / OIDC) — public/private/unlisted/admin/owner
+- [x] MCP server — added `semantic_search` + `hybrid_search` tools (6 total).
-- [ ] shadcn/ui components + Shiki syntax highlighting (replace minimal markdown render)
+- [x] Web search — keyword/hybrid toggle (`?mode=hybrid`), hybrid reaches semantically-related docs keyword misses.
 
 ## Phase 3 — Async + Scale
 

README.md

@@ -51,12 +51,18 @@ bun run mcp                       # MCP server on stdio (pipe to an MCP client)
 
 ### Database
 
-Phase 1 uses Postgres FTS only. Schema is defined in `packages/db/src/schema.ts`
+Schema is defined in `packages/db/src/schema.ts` (`documents` with a weighted
-(a `documents` table with a `search_vector` generated `tsvector` column + GIN
+`search_vector` tsvector + GIN index, and `document_chunks` with an `embedding real[]`).
-index). Apply it with:
+The `pgvector` extension is **not available** on the shared imrnes Postgres, so
+semantic search stores vectors as `real[]` and ranks by in-app cosine similarity.
+
+Migrations live in `packages/db/drizzle/`. They were applied manually via `psql`
+(`drizzle-kit push` is unreliable under PgBouncer transaction pooling); to
+re-apply on a fresh DB:
 
 ```bash
-bunx --cwd packages/db drizzle-kit push
+psql $DATABASE_URL -f packages/db/drizzle/0000_grey_toro.sql
+psql $DATABASE_URL -f packages/db/drizzle/0001_document_chunks.sql
 ```
 
 > Note: on imrnes (PgBouncer `:6432`) a leaked `DATABASE_URL` shell var can
@@ -84,11 +90,13 @@ updated_at: 2026-08-19
 `body` shown in the UI is always read from the on-disk file (source of truth);
 the DB stores metadata + the search vector.
 
-## MCP tools (Phase 1)
+## MCP tools
 
 | Tool                  | Purpose                                          |
 | --------------------- | ------------------------------------------------ |
 | `search_documents`    | Postgres FTS over the corpus (ranked + snippet)  |
+| `semantic_search`     | Embedding/cosine search over chunked content     |
+| `hybrid_search`       | FTS + semantic fused via RRF                      |
 | `get_document`        | Full markdown body by slug                       |
 | `list_documents`      | List, optionally filtered by section             |
 | `get_related_documents` | Docs sharing tags with a given slug            |
@@ -99,10 +107,29 @@ Smoke test (in-memory transport, real JSON-RPC):
 bun --cwd apps/mcp run smoke
 ```
 
+## API (Phase 2)
+
+A tRPC v11 API is also exposed via Hono on **:4020** (all procedures mirror the
+MCP tools):
+
+```bash
+bun run api            # http://localhost:4020 (GET /health, POST/GET /trpc/*)
+```
+
+`bun run index` now also chunks + embeds (Phase 2 indexer). Requires `EMBED_*`
+vars in `.env` (see `.env.example`).
+
 ## Status
 
 **Phase 1 — MVP (DONE):** monorepo, Core, Web UI (home/doc/search), MCP server,
 Postgres FTS keyword search, content indexing.
 
-See `PHASES.md` for Phase 2–4 (pgvector semantic/hybrid search, tRPC/Hono API,
+**Phase 2 — Semantic + API (DONE):** embeddings provider (OpenRouter via 9router),
-auth, Redis/BullMQ background workers, revisions, scale-out).
+chunked `document_chunks`, `semanticSearch` + `hybridSearch` (RRF), tRPC/Hono API
+(`apps/api`, :4020), MCP `semantic_search`/`hybrid_search` tools, web hybrid toggle.
+
+> pgvector is **not installed** on the shared imrnes Postgres, so vector storage is
+> a `real[]` column with in-app cosine similarity (instant at KB scale). pgvector is
+> the Phase-4 scale-out path. See `PHASES.md`.
+
+See `PHASES.md` for Phase 3–4 (Redis/BullMQ, auth, revisions, scale-out).

apps/api/package.json

@@ -0,0 +1,23 @@
+{
+  "name": "@mcpedia/api",
+  "version": "0.1.0",
+  "private": true,
+  "type": "module",
+  "scripts": {
+    "dev": "bun run src/index.ts",
+    "start": "bun run src/index.ts",
+    "lint": "tsc --noEmit",
+    "typecheck": "tsc --noEmit"
+  },
+  "dependencies": {
+    "@hono/node-server": "^1.13.0",
+    "@mcpedia/config": "workspace:*",
+    "@mcpedia/core": "workspace:*",
+    "@trpc/server": "^11.0.0",
+    "hono": "^4.6.0",
+    "zod": "^3.23.8"
+  },
+  "devDependencies": {
+    "typescript": "^5.6.0"
+  }
+}

apps/api/src/index.ts

@@ -0,0 +1,26 @@
+import { serve } from "@hono/node-server";
+import { Hono } from "hono";
+import { fetchRequestHandler } from "@trpc/server/adapters/fetch";
+import { db } from "@mcpedia/db";
+import { appRouter } from "./router";
+import type { Context } from "./trpc";
+
+const app = new Hono();
+
+// Health check.
+app.get("/health", (c) => c.json({ ok: true }));
+
+// Mount tRPC at /trpc/*. The fetch adapter is the canonical Bun/Hono adapter.
+app.all("/trpc/*", (c) =>
+  fetchRequestHandler({
+    endpoint: "/trpc",
+    req: c.req.raw,
+    router: appRouter,
+    createContext: (): Context => ({ db }),
+  }),
+);
+
+const port = Number(process.env.API_PORT ?? 4020);
+serve({ fetch: app.fetch, port }, (info) => {
+  console.log(`MCPedia API listening on http://localhost:${info.port}`);
+});

apps/api/src/router.ts

@@ -0,0 +1,38 @@
+import { z } from "zod";
+import { publicProcedure, router } from "./trpc";
+import {
+  getDocument,
+  getRelated,
+  hybridSearch,
+  keywordSearch,
+  listDocuments,
+  semanticSearch,
+} from "@mcpedia/core";
+
+export const appRouter = router({
+  search: publicProcedure
+    .input(z.object({ q: z.string(), limit: z.number().int().min(1).max(50).default(20) }))
+    .query(({ input }) => keywordSearch(input.q, input.limit)),
+
+  semanticSearch: publicProcedure
+    .input(z.object({ q: z.string(), limit: z.number().int().min(1).max(50).default(10) }))
+    .query(async ({ input }) => semanticSearch(input.q, input.limit)),
+
+  hybridSearch: publicProcedure
+    .input(z.object({ q: z.string(), limit: z.number().int().min(1).max(50).default(10) }))
+    .query(async ({ input }) => hybridSearch(input.q, input.limit)),
+
+  getDocument: publicProcedure
+    .input(z.object({ slug: z.string() }))
+    .query(async ({ input }) => getDocument(input.slug)),
+
+  listDocuments: publicProcedure
+    .input(z.object({ section: z.string().optional(), status: z.string().optional() }).optional())
+    .query(async ({ input }) => listDocuments(input ?? {})),
+
+  related: publicProcedure
+    .input(z.object({ slug: z.string(), limit: z.number().int().min(1).max(20).default(5) }))
+    .query(async ({ input }) => getRelated(input.slug, input.limit)),
+});
+
+export type AppRouter = typeof appRouter;

apps/api/src/trpc.ts

@@ -0,0 +1,11 @@
+import { initTRPC } from "@trpc/server";
+import { db } from "@mcpedia/db";
+
+export interface Context {
+  db: typeof db;
+}
+
+export const t = initTRPC.context<Context>().create();
+
+export const router = t.router;
+export const publicProcedure = t.procedure;

apps/api/tsconfig.json

@@ -0,0 +1,11 @@
+{
+  "extends": "../../tsconfig.base.json",
+  "compilerOptions": {
+    "paths": {
+      "@mcpedia/db": ["../../packages/db/src/index.ts"],
+      "@mcpedia/db/schema": ["../../packages/db/src/schema.ts"],
+      "@mcpedia/*": ["../../packages/*"]
+    }
+  },
+  "include": ["src/**/*.ts"]
+}

apps/mcp/src/index.ts

@@ -1,8 +1,7 @@
 import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
 import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
 import { z } from "zod";
-import { listDocuments, getDocument, getRelated } from "@mcpedia/core";
+import { listDocuments, getDocument, getRelated, semanticSearch, hybridSearch, keywordSearch } from "@mcpedia/core";
-import { keywordSearch } from "@mcpedia/search";
 
 export function createMcpServer(): McpServer {
   const server = new McpServer({
@@ -84,6 +83,42 @@ export function createMcpServer(): McpServer {
     },
   );
 
+  server.registerTool(
+    "semantic_search",
+    {
+      description:
+        "Semantic (embedding) search across chunked document content. Best for conceptual/paraphrased queries that don't share exact keywords. Returns chunks ranked by cosine similarity.",
+      inputSchema: z.object({
+        query: z.string().describe("Natural-language query"),
+        limit: z.number().int().positive().max(50).optional(),
+      }),
+    },
+    async ({ query, limit }) => {
+      const hits = await semanticSearch(query, limit ?? 10);
+      return {
+        content: [{ type: "text", text: JSON.stringify(hits, null, 2) }],
+      };
+    },
+  );
+
+  server.registerTool(
+    "hybrid_search",
+    {
+      description:
+        "Hybrid search fusing full-text (Postgres FTS) and semantic (embedding) signals via Reciprocal Rank Fusion. Best general-purpose search.",
+      inputSchema: z.object({
+        query: z.string().describe("Free-text or natural-language query"),
+        limit: z.number().int().positive().max(50).optional(),
+      }),
+    },
+    async ({ query, limit }) => {
+      const hits = await hybridSearch(query, limit ?? 10);
+      return {
+        content: [{ type: "text", text: JSON.stringify(hits, null, 2) }],
+      };
+    },
+  );
+
   return server;
 }
 

apps/mcp/src/smoke.test.ts

@@ -17,8 +17,10 @@ async function main() {
   const expected = [
     "get_document",
     "get_related_documents",
+    "hybrid_search",
     "list_documents",
     "search_documents",
+    "semantic_search",
   ].sort();
   if (JSON.stringify(names) !== JSON.stringify(expected)) {
     throw new Error(`tool set mismatch: ${names.join(",")}`);
@@ -66,6 +68,30 @@ async function main() {
   if (docs.length !== 1) throw new Error("list_documents docs != 1");
   console.log("list_documents(section=docs) =>", docs.length, "doc");
 
+  // 6) semantic_search
+  const sem = await client.callTool({
+    name: "semantic_search",
+    arguments: { query: "websocket connection closing unexpectedly", limit: 5 },
+  });
+  const semHits = JSON.parse((sem.content as any)[0].text);
+  if (!Array.isArray(semHits) || semHits.length < 1) {
+    throw new Error("semantic_search returned no hits");
+  }
+  console.log(
+    `semantic_search => ${semHits.length} chunks, top: ${semHits[0].slug}@${semHits[0].score.toFixed(3)}`,
+  );
+
+  // 7) hybrid_search
+  const hyb = await client.callTool({
+    name: "hybrid_search",
+    arguments: { query: "websocket timeout debugging", limit: 5 },
+  });
+  const hybHits = JSON.parse((hyb.content as any)[0].text);
+  if (!Array.isArray(hybHits) || hybHits.length < 1) {
+    throw new Error("hybrid_search returned no hits");
+  }
+  console.log(`hybrid_search => ${hybHits.length} docs, top: ${hybHits[0].doc.slug}`);
+
   await client.close();
   await server.close();
   console.log("\nSMOKE OK");

apps/mcp/tsconfig.json

@@ -2,6 +2,8 @@
   "extends": "../../tsconfig.base.json",
   "compilerOptions": {
     "paths": {
+      "@mcpedia/db": ["../../packages/db/src/index.ts"],
+      "@mcpedia/db/schema": ["../../packages/db/src/schema.ts"],
       "@mcpedia/*": ["../../packages/*"]
     }
   },

apps/web/app/search/page.tsx

@@ -1,18 +1,45 @@
 import Link from "next/link";
-import { keywordSearch } from "@mcpedia/core";
+import { keywordSearch, hybridSearch } from "@mcpedia/core";
+
+type Mode = "keyword" | "hybrid";
 
 export default async function SearchPage({
   searchParams,
 }: {
-  searchParams: Promise<{ q?: string }>;
+  searchParams: Promise<{ q?: string; mode?: string }>;
 }) {
-  const { q } = await searchParams;
+  const { q, mode } = await searchParams;
   const query = q?.trim() ?? "";
-  const hits = query ? await keywordSearch(query, 30) : [];
+  const activeMode: Mode = mode === "hybrid" ? "hybrid" : "keyword";
+
+  const hits = query
+    ? activeMode === "hybrid"
+      ? await hybridSearch(query, 30)
+      : await keywordSearch(query, 30)
+    : [];
+
+  const toggle = (m: Mode) => `/search?q=${encodeURIComponent(query)}&mode=${m}`;
 
   return (
     <div className="space-y-4">
-      <h1 className="text-2xl font-semibold tracking-tight">Search</h1>
+      <div className="flex items-center justify-between">
+        <h1 className="text-2xl font-semibold tracking-tight">Search</h1>
+        <div className="flex rounded overflow-hidden border border-zinc-300 dark:border-zinc-700 text-sm">
+          <Link
+            href={toggle("keyword")}
+            className={`px-3 py-1.5 ${activeMode === "keyword" ? "bg-zinc-900 text-white dark:bg-zinc-100 dark:text-zinc-900" : "hover:bg-zinc-100 dark:hover:bg-zinc-800"}`}
+          >
+            Keyword
+          </Link>
+          <Link
+            href={toggle("hybrid")}
+            className={`px-3 py-1.5 ${activeMode === "hybrid" ? "bg-zinc-900 text-white dark:bg-zinc-100 dark:text-zinc-900" : "hover:bg-zinc-100 dark:hover:bg-zinc-800"}`}
+          >
+            Hybrid
+          </Link>
+        </div>
+      </div>
+
       <form method="get" className="flex gap-2">
         <input
           name="q"
@@ -20,6 +47,7 @@ export default async function SearchPage({
           placeholder="e.g. websocket contract typescript"
           className="flex-1 rounded border border-zinc-300 dark:border-zinc-700 bg-white dark:bg-zinc-900 px-3 py-2 text-sm"
         />
+        <input type="hidden" name="mode" value={activeMode} />
         <button
           type="submit"
           className="rounded bg-zinc-900 text-white dark:bg-zinc-100 dark:text-zinc-900 px-4 py-2 text-sm font-medium"
@@ -44,6 +72,9 @@ export default async function SearchPage({
             >
               {h.doc.title}
             </Link>
+            {/* Snippet comes from Postgres ts_headline (keyword mode) or our own
+                chunk content (hybrid mode) — both trusted, first-party data, not
+                user input. The only markup is <mark> from ts_headline. */}
             <p
               className="text-sm text-zinc-600 dark:text-zinc-400 mt-1"
               dangerouslySetInnerHTML={{ __html: h.snippet }}

bun.lock

@@ -5,11 +5,28 @@
     "": {
       "name": "mcpedia",
       "devDependencies": {
+        "@trpc/client": "^11.18.0",
+        "@types/node": "^26.2.0",
         "prettier": "^3.3.0",
         "turbo": "^2.5.0",
         "typescript": "^5.6.0",
       },
     },
+    "apps/api": {
+      "name": "@mcpedia/api",
+      "version": "0.1.0",
+      "dependencies": {
+        "@hono/node-server": "^1.13.0",
+        "@mcpedia/config": "workspace:*",
+        "@mcpedia/core": "workspace:*",
+        "@trpc/server": "^11.0.0",
+        "hono": "^4.6.0",
+        "zod": "^3.23.8",
+      },
+      "devDependencies": {
+        "typescript": "^5.6.0",
+      },
+    },
     "apps/mcp": {
       "name": "@mcpedia/mcp",
       "version": "0.1.0",
@@ -63,6 +80,7 @@
       "dependencies": {
         "@mcpedia/config": "workspace:*",
         "@mcpedia/db": "workspace:*",
+        "@mcpedia/embeddings": "workspace:*",
         "@mcpedia/parser": "workspace:*",
         "@mcpedia/search": "workspace:*",
         "@mcpedia/types": "workspace:*",
@@ -82,6 +100,17 @@
         "drizzle-kit": "^0.30.0",
       },
     },
+    "packages/embeddings": {
+      "name": "@mcpedia/embeddings",
+      "version": "0.1.0",
+      "dependencies": {
+        "@mcpedia/config": "workspace:*",
+        "@mcpedia/types": "workspace:*",
+      },
+      "devDependencies": {
+        "typescript": "^5.6.0",
+      },
+    },
     "packages/parser": {
       "name": "@mcpedia/parser",
       "version": "0.1.0",
@@ -95,6 +124,7 @@
       "version": "0.1.0",
       "dependencies": {
         "@mcpedia/db": "workspace:*",
+        "@mcpedia/embeddings": "workspace:*",
         "@mcpedia/types": "workspace:*",
         "drizzle-orm": "^0.38.0",
       },
@@ -110,6 +140,7 @@
         "@mcpedia/config": "workspace:*",
         "@mcpedia/core": "workspace:*",
         "@mcpedia/db": "workspace:*",
+        "@mcpedia/embeddings": "workspace:*",
         "@mcpedia/parser": "workspace:*",
         "@mcpedia/search": "workspace:*",
         "drizzle-orm": "^0.38.0",
@@ -228,7 +259,7 @@
 
     "@eslint/plugin-kit": ["@eslint/plugin-kit@0.4.1", "", { "dependencies": { "@eslint/core": "^0.17.0", "levn": "^0.4.1" } }, "sha512-43/qtrDUokr7LJqoF2c3+RInu/t4zfrpYdoSDfYyhg52rwLV6TnOvdG4fXm7IkSB3wErkcmJS9iEhjVtOSEjjA=="],
 
-    "@hono/node-server": ["@hono/node-server@2.1.1", "", { "peerDependencies": { "hono": "^4" } }, "sha512-ELuehkj5VCBdgEw9zs+ivkKwyzzUCSQuE96YmiPvn1ECBoZCczbFXJLeEGMTYjphP6gydh4pHMqEYPVMYUVgQg=="],
+    "@hono/node-server": ["@hono/node-server@1.19.17", "", { "peerDependencies": { "hono": "^4" } }, "sha512-dSneS5qhiauZWGDCeK4o695Xd9nUNjviSZCMQrj10eetr8Uln1ucn6bbphOM6UynAMMtNIzZNSpL9vnASJwrPQ=="],
 
     "@humanfs/core": ["@humanfs/core@0.19.2", "", { "dependencies": { "@humanfs/types": "^0.15.0" } }, "sha512-UhXNm+CFMWcbChXywFwkmhqjs3PRCmcSa/hfBgLIb7oQ5HNb1wS0icWsGtSAUNgefHeI+eBrA8I1fxmbHsGdvA=="],
 
@@ -304,12 +335,16 @@
 
     "@jridgewell/trace-mapping": ["@jridgewell/trace-mapping@0.3.31", "", { "dependencies": { "@jridgewell/resolve-uri": "^3.1.0", "@jridgewell/sourcemap-codec": "^1.4.14" } }, "sha512-zzNR+SdQSDJzc8joaeP8QQoCQr8NuYx2dIIytl1QeBEZHJ9uW6hebsrYgbz8hJwUQao3TWCMtmfV8Nu1twOLAw=="],
 
+    "@mcpedia/api": ["@mcpedia/api@workspace:apps/api"],
+
     "@mcpedia/config": ["@mcpedia/config@workspace:packages/config"],
 
     "@mcpedia/core": ["@mcpedia/core@workspace:packages/core"],
 
     "@mcpedia/db": ["@mcpedia/db@workspace:packages/db"],
 
+    "@mcpedia/embeddings": ["@mcpedia/embeddings@workspace:packages/embeddings"],
+
     "@mcpedia/mcp": ["@mcpedia/mcp@workspace:apps/mcp"],
 
     "@mcpedia/parser": ["@mcpedia/parser@workspace:packages/parser"],
@@ -390,6 +425,10 @@
 
     "@tailwindcss/postcss": ["@tailwindcss/postcss@4.3.3", "", { "dependencies": { "@alloc/quick-lru": "^5.2.0", "@tailwindcss/node": "4.3.3", "@tailwindcss/oxide": "4.3.3", "postcss": "^8.5.16", "tailwindcss": "4.3.3" } }, "sha512-JTSZZGQi1AyKirbLN3azmjVzef92tcX7h+iSqPdaeStyFpGpDlKvvpxeOE8njhbUanbRwr3z8DyzhICWnMtQeg=="],
 
+    "@trpc/client": ["@trpc/client@11.18.0", "", { "peerDependencies": { "@trpc/server": "11.18.0", "typescript": ">=5.7.2" }, "bin": { "intent": "bin/intent.js" } }, "sha512-wOqeg3Fvl25V1ZisQhUD3K8G60ZJDlSGJNSyeXrLH24xAo5w6GSR2Kzb1cSNY9Y+IQ2YZvYGZstBU+V/ulo/ow=="],
+
+    "@trpc/server": ["@trpc/server@11.18.0", "", { "peerDependencies": { "typescript": ">=5.7.2" }, "bin": { "intent": "bin/intent.js" } }, "sha512-JAvXOuNTxgXjIDfQaOvDq1j66LMNfDJUH1IU7Slfn8EvRv2EkH6ehu3A7zpYhjO0syHHiYg77v2lG2JFJgvw7Q=="],
+
     "@turbo/darwin-64": ["@turbo/darwin-64@2.10.11", "", { "os": "darwin", "cpu": "x64" }, "sha512-v3R+1R/Ysozyo+p7Ri8MCIbndOvYt3DgPFrGLhrhQHfvyvbxyH3WyJj+A/2JTNmNleuAlh3JUyCV0iSVHIONTA=="],
 
     "@turbo/darwin-arm64": ["@turbo/darwin-arm64@2.10.11", "", { "os": "darwin", "cpu": "arm64" }, "sha512-R0a0CvGAeYYsBgPIgFNB3agGXh6qukjduNhFlwVVX1Ss2IdBJLXmgjytNGmo084bLKS0B6UdLRwKhXMHKKaObQ=="],
@@ -420,7 +459,7 @@
 
     "@types/ms": ["@types/ms@2.1.0", "", {}, "sha512-GsCCIZDE/p3i96vtEqx+7dBUGXrc7zeSK3wwPHIaRThS+9OhWIXRqzs4d6k1SVU8g91DrNRWxWUGhp5KXQb2VA=="],
 
-    "@types/node": ["@types/node@20.19.43", "", { "dependencies": { "undici-types": "~6.21.0" } }, "sha512-6oYBAi5ikg4Pl+kGsoYtawUMBT2zZMCvPNF7pVLnHZfd1zf38DRiWn/gT01RYCdUqkv7Fhr+C9ot4/tb+2sVvA=="],
+    "@types/node": ["@types/node@26.2.0", "", { "dependencies": { "undici-types": "~8.3.0" } }, "sha512-5IviulTZeRNp2vAJ514cc/HUlY5nZ9fCbq9DMyC52BrhFZACo3nI0R7qBxhQmo/d27NFe96ur/b7Wwxklda+kg=="],
 
     "@types/react": ["@types/react@19.2.18", "", { "dependencies": { "csstype": "^3.2.2" } }, "sha512-AnzbBERsrLKtk2XSfTbYRLjQPdy116Sty4q+T+Bp3IC4l6jNBvreVPAHmpq9qhXQM7CXZPjLVmGMw9sy+hxQ3w=="],
 
@@ -1302,7 +1341,7 @@
 
     "unbox-primitive": ["unbox-primitive@1.1.0", "", { "dependencies": { "call-bound": "^1.0.3", "has-bigints": "^1.0.2", "has-symbols": "^1.1.0", "which-boxed-primitive": "^1.1.1" } }, "sha512-nWJ91DjeOkej/TA8pXQ3myruKpKEYgqvpw9lz4OPHj/NWFNluYrjbz9j01CJ8yKQd2g4jFoOkINCTW2I5LEEyw=="],
 
-    "undici-types": ["undici-types@6.21.0", "", {}, "sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ=="],
+    "undici-types": ["undici-types@8.3.0", "", {}, "sha512-j375ScV60dom+YkPFIfTLcOiPxkN/buHz5GobjLhixFuANaNs3C9l4GmrWqejgXWJ7BbJcFYpTEUkS1Ge8bpZQ=="],
 
     "unified": ["unified@11.0.5", "", { "dependencies": { "@types/unist": "^3.0.0", "bail": "^2.0.0", "devlop": "^1.0.0", "extend": "^3.0.0", "is-plain-obj": "^4.0.0", "trough": "^2.0.0", "vfile": "^6.0.0" } }, "sha512-xKvGhPWw3k84Qjh8bI3ZeJjqnyadK+GEFtazSfZv/rKeTkTjOJho6mFqh2SM96iIcZokxiOpg78GazTSg8+KHA=="],
 
@@ -1374,6 +1413,14 @@
 
     "@img/sharp-wasm32/@emnapi/runtime": ["@emnapi/runtime@1.11.3", "", { "dependencies": { "tslib": "^2.4.0" } }, "sha512-Xz4Tpyki7XyrpbUK1jR1AhdAdaXyhhY4lZ3neLodmhpuWfy2PAQN5B46sAiU4liOXGLkHypn/qU+jvfWSCYYLA=="],
 
+    "@mcpedia/db/@types/node": ["@types/node@20.19.43", "", { "dependencies": { "undici-types": "~6.21.0" } }, "sha512-6oYBAi5ikg4Pl+kGsoYtawUMBT2zZMCvPNF7pVLnHZfd1zf38DRiWn/gT01RYCdUqkv7Fhr+C9ot4/tb+2sVvA=="],
+
+    "@mcpedia/mcp/@types/node": ["@types/node@20.19.43", "", { "dependencies": { "undici-types": "~6.21.0" } }, "sha512-6oYBAi5ikg4Pl+kGsoYtawUMBT2zZMCvPNF7pVLnHZfd1zf38DRiWn/gT01RYCdUqkv7Fhr+C9ot4/tb+2sVvA=="],
+
+    "@mcpedia/web/@types/node": ["@types/node@20.19.43", "", { "dependencies": { "undici-types": "~6.21.0" } }, "sha512-6oYBAi5ikg4Pl+kGsoYtawUMBT2zZMCvPNF7pVLnHZfd1zf38DRiWn/gT01RYCdUqkv7Fhr+C9ot4/tb+2sVvA=="],
+
+    "@modelcontextprotocol/sdk/@hono/node-server": ["@hono/node-server@2.1.1", "", { "peerDependencies": { "hono": "^4" } }, "sha512-ELuehkj5VCBdgEw9zs+ivkKwyzzUCSQuE96YmiPvn1ECBoZCczbFXJLeEGMTYjphP6gydh4pHMqEYPVMYUVgQg=="],
+
     "@modelcontextprotocol/sdk/zod": ["zod@4.4.3", "", {}, "sha512-ytENFjIJFl2UwYglde2jchW2Hwm4GJFLDiSXWdTrJQBIN9Fcyp7n4DhxJEiWNAJMV1/BqWfW/kkg71UDcHJyTQ=="],
 
     "@next/eslint-plugin-next/@eslint-community/eslint-utils": ["@eslint-community/eslint-utils@4.9.1", "", { "dependencies": { "eslint-visitor-keys": "^3.4.3" }, "peerDependencies": { "eslint": "^6.0.0 || ^7.0.0 || >=8.0.0" } }, "sha512-phrYmNiYppR7znFEdqgfWHXR6NCkZEK7hwWDHZUjit/2/U0r6XvkDl0SYnoM51Hq7FhCGdLDT6zxCCOY1hexsQ=="],
@@ -1474,6 +1521,12 @@
 
     "@eslint/eslintrc/js-yaml/argparse": ["argparse@2.0.1", "", {}, "sha512-8+9WqebbFzpX9OR+Wa6O29asIogeRMzcGtAINdpMHHyAg10f05aSFVBbcEqGf/PXw1EjAZ+q2/bEBg3DvurK3Q=="],
 
+    "@mcpedia/db/@types/node/undici-types": ["undici-types@6.21.0", "", {}, "sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ=="],
+
+    "@mcpedia/mcp/@types/node/undici-types": ["undici-types@6.21.0", "", {}, "sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ=="],
+
+    "@mcpedia/web/@types/node/undici-types": ["undici-types@6.21.0", "", {}, "sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ=="],
+
     "@next/eslint-plugin-next/@eslint-community/eslint-utils/eslint-visitor-keys": ["eslint-visitor-keys@3.4.3", "", {}, "sha512-wpc+LXeiyiisxPlEkUzU6svyS1frIO3Mgxj1fdy7Pm8Ygzguax2N3Fa/D/ag1WqbOprdI+uY6wMUl8/a2G+iag=="],
 
     "@typescript-eslint/typescript-estree/minimatch/brace-expansion": ["brace-expansion@5.0.9", "", { "dependencies": { "balanced-match": "^4.0.2" } }, "sha512-ScQ4IuvIEF1TMlP7Zt+vjJ//9zlPb2SDcxWxM3bk8s6t6GGdJ7KO1dCcTidOPJKePW30LE/2cT7wCyPho9/Wxg=="],

package.json

@@ -14,11 +14,14 @@
     "lint": "turbo run lint",
     "typecheck": "turbo run typecheck",
     "index": "bun run scripts/indexer.ts",
-    "mcp": "bun --cwd apps/mcp run start"
+    "mcp": "bun --cwd apps/mcp run start",
+    "api": "bun --cwd apps/api run dev"
   },
   "devDependencies": {
+    "@trpc/client": "^11.18.0",
+    "@types/node": "^26.2.0",
+    "prettier": "^3.3.0",
     "turbo": "^2.5.0",
-    "typescript": "^5.6.0",
+    "typescript": "^5.6.0"
-    "prettier": "^3.3.0"
   }
 }

packages/config/src/index.ts

@@ -36,6 +36,10 @@ export const CONTENT_ROOT =
 
 export const DATABASE_URL = process.env.DATABASE_URL ?? "";
 
+export const EMBED_BASE_URL = process.env.EMBED_BASE_URL ?? "";
+export const EMBED_API_KEY = process.env.EMBED_API_KEY ?? "";
+export const EMBED_MODEL = process.env.EMBED_MODEL ?? "";
+
 if (!DATABASE_URL) {
   // Fail fast with an explicit message instead of a cryptic driver error.
   throw new Error(

packages/core/package.json

@@ -10,6 +10,7 @@
   "dependencies": {
     "@mcpedia/config": "workspace:*",
     "@mcpedia/db": "workspace:*",
+    "@mcpedia/embeddings": "workspace:*",
     "@mcpedia/parser": "workspace:*",
     "@mcpedia/search": "workspace:*",
     "@mcpedia/types": "workspace:*",

packages/core/src/document.service.ts

@@ -1,6 +1,6 @@
 import { and, eq, sql } from "drizzle-orm";
 import { db } from "@mcpedia/db";
-import { documents } from "@mcpedia/db/schema";
+import { documentChunks, documents } from "@mcpedia/db/schema";
 import { CONTENT_ROOT } from "@mcpedia/config";
 import { existsSync, readFileSync } from "node:fs";
 import { join } from "node:path";
@@ -8,9 +8,12 @@ import type {
   Document,
   DocumentMeta,
 } from "@mcpedia/types";
+import { chunkText, embedChunks, createEmbeddingProvider } from "@mcpedia/embeddings";
 import { readContentFile } from "./content.service";
 import { toMeta } from "./row-map";
 
+const embedder = createEmbeddingProvider();
+
 export async function listDocuments(opts: {
   section?: string;
   status?: string;
@@ -56,3 +59,40 @@ export async function getRelated(slug: string, limit = 5): Promise<DocumentMeta[
 }
 
 export { readContentFile };
+
+/**
+ * Chunk a document body, embed the chunks, and upsert them into
+ * `document_chunks` (replacing any prior chunks for the same slug).
+ * Failures are thrown so the caller can decide whether to abort the index.
+ */
+export async function indexChunks(slug: string, body: string): Promise<number> {
+  const [doc] = await db
+    .select({ id: documents.id })
+    .from(documents)
+    .where(eq(documents.slug, slug));
+  if (!doc) return 0;
+
+  const chunks = chunkText(body, { size: 1000, overlap: 150 });
+  if (chunks.length === 0) return 0;
+
+  const vectors = await embedChunks(embedder, chunks, 16);
+  if (vectors.length !== chunks.length) {
+    throw new Error(
+      `chunk/embedding count mismatch for ${slug}: ${chunks.length} vs ${vectors.length}`,
+    );
+  }
+
+  // Replace existing chunks for this doc in one transaction.
+  await db.delete(documentChunks).where(eq(documentChunks.slug, slug));
+  await db.insert(documentChunks).values(
+    chunks.map((content: string, i: number) => ({
+      documentId: doc.id,
+      slug,
+      chunkIndex: i,
+      content,
+      embedding: vectors[i],
+    })),
+  );
+  return chunks.length;
+}
+

packages/core/src/search.service.ts

@@ -1 +1,8 @@
-export { keywordSearch, toTsQuery } from "@mcpedia/search";
+export {
+  keywordSearch,
+  semanticSearch,
+  hybridSearch,
+  toTsQuery,
+  cosine,
+} from "@mcpedia/search";
+export type { ChunkHit } from "@mcpedia/search";

packages/db/drizzle/0001_document_chunks.sql

@@ -0,0 +1,13 @@
+CREATE TABLE "document_chunks" (
+	"id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL,
+	"document_id" text NOT NULL,
+	"slug" text NOT NULL,
+	"chunk_index" integer NOT NULL,
+	"content" text NOT NULL,
+	"embedding" real[],
+	"created_at" timestamp with time zone DEFAULT now() NOT NULL
+);
+--> statement-breakpoint
+CREATE INDEX "document_chunks_slug_idx" ON "document_chunks" USING btree ("slug");
+--> statement-breakpoint
+ALTER TABLE "document_chunks" ADD CONSTRAINT "document_chunks_document_id_documents_id_fk" FOREIGN KEY ("document_id") REFERENCES "public"."documents"("id") ON DELETE cascade;

packages/db/drizzle/meta/0001_snapshot.json

@@ -0,0 +1,260 @@
+{
+  "id": "fd210ce3-31f0-434d-9e80-6f5a0bb07504",
+  "prevId": "249f80c0-d953-42f2-a98b-6701e2115856",
+  "version": "7",
+  "dialect": "postgresql",
+  "tables": {
+    "public.document_chunks": {
+      "name": "document_chunks",
+      "schema": "",
+      "columns": {
+        "id": {
+          "name": "id",
+          "type": "uuid",
+          "primaryKey": true,
+          "notNull": true,
+          "default": "gen_random_uuid()"
+        },
+        "document_id": {
+          "name": "document_id",
+          "type": "text",
+          "primaryKey": false,
+          "notNull": true
+        },
+        "slug": {
+          "name": "slug",
+          "type": "text",
+          "primaryKey": false,
+          "notNull": true
+        },
+        "chunk_index": {
+          "name": "chunk_index",
+          "type": "integer",
+          "primaryKey": false,
+          "notNull": true
+        },
+        "content": {
+          "name": "content",
+          "type": "text",
+          "primaryKey": false,
+          "notNull": true
+        },
+        "embedding": {
+          "name": "embedding",
+          "type": "vector(2048)",
+          "primaryKey": false,
+          "notNull": false
+        },
+        "created_at": {
+          "name": "created_at",
+          "type": "timestamp with time zone",
+          "primaryKey": false,
+          "notNull": true,
+          "default": "now()"
+        }
+      },
+      "indexes": {
+        "document_chunks_embedding_idx": {
+          "name": "document_chunks_embedding_idx",
+          "columns": [
+            {
+              "expression": "embedding",
+              "isExpression": false,
+              "asc": true,
+              "nulls": "last",
+              "opclass": "vector_cosine_ops"
+            }
+          ],
+          "isUnique": false,
+          "concurrently": false,
+          "method": "hnsw",
+          "with": {}
+        },
+        "document_chunks_slug_idx": {
+          "name": "document_chunks_slug_idx",
+          "columns": [
+            {
+              "expression": "slug",
+              "isExpression": false,
+              "asc": true,
+              "nulls": "last"
+            }
+          ],
+          "isUnique": false,
+          "concurrently": false,
+          "method": "btree",
+          "with": {}
+        }
+      },
+      "foreignKeys": {
+        "document_chunks_document_id_documents_id_fk": {
+          "name": "document_chunks_document_id_documents_id_fk",
+          "tableFrom": "document_chunks",
+          "tableTo": "documents",
+          "columnsFrom": [
+            "document_id"
+          ],
+          "columnsTo": [
+            "id"
+          ],
+          "onDelete": "cascade",
+          "onUpdate": "no action"
+        }
+      },
+      "compositePrimaryKeys": {},
+      "uniqueConstraints": {},
+      "policies": {},
+      "checkConstraints": {},
+      "isRLSEnabled": false
+    },
+    "public.documents": {
+      "name": "documents",
+      "schema": "",
+      "columns": {
+        "id": {
+          "name": "id",
+          "type": "text",
+          "primaryKey": true,
+          "notNull": true
+        },
+        "slug": {
+          "name": "slug",
+          "type": "text",
+          "primaryKey": false,
+          "notNull": true
+        },
+        "title": {
+          "name": "title",
+          "type": "text",
+          "primaryKey": false,
+          "notNull": true
+        },
+        "type": {
+          "name": "type",
+          "type": "text",
+          "primaryKey": false,
+          "notNull": true
+        },
+        "section": {
+          "name": "section",
+          "type": "text",
+          "primaryKey": false,
+          "notNull": true
+        },
+        "status": {
+          "name": "status",
+          "type": "text",
+          "primaryKey": false,
+          "notNull": true,
+          "default": "'published'"
+        },
+        "author": {
+          "name": "author",
+          "type": "text",
+          "primaryKey": false,
+          "notNull": true,
+          "default": "''"
+        },
+        "tags": {
+          "name": "tags",
+          "type": "text[]",
+          "primaryKey": false,
+          "notNull": true,
+          "default": "'{}'"
+        },
+        "path": {
+          "name": "path",
+          "type": "text",
+          "primaryKey": false,
+          "notNull": true
+        },
+        "body": {
+          "name": "body",
+          "type": "text",
+          "primaryKey": false,
+          "notNull": true,
+          "default": "''"
+        },
+        "search_vector": {
+          "name": "search_vector",
+          "type": "tsvector",
+          "primaryKey": false,
+          "notNull": true,
+          "generated": {
+            "as": "setweight(to_tsvector('simple', coalesce(\"documents\".\"title\", '')), 'A') || setweight(to_tsvector('simple', coalesce(\"documents\".\"body\", '')), 'B')",
+            "type": "stored"
+          }
+        },
+        "created_at": {
+          "name": "created_at",
+          "type": "timestamp with time zone",
+          "primaryKey": false,
+          "notNull": true
+        },
+        "updated_at": {
+          "name": "updated_at",
+          "type": "timestamp with time zone",
+          "primaryKey": false,
+          "notNull": true
+        }
+      },
+      "indexes": {
+        "documents_search_idx": {
+          "name": "documents_search_idx",
+          "columns": [
+            {
+              "expression": "search_vector",
+              "isExpression": false,
+              "asc": true,
+              "nulls": "last"
+            }
+          ],
+          "isUnique": false,
+          "concurrently": false,
+          "method": "gin",
+          "with": {}
+        },
+        "documents_section_idx": {
+          "name": "documents_section_idx",
+          "columns": [
+            {
+              "expression": "section",
+              "isExpression": false,
+              "asc": true,
+              "nulls": "last"
+            }
+          ],
+          "isUnique": false,
+          "concurrently": false,
+          "method": "btree",
+          "with": {}
+        }
+      },
+      "foreignKeys": {},
+      "compositePrimaryKeys": {},
+      "uniqueConstraints": {
+        "documents_slug_unique": {
+          "name": "documents_slug_unique",
+          "nullsNotDistinct": false,
+          "columns": [
+            "slug"
+          ]
+        }
+      },
+      "policies": {},
+      "checkConstraints": {},
+      "isRLSEnabled": false
+    }
+  },
+  "enums": {},
+  "schemas": {},
+  "sequences": {},
+  "roles": {},
+  "policies": {},
+  "views": {},
+  "_meta": {
+    "columns": {},
+    "schemas": {},
+    "tables": {}
+  }
+}

packages/db/drizzle/meta/_journal.json

@@ -8,6 +8,13 @@
       "when": 1787133375079,
       "tag": "0000_grey_toro",
       "breakpoints": true
+    },
+    {
+      "idx": 1,
+      "version": "7",
+      "when": 1787137149735,
+      "tag": "0001_document_chunks",
+      "breakpoints": true
     }
   ]
-}
+}

packages/db/package.json

@@ -4,7 +4,7 @@
   "private": true,
   "type": "module",
   "exports": {
-    ".": "./src/client.ts",
+    ".": "./src/index.ts",
     "./schema": "./src/schema.ts"
   },
   "dependencies": {

packages/db/src/index.ts

@@ -0,0 +1,2 @@
+export { db, schema, client } from "./client";
+export * from "./schema";

packages/db/src/schema.ts

@@ -4,8 +4,11 @@ import {
   index,
   integer,
   pgTable,
+  real,
   text,
   timestamp,
+  uuid,
+  vector,
 } from "drizzle-orm/pg-core";
 
 // tsvector isn't a first-class drizzle type; wrap the raw Postgres type.
@@ -46,14 +49,33 @@ export const documents = pgTable(
   }),
 );
 
-// Phase 2 (semantic search) — defined here for reference, NOT created yet:
+// Phase 2: semantic search chunks. Each row is an embedded slice of a document
-// export const documentChunks = pgTable("document_chunks", {
+// body. `embedding` is a plain float array (real[]). We compute cosine
-//   id: text("id").primaryKey(),
+// similarity in the application layer — pgvector isn't available on the shared
-//   documentId: text("document_id").notNull().references(() => documents.id, { onDelete: "cascade" }),
+// imrnes Postgres, and brute-force cosine is instant for a KB-sized corpus.
-//   content: text("content").notNull(),
+// (pgvector/HNSW is the Phase-4 scale-out path.)
-//   position: integer("position").notNull(),
+export const documentChunks = pgTable(
-//   embedding: customType<{ data: number[] }>({ dataType: () => "vector(1536)" })("embedding"),
+  "document_chunks",
-// });
+  {
+    id: uuid("id").primaryKey().defaultRandom(),
+    documentId: text("document_id")
+      .notNull()
+      .references(() => documents.id, { onDelete: "cascade" }),
+    slug: text("slug").notNull(),
+    chunkIndex: integer("chunk_index").notNull(),
+    content: text("content").notNull(),
+    embedding: real("embedding").array(),
+    createdAt: timestamp("created_at", { withTimezone: true })
+      .notNull()
+      .defaultNow(),
+  },
+  (t) => ({
+    slugIdx: index("document_chunks_slug_idx").on(t.slug),
+  }),
+);
+
+export type DocumentChunkRow = typeof documentChunks.$inferSelect;
+export type NewDocumentChunkRow = typeof documentChunks.$inferInsert;
 
 export type DocumentRow = typeof documents.$inferSelect;
 export type NewDocumentRow = typeof documents.$inferInsert;

packages/embeddings/package.json

@@ -0,0 +1,17 @@
+{
+  "name": "@mcpedia/embeddings",
+  "version": "0.1.0",
+  "private": true,
+  "type": "module",
+  "main": "./src/index.ts",
+  "exports": {
+    ".": "./src/index.ts"
+  },
+  "dependencies": {
+    "@mcpedia/config": "workspace:*",
+    "@mcpedia/types": "workspace:*"
+  },
+  "devDependencies": {
+    "typescript": "^5.6.0"
+  }
+}

packages/embeddings/src/chunk.ts

@@ -0,0 +1,48 @@
+import type { EmbeddingProvider } from "./provider";
+
+/**
+ * Split text into overlapping chunks for embedding. Keeps paragraphs/words
+ * intact where possible; never splits a chunk mid-word by more than `overlap`.
+ */
+export function chunkText(
+  text: string,
+  opts: { size?: number; overlap?: number } = {},
+): string[] {
+  const size = opts.size ?? 1000;
+  const overlap = opts.overlap ?? 150;
+  const clean = text.replace(/\r\n/g, "\n").trim();
+  if (!clean) return [];
+  if (clean.length <= size) return [clean];
+
+  const chunks: string[] = [];
+  let start = 0;
+  while (start < clean.length) {
+    let end = Math.min(start + size, clean.length);
+    // Prefer to break on a newline/space near the boundary.
+    if (end < clean.length) {
+      const nl = clean.lastIndexOf("\n", end);
+      const sp = clean.lastIndexOf(" ", end);
+      const breakAt = nl > start + size * 0.5 ? nl : sp > start + size * 0.5 ? sp : end;
+      if (breakAt > start) end = breakAt;
+    }
+    chunks.push(clean.slice(start, end).trim());
+    if (end >= clean.length) break;
+    start = Math.max(end - overlap, start + 1);
+  }
+  return chunks.filter(Boolean);
+}
+
+/** Embed a list of chunks in batches to avoid oversized requests. */
+export async function embedChunks(
+  provider: EmbeddingProvider,
+  chunks: string[],
+  batchSize = 16,
+): Promise<number[][]> {
+  const out: number[][] = [];
+  for (let i = 0; i < chunks.length; i += batchSize) {
+    const batch = chunks.slice(i, i + batchSize);
+    const vecs = await provider.embed(batch);
+    out.push(...vecs);
+  }
+  return out;
+}

packages/embeddings/src/index.ts

@@ -0,0 +1,2 @@
+export * from "./provider";
+export * from "./chunk";

packages/embeddings/src/provider.ts

@@ -0,0 +1,82 @@
+import {
+  EMBED_API_KEY,
+  EMBED_BASE_URL,
+  EMBED_MODEL,
+} from "@mcpedia/config";
+
+export interface EmbeddingProvider {
+  /** Embed a batch of texts into vectors of fixed dimension. */
+  embed(texts: string[]): Promise<number[][]>;
+  readonly model: string;
+  readonly dimensions: number;
+}
+
+/** Pinned embedding dimension for the configured OpenRouter model. */
+export const EMBED_DIM = 2048;
+
+/**
+ * OpenRouter embeddings provider (we route through 9router's OpenAI-compatible
+ * /v1 endpoint). `encoding_format: "float"` is REQUIRED — the Nvidia-backed
+ * model rejects base64.
+ */
+export class OpenRouterEmbeddingProvider implements EmbeddingProvider {
+  readonly model: string;
+  private readonly baseUrl: string;
+  private readonly apiKey: string;
+
+  constructor(opts?: {
+    baseUrl?: string;
+    apiKey?: string;
+    model?: string;
+  }) {
+    this.baseUrl = (opts?.baseUrl ?? EMBED_BASE_URL).replace(/\/$/, "");
+    this.apiKey = opts?.apiKey ?? EMBED_API_KEY;
+    this.model = opts?.model ?? EMBED_MODEL;
+    if (!this.baseUrl || !this.apiKey || !this.model) {
+      throw new Error(
+        "OpenRouterEmbeddingProvider: missing EMBED_BASE_URL / EMBED_API_KEY / EMBED_MODEL",
+      );
+    }
+  }
+
+  get dimensions(): number {
+    return EMBED_DIM;
+  }
+
+  async embed(texts: string[]): Promise<number[][]> {
+    if (texts.length === 0) return [];
+    const res = await fetch(`${this.baseUrl}/embeddings`, {
+      method: "POST",
+      headers: {
+        "Content-Type": "application/json",
+        Authorization: `Bearer ${this.apiKey}`,
+      },
+      body: JSON.stringify({
+        model: this.model,
+        input: texts,
+        encoding_format: "float",
+      }),
+    });
+    if (!res.ok) {
+      const body = await res.text().catch(() => "");
+      throw new Error(
+        `embedding request failed (${res.status}): ${body.slice(0, 300)}`,
+      );
+    }
+    const json = (await res.json()) as {
+      data?: { embedding: number[] }[];
+    };
+    const data = json.data;
+    if (!data || data.length !== texts.length) {
+      throw new Error(
+        `embedding response mismatch: expected ${texts.length}, got ${data?.length ?? 0}`,
+      );
+    }
+    return data.map((d) => d.embedding);
+  }
+}
+
+/** Default singleton provider. */
+export function createEmbeddingProvider(): EmbeddingProvider {
+  return new OpenRouterEmbeddingProvider();
+}

packages/search/package.json

@@ -8,6 +8,7 @@
   },
   "dependencies": {
     "@mcpedia/db": "workspace:*",
+    "@mcpedia/embeddings": "workspace:*",
     "@mcpedia/types": "workspace:*",
     "drizzle-orm": "^0.38.0"
   }

packages/search/src/index.ts

@@ -1,6 +1,7 @@
 import { db } from "@mcpedia/db";
-import { documents, type DocumentRow } from "@mcpedia/db/schema";
+import { documents, documentChunks, type DocumentRow } from "@mcpedia/db/schema";
 import { and, eq, sql } from "drizzle-orm";
+import { createEmbeddingProvider } from "@mcpedia/embeddings";
 import type {
   DocSection,
   DocStatus,
@@ -9,6 +10,23 @@ import type {
   SearchHit,
 } from "@mcpedia/types";
 
+const embedder = createEmbeddingProvider();
+
+/** Cosine similarity between two equal-length vectors. */
+export function cosine(a: number[], b: number[]): number {
+  if (a.length === 0 || a.length !== b.length) return 0;
+  let dot = 0;
+  let na = 0;
+  let nb = 0;
+  for (let i = 0; i < a.length; i++) {
+    dot += a[i] * b[i];
+    na += a[i] * a[i];
+    nb += b[i] * b[i];
+  }
+  const denom = Math.sqrt(na) * Math.sqrt(nb);
+  return denom === 0 ? 0 : dot / denom;
+}
+
 const VALID_SECTIONS: DocSection[] = ["docs", "writeups", "research", "notes"];
 const VALID_TYPES: DocType[] = ["documentation", "writeup", "research", "note"];
 
@@ -43,6 +61,99 @@ export function toTsQuery(q: string): string {
   return terms.map((t) => `${t}:*`).join(" & ");
 }
 
+export interface ChunkHit {
+  slug: string;
+  chunkIndex: number;
+  content: string;
+  score: number;
+}
+
+/**
+ * Semantic search: embed the query, then rank document chunks by cosine
+ * similarity. Cosine is computed in the app layer (pgvector isn't available on
+ * the shared imrnes Postgres); for a KB-sized corpus this is instant.
+ */
+export async function semanticSearch(q: string, limit = 10): Promise<ChunkHit[]> {
+  const query = q.trim();
+  if (!query) return [];
+  const [vec] = await embedder.embed([query]);
+  if (!vec || vec.length === 0) return [];
+
+  const rows = await db
+    .select({
+      slug: documentChunks.slug,
+      chunkIndex: documentChunks.chunkIndex,
+      content: documentChunks.content,
+      embedding: documentChunks.embedding,
+    })
+    .from(documentChunks)
+    .where(sql`${documentChunks.embedding} IS NOT NULL`);
+
+  return rows
+    .map((r) => ({
+      slug: r.slug,
+      chunkIndex: r.chunkIndex,
+      content: r.content,
+      score: cosine(vec, (r.embedding ?? []) as number[]),
+    }))
+    .filter((h) => h.score > 0)
+    .sort((a, b) => b.score - a.score)
+    .slice(0, limit);
+}
+
+/**
+ * Hybrid search: run FTS (ts_rank) and semantic (cosine) in parallel, then fuse
+ * with Reciprocal Rank Fusion (RRF, k=60). Returns merged document-level hits.
+ */
+export async function hybridSearch(q: string, limit = 10): Promise<SearchHit[]> {
+  const [fts, sem] = await Promise.all([keywordSearch(q, limit * 2), semanticSearch(q, limit * 2)]);
+  const k = 60;
+  const fused = new Map<string, { score: number; snippet: string; chunk: string }>();
+
+  fts.forEach((hit, i) => {
+    const rrf = 1 / (k + i + 1);
+    fused.set(hit.doc.slug, {
+      score: (fused.get(hit.doc.slug)?.score ?? 0) + rrf,
+      snippet: hit.snippet,
+      chunk: "",
+    });
+  });
+  sem.forEach((hit, i) => {
+    const rrf = 1 / (k + i + 1);
+    const prev = fused.get(hit.slug);
+    fused.set(hit.slug, {
+      score: (prev?.score ?? 0) + rrf,
+      snippet: prev?.snippet ?? hit.content.slice(0, 160),
+      chunk: prev?.chunk || hit.content,
+    });
+  });
+
+  const slugs = [...fused.entries()]
+    .sort((a, b) => b[1].score - a[1].score)
+    .slice(0, limit)
+    .map(([slug]) => slug);
+
+  if (slugs.length === 0) return [];
+  const rows = await db
+    .select()
+    .from(documents)
+    .where(and(eq(documents.status, "published"), sql`${documents.slug} IN ${slugs}`));
+
+  const bySlug = new Map(rows.map((r) => [r.slug, r]));
+  return slugs
+    .map((slug, i) => {
+      const row = bySlug.get(slug);
+      if (!row) return null;
+      const m = fused.get(slug)!;
+      return {
+        doc: toMeta(row),
+        rank: m.score,
+        snippet: m.snippet,
+      } as SearchHit;
+    })
+    .filter((x): x is SearchHit => x !== null);
+}
+
 /**
  * Postgres FTS keyword search over published documents.
  * Ranks by ts_rank and returns a headline snippet for display.

scripts/indexer.ts

@@ -2,12 +2,13 @@ import { db } from "@mcpedia/db";
 import { documents } from "@mcpedia/db/schema";
 import { parseFile } from "@mcpedia/parser";
 import { CONTENT_ROOT } from "@mcpedia/config";
-import { listContentFiles } from "@mcpedia/core";
+import { listContentFiles, indexChunks } from "@mcpedia/core";
 import { join } from "node:path";
 
 async function main() {
   const files = listContentFiles();
   let indexed = 0;
+  let chunked = 0;
   for (const rel of files) {
     const abs = join(CONTENT_ROOT, rel);
     const { meta, body } = parseFile(abs, rel);
@@ -47,8 +48,20 @@ async function main() {
       });
     indexed++;
     console.log(`  indexed ${rel}`);
+
+    // Phase 2: chunk + embed for semantic search.
+    try {
+      const n = await indexChunks(meta.slug, body);
+      chunked += n;
+      console.log(`    embedded ${n} chunks`);
+    } catch (err) {
+      console.error(
+        `    embed FAILED for ${meta.slug}: ${err instanceof Error ? err.message : err}`,
+      );
+      // Don't abort the whole index over one doc's embedding failure.
+    }
   }
-  console.log(`indexed ${indexed} documents`);
+  console.log(`indexed ${indexed} documents, ${chunked} chunks embedded`);
 }
 
 main()

scripts/package.json

@@ -7,6 +7,7 @@
     "@mcpedia/config": "workspace:*",
     "@mcpedia/core": "workspace:*",
     "@mcpedia/db": "workspace:*",
+    "@mcpedia/embeddings": "workspace:*",
     "@mcpedia/parser": "workspace:*",
     "@mcpedia/search": "workspace:*",
     "drizzle-orm": "^0.38.0",