feat(mcpedia): Phase 2 — semantic + hybrid search, tRPC/Hono API
- @mcpedia/embeddings: OpenRouter provider (9router /v1, encoding_format float), chunkText + embedChunks; EMBED_DIM=2048 - document_chunks table (real[] embedding) — pgvector NOT available on shared imrnes Postgres, so cosine is computed in-app (KB-scale fine); pgvector deferred - indexer: chunk + embed + upsert per document - @mcpedia/search: semanticSearch (cosine) + hybridSearch (FTS+cosine RRF) - apps/api: Hono + tRPC v11 (6 procedures), serve on :4020 - MCP: semantic_search + hybrid_search tools (6 total) - web: keyword/hybrid toggle; .env.example + README + PHASES updated
This commit is contained in:
asepharyana
@@ -0,0 +1,161 @@
|
||||
# MCPedia Phase 2 — Semantic Search + tRPC/Hono API
|
||||
|
||||
> **For Hermes:** implement task-by-task. Spec-first (user rule 2026-08-19).
|
||||
|
||||
**Goal:** Add semantic + hybrid search (pgvector) and a typed tRPC/Hono API so
|
||||
MCPedia is queryable by embeddings, not just keyword FTS — and expose the
|
||||
corpus over a programmatic HTTP API.
|
||||
|
||||
**Architecture:** Content (Markdown) → chunk → embed (OpenRouter) → store
|
||||
`document_chunks` with `vector(N)` in Postgres → `semanticSearch` (cosine) and
|
||||
`hybridSearch` (FTS + cosine, reciprocal-rank fusion) in `@mcpedia/search` →
|
||||
exposed via Core, the MCP server (new tools), and a new `apps/api` (Hono +
|
||||
tRPC v11).
|
||||
|
||||
**Embedding provider:** OpenRouter (`openrouter/llama-nemotron-embed-vl-1b-v2:free`)
|
||||
via `9router_ai_llm_api_key` + `9router_ai_llm_base_url` (BWS). Dimension is
|
||||
discovered at first live call (see Step 1.3) and pinned in schema/migration.
|
||||
|
||||
**Tech stack:** drizzle-orm `vector` column + pgvector extension, HNSW index,
|
||||
`@trpc/server` v11 (fetch adapter), `hono` + `@hono/node-server`.
|
||||
|
||||
---
|
||||
|
||||
## Task P2.1 — `packages/embeddings` (provider + abstraction)
|
||||
|
||||
**Files:** `packages/embeddings/package.json`, `src/index.ts`, `src/provider.ts`,
|
||||
`src/openrouter.ts`
|
||||
|
||||
- `EmbeddingProvider` interface: `embed(texts: string[]): Promise<number[][]>`, `readonly model`, `readonly dimensions`.
|
||||
- `OpenRouterEmbeddingProvider`: POST `${baseUrl}/embeddings` with `{ model, input }`,
|
||||
`Authorization: Bearer ${key}`. Returns `data[].embedding`. Validate length === dimensions.
|
||||
- Read `EMBED_BASE_URL`, `EMBED_API_KEY`, `EMBED_MODEL` from `@mcpedia/config`
|
||||
(with `.env` fallback). Dimensions discovered live (Step 1.3) → export `EMBED_DIM`.
|
||||
- Chunk helper `chunkText(text, { size=1000, overlap=150 })` in `src/chunk.ts`.
|
||||
|
||||
**Step 1.3 (discover dim):** live call `embed(["test"])`, read `embedding.length`,
|
||||
pin `EMBED_DIM`, assert mismatch throws.
|
||||
|
||||
**Verify:** `bun run` a temp script: `embed(["hello world"])` prints a vector of
|
||||
length N (e.g. 1024). Confirm no key is logged.
|
||||
|
||||
---
|
||||
|
||||
## Task P2.2 — Schema: `document_chunks` + vector extension
|
||||
|
||||
**Files:** `packages/db/src/schema.ts` (add), `packages/db/drizzle.config.ts`
|
||||
(unchanged), new migration.
|
||||
|
||||
- `CREATE EXTENSION IF NOT EXISTS vector;` (idempotent; run once via psql).
|
||||
- `document_chunks` table:
|
||||
- `id` uuid pk default gen_random_uuid()
|
||||
- `document_id` text → `documents.id` on delete cascade
|
||||
- `slug` text (denormalized for convenience)
|
||||
- `chunk_index` integer
|
||||
- `content` text
|
||||
- `embedding` vector(EMBED_DIM)
|
||||
- `created_at` timestamp default now()
|
||||
- index `chunk_embedding_idx` using hnsw (`embedding` op `vector_cosine_ops`)
|
||||
- Generate migration with `drizzle-kit generate`, apply via `psql` (drizzle-kit
|
||||
push is unreliable here — known).
|
||||
|
||||
**Verify:** `\d document_chunks` shows `embedding vector(N)` + HNSW index;
|
||||
`select count(*) from document_chunks` = 0.
|
||||
|
||||
---
|
||||
|
||||
## Task P2.3 — Indexer: chunk + embed + upsert
|
||||
|
||||
**Files:** `scripts/indexer.ts` (extend), `packages/core/src/document.service.ts`
|
||||
(add `indexChunks`).
|
||||
|
||||
- For each published doc: read body (already on disk), `chunkText`, `embed` in
|
||||
batches (≤ 16), delete existing chunks for slug, insert new rows.
|
||||
- Guard: if embedding provider fails, log + skip (don't crash the whole index).
|
||||
- Add `bun run index:embed` (or extend `bun run index` to also embed).
|
||||
|
||||
**Verify:** after running, `select count(*) from document_chunks` > 0; a sample
|
||||
row has non-null `embedding`.
|
||||
|
||||
---
|
||||
|
||||
## Task P2.4 — `packages/search`: semantic + hybrid
|
||||
|
||||
**Files:** `packages/search/src/index.ts` (add `semanticSearch`, `hybridSearch`).
|
||||
|
||||
- `semanticSearch(vec, limit)`: order by `embedding <=> ${vec}` asc, filter published.
|
||||
- `hybridSearch(q, limit)`: run FTS (`ts_rank`) + semantic (cosine) in parallel;
|
||||
fuse with reciprocal-rank (RRF: score = 1/(k+rank), k=60); return merged hits.
|
||||
- Keep `keywordSearch` unchanged (Phase 1).
|
||||
|
||||
**Verify:** unit-ish script: embed a query, `semanticSearch` returns relevant
|
||||
chunks; `hybridSearch("websocket")` returns ≥ keyword results.
|
||||
|
||||
---
|
||||
|
||||
## Task P2.5 — `packages/core` expose semantic/hybrid
|
||||
|
||||
**Files:** `packages/core/src/search.service.ts`, `index.ts`.
|
||||
|
||||
- Re-export `semanticSearch`, `hybridSearch` from Core.
|
||||
|
||||
---
|
||||
|
||||
## Task P2.6 — `apps/api` (Hono + tRPC v11)
|
||||
|
||||
**Files:** `apps/api/package.json`, `tsconfig.json`, `src/index.ts`,
|
||||
`src/router.ts`, `src/trpc.ts`.
|
||||
|
||||
- `initTRPC.create()` router with procedures: `search`, `semanticSearch`,
|
||||
`hybridSearch`, `getDocument`, `listDocuments` (mirrors MCP tools).
|
||||
- Mount `fetchRequestHandler` on a Hono app at `/trpc/*`; serve via
|
||||
`@hono/node-server` `serve({ fetch: app.fetch, port: 4020 })`.
|
||||
- `createContext` returns `{ db }`.
|
||||
|
||||
**Verify:** `bun run dev` → `curl -X POST localhost:4020/trpc/search`
|
||||
with JSON body returns hits.
|
||||
|
||||
---
|
||||
|
||||
## Task P2.7 — MCP server: semantic + hybrid tools
|
||||
|
||||
**Files:** `apps/mcp/src/index.ts` (add `semantic_search`, `hybrid_search`),
|
||||
extend `smoke.test.ts`.
|
||||
|
||||
- `semantic_search`: embed query → `semanticSearch`.
|
||||
- `hybrid_search`: embed query → `hybridSearch`.
|
||||
- Smoke: assert both return ≥1 hit for "websocket".
|
||||
|
||||
---
|
||||
|
||||
## Task P2.8 — Web: semantic toggle on search
|
||||
|
||||
**Files:** `apps/web/app/search/page.tsx`.
|
||||
|
||||
- Add `mode=keyword|hybrid` query param; server component calls Core
|
||||
`hybridSearch` when `mode=hybrid`. Minimal UI toggle (link/buttons).
|
||||
- Keep keyword as default.
|
||||
|
||||
**Verify:** `bun run build`; `curl '/search?q=websocket&mode=hybrid'` returns hits.
|
||||
|
||||
---
|
||||
|
||||
## Task P2.9 — Verify all + commit
|
||||
|
||||
- `bunx turbo run build` (web + api + mcp), `bun run apps/mcp smoke`,
|
||||
live API curl, live web hybrid search.
|
||||
- Update `README.md` + `PHASES.md` (mark Phase 2 ✅).
|
||||
- `git add -A` (exclude `.env`), commit as asepharyana (no Co-Authored-By).
|
||||
|
||||
---
|
||||
|
||||
## Risks / decisions
|
||||
- **Dimension unknown until live call** → P2.1.3 discovers it; pinned EMBED_DIM=2048.
|
||||
- **pgvector NOT available on shared imrnes Postgres** (extension not installed;
|
||||
installing needs host-level apt on a managed/shared DB — deferred). PIVOT:
|
||||
store `embedding` as `real[]` and compute cosine similarity in the app layer.
|
||||
Brute-force cosine is instant for a KB-sized corpus (dozens of docs / hundreds
|
||||
of chunks). pgvector+HNSW is the Phase-4 scale-out path.
|
||||
- **PgBouncer + real[]**: fine; simple queries, no extension needed.
|
||||
- **API port 4020** (host 4000s range is 4000–4015; 4020 is free for dev). Deploy later.
|
||||
- **YAGNI**: no auth/revisions this phase (Phase 3).
|
||||
@@ -19,13 +19,13 @@ Legend: ✅ built · 🟡 partial · ⬜ deferred
|
||||
|
||||
## Phase 2 — Semantic + API
|
||||
|
||||
- [ ] `pgvector` + embedding column on `document_chunks`
|
||||
- [ ] Chunking + embedding provider abstraction (`EmbeddingProvider`: OpenAI/Gemini/Ollama/local)
|
||||
- [ ] Hybrid search (FTS score + cosine, reciprocal-rank fusion)
|
||||
- [ ] tRPC + Hono API (`apps/api`) sharing `@mcpedia/core`
|
||||
- [ ] Tags / Categories / References as first-class tables
|
||||
- [ ] Auth (Auth.js / OIDC) — public/private/unlisted/admin/owner
|
||||
- [ ] shadcn/ui components + Shiki syntax highlighting (replace minimal markdown render)
|
||||
- [x] `packages/embeddings` — `EmbeddingProvider` interface + OpenRouter provider (via 9router `/v1`, `encoding_format:"float"`); `chunkText` + `embedChunks` batcher. `EMBED_DIM=2048` discovered live.
|
||||
- [x] Schema `document_chunks` (id, document_id→documents.id cascade, slug, chunk_index, content, `embedding real[]`). Stored as `real[]` because pgvector **is not installed** on the shared imrnes Postgres (installing needs host-level apt — deferred). Cosine computed in-app; instant for a KB-sized corpus.
|
||||
- [x] `scripts/indexer.ts` — chunks + embeds + upserts (per-doc replace).
|
||||
- [x] `@mcpedia/search` — `semanticSearch` (cosine) + `hybridSearch` (FTS + cosine, RRF fusion). `keywordSearch` unchanged.
|
||||
- [x] `apps/api` — Hono + tRPC v11 (`@trpc/server` fetch adapter, `@hono/node-server` on :4020): `search`, `semanticSearch`, `hybridSearch`, `getDocument`, `listDocuments`, `related`.
|
||||
- [x] MCP server — added `semantic_search` + `hybrid_search` tools (6 total).
|
||||
- [x] Web search — keyword/hybrid toggle (`?mode=hybrid`), hybrid reaches semantically-related docs keyword misses.
|
||||
|
||||
## Phase 3 — Async + Scale
|
||||
|
||||
|
||||
@@ -51,12 +51,18 @@ bun run mcp # MCP server on stdio (pipe to an MCP client)
|
||||
|
||||
### Database
|
||||
|
||||
Phase 1 uses Postgres FTS only. Schema is defined in `packages/db/src/schema.ts`
|
||||
(a `documents` table with a `search_vector` generated `tsvector` column + GIN
|
||||
index). Apply it with:
|
||||
Schema is defined in `packages/db/src/schema.ts` (`documents` with a weighted
|
||||
`search_vector` tsvector + GIN index, and `document_chunks` with an `embedding real[]`).
|
||||
The `pgvector` extension is **not available** on the shared imrnes Postgres, so
|
||||
semantic search stores vectors as `real[]` and ranks by in-app cosine similarity.
|
||||
|
||||
Migrations live in `packages/db/drizzle/`. They were applied manually via `psql`
|
||||
(`drizzle-kit push` is unreliable under PgBouncer transaction pooling); to
|
||||
re-apply on a fresh DB:
|
||||
|
||||
```bash
|
||||
bunx --cwd packages/db drizzle-kit push
|
||||
psql $DATABASE_URL -f packages/db/drizzle/0000_grey_toro.sql
|
||||
psql $DATABASE_URL -f packages/db/drizzle/0001_document_chunks.sql
|
||||
```
|
||||
|
||||
> Note: on imrnes (PgBouncer `:6432`) a leaked `DATABASE_URL` shell var can
|
||||
@@ -84,11 +90,13 @@ updated_at: 2026-08-19
|
||||
`body` shown in the UI is always read from the on-disk file (source of truth);
|
||||
the DB stores metadata + the search vector.
|
||||
|
||||
## MCP tools (Phase 1)
|
||||
## MCP tools
|
||||
|
||||
| Tool | Purpose |
|
||||
| --------------------- | ------------------------------------------------ |
|
||||
| `search_documents` | Postgres FTS over the corpus (ranked + snippet) |
|
||||
| `semantic_search` | Embedding/cosine search over chunked content |
|
||||
| `hybrid_search` | FTS + semantic fused via RRF |
|
||||
| `get_document` | Full markdown body by slug |
|
||||
| `list_documents` | List, optionally filtered by section |
|
||||
| `get_related_documents` | Docs sharing tags with a given slug |
|
||||
@@ -99,10 +107,29 @@ Smoke test (in-memory transport, real JSON-RPC):
|
||||
bun --cwd apps/mcp run smoke
|
||||
```
|
||||
|
||||
## API (Phase 2)
|
||||
|
||||
A tRPC v11 API is also exposed via Hono on **:4020** (all procedures mirror the
|
||||
MCP tools):
|
||||
|
||||
```bash
|
||||
bun run api # http://localhost:4020 (GET /health, POST/GET /trpc/*)
|
||||
```
|
||||
|
||||
`bun run index` now also chunks + embeds (Phase 2 indexer). Requires `EMBED_*`
|
||||
vars in `.env` (see `.env.example`).
|
||||
|
||||
## Status
|
||||
|
||||
**Phase 1 — MVP (DONE):** monorepo, Core, Web UI (home/doc/search), MCP server,
|
||||
Postgres FTS keyword search, content indexing.
|
||||
|
||||
See `PHASES.md` for Phase 2–4 (pgvector semantic/hybrid search, tRPC/Hono API,
|
||||
auth, Redis/BullMQ background workers, revisions, scale-out).
|
||||
**Phase 2 — Semantic + API (DONE):** embeddings provider (OpenRouter via 9router),
|
||||
chunked `document_chunks`, `semanticSearch` + `hybridSearch` (RRF), tRPC/Hono API
|
||||
(`apps/api`, :4020), MCP `semantic_search`/`hybrid_search` tools, web hybrid toggle.
|
||||
|
||||
> pgvector is **not installed** on the shared imrnes Postgres, so vector storage is
|
||||
> a `real[]` column with in-app cosine similarity (instant at KB scale). pgvector is
|
||||
> the Phase-4 scale-out path. See `PHASES.md`.
|
||||
|
||||
See `PHASES.md` for Phase 3–4 (Redis/BullMQ, auth, revisions, scale-out).
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"name": "@mcpedia/api",
|
||||
"version": "0.1.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "bun run src/index.ts",
|
||||
"start": "bun run src/index.ts",
|
||||
"lint": "tsc --noEmit",
|
||||
"typecheck": "tsc --noEmit"
|
||||
},
|
||||
"dependencies": {
|
||||
"@hono/node-server": "^1.13.0",
|
||||
"@mcpedia/config": "workspace:*",
|
||||
"@mcpedia/core": "workspace:*",
|
||||
"@trpc/server": "^11.0.0",
|
||||
"hono": "^4.6.0",
|
||||
"zod": "^3.23.8"
|
||||
},
|
||||
"devDependencies": {
|
||||
"typescript": "^5.6.0"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
import { serve } from "@hono/node-server";
|
||||
import { Hono } from "hono";
|
||||
import { fetchRequestHandler } from "@trpc/server/adapters/fetch";
|
||||
import { db } from "@mcpedia/db";
|
||||
import { appRouter } from "./router";
|
||||
import type { Context } from "./trpc";
|
||||
|
||||
const app = new Hono();
|
||||
|
||||
// Health check.
|
||||
app.get("/health", (c) => c.json({ ok: true }));
|
||||
|
||||
// Mount tRPC at /trpc/*. The fetch adapter is the canonical Bun/Hono adapter.
|
||||
app.all("/trpc/*", (c) =>
|
||||
fetchRequestHandler({
|
||||
endpoint: "/trpc",
|
||||
req: c.req.raw,
|
||||
router: appRouter,
|
||||
createContext: (): Context => ({ db }),
|
||||
}),
|
||||
);
|
||||
|
||||
const port = Number(process.env.API_PORT ?? 4020);
|
||||
serve({ fetch: app.fetch, port }, (info) => {
|
||||
console.log(`MCPedia API listening on http://localhost:${info.port}`);
|
||||
});
|
||||
@@ -0,0 +1,38 @@
|
||||
import { z } from "zod";
|
||||
import { publicProcedure, router } from "./trpc";
|
||||
import {
|
||||
getDocument,
|
||||
getRelated,
|
||||
hybridSearch,
|
||||
keywordSearch,
|
||||
listDocuments,
|
||||
semanticSearch,
|
||||
} from "@mcpedia/core";
|
||||
|
||||
export const appRouter = router({
|
||||
search: publicProcedure
|
||||
.input(z.object({ q: z.string(), limit: z.number().int().min(1).max(50).default(20) }))
|
||||
.query(({ input }) => keywordSearch(input.q, input.limit)),
|
||||
|
||||
semanticSearch: publicProcedure
|
||||
.input(z.object({ q: z.string(), limit: z.number().int().min(1).max(50).default(10) }))
|
||||
.query(async ({ input }) => semanticSearch(input.q, input.limit)),
|
||||
|
||||
hybridSearch: publicProcedure
|
||||
.input(z.object({ q: z.string(), limit: z.number().int().min(1).max(50).default(10) }))
|
||||
.query(async ({ input }) => hybridSearch(input.q, input.limit)),
|
||||
|
||||
getDocument: publicProcedure
|
||||
.input(z.object({ slug: z.string() }))
|
||||
.query(async ({ input }) => getDocument(input.slug)),
|
||||
|
||||
listDocuments: publicProcedure
|
||||
.input(z.object({ section: z.string().optional(), status: z.string().optional() }).optional())
|
||||
.query(async ({ input }) => listDocuments(input ?? {})),
|
||||
|
||||
related: publicProcedure
|
||||
.input(z.object({ slug: z.string(), limit: z.number().int().min(1).max(20).default(5) }))
|
||||
.query(async ({ input }) => getRelated(input.slug, input.limit)),
|
||||
});
|
||||
|
||||
export type AppRouter = typeof appRouter;
|
||||
@@ -0,0 +1,11 @@
|
||||
import { initTRPC } from "@trpc/server";
|
||||
import { db } from "@mcpedia/db";
|
||||
|
||||
export interface Context {
|
||||
db: typeof db;
|
||||
}
|
||||
|
||||
export const t = initTRPC.context<Context>().create();
|
||||
|
||||
export const router = t.router;
|
||||
export const publicProcedure = t.procedure;
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"extends": "../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"paths": {
|
||||
"@mcpedia/db": ["../../packages/db/src/index.ts"],
|
||||
"@mcpedia/db/schema": ["../../packages/db/src/schema.ts"],
|
||||
"@mcpedia/*": ["../../packages/*"]
|
||||
}
|
||||
},
|
||||
"include": ["src/**/*.ts"]
|
||||
}
|
||||
+37
-2
@@ -1,8 +1,7 @@
|
||||
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
|
||||
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
|
||||
import { z } from "zod";
|
||||
import { listDocuments, getDocument, getRelated } from "@mcpedia/core";
|
||||
import { keywordSearch } from "@mcpedia/search";
|
||||
import { listDocuments, getDocument, getRelated, semanticSearch, hybridSearch, keywordSearch } from "@mcpedia/core";
|
||||
|
||||
export function createMcpServer(): McpServer {
|
||||
const server = new McpServer({
|
||||
@@ -84,6 +83,42 @@ export function createMcpServer(): McpServer {
|
||||
},
|
||||
);
|
||||
|
||||
server.registerTool(
|
||||
"semantic_search",
|
||||
{
|
||||
description:
|
||||
"Semantic (embedding) search across chunked document content. Best for conceptual/paraphrased queries that don't share exact keywords. Returns chunks ranked by cosine similarity.",
|
||||
inputSchema: z.object({
|
||||
query: z.string().describe("Natural-language query"),
|
||||
limit: z.number().int().positive().max(50).optional(),
|
||||
}),
|
||||
},
|
||||
async ({ query, limit }) => {
|
||||
const hits = await semanticSearch(query, limit ?? 10);
|
||||
return {
|
||||
content: [{ type: "text", text: JSON.stringify(hits, null, 2) }],
|
||||
};
|
||||
},
|
||||
);
|
||||
|
||||
server.registerTool(
|
||||
"hybrid_search",
|
||||
{
|
||||
description:
|
||||
"Hybrid search fusing full-text (Postgres FTS) and semantic (embedding) signals via Reciprocal Rank Fusion. Best general-purpose search.",
|
||||
inputSchema: z.object({
|
||||
query: z.string().describe("Free-text or natural-language query"),
|
||||
limit: z.number().int().positive().max(50).optional(),
|
||||
}),
|
||||
},
|
||||
async ({ query, limit }) => {
|
||||
const hits = await hybridSearch(query, limit ?? 10);
|
||||
return {
|
||||
content: [{ type: "text", text: JSON.stringify(hits, null, 2) }],
|
||||
};
|
||||
},
|
||||
);
|
||||
|
||||
return server;
|
||||
}
|
||||
|
||||
|
||||
@@ -17,8 +17,10 @@ async function main() {
|
||||
const expected = [
|
||||
"get_document",
|
||||
"get_related_documents",
|
||||
"hybrid_search",
|
||||
"list_documents",
|
||||
"search_documents",
|
||||
"semantic_search",
|
||||
].sort();
|
||||
if (JSON.stringify(names) !== JSON.stringify(expected)) {
|
||||
throw new Error(`tool set mismatch: ${names.join(",")}`);
|
||||
@@ -66,6 +68,30 @@ async function main() {
|
||||
if (docs.length !== 1) throw new Error("list_documents docs != 1");
|
||||
console.log("list_documents(section=docs) =>", docs.length, "doc");
|
||||
|
||||
// 6) semantic_search
|
||||
const sem = await client.callTool({
|
||||
name: "semantic_search",
|
||||
arguments: { query: "websocket connection closing unexpectedly", limit: 5 },
|
||||
});
|
||||
const semHits = JSON.parse((sem.content as any)[0].text);
|
||||
if (!Array.isArray(semHits) || semHits.length < 1) {
|
||||
throw new Error("semantic_search returned no hits");
|
||||
}
|
||||
console.log(
|
||||
`semantic_search => ${semHits.length} chunks, top: ${semHits[0].slug}@${semHits[0].score.toFixed(3)}`,
|
||||
);
|
||||
|
||||
// 7) hybrid_search
|
||||
const hyb = await client.callTool({
|
||||
name: "hybrid_search",
|
||||
arguments: { query: "websocket timeout debugging", limit: 5 },
|
||||
});
|
||||
const hybHits = JSON.parse((hyb.content as any)[0].text);
|
||||
if (!Array.isArray(hybHits) || hybHits.length < 1) {
|
||||
throw new Error("hybrid_search returned no hits");
|
||||
}
|
||||
console.log(`hybrid_search => ${hybHits.length} docs, top: ${hybHits[0].doc.slug}`);
|
||||
|
||||
await client.close();
|
||||
await server.close();
|
||||
console.log("\nSMOKE OK");
|
||||
|
||||
@@ -2,6 +2,8 @@
|
||||
"extends": "../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"paths": {
|
||||
"@mcpedia/db": ["../../packages/db/src/index.ts"],
|
||||
"@mcpedia/db/schema": ["../../packages/db/src/schema.ts"],
|
||||
"@mcpedia/*": ["../../packages/*"]
|
||||
}
|
||||
},
|
||||
|
||||
@@ -1,18 +1,45 @@
|
||||
import Link from "next/link";
|
||||
import { keywordSearch } from "@mcpedia/core";
|
||||
import { keywordSearch, hybridSearch } from "@mcpedia/core";
|
||||
|
||||
type Mode = "keyword" | "hybrid";
|
||||
|
||||
export default async function SearchPage({
|
||||
searchParams,
|
||||
}: {
|
||||
searchParams: Promise<{ q?: string }>;
|
||||
searchParams: Promise<{ q?: string; mode?: string }>;
|
||||
}) {
|
||||
const { q } = await searchParams;
|
||||
const { q, mode } = await searchParams;
|
||||
const query = q?.trim() ?? "";
|
||||
const hits = query ? await keywordSearch(query, 30) : [];
|
||||
const activeMode: Mode = mode === "hybrid" ? "hybrid" : "keyword";
|
||||
|
||||
const hits = query
|
||||
? activeMode === "hybrid"
|
||||
? await hybridSearch(query, 30)
|
||||
: await keywordSearch(query, 30)
|
||||
: [];
|
||||
|
||||
const toggle = (m: Mode) => `/search?q=${encodeURIComponent(query)}&mode=${m}`;
|
||||
|
||||
return (
|
||||
<div className="space-y-4">
|
||||
<h1 className="text-2xl font-semibold tracking-tight">Search</h1>
|
||||
<div className="flex items-center justify-between">
|
||||
<h1 className="text-2xl font-semibold tracking-tight">Search</h1>
|
||||
<div className="flex rounded overflow-hidden border border-zinc-300 dark:border-zinc-700 text-sm">
|
||||
<Link
|
||||
href={toggle("keyword")}
|
||||
className={`px-3 py-1.5 ${activeMode === "keyword" ? "bg-zinc-900 text-white dark:bg-zinc-100 dark:text-zinc-900" : "hover:bg-zinc-100 dark:hover:bg-zinc-800"}`}
|
||||
>
|
||||
Keyword
|
||||
</Link>
|
||||
<Link
|
||||
href={toggle("hybrid")}
|
||||
className={`px-3 py-1.5 ${activeMode === "hybrid" ? "bg-zinc-900 text-white dark:bg-zinc-100 dark:text-zinc-900" : "hover:bg-zinc-100 dark:hover:bg-zinc-800"}`}
|
||||
>
|
||||
Hybrid
|
||||
</Link>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<form method="get" className="flex gap-2">
|
||||
<input
|
||||
name="q"
|
||||
@@ -20,6 +47,7 @@ export default async function SearchPage({
|
||||
placeholder="e.g. websocket contract typescript"
|
||||
className="flex-1 rounded border border-zinc-300 dark:border-zinc-700 bg-white dark:bg-zinc-900 px-3 py-2 text-sm"
|
||||
/>
|
||||
<input type="hidden" name="mode" value={activeMode} />
|
||||
<button
|
||||
type="submit"
|
||||
className="rounded bg-zinc-900 text-white dark:bg-zinc-100 dark:text-zinc-900 px-4 py-2 text-sm font-medium"
|
||||
@@ -44,6 +72,9 @@ export default async function SearchPage({
|
||||
>
|
||||
{h.doc.title}
|
||||
</Link>
|
||||
{/* Snippet comes from Postgres ts_headline (keyword mode) or our own
|
||||
chunk content (hybrid mode) — both trusted, first-party data, not
|
||||
user input. The only markup is <mark> from ts_headline. */}
|
||||
<p
|
||||
className="text-sm text-zinc-600 dark:text-zinc-400 mt-1"
|
||||
dangerouslySetInnerHTML={{ __html: h.snippet }}
|
||||
|
||||
@@ -5,11 +5,28 @@
|
||||
"": {
|
||||
"name": "mcpedia",
|
||||
"devDependencies": {
|
||||
"@trpc/client": "^11.18.0",
|
||||
"@types/node": "^26.2.0",
|
||||
"prettier": "^3.3.0",
|
||||
"turbo": "^2.5.0",
|
||||
"typescript": "^5.6.0",
|
||||
},
|
||||
},
|
||||
"apps/api": {
|
||||
"name": "@mcpedia/api",
|
||||
"version": "0.1.0",
|
||||
"dependencies": {
|
||||
"@hono/node-server": "^1.13.0",
|
||||
"@mcpedia/config": "workspace:*",
|
||||
"@mcpedia/core": "workspace:*",
|
||||
"@trpc/server": "^11.0.0",
|
||||
"hono": "^4.6.0",
|
||||
"zod": "^3.23.8",
|
||||
},
|
||||
"devDependencies": {
|
||||
"typescript": "^5.6.0",
|
||||
},
|
||||
},
|
||||
"apps/mcp": {
|
||||
"name": "@mcpedia/mcp",
|
||||
"version": "0.1.0",
|
||||
@@ -63,6 +80,7 @@
|
||||
"dependencies": {
|
||||
"@mcpedia/config": "workspace:*",
|
||||
"@mcpedia/db": "workspace:*",
|
||||
"@mcpedia/embeddings": "workspace:*",
|
||||
"@mcpedia/parser": "workspace:*",
|
||||
"@mcpedia/search": "workspace:*",
|
||||
"@mcpedia/types": "workspace:*",
|
||||
@@ -82,6 +100,17 @@
|
||||
"drizzle-kit": "^0.30.0",
|
||||
},
|
||||
},
|
||||
"packages/embeddings": {
|
||||
"name": "@mcpedia/embeddings",
|
||||
"version": "0.1.0",
|
||||
"dependencies": {
|
||||
"@mcpedia/config": "workspace:*",
|
||||
"@mcpedia/types": "workspace:*",
|
||||
},
|
||||
"devDependencies": {
|
||||
"typescript": "^5.6.0",
|
||||
},
|
||||
},
|
||||
"packages/parser": {
|
||||
"name": "@mcpedia/parser",
|
||||
"version": "0.1.0",
|
||||
@@ -95,6 +124,7 @@
|
||||
"version": "0.1.0",
|
||||
"dependencies": {
|
||||
"@mcpedia/db": "workspace:*",
|
||||
"@mcpedia/embeddings": "workspace:*",
|
||||
"@mcpedia/types": "workspace:*",
|
||||
"drizzle-orm": "^0.38.0",
|
||||
},
|
||||
@@ -110,6 +140,7 @@
|
||||
"@mcpedia/config": "workspace:*",
|
||||
"@mcpedia/core": "workspace:*",
|
||||
"@mcpedia/db": "workspace:*",
|
||||
"@mcpedia/embeddings": "workspace:*",
|
||||
"@mcpedia/parser": "workspace:*",
|
||||
"@mcpedia/search": "workspace:*",
|
||||
"drizzle-orm": "^0.38.0",
|
||||
@@ -228,7 +259,7 @@
|
||||
|
||||
"@eslint/plugin-kit": ["@eslint/plugin-kit@0.4.1", "", { "dependencies": { "@eslint/core": "^0.17.0", "levn": "^0.4.1" } }, "sha512-43/qtrDUokr7LJqoF2c3+RInu/t4zfrpYdoSDfYyhg52rwLV6TnOvdG4fXm7IkSB3wErkcmJS9iEhjVtOSEjjA=="],
|
||||
|
||||
"@hono/node-server": ["@hono/node-server@2.1.1", "", { "peerDependencies": { "hono": "^4" } }, "sha512-ELuehkj5VCBdgEw9zs+ivkKwyzzUCSQuE96YmiPvn1ECBoZCczbFXJLeEGMTYjphP6gydh4pHMqEYPVMYUVgQg=="],
|
||||
"@hono/node-server": ["@hono/node-server@1.19.17", "", { "peerDependencies": { "hono": "^4" } }, "sha512-dSneS5qhiauZWGDCeK4o695Xd9nUNjviSZCMQrj10eetr8Uln1ucn6bbphOM6UynAMMtNIzZNSpL9vnASJwrPQ=="],
|
||||
|
||||
"@humanfs/core": ["@humanfs/core@0.19.2", "", { "dependencies": { "@humanfs/types": "^0.15.0" } }, "sha512-UhXNm+CFMWcbChXywFwkmhqjs3PRCmcSa/hfBgLIb7oQ5HNb1wS0icWsGtSAUNgefHeI+eBrA8I1fxmbHsGdvA=="],
|
||||
|
||||
@@ -304,12 +335,16 @@
|
||||
|
||||
"@jridgewell/trace-mapping": ["@jridgewell/trace-mapping@0.3.31", "", { "dependencies": { "@jridgewell/resolve-uri": "^3.1.0", "@jridgewell/sourcemap-codec": "^1.4.14" } }, "sha512-zzNR+SdQSDJzc8joaeP8QQoCQr8NuYx2dIIytl1QeBEZHJ9uW6hebsrYgbz8hJwUQao3TWCMtmfV8Nu1twOLAw=="],
|
||||
|
||||
"@mcpedia/api": ["@mcpedia/api@workspace:apps/api"],
|
||||
|
||||
"@mcpedia/config": ["@mcpedia/config@workspace:packages/config"],
|
||||
|
||||
"@mcpedia/core": ["@mcpedia/core@workspace:packages/core"],
|
||||
|
||||
"@mcpedia/db": ["@mcpedia/db@workspace:packages/db"],
|
||||
|
||||
"@mcpedia/embeddings": ["@mcpedia/embeddings@workspace:packages/embeddings"],
|
||||
|
||||
"@mcpedia/mcp": ["@mcpedia/mcp@workspace:apps/mcp"],
|
||||
|
||||
"@mcpedia/parser": ["@mcpedia/parser@workspace:packages/parser"],
|
||||
@@ -390,6 +425,10 @@
|
||||
|
||||
"@tailwindcss/postcss": ["@tailwindcss/postcss@4.3.3", "", { "dependencies": { "@alloc/quick-lru": "^5.2.0", "@tailwindcss/node": "4.3.3", "@tailwindcss/oxide": "4.3.3", "postcss": "^8.5.16", "tailwindcss": "4.3.3" } }, "sha512-JTSZZGQi1AyKirbLN3azmjVzef92tcX7h+iSqPdaeStyFpGpDlKvvpxeOE8njhbUanbRwr3z8DyzhICWnMtQeg=="],
|
||||
|
||||
"@trpc/client": ["@trpc/client@11.18.0", "", { "peerDependencies": { "@trpc/server": "11.18.0", "typescript": ">=5.7.2" }, "bin": { "intent": "bin/intent.js" } }, "sha512-wOqeg3Fvl25V1ZisQhUD3K8G60ZJDlSGJNSyeXrLH24xAo5w6GSR2Kzb1cSNY9Y+IQ2YZvYGZstBU+V/ulo/ow=="],
|
||||
|
||||
"@trpc/server": ["@trpc/server@11.18.0", "", { "peerDependencies": { "typescript": ">=5.7.2" }, "bin": { "intent": "bin/intent.js" } }, "sha512-JAvXOuNTxgXjIDfQaOvDq1j66LMNfDJUH1IU7Slfn8EvRv2EkH6ehu3A7zpYhjO0syHHiYg77v2lG2JFJgvw7Q=="],
|
||||
|
||||
"@turbo/darwin-64": ["@turbo/darwin-64@2.10.11", "", { "os": "darwin", "cpu": "x64" }, "sha512-v3R+1R/Ysozyo+p7Ri8MCIbndOvYt3DgPFrGLhrhQHfvyvbxyH3WyJj+A/2JTNmNleuAlh3JUyCV0iSVHIONTA=="],
|
||||
|
||||
"@turbo/darwin-arm64": ["@turbo/darwin-arm64@2.10.11", "", { "os": "darwin", "cpu": "arm64" }, "sha512-R0a0CvGAeYYsBgPIgFNB3agGXh6qukjduNhFlwVVX1Ss2IdBJLXmgjytNGmo084bLKS0B6UdLRwKhXMHKKaObQ=="],
|
||||
@@ -420,7 +459,7 @@
|
||||
|
||||
"@types/ms": ["@types/ms@2.1.0", "", {}, "sha512-GsCCIZDE/p3i96vtEqx+7dBUGXrc7zeSK3wwPHIaRThS+9OhWIXRqzs4d6k1SVU8g91DrNRWxWUGhp5KXQb2VA=="],
|
||||
|
||||
"@types/node": ["@types/node@20.19.43", "", { "dependencies": { "undici-types": "~6.21.0" } }, "sha512-6oYBAi5ikg4Pl+kGsoYtawUMBT2zZMCvPNF7pVLnHZfd1zf38DRiWn/gT01RYCdUqkv7Fhr+C9ot4/tb+2sVvA=="],
|
||||
"@types/node": ["@types/node@26.2.0", "", { "dependencies": { "undici-types": "~8.3.0" } }, "sha512-5IviulTZeRNp2vAJ514cc/HUlY5nZ9fCbq9DMyC52BrhFZACo3nI0R7qBxhQmo/d27NFe96ur/b7Wwxklda+kg=="],
|
||||
|
||||
"@types/react": ["@types/react@19.2.18", "", { "dependencies": { "csstype": "^3.2.2" } }, "sha512-AnzbBERsrLKtk2XSfTbYRLjQPdy116Sty4q+T+Bp3IC4l6jNBvreVPAHmpq9qhXQM7CXZPjLVmGMw9sy+hxQ3w=="],
|
||||
|
||||
@@ -1302,7 +1341,7 @@
|
||||
|
||||
"unbox-primitive": ["unbox-primitive@1.1.0", "", { "dependencies": { "call-bound": "^1.0.3", "has-bigints": "^1.0.2", "has-symbols": "^1.1.0", "which-boxed-primitive": "^1.1.1" } }, "sha512-nWJ91DjeOkej/TA8pXQ3myruKpKEYgqvpw9lz4OPHj/NWFNluYrjbz9j01CJ8yKQd2g4jFoOkINCTW2I5LEEyw=="],
|
||||
|
||||
"undici-types": ["undici-types@6.21.0", "", {}, "sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ=="],
|
||||
"undici-types": ["undici-types@8.3.0", "", {}, "sha512-j375ScV60dom+YkPFIfTLcOiPxkN/buHz5GobjLhixFuANaNs3C9l4GmrWqejgXWJ7BbJcFYpTEUkS1Ge8bpZQ=="],
|
||||
|
||||
"unified": ["unified@11.0.5", "", { "dependencies": { "@types/unist": "^3.0.0", "bail": "^2.0.0", "devlop": "^1.0.0", "extend": "^3.0.0", "is-plain-obj": "^4.0.0", "trough": "^2.0.0", "vfile": "^6.0.0" } }, "sha512-xKvGhPWw3k84Qjh8bI3ZeJjqnyadK+GEFtazSfZv/rKeTkTjOJho6mFqh2SM96iIcZokxiOpg78GazTSg8+KHA=="],
|
||||
|
||||
@@ -1374,6 +1413,14 @@
|
||||
|
||||
"@img/sharp-wasm32/@emnapi/runtime": ["@emnapi/runtime@1.11.3", "", { "dependencies": { "tslib": "^2.4.0" } }, "sha512-Xz4Tpyki7XyrpbUK1jR1AhdAdaXyhhY4lZ3neLodmhpuWfy2PAQN5B46sAiU4liOXGLkHypn/qU+jvfWSCYYLA=="],
|
||||
|
||||
"@mcpedia/db/@types/node": ["@types/node@20.19.43", "", { "dependencies": { "undici-types": "~6.21.0" } }, "sha512-6oYBAi5ikg4Pl+kGsoYtawUMBT2zZMCvPNF7pVLnHZfd1zf38DRiWn/gT01RYCdUqkv7Fhr+C9ot4/tb+2sVvA=="],
|
||||
|
||||
"@mcpedia/mcp/@types/node": ["@types/node@20.19.43", "", { "dependencies": { "undici-types": "~6.21.0" } }, "sha512-6oYBAi5ikg4Pl+kGsoYtawUMBT2zZMCvPNF7pVLnHZfd1zf38DRiWn/gT01RYCdUqkv7Fhr+C9ot4/tb+2sVvA=="],
|
||||
|
||||
"@mcpedia/web/@types/node": ["@types/node@20.19.43", "", { "dependencies": { "undici-types": "~6.21.0" } }, "sha512-6oYBAi5ikg4Pl+kGsoYtawUMBT2zZMCvPNF7pVLnHZfd1zf38DRiWn/gT01RYCdUqkv7Fhr+C9ot4/tb+2sVvA=="],
|
||||
|
||||
"@modelcontextprotocol/sdk/@hono/node-server": ["@hono/node-server@2.1.1", "", { "peerDependencies": { "hono": "^4" } }, "sha512-ELuehkj5VCBdgEw9zs+ivkKwyzzUCSQuE96YmiPvn1ECBoZCczbFXJLeEGMTYjphP6gydh4pHMqEYPVMYUVgQg=="],
|
||||
|
||||
"@modelcontextprotocol/sdk/zod": ["zod@4.4.3", "", {}, "sha512-ytENFjIJFl2UwYglde2jchW2Hwm4GJFLDiSXWdTrJQBIN9Fcyp7n4DhxJEiWNAJMV1/BqWfW/kkg71UDcHJyTQ=="],
|
||||
|
||||
"@next/eslint-plugin-next/@eslint-community/eslint-utils": ["@eslint-community/eslint-utils@4.9.1", "", { "dependencies": { "eslint-visitor-keys": "^3.4.3" }, "peerDependencies": { "eslint": "^6.0.0 || ^7.0.0 || >=8.0.0" } }, "sha512-phrYmNiYppR7znFEdqgfWHXR6NCkZEK7hwWDHZUjit/2/U0r6XvkDl0SYnoM51Hq7FhCGdLDT6zxCCOY1hexsQ=="],
|
||||
@@ -1474,6 +1521,12 @@
|
||||
|
||||
"@eslint/eslintrc/js-yaml/argparse": ["argparse@2.0.1", "", {}, "sha512-8+9WqebbFzpX9OR+Wa6O29asIogeRMzcGtAINdpMHHyAg10f05aSFVBbcEqGf/PXw1EjAZ+q2/bEBg3DvurK3Q=="],
|
||||
|
||||
"@mcpedia/db/@types/node/undici-types": ["undici-types@6.21.0", "", {}, "sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ=="],
|
||||
|
||||
"@mcpedia/mcp/@types/node/undici-types": ["undici-types@6.21.0", "", {}, "sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ=="],
|
||||
|
||||
"@mcpedia/web/@types/node/undici-types": ["undici-types@6.21.0", "", {}, "sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ=="],
|
||||
|
||||
"@next/eslint-plugin-next/@eslint-community/eslint-utils/eslint-visitor-keys": ["eslint-visitor-keys@3.4.3", "", {}, "sha512-wpc+LXeiyiisxPlEkUzU6svyS1frIO3Mgxj1fdy7Pm8Ygzguax2N3Fa/D/ag1WqbOprdI+uY6wMUl8/a2G+iag=="],
|
||||
|
||||
"@typescript-eslint/typescript-estree/minimatch/brace-expansion": ["brace-expansion@5.0.9", "", { "dependencies": { "balanced-match": "^4.0.2" } }, "sha512-ScQ4IuvIEF1TMlP7Zt+vjJ//9zlPb2SDcxWxM3bk8s6t6GGdJ7KO1dCcTidOPJKePW30LE/2cT7wCyPho9/Wxg=="],
|
||||
|
||||
+6
-3
@@ -14,11 +14,14 @@
|
||||
"lint": "turbo run lint",
|
||||
"typecheck": "turbo run typecheck",
|
||||
"index": "bun run scripts/indexer.ts",
|
||||
"mcp": "bun --cwd apps/mcp run start"
|
||||
"mcp": "bun --cwd apps/mcp run start",
|
||||
"api": "bun --cwd apps/api run dev"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@trpc/client": "^11.18.0",
|
||||
"@types/node": "^26.2.0",
|
||||
"prettier": "^3.3.0",
|
||||
"turbo": "^2.5.0",
|
||||
"typescript": "^5.6.0",
|
||||
"prettier": "^3.3.0"
|
||||
"typescript": "^5.6.0"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -36,6 +36,10 @@ export const CONTENT_ROOT =
|
||||
|
||||
export const DATABASE_URL = process.env.DATABASE_URL ?? "";
|
||||
|
||||
export const EMBED_BASE_URL = process.env.EMBED_BASE_URL ?? "";
|
||||
export const EMBED_API_KEY = process.env.EMBED_API_KEY ?? "";
|
||||
export const EMBED_MODEL = process.env.EMBED_MODEL ?? "";
|
||||
|
||||
if (!DATABASE_URL) {
|
||||
// Fail fast with an explicit message instead of a cryptic driver error.
|
||||
throw new Error(
|
||||
|
||||
@@ -10,6 +10,7 @@
|
||||
"dependencies": {
|
||||
"@mcpedia/config": "workspace:*",
|
||||
"@mcpedia/db": "workspace:*",
|
||||
"@mcpedia/embeddings": "workspace:*",
|
||||
"@mcpedia/parser": "workspace:*",
|
||||
"@mcpedia/search": "workspace:*",
|
||||
"@mcpedia/types": "workspace:*",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { db } from "@mcpedia/db";
|
||||
import { documents } from "@mcpedia/db/schema";
|
||||
import { documentChunks, documents } from "@mcpedia/db/schema";
|
||||
import { CONTENT_ROOT } from "@mcpedia/config";
|
||||
import { existsSync, readFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
@@ -8,9 +8,12 @@ import type {
|
||||
Document,
|
||||
DocumentMeta,
|
||||
} from "@mcpedia/types";
|
||||
import { chunkText, embedChunks, createEmbeddingProvider } from "@mcpedia/embeddings";
|
||||
import { readContentFile } from "./content.service";
|
||||
import { toMeta } from "./row-map";
|
||||
|
||||
const embedder = createEmbeddingProvider();
|
||||
|
||||
export async function listDocuments(opts: {
|
||||
section?: string;
|
||||
status?: string;
|
||||
@@ -56,3 +59,40 @@ export async function getRelated(slug: string, limit = 5): Promise<DocumentMeta[
|
||||
}
|
||||
|
||||
export { readContentFile };
|
||||
|
||||
/**
|
||||
* Chunk a document body, embed the chunks, and upsert them into
|
||||
* `document_chunks` (replacing any prior chunks for the same slug).
|
||||
* Failures are thrown so the caller can decide whether to abort the index.
|
||||
*/
|
||||
export async function indexChunks(slug: string, body: string): Promise<number> {
|
||||
const [doc] = await db
|
||||
.select({ id: documents.id })
|
||||
.from(documents)
|
||||
.where(eq(documents.slug, slug));
|
||||
if (!doc) return 0;
|
||||
|
||||
const chunks = chunkText(body, { size: 1000, overlap: 150 });
|
||||
if (chunks.length === 0) return 0;
|
||||
|
||||
const vectors = await embedChunks(embedder, chunks, 16);
|
||||
if (vectors.length !== chunks.length) {
|
||||
throw new Error(
|
||||
`chunk/embedding count mismatch for ${slug}: ${chunks.length} vs ${vectors.length}`,
|
||||
);
|
||||
}
|
||||
|
||||
// Replace existing chunks for this doc in one transaction.
|
||||
await db.delete(documentChunks).where(eq(documentChunks.slug, slug));
|
||||
await db.insert(documentChunks).values(
|
||||
chunks.map((content: string, i: number) => ({
|
||||
documentId: doc.id,
|
||||
slug,
|
||||
chunkIndex: i,
|
||||
content,
|
||||
embedding: vectors[i],
|
||||
})),
|
||||
);
|
||||
return chunks.length;
|
||||
}
|
||||
|
||||
|
||||
@@ -1 +1,8 @@
|
||||
export { keywordSearch, toTsQuery } from "@mcpedia/search";
|
||||
export {
|
||||
keywordSearch,
|
||||
semanticSearch,
|
||||
hybridSearch,
|
||||
toTsQuery,
|
||||
cosine,
|
||||
} from "@mcpedia/search";
|
||||
export type { ChunkHit } from "@mcpedia/search";
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
CREATE TABLE "document_chunks" (
|
||||
"id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL,
|
||||
"document_id" text NOT NULL,
|
||||
"slug" text NOT NULL,
|
||||
"chunk_index" integer NOT NULL,
|
||||
"content" text NOT NULL,
|
||||
"embedding" real[],
|
||||
"created_at" timestamp with time zone DEFAULT now() NOT NULL
|
||||
);
|
||||
--> statement-breakpoint
|
||||
CREATE INDEX "document_chunks_slug_idx" ON "document_chunks" USING btree ("slug");
|
||||
--> statement-breakpoint
|
||||
ALTER TABLE "document_chunks" ADD CONSTRAINT "document_chunks_document_id_documents_id_fk" FOREIGN KEY ("document_id") REFERENCES "public"."documents"("id") ON DELETE cascade;
|
||||
@@ -0,0 +1,260 @@
|
||||
{
|
||||
"id": "fd210ce3-31f0-434d-9e80-6f5a0bb07504",
|
||||
"prevId": "249f80c0-d953-42f2-a98b-6701e2115856",
|
||||
"version": "7",
|
||||
"dialect": "postgresql",
|
||||
"tables": {
|
||||
"public.document_chunks": {
|
||||
"name": "document_chunks",
|
||||
"schema": "",
|
||||
"columns": {
|
||||
"id": {
|
||||
"name": "id",
|
||||
"type": "uuid",
|
||||
"primaryKey": true,
|
||||
"notNull": true,
|
||||
"default": "gen_random_uuid()"
|
||||
},
|
||||
"document_id": {
|
||||
"name": "document_id",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"slug": {
|
||||
"name": "slug",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"chunk_index": {
|
||||
"name": "chunk_index",
|
||||
"type": "integer",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"content": {
|
||||
"name": "content",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"embedding": {
|
||||
"name": "embedding",
|
||||
"type": "vector(2048)",
|
||||
"primaryKey": false,
|
||||
"notNull": false
|
||||
},
|
||||
"created_at": {
|
||||
"name": "created_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "now()"
|
||||
}
|
||||
},
|
||||
"indexes": {
|
||||
"document_chunks_embedding_idx": {
|
||||
"name": "document_chunks_embedding_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "embedding",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last",
|
||||
"opclass": "vector_cosine_ops"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"concurrently": false,
|
||||
"method": "hnsw",
|
||||
"with": {}
|
||||
},
|
||||
"document_chunks_slug_idx": {
|
||||
"name": "document_chunks_slug_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "slug",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
}
|
||||
},
|
||||
"foreignKeys": {
|
||||
"document_chunks_document_id_documents_id_fk": {
|
||||
"name": "document_chunks_document_id_documents_id_fk",
|
||||
"tableFrom": "document_chunks",
|
||||
"tableTo": "documents",
|
||||
"columnsFrom": [
|
||||
"document_id"
|
||||
],
|
||||
"columnsTo": [
|
||||
"id"
|
||||
],
|
||||
"onDelete": "cascade",
|
||||
"onUpdate": "no action"
|
||||
}
|
||||
},
|
||||
"compositePrimaryKeys": {},
|
||||
"uniqueConstraints": {},
|
||||
"policies": {},
|
||||
"checkConstraints": {},
|
||||
"isRLSEnabled": false
|
||||
},
|
||||
"public.documents": {
|
||||
"name": "documents",
|
||||
"schema": "",
|
||||
"columns": {
|
||||
"id": {
|
||||
"name": "id",
|
||||
"type": "text",
|
||||
"primaryKey": true,
|
||||
"notNull": true
|
||||
},
|
||||
"slug": {
|
||||
"name": "slug",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"title": {
|
||||
"name": "title",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"type": {
|
||||
"name": "type",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"section": {
|
||||
"name": "section",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"status": {
|
||||
"name": "status",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "'published'"
|
||||
},
|
||||
"author": {
|
||||
"name": "author",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "''"
|
||||
},
|
||||
"tags": {
|
||||
"name": "tags",
|
||||
"type": "text[]",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "'{}'"
|
||||
},
|
||||
"path": {
|
||||
"name": "path",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"body": {
|
||||
"name": "body",
|
||||
"type": "text",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"default": "''"
|
||||
},
|
||||
"search_vector": {
|
||||
"name": "search_vector",
|
||||
"type": "tsvector",
|
||||
"primaryKey": false,
|
||||
"notNull": true,
|
||||
"generated": {
|
||||
"as": "setweight(to_tsvector('simple', coalesce(\"documents\".\"title\", '')), 'A') || setweight(to_tsvector('simple', coalesce(\"documents\".\"body\", '')), 'B')",
|
||||
"type": "stored"
|
||||
}
|
||||
},
|
||||
"created_at": {
|
||||
"name": "created_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
},
|
||||
"updated_at": {
|
||||
"name": "updated_at",
|
||||
"type": "timestamp with time zone",
|
||||
"primaryKey": false,
|
||||
"notNull": true
|
||||
}
|
||||
},
|
||||
"indexes": {
|
||||
"documents_search_idx": {
|
||||
"name": "documents_search_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "search_vector",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"concurrently": false,
|
||||
"method": "gin",
|
||||
"with": {}
|
||||
},
|
||||
"documents_section_idx": {
|
||||
"name": "documents_section_idx",
|
||||
"columns": [
|
||||
{
|
||||
"expression": "section",
|
||||
"isExpression": false,
|
||||
"asc": true,
|
||||
"nulls": "last"
|
||||
}
|
||||
],
|
||||
"isUnique": false,
|
||||
"concurrently": false,
|
||||
"method": "btree",
|
||||
"with": {}
|
||||
}
|
||||
},
|
||||
"foreignKeys": {},
|
||||
"compositePrimaryKeys": {},
|
||||
"uniqueConstraints": {
|
||||
"documents_slug_unique": {
|
||||
"name": "documents_slug_unique",
|
||||
"nullsNotDistinct": false,
|
||||
"columns": [
|
||||
"slug"
|
||||
]
|
||||
}
|
||||
},
|
||||
"policies": {},
|
||||
"checkConstraints": {},
|
||||
"isRLSEnabled": false
|
||||
}
|
||||
},
|
||||
"enums": {},
|
||||
"schemas": {},
|
||||
"sequences": {},
|
||||
"roles": {},
|
||||
"policies": {},
|
||||
"views": {},
|
||||
"_meta": {
|
||||
"columns": {},
|
||||
"schemas": {},
|
||||
"tables": {}
|
||||
}
|
||||
}
|
||||
@@ -8,6 +8,13 @@
|
||||
"when": 1787133375079,
|
||||
"tag": "0000_grey_toro",
|
||||
"breakpoints": true
|
||||
},
|
||||
{
|
||||
"idx": 1,
|
||||
"version": "7",
|
||||
"when": 1787137149735,
|
||||
"tag": "0001_document_chunks",
|
||||
"breakpoints": true
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"exports": {
|
||||
".": "./src/client.ts",
|
||||
".": "./src/index.ts",
|
||||
"./schema": "./src/schema.ts"
|
||||
},
|
||||
"dependencies": {
|
||||
|
||||
@@ -0,0 +1,2 @@
|
||||
export { db, schema, client } from "./client";
|
||||
export * from "./schema";
|
||||
@@ -4,8 +4,11 @@ import {
|
||||
index,
|
||||
integer,
|
||||
pgTable,
|
||||
real,
|
||||
text,
|
||||
timestamp,
|
||||
uuid,
|
||||
vector,
|
||||
} from "drizzle-orm/pg-core";
|
||||
|
||||
// tsvector isn't a first-class drizzle type; wrap the raw Postgres type.
|
||||
@@ -46,14 +49,33 @@ export const documents = pgTable(
|
||||
}),
|
||||
);
|
||||
|
||||
// Phase 2 (semantic search) — defined here for reference, NOT created yet:
|
||||
// export const documentChunks = pgTable("document_chunks", {
|
||||
// id: text("id").primaryKey(),
|
||||
// documentId: text("document_id").notNull().references(() => documents.id, { onDelete: "cascade" }),
|
||||
// content: text("content").notNull(),
|
||||
// position: integer("position").notNull(),
|
||||
// embedding: customType<{ data: number[] }>({ dataType: () => "vector(1536)" })("embedding"),
|
||||
// });
|
||||
// Phase 2: semantic search chunks. Each row is an embedded slice of a document
|
||||
// body. `embedding` is a plain float array (real[]). We compute cosine
|
||||
// similarity in the application layer — pgvector isn't available on the shared
|
||||
// imrnes Postgres, and brute-force cosine is instant for a KB-sized corpus.
|
||||
// (pgvector/HNSW is the Phase-4 scale-out path.)
|
||||
export const documentChunks = pgTable(
|
||||
"document_chunks",
|
||||
{
|
||||
id: uuid("id").primaryKey().defaultRandom(),
|
||||
documentId: text("document_id")
|
||||
.notNull()
|
||||
.references(() => documents.id, { onDelete: "cascade" }),
|
||||
slug: text("slug").notNull(),
|
||||
chunkIndex: integer("chunk_index").notNull(),
|
||||
content: text("content").notNull(),
|
||||
embedding: real("embedding").array(),
|
||||
createdAt: timestamp("created_at", { withTimezone: true })
|
||||
.notNull()
|
||||
.defaultNow(),
|
||||
},
|
||||
(t) => ({
|
||||
slugIdx: index("document_chunks_slug_idx").on(t.slug),
|
||||
}),
|
||||
);
|
||||
|
||||
export type DocumentChunkRow = typeof documentChunks.$inferSelect;
|
||||
export type NewDocumentChunkRow = typeof documentChunks.$inferInsert;
|
||||
|
||||
export type DocumentRow = typeof documents.$inferSelect;
|
||||
export type NewDocumentRow = typeof documents.$inferInsert;
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"name": "@mcpedia/embeddings",
|
||||
"version": "0.1.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "./src/index.ts",
|
||||
"exports": {
|
||||
".": "./src/index.ts"
|
||||
},
|
||||
"dependencies": {
|
||||
"@mcpedia/config": "workspace:*",
|
||||
"@mcpedia/types": "workspace:*"
|
||||
},
|
||||
"devDependencies": {
|
||||
"typescript": "^5.6.0"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,48 @@
|
||||
import type { EmbeddingProvider } from "./provider";
|
||||
|
||||
/**
|
||||
* Split text into overlapping chunks for embedding. Keeps paragraphs/words
|
||||
* intact where possible; never splits a chunk mid-word by more than `overlap`.
|
||||
*/
|
||||
export function chunkText(
|
||||
text: string,
|
||||
opts: { size?: number; overlap?: number } = {},
|
||||
): string[] {
|
||||
const size = opts.size ?? 1000;
|
||||
const overlap = opts.overlap ?? 150;
|
||||
const clean = text.replace(/\r\n/g, "\n").trim();
|
||||
if (!clean) return [];
|
||||
if (clean.length <= size) return [clean];
|
||||
|
||||
const chunks: string[] = [];
|
||||
let start = 0;
|
||||
while (start < clean.length) {
|
||||
let end = Math.min(start + size, clean.length);
|
||||
// Prefer to break on a newline/space near the boundary.
|
||||
if (end < clean.length) {
|
||||
const nl = clean.lastIndexOf("\n", end);
|
||||
const sp = clean.lastIndexOf(" ", end);
|
||||
const breakAt = nl > start + size * 0.5 ? nl : sp > start + size * 0.5 ? sp : end;
|
||||
if (breakAt > start) end = breakAt;
|
||||
}
|
||||
chunks.push(clean.slice(start, end).trim());
|
||||
if (end >= clean.length) break;
|
||||
start = Math.max(end - overlap, start + 1);
|
||||
}
|
||||
return chunks.filter(Boolean);
|
||||
}
|
||||
|
||||
/** Embed a list of chunks in batches to avoid oversized requests. */
|
||||
export async function embedChunks(
|
||||
provider: EmbeddingProvider,
|
||||
chunks: string[],
|
||||
batchSize = 16,
|
||||
): Promise<number[][]> {
|
||||
const out: number[][] = [];
|
||||
for (let i = 0; i < chunks.length; i += batchSize) {
|
||||
const batch = chunks.slice(i, i + batchSize);
|
||||
const vecs = await provider.embed(batch);
|
||||
out.push(...vecs);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
export * from "./provider";
|
||||
export * from "./chunk";
|
||||
@@ -0,0 +1,82 @@
|
||||
import {
|
||||
EMBED_API_KEY,
|
||||
EMBED_BASE_URL,
|
||||
EMBED_MODEL,
|
||||
} from "@mcpedia/config";
|
||||
|
||||
export interface EmbeddingProvider {
|
||||
/** Embed a batch of texts into vectors of fixed dimension. */
|
||||
embed(texts: string[]): Promise<number[][]>;
|
||||
readonly model: string;
|
||||
readonly dimensions: number;
|
||||
}
|
||||
|
||||
/** Pinned embedding dimension for the configured OpenRouter model. */
|
||||
export const EMBED_DIM = 2048;
|
||||
|
||||
/**
|
||||
* OpenRouter embeddings provider (we route through 9router's OpenAI-compatible
|
||||
* /v1 endpoint). `encoding_format: "float"` is REQUIRED — the Nvidia-backed
|
||||
* model rejects base64.
|
||||
*/
|
||||
export class OpenRouterEmbeddingProvider implements EmbeddingProvider {
|
||||
readonly model: string;
|
||||
private readonly baseUrl: string;
|
||||
private readonly apiKey: string;
|
||||
|
||||
constructor(opts?: {
|
||||
baseUrl?: string;
|
||||
apiKey?: string;
|
||||
model?: string;
|
||||
}) {
|
||||
this.baseUrl = (opts?.baseUrl ?? EMBED_BASE_URL).replace(/\/$/, "");
|
||||
this.apiKey = opts?.apiKey ?? EMBED_API_KEY;
|
||||
this.model = opts?.model ?? EMBED_MODEL;
|
||||
if (!this.baseUrl || !this.apiKey || !this.model) {
|
||||
throw new Error(
|
||||
"OpenRouterEmbeddingProvider: missing EMBED_BASE_URL / EMBED_API_KEY / EMBED_MODEL",
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
get dimensions(): number {
|
||||
return EMBED_DIM;
|
||||
}
|
||||
|
||||
async embed(texts: string[]): Promise<number[][]> {
|
||||
if (texts.length === 0) return [];
|
||||
const res = await fetch(`${this.baseUrl}/embeddings`, {
|
||||
method: "POST",
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
Authorization: `Bearer ${this.apiKey}`,
|
||||
},
|
||||
body: JSON.stringify({
|
||||
model: this.model,
|
||||
input: texts,
|
||||
encoding_format: "float",
|
||||
}),
|
||||
});
|
||||
if (!res.ok) {
|
||||
const body = await res.text().catch(() => "");
|
||||
throw new Error(
|
||||
`embedding request failed (${res.status}): ${body.slice(0, 300)}`,
|
||||
);
|
||||
}
|
||||
const json = (await res.json()) as {
|
||||
data?: { embedding: number[] }[];
|
||||
};
|
||||
const data = json.data;
|
||||
if (!data || data.length !== texts.length) {
|
||||
throw new Error(
|
||||
`embedding response mismatch: expected ${texts.length}, got ${data?.length ?? 0}`,
|
||||
);
|
||||
}
|
||||
return data.map((d) => d.embedding);
|
||||
}
|
||||
}
|
||||
|
||||
/** Default singleton provider. */
|
||||
export function createEmbeddingProvider(): EmbeddingProvider {
|
||||
return new OpenRouterEmbeddingProvider();
|
||||
}
|
||||
@@ -8,6 +8,7 @@
|
||||
},
|
||||
"dependencies": {
|
||||
"@mcpedia/db": "workspace:*",
|
||||
"@mcpedia/embeddings": "workspace:*",
|
||||
"@mcpedia/types": "workspace:*",
|
||||
"drizzle-orm": "^0.38.0"
|
||||
}
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import { db } from "@mcpedia/db";
|
||||
import { documents, type DocumentRow } from "@mcpedia/db/schema";
|
||||
import { documents, documentChunks, type DocumentRow } from "@mcpedia/db/schema";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { createEmbeddingProvider } from "@mcpedia/embeddings";
|
||||
import type {
|
||||
DocSection,
|
||||
DocStatus,
|
||||
@@ -9,6 +10,23 @@ import type {
|
||||
SearchHit,
|
||||
} from "@mcpedia/types";
|
||||
|
||||
const embedder = createEmbeddingProvider();
|
||||
|
||||
/** Cosine similarity between two equal-length vectors. */
|
||||
export function cosine(a: number[], b: number[]): number {
|
||||
if (a.length === 0 || a.length !== b.length) return 0;
|
||||
let dot = 0;
|
||||
let na = 0;
|
||||
let nb = 0;
|
||||
for (let i = 0; i < a.length; i++) {
|
||||
dot += a[i] * b[i];
|
||||
na += a[i] * a[i];
|
||||
nb += b[i] * b[i];
|
||||
}
|
||||
const denom = Math.sqrt(na) * Math.sqrt(nb);
|
||||
return denom === 0 ? 0 : dot / denom;
|
||||
}
|
||||
|
||||
const VALID_SECTIONS: DocSection[] = ["docs", "writeups", "research", "notes"];
|
||||
const VALID_TYPES: DocType[] = ["documentation", "writeup", "research", "note"];
|
||||
|
||||
@@ -43,6 +61,99 @@ export function toTsQuery(q: string): string {
|
||||
return terms.map((t) => `${t}:*`).join(" & ");
|
||||
}
|
||||
|
||||
export interface ChunkHit {
|
||||
slug: string;
|
||||
chunkIndex: number;
|
||||
content: string;
|
||||
score: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Semantic search: embed the query, then rank document chunks by cosine
|
||||
* similarity. Cosine is computed in the app layer (pgvector isn't available on
|
||||
* the shared imrnes Postgres); for a KB-sized corpus this is instant.
|
||||
*/
|
||||
export async function semanticSearch(q: string, limit = 10): Promise<ChunkHit[]> {
|
||||
const query = q.trim();
|
||||
if (!query) return [];
|
||||
const [vec] = await embedder.embed([query]);
|
||||
if (!vec || vec.length === 0) return [];
|
||||
|
||||
const rows = await db
|
||||
.select({
|
||||
slug: documentChunks.slug,
|
||||
chunkIndex: documentChunks.chunkIndex,
|
||||
content: documentChunks.content,
|
||||
embedding: documentChunks.embedding,
|
||||
})
|
||||
.from(documentChunks)
|
||||
.where(sql`${documentChunks.embedding} IS NOT NULL`);
|
||||
|
||||
return rows
|
||||
.map((r) => ({
|
||||
slug: r.slug,
|
||||
chunkIndex: r.chunkIndex,
|
||||
content: r.content,
|
||||
score: cosine(vec, (r.embedding ?? []) as number[]),
|
||||
}))
|
||||
.filter((h) => h.score > 0)
|
||||
.sort((a, b) => b.score - a.score)
|
||||
.slice(0, limit);
|
||||
}
|
||||
|
||||
/**
|
||||
* Hybrid search: run FTS (ts_rank) and semantic (cosine) in parallel, then fuse
|
||||
* with Reciprocal Rank Fusion (RRF, k=60). Returns merged document-level hits.
|
||||
*/
|
||||
export async function hybridSearch(q: string, limit = 10): Promise<SearchHit[]> {
|
||||
const [fts, sem] = await Promise.all([keywordSearch(q, limit * 2), semanticSearch(q, limit * 2)]);
|
||||
const k = 60;
|
||||
const fused = new Map<string, { score: number; snippet: string; chunk: string }>();
|
||||
|
||||
fts.forEach((hit, i) => {
|
||||
const rrf = 1 / (k + i + 1);
|
||||
fused.set(hit.doc.slug, {
|
||||
score: (fused.get(hit.doc.slug)?.score ?? 0) + rrf,
|
||||
snippet: hit.snippet,
|
||||
chunk: "",
|
||||
});
|
||||
});
|
||||
sem.forEach((hit, i) => {
|
||||
const rrf = 1 / (k + i + 1);
|
||||
const prev = fused.get(hit.slug);
|
||||
fused.set(hit.slug, {
|
||||
score: (prev?.score ?? 0) + rrf,
|
||||
snippet: prev?.snippet ?? hit.content.slice(0, 160),
|
||||
chunk: prev?.chunk || hit.content,
|
||||
});
|
||||
});
|
||||
|
||||
const slugs = [...fused.entries()]
|
||||
.sort((a, b) => b[1].score - a[1].score)
|
||||
.slice(0, limit)
|
||||
.map(([slug]) => slug);
|
||||
|
||||
if (slugs.length === 0) return [];
|
||||
const rows = await db
|
||||
.select()
|
||||
.from(documents)
|
||||
.where(and(eq(documents.status, "published"), sql`${documents.slug} IN ${slugs}`));
|
||||
|
||||
const bySlug = new Map(rows.map((r) => [r.slug, r]));
|
||||
return slugs
|
||||
.map((slug, i) => {
|
||||
const row = bySlug.get(slug);
|
||||
if (!row) return null;
|
||||
const m = fused.get(slug)!;
|
||||
return {
|
||||
doc: toMeta(row),
|
||||
rank: m.score,
|
||||
snippet: m.snippet,
|
||||
} as SearchHit;
|
||||
})
|
||||
.filter((x): x is SearchHit => x !== null);
|
||||
}
|
||||
|
||||
/**
|
||||
* Postgres FTS keyword search over published documents.
|
||||
* Ranks by ts_rank and returns a headline snippet for display.
|
||||
|
||||
+15
-2
@@ -2,12 +2,13 @@ import { db } from "@mcpedia/db";
|
||||
import { documents } from "@mcpedia/db/schema";
|
||||
import { parseFile } from "@mcpedia/parser";
|
||||
import { CONTENT_ROOT } from "@mcpedia/config";
|
||||
import { listContentFiles } from "@mcpedia/core";
|
||||
import { listContentFiles, indexChunks } from "@mcpedia/core";
|
||||
import { join } from "node:path";
|
||||
|
||||
async function main() {
|
||||
const files = listContentFiles();
|
||||
let indexed = 0;
|
||||
let chunked = 0;
|
||||
for (const rel of files) {
|
||||
const abs = join(CONTENT_ROOT, rel);
|
||||
const { meta, body } = parseFile(abs, rel);
|
||||
@@ -47,8 +48,20 @@ async function main() {
|
||||
});
|
||||
indexed++;
|
||||
console.log(` indexed ${rel}`);
|
||||
|
||||
// Phase 2: chunk + embed for semantic search.
|
||||
try {
|
||||
const n = await indexChunks(meta.slug, body);
|
||||
chunked += n;
|
||||
console.log(` embedded ${n} chunks`);
|
||||
} catch (err) {
|
||||
console.error(
|
||||
` embed FAILED for ${meta.slug}: ${err instanceof Error ? err.message : err}`,
|
||||
);
|
||||
// Don't abort the whole index over one doc's embedding failure.
|
||||
}
|
||||
}
|
||||
console.log(`indexed ${indexed} documents`);
|
||||
console.log(`indexed ${indexed} documents, ${chunked} chunks embedded`);
|
||||
}
|
||||
|
||||
main()
|
||||
|
||||
@@ -7,6 +7,7 @@
|
||||
"@mcpedia/config": "workspace:*",
|
||||
"@mcpedia/core": "workspace:*",
|
||||
"@mcpedia/db": "workspace:*",
|
||||
"@mcpedia/embeddings": "workspace:*",
|
||||
"@mcpedia/parser": "workspace:*",
|
||||
"@mcpedia/search": "workspace:*",
|
||||
"drizzle-orm": "^0.38.0",
|
||||
|
||||
Reference in New Issue
Block a user