refactor: migrate messageStore to drizzle-orm

- Replace all raw SQL queries in messageStore.ts with Drizzle ORM queries - Remove DatabaseAdapter dependency from messageStore functions - Update all function signatures to be async and remove db parameter - Functions now use getDatabase() internally for database access - Update all call sites in messageCapture.ts, attachmentUploader.ts, aiAnalyzer.ts, webserver.ts, and index.ts - All functions remain backward compatible in behavior - TypeScript typecheck passes with no errors - All tests pass (11 passed)
2026-05-14 15:41:11 +07:00
parent dfe3444018
commit 1c4b0afbce
7 changed files with 868 additions and 192 deletions
--- a/docs/superpowers/plans/2026-05-14-drizzle-orm-migration.md
+++ b/docs/superpowers/plans/2026-05-14-drizzle-orm-migration.md
@@ -0,0 +1,704 @@
+# Drizzle ORM Migration Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Replace raw SQL queries and manual database adapter with Drizzle ORM, providing type-safe database operations, automatic migrations, and better maintainability while supporting both SQLite and PostgreSQL.
+
+**Architecture:** Replace the custom DatabaseAdapter pattern with Drizzle ORM's unified API. Define schema using Drizzle's TypeScript schema definitions. Replace all raw SQL queries in muxer-queue.ts and messageStore.ts with Drizzle query builder. Use Drizzle migrations for schema management. Maintain backward compatibility with existing data.
+
+**Tech Stack:** drizzle-orm, drizzle-kit, better-sqlite3 (SQLite), postgres (PostgreSQL), TypeScript
+
+---
+
+## File Structure
+
+**New files to create:**
+- `src/database/schema.ts` — Drizzle schema definitions for all tables
+- `src/database/drizzle.ts` — Drizzle database client initialization
+- `drizzle.config.ts` — Drizzle Kit configuration
+- `drizzle/migrations/` — Auto-generated migration files
+
+**Modified files:**
+- `src/muxer-queue.ts` — Replace raw SQL with Drizzle queries
+- `src/moderation/messageStore.ts` — Replace raw SQL with Drizzle queries
+- `src/database/adapter.ts` — Remove (no longer needed)
+- `src/database/postgres.ts` — Remove (Drizzle handles this)
+- `src/database/migrations.ts` — Remove (Drizzle handles this)
+- `src/index.ts` — Update database initialization
+- `src/webserver.ts` — Update database calls
+- `package.json` — Add drizzle-orm, drizzle-kit dependencies
+- `src/config.ts` — Keep PostgreSQL config variables
+
+---
+
+## Task 1: Add Drizzle Dependencies
+
+**Files:**
+- Modify: `package.json`
+
+- [ ] **Step 1: Add drizzle-orm and drizzle-kit**
+
+```bash
+cd /mnt/code/bete && pnpm add drizzle-orm
+```
+
+Expected: drizzle-orm installed
+
+- [ ] **Step 2: Add drizzle-kit as dev dependency**
+
+```bash
+cd /mnt/code/bete && pnpm add -D drizzle-kit
+```
+
+Expected: drizzle-kit installed
+
+- [ ] **Step 3: Verify installation**
+
+```bash
+cd /mnt/code/bete && pnpm list drizzle-orm drizzle-kit
+```
+
+Expected: Both packages listed with versions
+
+- [ ] **Step 4: Commit**
+
+```bash
+git add package.json pnpm-lock.yaml
+git commit -m "feat: add drizzle-orm and drizzle-kit dependencies"
+```
+
+---
+
+## Task 2: Create Drizzle Schema Definitions
+
+**Files:**
+- Create: `src/database/schema.ts`
+
+- [ ] **Step 1: Create schema.ts with table definitions**
+
+```typescript
+import { pgTable, text, integer, bigint, real, index, foreignKey } from "drizzle-orm/pg-core";
+import { sqliteTable, SQLiteInteger, SQLiteText } from "drizzle-orm/sqlite-core";
+import { config } from "../config";
+
+// Determine which table function to use based on database type
+const tableFactory = config.DATABASE_TYPE === "postgres" ? pgTable : sqliteTable;
+
+// Muxer Jobs Table
+export const muxerJobs = tableFactory("muxer_jobs", {
+  id: text("id").primaryKey(),
+  data: text("data").notNull(),
+  status: text("status", { enum: ["pending", "processing", "completed", "failed"] }).notNull().default("pending"),
+  attempts: integer("attempts").notNull().default(0),
+  maxAttempts: integer("maxAttempts").notNull().default(3),
+  createdAt: bigint("createdAt", { mode: "number" }).notNull(),
+  updatedAt: bigint("updatedAt", { mode: "number" }).notNull(),
+  error: text("error"),
+}, (table) => ({
+  statusIdx: index("idx_muxer_jobs_status").on(table.status),
+  createdAtIdx: index("idx_muxer_jobs_createdAt").on(table.createdAt),
+}));
+
+// Messages Table
+export const messages = tableFactory("messages", {
+  id: text("id").primaryKey(),
+  guild_id: text("guild_id").notNull(),
+  channel_id: text("channel_id").notNull(),
+  thread_id: text("thread_id"),
+  user_id: text("user_id").notNull(),
+  username: text("username").notNull(),
+  avatar_url: text("avatar_url"),
+  content: text("content").notNull(),
+  edited_content: text("edited_content"),
+  created_at: bigint("created_at", { mode: "number" }).notNull(),
+  edited_at: bigint("edited_at", { mode: "number" }),
+  deleted_at: bigint("deleted_at", { mode: "number" }),
+  type: text("type", { enum: ["text", "edited", "deleted"] }).notNull().default("text"),
+  metadata: text("metadata"),
+  ai_status: text("ai_status", { enum: ["pending", "clean", "warn", "flagged", "error"] }).notNull().default("pending"),
+  ai_moderation_flags: text("ai_moderation_flags"),
+  ai_moderation_score: real("ai_moderation_score"),
+  ai_moderation_raw: text("ai_moderation_raw"),
+  ai_analysis: text("ai_analysis"),
+  ai_analyzed_at: bigint("ai_analyzed_at", { mode: "number" }),
+  ai_error: text("ai_error"),
+}, (table) => ({
+  channelIdx: index("idx_messages_channel").on(table.channel_id),
+  userIdx: index("idx_messages_user").on(table.user_id),
+  createdIdx: index("idx_messages_created").on(table.created_at),
+  threadIdx: index("idx_messages_thread").on(table.thread_id),
+}));
+
+// Attachments Table
+export const attachments = tableFactory("attachments", {
+  id: text("id").primaryKey(),
+  message_id: text("message_id").notNull(),
+  guild_id: text("guild_id").notNull(),
+  channel_id: text("channel_id").notNull(),
+  thread_id: text("thread_id"),
+  user_id: text("user_id").notNull(),
+  filename: text("filename").notNull(),
+  size: integer("size").notNull(),
+  type: text("type").notNull(),
+  discord_url: text("discord_url").notNull(),
+  uploaded_url: text("uploaded_url"),
+  upload_status: text("upload_status", { enum: ["pending", "uploaded", "failed"] }).notNull().default("pending"),
+  upload_error: text("upload_error"),
+  created_at: bigint("created_at", { mode: "number" }).notNull(),
+  uploaded_at: bigint("uploaded_at", { mode: "number" }),
+}, (table) => ({
+  channelIdx: index("idx_attachments_channel").on(table.channel_id),
+  messageIdx: index("idx_attachments_message").on(table.message_id),
+  statusIdx: index("idx_attachments_status").on(table.upload_status),
+  fk: foreignKey({
+    columns: [table.message_id],
+    foreignColumns: [messages.id],
+  }).onDelete("cascade"),
+}));
+
+// UI State Table
+export const uiState = tableFactory("ui_state", {
+  key: text("key").primaryKey(),
+  value: text("value").notNull(),
+  updated_at: bigint("updated_at", { mode: "number" }).notNull(),
+});
+```
+
+- [ ] **Step 2: Run typecheck**
+
+```bash
+cd /mnt/code/bete && pnpm run typecheck
+```
+
+Expected: No TypeScript errors
+
+- [ ] **Step 3: Commit**
+
+```bash
+git add src/database/schema.ts
+git commit -m "feat: create drizzle schema definitions"
+```
+
+---
+
+## Task 3: Create Drizzle Configuration
+
+**Files:**
+- Create: `drizzle.config.ts`
+
+- [ ] **Step 1: Create drizzle.config.ts**
+
+```typescript
+import { defineConfig } from "drizzle-kit";
+import { config } from "./src/config";
+
+export default defineConfig({
+  schema: "./src/database/schema.ts",
+  out: "./drizzle/migrations",
+  dialect: config.DATABASE_TYPE === "postgres" ? "postgresql" : "sqlite",
+  dbCredentials: config.DATABASE_TYPE === "postgres" 
+    ? {
+        host: config.POSTGRES_HOST,
+        port: config.POSTGRES_PORT,
+        user: config.POSTGRES_USER,
+        password: config.POSTGRES_PASSWORD,
+        database: config.POSTGRES_DB,
+      }
+    : {
+        url: `file:./.muxer-queue.db`,
+      },
+});
+```
+
+- [ ] **Step 2: Add migration scripts to package.json**
+
+```json
+"scripts": {
+  "db:generate": "drizzle-kit generate",
+  "db:migrate": "drizzle-kit migrate",
+  "db:studio": "drizzle-kit studio"
+}
+```
+
+- [ ] **Step 3: Generate initial migration**
+
+```bash
+cd /mnt/code/bete && pnpm run db:generate
+```
+
+Expected: Migration files created in drizzle/migrations/
+
+- [ ] **Step 4: Commit**
+
+```bash
+git add drizzle.config.ts package.json drizzle/
+git commit -m "feat: add drizzle configuration and initial migrations"
+```
+
+---
+
+## Task 4: Create Drizzle Database Client
+
+**Files:**
+- Create: `src/database/drizzle.ts`
+
+- [ ] **Step 1: Create drizzle.ts**
+
+```typescript
+import { drizzle } from "drizzle-orm/node-postgres";
+import { drizzle as drizzleSqlite } from "drizzle-orm/better-sqlite3";
+import Database from "better-sqlite3";
+import { Pool } from "pg";
+import { config } from "../config";
+import { createChildLogger } from "../logger";
+import * as schema from "./schema";
+
+const logger = createChildLogger("drizzle");
+
+let db: ReturnType<typeof drizzle> | null = null;
+
+export async function initializeDatabase() {
+  if (db) return db;
+
+  if (config.DATABASE_TYPE === "postgres") {
+    const pool = new Pool({
+      host: config.POSTGRES_HOST,
+      port: config.POSTGRES_PORT,
+      user: config.POSTGRES_USER,
+      password: config.POSTGRES_PASSWORD,
+      database: config.POSTGRES_DB,
+      min: config.POSTGRES_POOL_MIN,
+      max: config.POSTGRES_POOL_MAX,
+    });
+
+    db = drizzle(pool, { schema });
+    logger.info("PostgreSQL database initialized");
+  } else {
+    const sqlite = new Database(".muxer-queue.db");
+    sqlite.pragma("journal_mode = WAL");
+    db = drizzleSqlite(sqlite, { schema });
+    logger.info("SQLite database initialized");
+  }
+
+  return db;
+}
+
+export function getDatabase() {
+  if (!db) {
+    throw new Error("Database not initialized. Call initializeDatabase() first.");
+  }
+  return db;
+}
+
+export async function closeDatabase() {
+  if (db) {
+    // Drizzle doesn't have a close method, but we can close the underlying connection
+    if (config.DATABASE_TYPE === "postgres") {
+      // Pool will be closed when the process exits
+      logger.info("PostgreSQL connection pool will close on process exit");
+    } else {
+      logger.info("SQLite database closed");
+    }
+    db = null;
+  }
+}
+```
+
+- [ ] **Step 2: Run typecheck**
+
+```bash
+cd /mnt/code/bete && pnpm run typecheck
+```
+
+Expected: No TypeScript errors
+
+- [ ] **Step 3: Commit**
+
+```bash
+git add src/database/drizzle.ts
+git commit -m "feat: create drizzle database client"
+```
+
+---
+
+## Task 5: Migrate muxer-queue.ts to Drizzle
+
+**Files:**
+- Modify: `src/muxer-queue.ts`
+
+- [ ] **Step 1: Replace imports**
+
+Replace:
+```typescript
+import { getDatabase, DatabaseAdapter } from "./database/adapter";
+```
+
+With:
+```typescript
+import { getDatabase, initializeDatabase } from "./database/drizzle";
+import { muxerJobs } from "./database/schema";
+import { eq, asc, desc } from "drizzle-orm";
+```
+
+- [ ] **Step 2: Replace enqueueMuxerJob function**
+
+Replace raw SQL with:
+```typescript
+export async function enqueueMuxerJob(data: MuxerJobData): Promise<string> {
+  try {
+    const db = getDatabase();
+    const jobId = `${data.userId}-${data.sessionId}`;
+    const now = Date.now();
+
+    await db.insert(muxerJobs).values({
+      id: jobId,
+      data: JSON.stringify(data),
+      status: "pending",
+      attempts: 0,
+      maxAttempts: 3,
+      createdAt: now,
+      updatedAt: now,
+    }).onConflictDoNothing();
+
+    logger.info({ jobId, userId: data.userId }, "Muxer job enqueued");
+    return jobId;
+  } catch (error) {
+    logger.error({ error: error instanceof Error ? error.message : String(error) }, "Failed to enqueue muxer job");
+    throw error;
+  }
+}
+```
+
+- [ ] **Step 3: Replace getPendingJobs function**
+
+```typescript
+export async function getPendingJobs(): Promise<StoredJob[]> {
+  const db = getDatabase();
+  const rows = await db
+    .select()
+    .from(muxerJobs)
+    .where(eq(muxerJobs.status, "pending"))
+    .orderBy(asc(muxerJobs.createdAt))
+    .limit(10);
+
+  return rows.map((row) => ({
+    ...row,
+    status: row.status as "pending" | "processing" | "completed" | "failed",
+  }));
+}
+```
+
+- [ ] **Step 4: Replace updateJobStatus function**
+
+```typescript
+export async function updateJobStatus(
+  jobId: string,
+  status: "processing" | "completed" | "failed",
+  error?: string,
+): Promise<void> {
+  const db = getDatabase();
+  const now = Date.now();
+
+  if (status === "failed") {
+    await db
+      .update(muxerJobs)
+      .set({
+        status,
+        attempts: muxerJobs.attempts + 1,
+        updatedAt: now,
+        error: error || null,
+      })
+      .where(eq(muxerJobs.id, jobId));
+  } else {
+    await db
+      .update(muxerJobs)
+      .set({ status, updatedAt: now })
+      .where(eq(muxerJobs.id, jobId));
+  }
+
+  logger.info({ jobId, status, error }, "Job status updated");
+}
+```
+
+- [ ] **Step 5: Replace remaining functions similarly**
+
+Replace `retryFailedJob`, `cleanupCompletedJobs`, `getJobStats` with Drizzle equivalents
+
+- [ ] **Step 6: Update getPersistedValue and setPersistedValue**
+
+Use Drizzle's uiState table instead of raw SQL
+
+- [ ] **Step 7: Run tests**
+
+```bash
+cd /mnt/code/bete && pnpm run test
+```
+
+Expected: All tests pass
+
+- [ ] **Step 8: Commit**
+
+```bash
+git add src/muxer-queue.ts
+git commit -m "refactor: migrate muxer-queue to drizzle-orm"
+```
+
+---
+
+## Task 6: Migrate messageStore.ts to Drizzle
+
+**Files:**
+- Modify: `src/moderation/messageStore.ts`
+
+- [ ] **Step 1: Replace imports**
+
+```typescript
+import { getDatabase } from "../database/drizzle";
+import { messages, attachments } from "../database/schema";
+import { eq, or, desc, and } from "drizzle-orm";
+```
+
+- [ ] **Step 2: Replace insertMessage function**
+
+```typescript
+export async function insertMessage(message: MessageRecord): Promise<void> {
+  try {
+    const db = getDatabase();
+    await db.insert(messages).values(message).onConflictDoNothing();
+    logger.debug({ messageId: message.id }, "Message inserted");
+  } catch (error) {
+    logger.error({ messageId: message.id, error: error instanceof Error ? error.message : String(error) }, "Failed to insert message");
+    throw error;
+  }
+}
+```
+
+- [ ] **Step 3: Replace updateMessageAsEdited function**
+
+```typescript
+export async function updateMessageAsEdited(
+  messageId: string,
+  editedContent: string,
+  editedAt: number,
+): Promise<void> {
+  try {
+    const db = getDatabase();
+    await db
+      .update(messages)
+      .set({ edited_content: editedContent, edited_at: editedAt, type: "edited" })
+      .where(eq(messages.id, messageId));
+    logger.debug({ messageId }, "Message marked as edited");
+  } catch (error) {
+    logger.error({ messageId, error: error instanceof Error ? error.message : String(error) }, "Failed to update message as edited");
+    throw error;
+  }
+}
+```
+
+- [ ] **Step 4: Replace getMessagesByChannel function**
+
+```typescript
+export async function getMessagesByChannel(
+  channelId: string,
+  limit: number = 50,
+  offset: number = 0,
+): Promise<MessageRecord[]> {
+  try {
+    const db = getDatabase();
+    return await db
+      .select()
+      .from(messages)
+      .where(or(eq(messages.channel_id, channelId), eq(messages.thread_id, channelId)))
+      .orderBy(desc(messages.created_at))
+      .limit(limit)
+      .offset(offset);
+  } catch (error) {
+    logger.error({ channelId, error: error instanceof Error ? error.message : String(error) }, "Failed to get messages by channel");
+    throw error;
+  }
+}
+```
+
+- [ ] **Step 5: Replace attachment functions similarly**
+
+Replace `insertAttachment`, `getAttachmentsByChannel`, `updateAttachmentAsUploaded`, `updateAttachmentAsFailedUpload` with Drizzle equivalents
+
+- [ ] **Step 6: Replace AI analysis functions**
+
+Replace `updateMessageAIAnalysis`, `getPendingAIAnalysisMessages`, `getMessageById` with Drizzle equivalents
+
+- [ ] **Step 7: Update function signatures**
+
+Remove `db: DatabaseAdapter` parameter from all functions since they now use `getDatabase()` internally
+
+- [ ] **Step 8: Run tests**
+
+```bash
+cd /mnt/code/bete && pnpm run test
+```
+
+Expected: All tests pass
+
+- [ ] **Step 9: Commit**
+
+```bash
+git add src/moderation/messageStore.ts
+git commit -m "refactor: migrate messageStore to drizzle-orm"
+```
+
+---
+
+## Task 7: Update Application Initialization
+
+**Files:**
+- Modify: `src/index.ts`
+- Modify: `src/webserver.ts`
+
+- [ ] **Step 1: Update src/index.ts imports**
+
+Replace:
+```typescript
+import { getDatabase } from "./database/adapter";
+```
+
+With:
+```typescript
+import { initializeDatabase } from "./database/drizzle";
+```
+
+- [ ] **Step 2: Update database initialization in index.ts**
+
+```typescript
+const db = await initializeDatabase();
+logger.info({ type: config.DATABASE_TYPE }, "Database initialized");
+```
+
+- [ ] **Step 3: Update src/webserver.ts**
+
+Replace any `getDatabase()` calls with the new Drizzle client
+
+- [ ] **Step 4: Run typecheck**
+
+```bash
+cd /mnt/code/bete && pnpm run typecheck
+```
+
+Expected: No TypeScript errors
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add src/index.ts src/webserver.ts
+git commit -m "feat: update application initialization for drizzle"
+```
+
+---
+
+## Task 8: Remove Old Database Files
+
+**Files:**
+- Delete: `src/database/adapter.ts`
+- Delete: `src/database/postgres.ts`
+- Delete: `src/database/migrations.ts`
+
+- [ ] **Step 1: Remove old adapter files**
+
+```bash
+cd /mnt/code/bete && rm src/database/adapter.ts src/database/postgres.ts src/database/migrations.ts
+```
+
+- [ ] **Step 2: Verify no imports remain**
+
+```bash
+grep -r "database/adapter\|database/postgres\|database/migrations" src/ --include="*.ts"
+```
+
+Expected: No results
+
+- [ ] **Step 3: Commit**
+
+```bash
+git add -A
+git commit -m "refactor: remove old database adapter files"
+```
+
+---
+
+## Task 9: Final Testing and Verification
+
+**Files:**
+- Test all functionality
+
+- [ ] **Step 1: Run full test suite**
+
+```bash
+cd /mnt/code/bete && pnpm run test
+```
+
+Expected: All tests pass
+
+- [ ] **Step 2: Type check**
+
+```bash
+cd /mnt/code/bete && pnpm run typecheck
+```
+
+Expected: No TypeScript errors
+
+- [ ] **Step 3: Lint**
+
+```bash
+cd /mnt/code/bete && pnpm run lint
+```
+
+Expected: No linting errors
+
+- [ ] **Step 4: Test startup with SQLite**
+
+```bash
+cd /mnt/code/bete && timeout 10 pnpm run dev || true
+```
+
+Expected: Bot starts successfully, logs show "Database initialized"
+
+- [ ] **Step 5: Verify git status**
+
+```bash
+git status
+```
+
+Expected: Clean working tree
+
+- [ ] **Step 6: Final commit if needed**
+
+```bash
+git add -A
+git commit -m "feat: complete drizzle-orm migration"
+```
+
+---
+
+## Spec Coverage Checklist
+
+- ✅ Replace raw SQL with Drizzle ORM
+- ✅ Type-safe database operations
+- ✅ Support both SQLite and PostgreSQL
+- ✅ Automatic schema migrations
+- ✅ All existing functionality preserved
+- ✅ Backward compatible with existing data
+- ✅ Cleaner, more maintainable code
+- ✅ Better error handling
+- ✅ Tests passing
+- ✅ No TypeScript errors
+
+---
+
+Plan complete and saved to `/mnt/code/bete/docs/superpowers/plans/2026-05-14-drizzle-orm-migration.md`.
+
+**Two execution options:**
+
+**1. Subagent-Driven (recommended)** - I dispatch a fresh subagent per task, review between tasks, fast iteration
+
+**2. Inline Execution** - Execute tasks in this session using executing-plans, batch execution with checkpoints
+
+Which approach would you prefer?