feat(gateway): refactor term extraction and scoring logic into textSignals.ts for reuse

.env.example

@@ -134,4 +134,7 @@ RETENTION_DRY_RUN=true                  # Dry-run: log but do not delete (defaul
 AUTO_MIGRATE_ON_STARTUP=true            # Run database migrations on startup (default: true)
 
 # === Worker Pool ===
-# PISCINA_MAX_THREADS=4                   # Worker thread pool size (optional, defaults to CPU cores)
+# Text and media AI-analysis batches run on separate Piscina pools so a slow
+# image/vision batch can never queue-block fast text-only batches.
+# PISCINA_MAX_THREADS=4                   # Text-analysis worker pool size (optional, defaults to CPU cores)
+# PISCINA_MEDIA_MAX_THREADS=2             # Media-analysis worker pool size (optional, default 2)

.hermes/plans/2026-08-31_video-receive-eager-selfbot-connection-spec.md

@@ -0,0 +1,93 @@
+# Spec: Fix Video Capture — Eagerly Establish the Selfbot Voice Connection at Join Time
+
+Status: PLANNED
+Date: 2026-08-31
+Author: Hermes
+Related: `.hermes/plans/2026-08-31_video-receive-phaseC-spec.md` (Phase C build, made Option A this fix)
+
+## Symptom (from live logs, 2026-08-31 ~12:34)
+A user was actively screen-sharing + on camera in the recorded voice channel.
+The gateway recorded MANY users' audio (.ogg) fine, but video capture produced
+nothing. The only video signal in `journalctl -u gmw-discord-gateway` was:
+
+```
+[VOICE (guild:2)]: Sending voice state update: {"self_mute":false,...,"flags":2}
+[VOICE] received voice state update: {member hunterz ...}   # OTHER user, not bot
+[VOICE] connection? true, guild session channel
+[VOICE (guild:2)]: Setting sessionId <S> (stored as "undefined")
+[VOICE (guild:2)]: Authenticated with sessionId <S>         # debug print only
+[VOICE (guild:2)]: Authenticate failed - VOICE_CONNECTION_TIMEOUT   # +15s
+video-recorder: userId=..., "Connection not established within 15 seconds."
+```
+
+## Root cause (verified against discord.js-selfbot-v13 3.7.1 source)
+The gateway records audio via `@discordjs/voice` (`joinVoiceChannel` + adapter).
+Video receive lives on the SEPARATE selfbot `ClientVoiceManager.connection`
+(a singleton `VoiceConnection`). `videoRecorder.ts` currently calls
+`client.voice.joinChannel(channel)` LAZILY — only when a `voiceStateUpdate`
+shows `newState.streaming === true`.
+
+At that moment the bot is ALREADY connected to the channel via @discordjs/voice.
+A selfbot `joinChannel` then does `VoiceConnection.authenticate()` →
+`sendVoiceStateUpdate()`, and waits for a fresh `VOICE_SERVER_UPDATE`
+(`setTokenAndEndpoint`) + `VOICE_STATE_UPDATE` (`setSessionId`) to reach
+`checkAuthenticated()` (needs token+endpoint+sessionId). Because the bot is
+already in an established voice session, Discord does NOT emit a new
+`VOICE_SERVER_UPDATE` for the lazy selfbot re-join → token/endpoint never set →
+15s `VOICE_CONNECTION_TIMEOUT`.
+
+This is fatal to video: `joinStreamConnection(userId)` (STREAM_WATCH op 20) and
+`receiver.createVideoStream(userId, out)` (Recorder/ffmpeg) BOTH live on the
+parent selfbot `VoiceConnection` and require it `CONNECTED` (its own voice
+WS+UDP socket feeds `PacketHandler.push`, authenticated with
+`authentication.secret_key`).
+
+## Fix — establish the selfbot connection eagerly, at voice-join time
+The selfbot `VoiceConnection` must exist and be `CONNECTED` before any streamer
+appears. Establish it once, synchronously alongside the @discordjs/voice join in
+`recorder.startRecording`, so it rides the bot's FRESH voice join — when Discord
+DOES emit VOICE_SERVER_UPDATE. Then cache it and let `videoRecorder` reuse it.
+
+Ordering: in `startRecording`, after the @discordjs/voice `joinVoiceChannel`
+returns (and retries) — fire `ensureSelfbotVoice(channel)` best-effort:
+1. `await client.voice.joinChannel(channel, { selfMute:false, selfDeaf:false,
+   selfVideo:false })` (rejects ~VOICE_CONNECTION_TIMEOUT on failure → log +
+   return null; do NOT block audio).
+2. Cache the returned selfbot `VoiceConnection` keyed by guildId.
+3. Wire teardown: on `recorder` voice stop / destroyed → `untrackChannel` +
+   destroy the cached selfbot connection (`disconnect()`).
+
+`videoRecorder.startVideoRecording` then uses the cached selfbot connection:
+- If cached & `status === CONNECTED` → use it.
+- Else → fall back to a lazy `joinChannel` (still best-effort).
+
+## The two-connection coexistence risk (must verify live)
+@discordjs/voice (audio) and the selfbot `VoiceConnection` (video) each open
+their OWN low-level voice WS+UDP on the same session. The spec's original
+open-question flagged this. Mitigations:
+- Clear logging: `Selfbot voice connected (guild=...)`, plus a periodic
+  `djs/voice status` log so we can confirm audio stays `READY` while the selfbot
+  connection is up.
+- If Discord kicks/breaks the audio connection, logs will show
+  @discordjs/voice `Disconnected`/reconnect churn — we detect and pivot.
+
+## Files touched
+- `services/discord-gateway/src/modules/voice-recording/videoRecorder.ts`:
+  add `ensureSelfbotVoice(channel)` (return cached/connected), use it in
+  `startVideoRecording`, add `destroyGuildSelfbotVoice(guildId)`,
+  richer status logging.
+- `services/discord-gateway/src/modules/voice-recording/recorder.ts`: call
+  `ensureSelfbotVoice(channel)` after `joinVoiceChannel` (best-effort);
+  call `destroyGuildSelfbotVoice` on voice stop/destroy.
+- Tests: `tests/videoRecorder.test.ts` (update to assert eager-connection reuse
+  + status gating).
+
+## Verification
+1. `pnpm typecheck` + `pnpm build` + biome clean (discord-gateway).
+2. Tests green.
+3. Commit + push; CI `Build & Deploy (Nix)` green, service restarts.
+4. LIVE (deploy): join a channel with the bot → journal shows
+   `Selfbot voice connected` (parent CONNECTED). When a member streams →
+   `Sender signal screenshare` / `Video recorder ready` + a `.mkv` under
+   `<RECORDINGS_DIR>/<uid>/video-*.mkv`; playable via ffmpeg. Confirm audio
+   recording still flows (no djs/voice reconnect churn).

services/backend/src/shared/config/index.ts

@@ -193,6 +193,7 @@ export const configSchema = z
       .positive()
       .default(50),
     PISCINA_MAX_THREADS: z.coerce.number().int().positive().optional(),
+    PISCINA_MEDIA_MAX_THREADS: z.coerce.number().int().positive().optional(),
 
     // ── Voice Transcription ────────────────────────────────────────────────
     AI_VOICE_TRANSCRIPTION_ENABLED: z

services/discord-gateway/ARCHITECTURE.md

@@ -77,20 +77,26 @@ handles a whole batch (text + media split internally, parallel paths).
 
 - Main thread owns the LLM semaphore (`AI_LLM_MAX_CONCURRENT`, default 5) via
   `llmClient.withLlmConcurrency`.
-- Piscina pool (`PISCINA_MAX_THREADS`, default 4) runs the heavy LLM work off
-  the event loop; **each worker thread initializes its own pg Pool** (min 0,
-  grows to `POSTGRES_POOL_MAX`). See "Memory & connections" below.
+- Two Piscina pools run the heavy LLM work off the event loop: a text pool
+  (`PISCINA_MAX_THREADS`, default 4) and a dedicated media pool
+  (`PISCINA_MEDIA_MAX_THREADS`, default 2). A batch is routed to the media
+  pool if ANY of its messages carries an attachment/sticker/embed — this
+  keeps a slow image/vision batch from occupying every thread and blocking
+  unrelated text-only batches behind it. **Each worker thread (in either
+  pool) initializes its own pg Pool** (min 0, grows to `POSTGRES_POOL_MAX`).
+  See "Memory & connections" below.
 
 ## Memory & DB connections
 
 `MemoryMax=1G` (raised from 512M — live RSS sits at ~500 MiB, peak 508 MiB,
 so 512M left ~2% headroom and risked an OOM-kill restart). Host has 8 GB free.
 
-`POSTGRES_POOL_MIN=0` (default). The gateway = main process + up to 4 Piscina
-worker threads, each with its own pg Pool. With min:0 the pools stay empty
-until a query runs and drop idle clients afterward, instead of holding
-`(1 main + 4 workers) × 2 = 10` permanently-open idle connections against
-PgBouncer. The pool still grows on demand up to `POSTGRES_POOL_MAX`.
+`POSTGRES_POOL_MIN=0` (default). The gateway = main process + up to 4 text
+Piscina worker threads + up to 2 media Piscina worker threads, each with its
+own pg Pool. With min:0 the pools stay empty until a query runs and drop
+idle clients afterward, instead of holding `(1 main + 4 text + 2 media) × 2
+= 14` permanently-open idle connections against PgBouncer. The pool still
+grows on demand up to `POSTGRES_POOL_MAX`.
 
 ## Event channels (Redis pub/sub)
 

services/discord-gateway/src/app/bootstrap.ts

@@ -7,7 +7,10 @@ import {
   getAnalysisQueueStatus,
   startPendingAIAnalysisWorker,
 } from "../modules/ai-moderation/aiAnalyzer.js";
-import { workerPool } from "../modules/ai-moderation/circuitBreaker.js";
+import {
+  mediaWorkerPool,
+  textWorkerPool,
+} from "../modules/ai-moderation/circuitBreaker.js";
 import { registerChannelTopicCapture } from "../modules/channel-topic/index.js";
 import { CommandHandler } from "../modules/command-handler/commandHandler.js";
 import {
@@ -396,12 +399,25 @@ export async function initializeDiscordGateway() {
       if (typeof status.lastError === "string") {
         setGauge("ai_analysis_last_error_present", status.lastError ? 1 : 0);
       }
-      const pool = workerPool as unknown as {
-        _poolState?: { size: number; active: number };
-      };
-      if (pool._poolState) {
-        setGauge("ai_analysis_worker_threads", pool._poolState.size);
-        setGauge("ai_analysis_worker_threads_active", pool._poolState.active);
+      type PoolState = { _poolState?: { size: number; active: number } };
+      const textPool = textWorkerPool as unknown as PoolState;
+      const mediaPool = mediaWorkerPool as unknown as PoolState;
+      // Reported per queue (2026-08-31 text/media pool split) so the text
+      // and media backlogs are distinguishable in dashboards/alerts instead
+      // of one combined "worker threads" number.
+      if (textPool._poolState) {
+        setGauge("ai_analysis_worker_threads_text", textPool._poolState.size);
+        setGauge(
+          "ai_analysis_worker_threads_active_text",
+          textPool._poolState.active,
+        );
+      }
+      if (mediaPool._poolState) {
+        setGauge("ai_analysis_worker_threads_media", mediaPool._poolState.size);
+        setGauge(
+          "ai_analysis_worker_threads_active_media",
+          mediaPool._poolState.active,
+        );
       }
     } catch (err) {
       logger.warn({ error: String(err) }, "AI metrics collector failed");

services/discord-gateway/src/modules/ai-moderation/batchProcessor.ts

@@ -5,7 +5,7 @@ import { messageStore } from "../message-capture/messageStore.js";
 import type { MessageRecord } from "../message-capture/types.js";
 import { pickBatchWithinBudget as pickBatchWithinBudgetPure } from "./batchBudget.js";
 import { partitionBatchOutcome } from "./batchOutcomeClassifier.js";
-import { workerPool } from "./circuitBreaker.js";
+import { mediaWorkerPool, textWorkerPool } from "./circuitBreaker.js";
 import { estimateTokens } from "./conversationContext.js";
 import {
   conversationErrorCooldown,
@@ -14,6 +14,7 @@ import {
   resetConversationBatchFailures,
 } from "./conversationState.js";
 import { enqueueIndividualFallbacks } from "./individualFallbackProcessor.js";
+import { hasMediaContent } from "./mediaAnalysisClient.js";
 import {
   broadcastAnalysisCompleted,
   LAST_ERROR,
@@ -121,29 +122,31 @@ export async function skipAgeRestrictedMessages(
 // Batch pipeline
 // ---------------------------------------------------------------------------
 
-export async function processBatch(
+/**
+ * Runs one worker job (either the text-only or the media sub-batch of a
+ * conversation) end-to-end: dispatch → broadcast/save → fallback routing.
+ * Returns whether the *caller* should schedule the next debounce pass for
+ * this conversation (mirrors the old single-job semantics, now evaluated
+ * per queue).
+ *
+ * Broadcasting happens here, inside each queue's own call — NOT after
+ * waiting on the other queue. That's the actual fix for "text menunggu
+ * image": previously one mixed conversation batch made ONE worker call
+ * with both text and media targets, and `runModerationAnalysis` only
+ * resolves (so results only get saved/broadcast) once BOTH finish — so a
+ * fast text verdict sat unused until the slow vision/image verdict was
+ * ready too. Splitting into two independent jobs means the text queue
+ * saves+broadcasts its rows the moment IT finishes, regardless of how long
+ * the media queue takes.
+ */
+async function runQueueBatch(
+  pool: typeof textWorkerPool,
   conversationKey: string,
   messages: MessageRecord[],
-  processingStartedAt: number,
-): Promise<void> {
-  if (messages.length === 0) {
-    if (conversationProcessing.get(conversationKey) === processingStartedAt) {
-      conversationProcessing.delete(conversationKey);
-    }
-    return;
-  }
-  const cooldownUntil = conversationErrorCooldown.get(conversationKey) ?? 0;
-  if (Date.now() < cooldownUntil) {
-    if (conversationProcessing.get(conversationKey) === processingStartedAt) {
-      conversationProcessing.delete(conversationKey);
-    }
-    return;
-  }
-
+): Promise<boolean> {
   activeRequests++;
-  let shouldScheduleNext = false;
   try {
-    const result = (await workerPool.run({
+    const result = (await pool.run({
       type: "batch",
       conversationKey,
       messages,
@@ -191,7 +194,7 @@ export async function processBatch(
         },
         "Batch analysis failed, will retry after cooldown",
       );
-      return;
+      return false;
     }
 
     // Batch succeeded -- partition per-message outcome (2026-08-25).
@@ -281,20 +284,13 @@ export async function processBatch(
         conversationErrorCooldown.set(conversationKey, newCooldown);
       }
 
-      // Release the processing lock immediately so the cooldown timer controls retry
-      if (conversationProcessing.get(conversationKey) === processingStartedAt) {
-        conversationProcessing.delete(conversationKey);
-      }
-
       // Do NOT schedule next -- let the cooldown gate it
-      shouldScheduleNext = false;
+      return false;
     }
 
-    if (apiFailedMessages.length === 0) {
-      resetConversationBatchFailures(conversationKey);
-      conversationErrorCooldown.delete(conversationKey);
-    }
-    shouldScheduleNext = true;
+    resetConversationBatchFailures(conversationKey);
+    conversationErrorCooldown.delete(conversationKey);
+    return true;
   } catch (error) {
     recordConversationBatchFailure(conversationKey);
 
@@ -326,18 +322,63 @@ export async function processBatch(
       },
       "Analysis worker failed, will retry after cooldown",
     );
+    return false;
   } finally {
     activeRequests--;
+  }
+}
+
+export async function processBatch(
+  conversationKey: string,
+  messages: MessageRecord[],
+  processingStartedAt: number,
+): Promise<void> {
+  if (messages.length === 0) {
     if (conversationProcessing.get(conversationKey) === processingStartedAt) {
       conversationProcessing.delete(conversationKey);
     }
-    if (shouldScheduleNext) {
-      setImmediate(() => {
-        // Dynamic import to avoid circular dependency at module scope
-        import("./batchScheduler.js").then((m) =>
-          m.scheduleConversationAnalysis(conversationKey),
-        );
-      });
+    return;
+  }
+  const cooldownUntil = conversationErrorCooldown.get(conversationKey) ?? 0;
+  if (Date.now() < cooldownUntil) {
+    if (conversationProcessing.get(conversationKey) === processingStartedAt) {
+      conversationProcessing.delete(conversationKey);
     }
+    return;
+  }
+
+  // Split the batch itself — not just route it — so text and media never
+  // share one worker call. A conversation batch commonly mixes plain-text
+  // messages with an image/sticker from someone else; without this split,
+  // ALL of it (including the plain-text messages) would ride along on the
+  // media job and wait for vision analysis to finish. Each sub-batch is now
+  // dispatched to its own pool AND handled independently below, so the text
+  // queue's results land as soon as text analysis completes, full stop.
+  const textMessages = messages.filter((m) => !hasMediaContent(m));
+  const mediaMessages = messages.filter((m) => hasMediaContent(m));
+
+  const jobs: Promise<boolean>[] = [];
+  if (textMessages.length > 0) {
+    jobs.push(runQueueBatch(textWorkerPool, conversationKey, textMessages));
+  }
+  if (mediaMessages.length > 0) {
+    jobs.push(runQueueBatch(mediaWorkerPool, conversationKey, mediaMessages));
+  }
+
+  const outcomes = await Promise.allSettled(jobs);
+  const shouldScheduleNext = outcomes.every(
+    (o) => o.status === "fulfilled" && o.value,
+  );
+
+  if (conversationProcessing.get(conversationKey) === processingStartedAt) {
+    conversationProcessing.delete(conversationKey);
+  }
+  if (shouldScheduleNext) {
+    setImmediate(() => {
+      // Dynamic import to avoid circular dependency at module scope
+      import("./batchScheduler.js").then((m) =>
+        m.scheduleConversationAnalysis(conversationKey),
+      );
+    });
   }
 }

services/discord-gateway/src/modules/ai-moderation/circuitBreaker.ts

@@ -6,7 +6,16 @@ import { config } from "../../shared/config/config.js";
 import type { MessageRecord } from "../message-capture/types.js";
 
 // ---------------------------------------------------------------------------
-// Piscina worker pool (shared by batch + individual pipelines)
+// Piscina worker pools (2026-08-31: split text vs media)
+//
+// Both the batch and individual-fallback pipelines used to share ONE pool.
+// A conversation batch containing images/stickers/embeds routes through
+// runMediaBatch (download + vision + LLM — tens of seconds per batch), and
+// Piscina hands each worker thread exactly one task at a time. With a small
+// fixed thread count, a handful of slow media batches could occupy every
+// thread and leave fast text-only batches for OTHER conversations queued
+// behind them for the whole media duration. Text and media now get their
+// own dedicated pools so a media backlog can never starve text analysis.
 // ---------------------------------------------------------------------------
 
 function getAnalysisWorkerUrl(): URL {
@@ -25,12 +34,32 @@ function getAnalysisWorkerUrl(): URL {
   return candidates[2];
 }
 
-export const workerPool = new Piscina({
-  filename: fileURLToPath(getAnalysisWorkerUrl()),
+const analysisWorkerFilename = fileURLToPath(getAnalysisWorkerUrl());
+
+/** Dedicated pool for text-only batch/individual analysis jobs. */
+export const textWorkerPool = new Piscina({
+  filename: analysisWorkerFilename,
   execArgv: process.execArgv,
   maxThreads: config.PISCINA_MAX_THREADS ?? availableParallelism(),
 });
 
+/**
+ * Dedicated pool for jobs whose batch contains at least one message with
+ * media (attachments/stickers/embeds). Kept separate so slow image/vision
+ * analysis never blocks the text pool above.
+ */
+export const mediaWorkerPool = new Piscina({
+  filename: analysisWorkerFilename,
+  execArgv: process.execArgv,
+  maxThreads: config.PISCINA_MEDIA_MAX_THREADS ?? 2,
+});
+
+/**
+ * @deprecated Use `textWorkerPool` or `mediaWorkerPool` directly. Kept as an
+ * alias to the text pool only for anything not yet migrated.
+ */
+export const workerPool = textWorkerPool;
+
 /**
  * Gets the conversation key for a message (thread_id or channel_id).
  */

services/discord-gateway/src/modules/ai-moderation/individualFallbackProcessor.ts

@@ -6,9 +6,14 @@ import type {
   AnalysisResult,
   MessageRecord,
 } from "../message-capture/types.js";
-import { getConversationKey, workerPool } from "./circuitBreaker.js";
+import {
+  getConversationKey,
+  mediaWorkerPool,
+  textWorkerPool,
+} from "./circuitBreaker.js";
 import { fireAlert } from "./conversationState.js";
 import { classifyIndividualWorkerResult } from "./fallbackResultClassifier.js";
+import { hasMediaContent } from "./mediaAnalysisClient.js";
 import {
   broadcastAnalysisCompleted,
   LAST_ERROR,
@@ -73,8 +78,12 @@ async function processIndividualFallback(
   let exhaustedOnIncomplete = false;
 
   try {
-    // Run the LLM-heavy work in the worker thread
-    const workerResult = (await workerPool.run({
+    // Run the LLM-heavy work in the worker thread — same text/media pool
+    // split as the batch pipeline (see circuitBreaker.ts), so a single
+    // media message falling back to individual analysis can't queue behind
+    // (or block) text-only fallbacks, and vice versa.
+    const pool = hasMediaContent(message) ? mediaWorkerPool : textWorkerPool;
+    const workerResult = (await pool.run({
       type: "individual",
       message,
       skipNormalAnalysis: false,

services/discord-gateway/src/modules/ai-moderation/termGlossary.ts

@@ -41,6 +41,14 @@ import {
   getTermDefinitionFromDb,
   setTermDefinitionInDb,
 } from "./termGlossaryStore.js";
+import {
+  cleanContent,
+  isKnownTerm,
+  isMostlyStopwords,
+  isNoiseWord,
+  scoreWord,
+  WORD_RE,
+} from "./textSignals.js";
 import { wikipediaSummary } from "./wikipediaClient.js";
 
 const log = createChildLogger("term-glossary");
@@ -96,100 +104,10 @@ async function acquireLiveSlot(): Promise<void> {
 // ---------------------------------------------------------------------------
 // Term extraction
 // ---------------------------------------------------------------------------
-
-/** Word tokenizer — letters/digits plus internal -_'· (handles "well-known",
- *  "node_modules", diacritics). */
-const WORD_RE = /[\p{L}\p{N}]+(?:[-_'’·][\p{L}\p{N}]+)*/gu;
-
-/** Removes URLs, Discord mentions/custom emoji, code fences, markdown noise. */
-function cleanContent(raw: string): string {
-  return raw
-    .replace(/https?:\/\/\S+/gi, " ")
-    .replace(/<@!?\d+>/g, " ")
-    .replace(/<#\d+>/g, " ")
-    .replace(/<a?:\w+:\d+>/g, " ")
-    .replace(/[`*_~|>[\]]/g, " ")
-    .replace(/[\p{Emoji}\p{Extended_Pictographic}]/gu, " ")
-    .replace(/\s+/g, " ")
-    .trim();
-}
-
-/** Filters out tokens that are useless as glossary candidates (numbers,
- *  repeated-char noise, mega-tokens). */
-function isNoiseWord(word: string): boolean {
-  if (word.length > 28) return true;
-  if (/^\d+$/.test(word)) return true;
-  const lower = word.toLowerCase();
-  // "aaaa…", "wwwwww" — single repeated character
-  if (/^(.)\1{2,}$/.test(lower)) return true;
-  // "wkwk", "hehe", "69" alternations — repeated 2–3 char base. "meme" is
-  // the one legit 4-letter word this matches; it is whitelisted below.
-  if (/^([a-z]{2,3})\1{1,}$/.test(lower)) return true;
-  return false;
-}
-
-/** Deterministic bonus for words that look like proper nouns or foreign. */
-function scoreWord(word: string): number {
-  let score = 1;
-  // Capitalized first letter (proper noun / title) but not ALL-CAPS acronyms
-  if (/^[A-Z]/.test(word) && !/^[A-Z]{2,}$/.test(word)) score += 3;
-  // Contains a letter outside basic latin → regional/foreign spelling
-  if (/[\p{L}]/u.test(word.replace(/[A-Za-z]/g, ""))) score += 2;
-  // Contains an internal apostrophe or hyphen → likely a named entity
-  if (/[-_'’·]/.test(word)) score += 2;
-  return score;
-}
-
-const STOPWORDS = new Set(
-  // ── Bahasa Indonesia ────────────────────────────────────────────────
-  (
-    " yang dan di ke dari ini itu dengan untuk pada dalam adalah akan telah sudah bisa dapat harus tidak juga saya kamu kita kami mereka dia aku kau gua lu lo gw gue elu anda kalian nya kah lah pun ya yah kan sih dong deh kok loh toh aja saja gitu gini begitu begini tapi tetapi namun atau karena sebab jika kalau bila maka supaya agar meski meskipun walau walaupun ketika saat setelah sebelum selama antara terhadap tentang mengenai bagi oleh secara sebagai seperti daripada tanpa hingga sampai sejLine truncated
-    // ── English ───────────────────────────────────────────────────────
-    " the a an and or but if then else for to in on at by with without from of is are was were be been being have has had do does did will would can could should may might must shall this that these those it its i you he she we they them their there here when where why how what which who whom whose only very just about above after before below under over into onto within upon against between among during through across along around behind beyond near off out up down now then so as not no yes okLine truncated
-    // ── Common net slang / acronyms the LLM already knows ──────────────
-    " lol omg wtf idk btw tbh imo aka fyi nsfw smh nvm asap afk brb gg wp ty np mb sry thx kk oke okk ygy frfr"
-  ).split(/\s+/),
-);
-
-/**
- * Words that are either already defined by the moderation rules, or are so
- * common (brands, tech vocabulary, project names) that a Wikipedia lookup is
- * a guaranteed miss/waste. Keeps the glossary focused on genuinely unknown
- * terms.
- */
-const KNOWN_SAFE_TERMS = new Set(
-  (
-    "discord youtube google facebook instagram twitter tiktok whatsapp telegram netflix spotify steam github gitlab bitbucket chatgpt openai anthropic claude deepseek gemini llama copilot cursor vscode vscodium jetbrains intellij pycharm webstorm sublime codeblocks" +
-    " docker kubernetes k8s linux ubuntu debian arch fedora manjaro kali windows macos android ios chrome firefox safari edge opera brave" +
-    " react nextjs next vue svelte angular node nodejs deno bun pnpm yarn npm javascript typescript python golang go rust java kotlin swift cplusplus cpp css html json xml yaml toml regex backend frontend database mysql postgres postgresql mongodb redis qdrant sqlite nosql graphql rest websocket webhook" +
-    " bug crash error debug fix issue pr merge commit push pull branch main master dev staging production server client app website web browser" +
-    " stream streaming video audio voice call camera screen share screenshare gameplay gaming game play steam epic xbox playstation nintendo switch console" +
-    " bot discordbot moderation moderator admin member user profile avatar channel server guild message chat dm reply forward embed sticker emoji role permission" +
-    " meme code coding ngoding programmer program developer engineer software hardware cpu gpu ram rom storage disk network internet wifi lan ip dns vpn proxy cloud aws azure gcp vercel netlify heroku railway render vps hosting domain ssl login logout register account password email username" +
-    " anime manga waifu husbando tsundere moe otaku wibu weeb otome isekai shonen seinen josei manga manhwa manhua doujin" +
-    " anjay wkwk wkwkwk gws gaskeun santuy njir baka woy woi hadeh astaga asu anjing bangsat ngehe asal alay lebay caper mabar" +
-    " asus bete imphnen impnhen ngab" +
-    " syahadat sholat shalat solat puasa zakat haji umrah doa tuhan nabi allah yesus muhammad hashem" +
-    " loli shota incest exhibition furry fursuit cosplay costume" +
-    " gaza palestine israel yahudi yahud israel palestina israeli" +
-    " hokkian mandarin arabic jawa sunda betawi minang bugis batak melayu inggris indonesia"
-  ).split(/\s+/),
-);
-
-function isKnownTerm(word: string): boolean {
-  return STOPWORDS.has(word) || KNOWN_SAFE_TERMS.has(word);
-}
-
-/** True when a quoted phrase is mostly filler words (skip it). */
-function isMostlyStopwords(phrase: string): boolean {
-  const words = phrase
-    .toLowerCase()
-    .split(/[^a-zà-öø-ÿ]+/i)
-    .filter(Boolean);
-  if (words.length === 0) return true;
-  const stopCount = words.filter((w) => STOPWORDS.has(w)).length;
-  return stopCount / words.length >= 0.6;
-}
+// Tokenizer, stopword/known-term filters, and word scoring now live in
+// textSignals.ts (2026-08-31) — shared with wikipediaClient.ts's search-query
+// extractor so both use the exact same "is this word worth looking up"
+// signals instead of two divergent copies.
 
 export interface ExtractGlossaryOptions {
   maxTerms?: number;

services/discord-gateway/src/modules/ai-moderation/textSignals.ts

@@ -0,0 +1,106 @@
+/**
+ * textSignals.ts
+ *
+ * Shared, LLM-free text scoring signals for deciding which words/phrases in
+ * a message are worth an external lookup (Wikipedia glossary term, Wikipedia
+ * search query, etc.).
+ *
+ * Extracted out of termGlossary.ts (2026-08-31) so wikipediaClient.ts's
+ * phrase-level search-query extractor can reuse the exact same
+ * stopword/known-term/scoring logic instead of maintaining a second,
+ * divergent copy. Everything here is pure CPU-side regex + scoring — no
+ * network or LLM call — so using it in more places never adds AI requests.
+ */
+
+/** Word tokenizer — letters/digits plus internal -_'· (handles "well-known",
+ *  "node_modules", diacritics). */
+export const WORD_RE = /[\p{L}\p{N}]+(?:[-_'’·][\p{L}\p{N}]+)*/gu;
+
+/** Removes URLs, Discord mentions/custom emoji, code fences, markdown noise. */
+export function cleanContent(raw: string): string {
+  return raw
+    .replace(/https?:\/\/\S+/gi, " ")
+    .replace(/<@!?\d+>/g, " ")
+    .replace(/<#\d+>/g, " ")
+    .replace(/<a?:\w+:\d+>/g, " ")
+    .replace(/[`*_~|>[\]]/g, " ")
+    .replace(/[\p{Emoji}\p{Extended_Pictographic}]/gu, " ")
+    .replace(/\s+/g, " ")
+    .trim();
+}
+
+/** Filters out tokens that are useless as lookup candidates (numbers,
+ *  repeated-char noise, mega-tokens). */
+export function isNoiseWord(word: string): boolean {
+  if (word.length > 28) return true;
+  if (/^\d+$/.test(word)) return true;
+  const lower = word.toLowerCase();
+  // "aaaa…", "wwwwww" — single repeated character
+  if (/^(.)\1{2,}$/.test(lower)) return true;
+  // "wkwk", "hehe", "69" alternations — repeated 2–3 char base. "meme" is
+  // the one legit 4-letter word this matches; it is whitelisted below.
+  if (/^([a-z]{2,3})\1{1,}$/.test(lower)) return true;
+  return false;
+}
+
+/** Deterministic bonus for words that look like proper nouns or foreign. */
+export function scoreWord(word: string): number {
+  let score = 1;
+  // Capitalized first letter (proper noun / title) but not ALL-CAPS acronyms
+  if (/^[A-Z]/.test(word) && !/^[A-Z]{2,}$/.test(word)) score += 3;
+  // Contains a letter outside basic latin → regional/foreign spelling
+  if (/[\p{L}]/u.test(word.replace(/[A-Za-z]/g, ""))) score += 2;
+  // Contains an internal apostrophe or hyphen → likely a named entity
+  if (/[-_'’·]/.test(word)) score += 2;
+  return score;
+}
+
+export const STOPWORDS = new Set(
+  // ── Bahasa Indonesia ────────────────────────────────────────────────
+  (
+    " yang dan di ke dari ini itu dengan untuk pada dalam adalah akan telah sudah bisa dapat harus tidak juga saya kamu kita kami mereka dia aku kau gua lu lo gw gue elu anda kalian nya kah lah pun ya yah kan sih dong deh kok loh toh aja saja gitu gini begitu begini tapi tetapi namun atau karena sebab jika kalau bila maka supaya agar meski meskipun walau walaupun ketika saat setelah sebelum selama antara terhadap tentang mengenai bagi oleh secara sebagai seperti daripada tanpa hingga sampai sejLine truncated
+    // ── English ───────────────────────────────────────────────────────
+    " the a an and or but if then else for to in on at by with without from of is are was were be been being have has had do does did will would can could should may might must shall this that these those it its i you he she we they them their there here when where why how what which who whom whose only very just about above after before below under over into onto within upon against between among during through across along around behind beyond near off out up down now then so as not no yes okLine truncated
+    // ── Common net slang / acronyms the LLM already knows ──────────────
+    " lol omg wtf idk btw tbh imo aka fyi nsfw smh nvm asap afk brb gg wp ty np mb sry thx kk oke okk ygy frfr"
+  ).split(/\s+/),
+);
+
+/**
+ * Words that are either already defined by the moderation rules, or are so
+ * common (brands, tech vocabulary, project names) that a Wikipedia lookup is
+ * a guaranteed miss/waste. Keeps lookups focused on genuinely unknown terms.
+ */
+export const KNOWN_SAFE_TERMS = new Set(
+  (
+    "discord youtube google facebook instagram twitter tiktok whatsapp telegram netflix spotify steam github gitlab bitbucket chatgpt openai anthropic claude deepseek gemini llama copilot cursor vscode vscodium jetbrains intellij pycharm webstorm sublime codeblocks" +
+    " docker kubernetes k8s linux ubuntu debian arch fedora manjaro kali windows macos android ios chrome firefox safari edge opera brave" +
+    " react nextjs next vue svelte angular node nodejs deno bun pnpm yarn npm javascript typescript python golang go rust java kotlin swift cplusplus cpp css html json xml yaml toml regex backend frontend database mysql postgres postgresql mongodb redis qdrant sqlite nosql graphql rest websocket webhook" +
+    " bug crash error debug fix issue pr merge commit push pull branch main master dev staging production server client app website web browser" +
+    " stream streaming video audio voice call camera screen share screenshare gameplay gaming game play steam epic xbox playstation nintendo switch console" +
+    " bot discordbot moderation moderator admin member user profile avatar channel server guild message chat dm reply forward embed sticker emoji role permission" +
+    " meme code coding ngoding programmer program developer engineer software hardware cpu gpu ram rom storage disk network internet wifi lan ip dns vpn proxy cloud aws azure gcp vercel netlify heroku railway render vps hosting domain ssl login logout register account password email username" +
+    " anime manga waifu husbando tsundere moe otaku wibu weeb otome isekai shonen seinen josei manga manhwa manhua doujin" +
+    " anjay wkwk wkwkwk gws gaskeun santuy njir baka woy woi hadeh astaga asu anjing bangsat ngehe asal alay lebay caper mabar" +
+    " asus bete imphnen impnhen ngab" +
+    " syahadat sholat shalat solat puasa zakat haji umrah doa tuhan nabi allah yesus muhammad hashem" +
+    " loli shota incest exhibition furry fursuit cosplay costume" +
+    " gaza palestine israel yahudi yahud israel palestina israeli" +
+    " hokkian mandarin arabic jawa sunda betawi minang bugis batak melayu inggris indonesia"
+  ).split(/\s+/),
+);
+
+export function isKnownTerm(word: string): boolean {
+  return STOPWORDS.has(word) || KNOWN_SAFE_TERMS.has(word);
+}
+
+/** True when a phrase is mostly filler words (skip it as a lookup candidate). */
+export function isMostlyStopwords(phrase: string): boolean {
+  const words = phrase
+    .toLowerCase()
+    .split(/[^a-zà-öø-ÿ]+/i)
+    .filter(Boolean);
+  if (words.length === 0) return true;
+  const stopCount = words.filter((w) => STOPWORDS.has(w)).length;
+  return stopCount / words.length >= 0.6;
+}

services/discord-gateway/src/modules/ai-moderation/wikipediaClient.ts

@@ -22,6 +22,12 @@ import { createChildLogger } from "@/shared/logger/index";
 import { createAbortControllerWithTimeout } from "@/shared/utils/index";
 import { config } from "../../shared/config/config.js";
 import { cacheGet, cacheSet, makeCacheKey } from "./cacheStore.js";
+import {
+  cleanContent,
+  isKnownTerm,
+  isMostlyStopwords,
+  scoreWord,
+} from "./textSignals.js";
 
 const log = createChildLogger("wikipedia-client");
 
@@ -206,73 +212,121 @@ export async function wikipediaSummary(
   }
 }
 
-/**
- * Extract meaningful search queries from message content.
- * Uses multiple strategies to find terms worth searching.
- * Returns up to 3 clean queries.
- */
-export function extractSearchQueries(content: string): string[] {
-  const queries = new Set<string>();
+export interface ExtractSearchQueryOptions {
+  maxQueries?: number;
+}
 
-  // 1. Quoted phrases (explicit user intent)
-  const quotedPhrases = content.match(/"([^"]+)"|'([^']+)'/g);
+/**
+ * Verbs that signal "find/watch/download X" — used only to find a phrase
+ * BOUNDARY. Unlike the old version, no trailing category word
+ * ("anime|film|series") is required, so coverage isn't capped by an
+ * enumerated category list. The capture stops at the first coordinating
+ * conjunction (or end of message) so "nonton X sama Y terus Z" yields the
+ * single entity X instead of swallowing the whole multi-entity tail into
+ * one unsearchable blob.
+ */
+const SEARCH_INTENT_VERBS =
+  /\b(?:nonton|tonton|rekomen(?:dasiin|dasikan)?|cari(?:in|kan)?|search|google|download|donlod|unduh|streaming|baca|dengerin|dengar(?:kan)?)\b\s+(.+?)(?=\s+(?:sama|dan|juga|terus|lalu|atau|and|or)\b|\s*[!?.]*$)/i;
+
+/** "apa itu X" / "arti X" / "what is X" — factual/definition intent. */
+const DEFINITION_INTENT =
+  /\b(?:apa\s+(?:itu|sih)|what\s+is|siapa\s+itu|who\s+is|arti(?:nya)?|meaning(?:\s+of)?|definisi(?:nya)?|definition(?:\s+of)?)\s+(.{3,80})/i;
+
+/** Multi-word capitalized runs — usually a title/named entity regardless of
+ *  surrounding verbs (e.g. "Attack on Titan", "One Piece"). No trigger word
+ *  needed at all. */
+const PROPER_NOUN_PHRASE = /\b([A-Z][\p{L}]*(?:\s+[A-Z][\p{L}]*){1,4})\b/gu;
+
+interface PhraseCandidate {
+  phrase: string;
+  score: number;
+}
+
+/**
+ * Scores a phrase with the SAME per-word signals as the term glossary
+ * (proper-noun casing, foreign spelling, hyphenation — see textSignals.ts),
+ * plus a bonus for the extraction strategy that surfaced it. Returns null
+ * for junk (mostly stopwords, or made entirely of known-safe terms like
+ * "Discord"/"Google" — those never need a Wikipedia lookup).
+ */
+function scorePhrase(rawPhrase: string, bonus: number): PhraseCandidate | null {
+  const clean = rawPhrase
+    .trim()
+    .replace(/^[^\p{L}\p{N}]+|[^\p{L}\p{N}]+$/gu, "");
+  if (clean.length < 2 || clean.length > 80) return null;
+  if (isMostlyStopwords(clean)) return null;
+
+  const words = clean.split(/\s+/);
+  let score = bonus;
+  let hasNonStopword = false;
+  for (const w of words) {
+    if (isKnownTerm(w.toLowerCase())) continue;
+    hasNonStopword = true;
+    score += scoreWord(w);
+  }
+  if (!hasNonStopword) return null;
+
+  return { phrase: clean, score };
+}
+
+/**
+ * Extracts candidate phrases worth searching on Wikipedia — scored by the
+ * same word-level signals as the term glossary, instead of requiring an
+ * exact match against a fixed, manually-maintained category/brand list.
+ * Pure CPU-side regex + scoring — no network or LLM call, so using this
+ * more broadly never adds AI requests.
+ *
+ * Returns up to `maxQueries` phrases (default 3), highest-scored first.
+ */
+export function extractSearchQueries(
+  content: string,
+  options: ExtractSearchQueryOptions = {},
+): string[] {
+  const maxQueries = options.maxQueries ?? 3;
+  const cleaned = cleanContent(content);
+  if (!cleaned) return [];
+
+  const candidates = new Map<string, PhraseCandidate>();
+  const addCandidate = (raw: string, bonus: number): void => {
+    const scored = scorePhrase(raw, bonus);
+    if (!scored) return;
+    const key = scored.phrase.toLowerCase();
+    const existing = candidates.get(key);
+    if (!existing || scored.score > existing.score) {
+      candidates.set(key, scored);
+    }
+  };
+
+  // 1. Quoted phrases — explicit user intent, strongest signal.
+  const quotedPhrases = cleaned.match(/"([^"]{2,80})"|'([^']{2,80})'/g);
   if (quotedPhrases) {
     for (const phrase of quotedPhrases) {
-      const clean = phrase.replace(/["']/g, "").trim();
-      if (clean.length >= 3) queries.add(clean);
+      addCandidate(phrase.replace(/["']/g, ""), 10);
     }
   }
 
-  // 2. "nonton X" pattern — extract the title
-  const nontonMatch = content.match(
-    /\b(nonton|tonton|rekomen|cari|search|google)\s+(.+?)(?:\s+(?:anime|kartun|film|movie|series|serial))?\s*[!?.]*$/i,
-  );
-  if (nontonMatch) {
-    const title = nontonMatch[2].trim();
-    if (title.length >= 2 && title.length <= 80) {
-      queries.add(title);
-    }
+  // 2. Definition/factual intent ("apa itu X", "arti X", "what is X").
+  const definitionMatch = cleaned.match(DEFINITION_INTENT);
+  if (definitionMatch) {
+    addCandidate(definitionMatch[1].replace(/[?!.]+$/, ""), 8);
   }
 
-  // 3. "X anime/film" pattern — title before category
-  const titleBeforeCategory = content.match(
-    /\b(\w[\w\s]{2,40})\s+(?:anime|kartun|film|movie|series|serial)\b/i,
-  );
-  if (titleBeforeCategory) {
-    const title = titleBeforeCategory[1].trim();
-    if (
-      title.length >= 3 &&
-      !/^(yang|yang|sama|dari|untuk|ini|itu|ada)$/i.test(title)
-    ) {
-      queries.add(title);
-    }
+  // 3. "nonton/cari/rekomen/... X" — phrase after an intent verb.
+  const intentMatch = cleaned.match(SEARCH_INTENT_VERBS);
+  if (intentMatch) {
+    addCandidate(intentMatch[1], 6);
   }
 
-  // 4. Standalone proper nouns (2+ words, capitalized) that look like titles
-  const properNouns = content.match(
-    /\b([A-Z][a-z]+(?:\s+[A-Z][a-z]+){1,4})\b/g,
-  );
-  if (properNouns) {
-    for (const noun of properNouns) {
-      // Skip common non-title proper nouns
-      const skip =
-        /^(Discord|YouTube|Google|Facebook|Instagram|Twitter|Github|ChatGPT|OpenAI|Claude|Telegram|WhatsApp|TikTok|Netflix|Spotify|Steam|Instagram)$/i;
-      if (!skip.test(noun) && noun.length >= 5) {
-        queries.add(noun);
-      }
-    }
+  // 4. Proper-noun phrases anywhere in the message — titles/named entities
+  //    surface here even with no trigger verb.
+  for (const m of cleaned.matchAll(PROPER_NOUN_PHRASE)) {
+    addCandidate(m[1], 0);
   }
 
-  // 5. Terms that suggest research intent
-  const researchTerms = content.match(
-    /\b(apa\s+(?:itu|sih)|what\s+is|siapa\s+itu|who\s+is|arti|meaning|definisi|definition)\s+(.{3,60})/i,
-  );
-  if (researchTerms) {
-    const term = researchTerms[2].trim().replace(/[?!.]+$/, "");
-    if (term.length >= 3) queries.add(term);
-  }
-
-  return Array.from(queries).slice(0, 3);
+  return Array.from(candidates.values())
+    .sort((a, b) => b.score - a.score)
+    .slice(0, maxQueries)
+    .map((c) => c.phrase);
 }
 
 /**

services/discord-gateway/src/shared/config/index.ts

@@ -319,11 +319,20 @@ export const configSchema = z
       .int()
       .positive()
       .default(50),
-    // Worker pool size. Default 4 (not availableParallelism) because each
-    // Piscina thread owns its own pLimit(5) semaphore — on big VPSes
-    // availableParallelism × 5 concurrent LLM calls would overwhelm the
-    // router. Keep threads modest; concurrency is capped per-thread anyway.
+    // Text-analysis worker pool size (2026-08-31: split from the media pool
+    // below so a slow image/vision batch can never occupy every thread and
+    // starve the far more common text-only batches). Default 4 (not
+    // availableParallelism) because each Piscina thread owns its own
+    // pLimit(5) semaphore — on big VPSes availableParallelism × 5 concurrent
+    // LLM calls would overwhelm the router. Keep threads modest; concurrency
+    // is capped per-thread anyway.
     PISCINA_MAX_THREADS: z.coerce.number().int().positive().default(4),
+    // Media-analysis worker pool size — dedicated threads for batches that
+    // contain images/stickers/embeds (download + vision + LLM, much slower
+    // than text). Kept small since media batches are less frequent and each
+    // one is long-running; sized independently from PISCINA_MAX_THREADS so
+    // tuning one never starves the other.
+    PISCINA_MEDIA_MAX_THREADS: z.coerce.number().int().positive().default(2),
 
     // ── Voice Transcription ────────────────────────────────────────────────
     AI_VOICE_TRANSCRIPTION_ENABLED: z

services/discord-gateway/tests/wikipediaSearchQueries.test.ts

@@ -0,0 +1,61 @@
+// ═══════════════════════════════════════════════════════════════════════════
+// extractSearchQueries — pure scoring-based extraction (no DB, Redis, or
+// network). Replaces the old fixed-regex/category-list version (2026-08-31);
+// these cases check the new scored extractor still covers what the old
+// pattern list covered, plus the generalization gains.
+// ═══════════════════════════════════════════════════════════════════════════
+import { describe, expect, it } from "vitest";
+import { extractSearchQueries } from "../src/modules/ai-moderation/wikipediaClient.js";
+
+describe("extractSearchQueries", () => {
+  it("returns [] for plain conversational text with no lookup-worthy content", () => {
+    expect(extractSearchQueries("iya bener banget sih wkwkwk")).toEqual([]);
+  });
+
+  it("extracts a quoted phrase as the top candidate", () => {
+    const queries = extractSearchQueries('dia bilang "kostum hewan" itu aneh');
+    expect(queries[0]).toBe("kostum hewan");
+  });
+
+  it("extracts the target of a definition question without a fixed keyword list", () => {
+    const queries = extractSearchQueries("apa itu shirkmaxxing?");
+    expect(queries).toContain("shirkmaxxing");
+  });
+
+  it("extracts an intent-verb phrase with NO trailing category word required", () => {
+    // Old regex required a trailing anime|kartun|film|movie|series|serial to
+    // even try; the new version doesn't need one at all.
+    const queries = extractSearchQueries("woy nonton Attack on Titan dong");
+    expect(queries.some((q) => /attack on titan/i.test(q))).toBe(true);
+  });
+
+  it("extracts a multi-word proper-noun title with no trigger verb at all", () => {
+    const queries = extractSearchQueries("Chrono Cross itu keren banget");
+    expect(queries).toContain("Chrono Cross");
+  });
+
+  it("skips known-safe brand terms instead of relying on a fixed skip-list copy", () => {
+    // "Discord" is in the shared KNOWN_SAFE_TERMS set (textSignals.ts), reused
+    // here instead of a second hardcoded skip-list.
+    const queries = extractSearchQueries("Discord lagi down nih parah");
+    expect(queries).not.toContain("Discord");
+  });
+
+  it("caps results at maxQueries, highest-scored first", () => {
+    const queries = extractSearchQueries(
+      'nonton Xenogears sama "Chrono Cross" terus Yakuza juga',
+      { maxQueries: 2 },
+    );
+    expect(queries.length).toBeLessThanOrEqual(2);
+    // Quoted phrase (bonus 10) should outrank the bare proper nouns.
+    expect(queries[0]).toBe("Chrono Cross");
+  });
+
+  it("strips URLs and mentions before extracting", () => {
+    const queries = extractSearchQueries(
+      'cek https://example.com/foo <@123456> "kafircel"',
+    );
+    expect(queries).toContain("kafircel");
+    expect(queries.some((q) => /example|123456/.test(q))).toBe(false);
+  });
+});