MythEclipse/GMW
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

perf(ai-moderation): pack more messages per LLM request (fewer API calls when busy)

Browse Source 
User insight: rather than many small per-batch API requests, pack many
messages into ONE request so a burst is analyzed with far fewer calls.

- AI_LLM_TEXT_BATCH_SIZE 20 -> 60 (one request now carries ~3x more messages).
- AI_ANALYSIS_MAX_TARGET_TOKENS 4000 -> 14000 (the scheduler's token-budget
  gate was trimming pending messages to ~20 before they reached the sub-batch
  splitter; raising it lets ~60 messages through to a single LLM call).
- AI_LLM_TEXT_ANALYSIS_TIMEOUT_MS 30000 -> 45000 (one larger call needs more
  headroom; gemini-flash-lite has a 1M-token context so 14k+8k is trivial).

Net effect when ramai: a 60-message burst = 1-2 API calls instead of 3+,
less semaphore contention, faster throughput.
This commit is contained in:
asepharyana
2026-08-16 19:00:29 +07:00
parent 0dff7770a1
commit 4cf5b87f2b
1 changed files with 3 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
services/discord-gateway/src/shared/config/index.ts
+3 -3
View File
@@ -183,7 +183,7 @@ export const configSchema = z
.int() .int()
.positive() .positive()
.default(1024), .default(1024),
AI_LLM_TEXT_BATCH_SIZE: z.coerce.number().int().positive().default(20), AI_LLM_TEXT_BATCH_SIZE: z.coerce.number().int().positive().default(60),
AI_LLM_MEDIA_ANALYSIS_TIMEOUT_MS: z.coerce AI_LLM_MEDIA_ANALYSIS_TIMEOUT_MS: z.coerce
.number() .number()
.int() .int()
@@ -205,7 +205,7 @@ export const configSchema = z
.number() .number()
.int() .int()
.positive() .positive()
.default(30000), .default(45000),
// Term glossary — per-word Wikipedia lookups (via SearXNG) for words the // Term glossary — per-word Wikipedia lookups (via SearXNG) for words the
// LLM may not know (slang, jargon, regional language, foreign terms). // LLM may not know (slang, jargon, regional language, foreign terms).
// Definitions are cached (in-memory + Redis) so repeat lookups are fast. // Definitions are cached (in-memory + Redis) so repeat lookups are fast.
@@ -236,7 +236,7 @@ export const configSchema = z
// ── AI Analysis Batch ─────────────────────────────────────────────── // ── AI Analysis Batch ───────────────────────────────────────────────
AI_ANALYSIS_MAX_BATCH_SIZE: z.coerce.number().int().positive().default(200), AI_ANALYSIS_MAX_BATCH_SIZE: z.coerce.number().int().positive().default(200),
AI_ANALYSIS_MAX_CONTEXT_TOKENS: z.coerce.number().positive().default(8000), AI_ANALYSIS_MAX_CONTEXT_TOKENS: z.coerce.number().positive().default(8000),
AI_ANALYSIS_MAX_TARGET_TOKENS: z.coerce.number().positive().default(4000), AI_ANALYSIS_MAX_TARGET_TOKENS: z.coerce.number().positive().default(14000),
AI_ANALYSIS_CONTEXT_MESSAGE_LIMIT: z.coerce AI_ANALYSIS_CONTEXT_MESSAGE_LIMIT: z.coerce
.number() .number()
.int() .int()