- In llmModerationClient.ts (inner retry), if OpenAI throws a 429 (or 401/403), throw p-retry's AbortError to immediately exit the 3-attempt inner retry loop.
- In aiAnalyzer.ts (outer retry), propagate the AbortError from runModerationAnalysis so the 2-attempt outer retry loop also aborts immediately.
- This ensures that a burst of 20 concurrent tasks hitting rate limits immediately returns the messages to the DB queue (as 'analysis_incomplete') and rapidly increments the individual circuit breaker, pausing processing and preventing a thundering herd instead of making 12 API calls per stuck message.
Infinite recovery loop (#new):
- Inside retryWithBackoff callback in processIndividualFallback, detect
'analysis_incomplete' in the LLM result and throw to trigger backoff.
- Track exhaustedOnIncomplete flag across retries.
- On final exhaustion: write terminal flag 'individual_analysis_exhausted'
to DB so the recovery query (which only looks for 'analysis_incomplete')
never picks this message up again.
- Transient failures (network/parse) are NOT written as exhausted; they
remain as 'analysis_incomplete' and are retried via the CB-throttled
recovery cycle.
Token budget zero-result deadlock (#10):
- If pickBatchWithinBudget returns [] because every candidate message
individually exceeds AI_ANALYSIS_MAX_TARGET_TOKENS, fall back to
messages.slice(0,1) so at least the first message is processed.
- Without this, messages would be permanently stuck as 'pending' because
every recovery tick would fetch them, trim to 0, and exit silently.
- Uses messages.slice(0,1) instead of messages[0]! to avoid the
forbidden noNonNullAssertion lint rule.
Stale state map memory leak (#9):
- startPendingAIAnalysisWorker now prunes conversationErrorCooldown and
conversationProcessing on every recovery interval tick.
- Cooldown entries past their expiry timestamp are deleted.
- Processing entries older than AI_ANALYSIS_PROCESSING_TIMEOUT_MS are
deleted (these represent stale locks from crashed processing runs).
- Prevents unbounded Map growth for long-running bots with many channels.
Batch/individual scheduling collision (#8):
- Build incompleteKeySet (Set<string>) from incompleteKeys before the
batch recovery loop.
- Batch recovery loop skips any key present in incompleteKeySet so a
conversation that has both 'pending' and 'analysis_incomplete' messages
is only targeted by the individual pipeline, not both simultaneously.
- Avoids the DB last-write-wins race where batch and individual pipelines
both update the same message rows concurrently.
#1+#5 - Individual fallback circuit breaker
- Add individualConsecutiveErrors + individualCooldownUntil (30s)
- On success: reset counter; on failure: increment + trip at
AI_ANALYSIS_INDIVIDUAL_CB_THRESHOLD (default 10) consecutive errors
- enqueueIndividualFallbacks checks CB before admitting any work
#1 - Individual fallback concurrency cap
- enqueueIndividualFallbacks enforces AI_ANALYSIS_INDIVIDUAL_MAX_CONCURRENT
(default 20); overflow stays as error/analysis_incomplete in DB and is
recovered by the recovery worker on the next interval
#3 - Unhandled rejection in async setTimeout
- scheduleConversationAnalysis no longer uses async arrow in setTimeout;
all async work is chained with .then()/.catch() explicitly
#4 - Recovery worker ignores individualInFlight
- Add individualInFlightByConversation Map<conversationKey, count>
- processIndividualFallback increments/decrements it in try/finally
- startPendingAIAnalysisWorker skips conversations present in the map
- Recovery worker also processes error/analysis_incomplete messages via
two new messageStore queries: getConversationKeysWithIncompleteAnalysis
and getIncompleteMessagesByConversation
#6 - pickBatchWithinBudget never called
- scheduleConversationAnalysis now calls pickBatchWithinBudget with
AI_ANALYSIS_MAX_TARGET_TOKENS (default 4000) + 50-token per-msg overhead
after fetching messages, before passing to processBatch
#7 - AI_PROCESSING_OVERLAP_MS 30s shorter than max LLM retry window
- Replace hardcoded 30 000 ms constant with configurable
AI_ANALYSIS_PROCESSING_TIMEOUT_MS (default 120 000 ms)
- LLM client: 30s timeout × 3 retries + backoff ≈ 90-100s; 120s is safe
New config keys:
AI_ANALYSIS_PROCESSING_TIMEOUT_MS (default: 120000)
AI_ANALYSIS_INDIVIDUAL_MAX_CONCURRENT (default: 20)
AI_ANALYSIS_INDIVIDUAL_CB_THRESHOLD (default: 10)
AI_ANALYSIS_MAX_TARGET_TOKENS (default: 4000)
New AnalysisQueueStatus fields:
activeIndividualRequests, individualInFlightCount,
individualCircuitBreakerActive
- After a batch LLM call, any result flagged analysis_incomplete is
immediately fanned out to an individual per-message fallback queue
- Batch hard-fail (result.ok=false) and unhandled exceptions now also
route all affected messages to the individual queue instead of waiting
behind the conversation error cooldown
- Individual queue runs fully parallel (fire-and-forget per message),
de-duplicated by a Set<messageId> so no double-processing
- processIndividualFallback runs in the main process (no worker pool IPC
overhead for a single-item call), with retryWithBackoff 2x/2-15s
- AnalysisQueueStatus gains activeIndividualRequests +
individualInFlightCount fields for dashboard observability
- Changed all import statements across the project to include the .js extension for consistency and to comply with ES module standards.
- Updated imports in various files including bootstrap.ts, shutdown.ts, config.ts, and many others.
- Ensured that all related modules and types are correctly imported with the new extension.
- Introduced a new `ScreenShareController` to manage YouTube screenshare functionality.
- Updated `DiscordPlayer` to track ownership of audio streams, preventing conflicts between music playback and screenshare.
- Added error handling for various states including voice connection checks and media busy states.
- Created unit tests for `ScreenShareController` and `DiscordPlayer` ownership rules to ensure correct functionality.
- Added documentation for the new media echo fix and screenshare design.
- Replace all raw SQL queries in messageStore.ts with Drizzle ORM queries
- Remove DatabaseAdapter dependency from messageStore functions
- Update all function signatures to be async and remove db parameter
- Functions now use getDatabase() internally for database access
- Update all call sites in messageCapture.ts, attachmentUploader.ts, aiAnalyzer.ts, webserver.ts, and index.ts
- All functions remain backward compatible in behavior
- TypeScript typecheck passes with no errors
- All tests pass (11 passed)
Reduce effective AI batch size so streaming requests finish before timeout. Keep token-based batching but cap each request to 80 messages or about 9k content tokens, and recursively split failed batches instead of marking the whole batch failed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Batch AI moderation by estimated token budget instead of fixed message count. Send as many messages as fit within an 80k token request budget while keeping one concurrent API request. Include message metadata and chronological conversation context so the model can judge provocation and replies from surrounding discussion.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Remove unclear-message and low-quality-message warning criteria because this is a casual group. Keep short, ambiguous, informal, and light profanity messages clean unless they target someone or provoke conflict.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Remove the channel topic/OOT rule from AI moderation criteria and renumber the remaining rules. WARN criteria no longer includes OOT.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Expand system prompt with complete community rules (9 sections)
- Add specific examples for each rule category
- Clarify WARN vs FLAGGED decision criteria
- Include all prohibited content types and behaviors
- Provide clear guidance for AI analyzer on rule enforcement
Community rules now cover:
1. Jaga Sikap dan Hormati Sesama
2. Hindari Konflik
3. Gunakan Channel Sesuai Topik
4. Konten Eksplisit Dilarang
5. Jaga Privasi
6. Profil yang Sopan
7. Dilarang Spam dan Penipuan
8. Langsung ke Inti Pertanyaan
9. Diskusi Berkualitas
This ensures AI analyzer makes consistent moderation decisions based on actual community rules.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add "warn" status between "clean" and "flagged" for minor violations
- Update AI analyzer system prompt with community rules and warn category
- Warn: profanity, OOT, tone issues - requires warning but not deletion
- Flagged: NSFW, illegal, hacking, scam, harassment, violence, SARA - requires review/deletion
- Update types to support warn status in MessageRecord and AIAnalysisUpdate
- Update client UI to show three panels: All Messages, Warned, Flagged
- Warned messages show in right-top panel for quick review
- Flagged messages show in right-bottom panel for moderation action
This resolves:
- Need to distinguish between minor and severe violations
- Moderators can now warn users before taking action
- Better moderation workflow with three-tier system
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Change runLLMAnalysis to accept array of texts instead of single text
- Batch up to 5 messages per AI request instead of 1 message per request
- drainQueue now collects batch before sending to AI API
- Reduces API calls by 5x and speeds up analysis significantly
- System prompt updated to handle batch JSON array responses
This resolves:
- Slow AI analysis (3 messages every 15 seconds)
- Too many API calls (one per message)
- Long queue backlog
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Fix fetchJson to extract JSON from streaming response text
- API returns text/event-stream with complete JSON object embedded
- Extract JSON by finding first { and last } in response
- Prevents "Unexpected non-whitespace character after JSON" parse errors
- Streaming response now properly parsed and analyzed
This resolves:
- AI analysis stuck on "[Streaming in progress...]"
- JSON parse failures on streaming responses
- AI analysis now completes successfully
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add activeRequests counter to track in-flight AI requests
- Limit concurrent requests to 1 (MAX_CONCURRENT_REQUESTS)
- drainQueue now waits if at max concurrency before processing next message
- Prevents overwhelming streaming LLM API with multiple concurrent requests
This resolves:
- AI LLM API overload from concurrent requests
- Streaming response conflicts
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
