- Removed "recharts" from package.json and its associated entries from pnpm-lock.yaml.
- Cleaned up unnecessary dependencies and peer dependencies related to recharts and other unused packages.
- Added new API endpoints for daily trend data and activity heatmap in analyticsRoutes.ts.
- Created new frontend components: ActivityChart, ControlBar, Heatmap, SummaryCards, TopicList, TrendChart, UserTable, and ViolatorTable for displaying analytics data.
- Implemented loading and empty states in the new components.
- Enhanced the existing moderation tests with remote fallback handling for Indonesian text normalization.
- Analytics panel now refreshes immediately when messages are created, updated, deleted, or analyzed
- WebSocket events (message_created, message_updated, message_deleted, message_analyzed) trigger custom event 'analytics_refresh'
- Analytics hook listens to 'analytics_refresh' event for instant updates
- Auto-refresh interval lowered from 60s to 5s as fallback
- Two-tier refresh: real-time push + 5s polling safety net
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Remove Review tab completely (was redundant with Messages flagged view)
- UI state now client-side only (localStorage) — no server API calls for tab/channel/guild selection
- Fixes dashboard crash when server is down — now loads fully client-side
- Live panel still uses server API for voice/media operations (only what needs it)
- Analytics hourly chart labels now show Jakarta time (WIB/UTC+7) instead of UTC
- Analytics formatTimeAgo uses Jakarta time reference
- Reduced tabs to 3: Live, Messages, Analytics
- Removed unused uiState API imports and server-side state fetching
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- New LivePanel component combines voice bridge, media player, screen share, and recordings
- Single page layout: voice controls → audio visualizer + active speakers → now playing → music/screen/recordings tabs
- Reduced tabs from 6 to 4: Live, Messages, Analytics, Review
- Sidebar updated with new tab structure
- Default tab changed from 'voice' to 'live'
- Cleaner compact layout with icon buttons and inline controls
- Recordings sub-panel with user avatars, status badges, and download buttons
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
MessageCard:
- Add sticker image preview (renders actual sticker from URL, not just text name)
- Show attachment thumbnails inline (up to 4 with overflow counter)
- Remove dead columns (ai_policy_version, ai_evidence) from UI
- Add severity color coding (critical=red, high=orange, medium=yellow, low=blue)
- Add relative time display ('2h ago' instead of full datetime)
- Better hover effects and visual hierarchy
MessagesPanel:
- Add stats bar showing total/clean/warn/flagged/error/pending/deleted/edited counts
- Add AI status filter buttons (all, clean, warn, flagged, error, pending)
- Improve search UX with inline search icon and clear button
- Show filtered count in tab labels
ImageGrid:
- Include sticker images (was only attachments + embeds before)
- Add kind badge overlay (sticker/attachment/embed)
- Show user avatar next to image caption
- Better sticker rendering (object-contain with padding)
Header:
- Add shield icon per tab
- Improve tab titles and add descriptive subtitles
API client:
- Remove dead fields from MessageRecord (ai_moderation_raw, ai_policy_version, ai_evidence)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Drop 3 columns that were written but never read:
- ai_moderation_raw: raw LLM JSON response (~KB per message, never consumed)
- ai_policy_version: hardcoded 'default-2026-05-30', never used for decisions
- ai_evidence: JSON evidence array, never read after write
Changes:
- schema.ts: remove columns from both Postgres and SQLite table definitions
- messageStore.ts: remove from AIAnalysisUpdate interface and SET clauses
- aiAnalyzer.ts: remove from individual fallback update calls
- aiAnalysisWorker.ts: remove raw write, add missing fields (categories, severity, etc.)
- types.ts: remove from MessageRecord interface
- analysisRoutes.ts: remove from reset analysis call
- New migration: src/database/migrations/001_drop_unused_ai_columns.sql
- Migration applied to live DB: 27 columns → 24 columns
Kept ai_error (useful for future debugging, currently 0 non-null)
Kept metadata (1.9MB total, used for AI sticker/embed evidence analysis)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Adjusted formatting in conversationContext.ts for better token estimation readability.
- Enhanced readability in indonesianTextNormalizer.ts by formatting multiline replacements.
- Reformatted badword lists and whitelists in indonesianTextNormalizer.ts for consistency.
- Improved function signatures in messageStore.ts for clarity.
- Reformatted messageCapture.ts to enhance readability of channel ID checks.
- Cleaned up error logging in messageStore.ts and retentionManager.ts for better clarity.
- Reformatted indonesianSlangLexicon.ts for consistent object formatting.
- Enhanced URL fetching regex patterns in urlFetcher.ts for better readability.
- Simplified query parameter destructuring in analyticsRoutes.ts for cleaner code.
- Improved test readability in autoDeleteManager.test.ts and indonesianTextNormalizer.test.ts by formatting expectations.
- Cleaned up whitespace in messageCaptureFilter.test.ts for consistency.
- New sticker-specific vision prompt that tells LLM stickers are cartoon/meme art, not real photos
- New text-only warning for stickers that fail to download — prevents flagging based on name alone
- Updated system prompt with dedicated sticker guidance section (looser standards for cartoon content)
- Filesystem-backed sticker cache (keyed by name, 7-day TTL, 100MB max with LRU eviction)
- Config: STICKER_CACHE_DIR and STICKER_CACHE_MAX_SIZE_MB with defaults
- Updated .env.example with auto-delete + sticker config docs
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Added configuration options for NVIDIA Nemotron API key, model, and base URL.
- Refactored badword detection to utilize NVIDIA API, with a fallback to a local badword list.
- Updated moderation functions to handle asynchronous operations for text evidence generation.
- Removed dependency on the `indonesian-badwords` package and implemented custom detection logic.
- Enhanced tests to accommodate asynchronous behavior and validate new detection methods.
- Derive severity from status/score when ai_severity missing
- Derive recommended_action from severity/status when field missing
- Upgrade skip logs from debug to info so skip reasons are visible
- Allow both 'flagged' and 'warn' status for eligibility
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Implemented `useAnalytics` hook for fetching and managing analytics data.
- Created `analyticsStore.ts` to handle database queries for hourly stats, topic trends, user leaderboard, and moderation stats.
- Added Express routes for analytics endpoints including overview, hourly stats, topic trends, user leaderboard, moderation stats, and top violators.
- Introduced a utility function `filterHits` for filtering specific terms in text.
- Add indonesian-badwords dependency for local lexical signal
- Add Indonesian slang lexicon with woy/woi/hadeh as safe casual terms
- Normalize Discord custom emoji <:name:id> to [emoji:name] in prompts
- Wire normalization evidence into both conversationContext and llmModerationClient prompts
- Harden system prompt: woy/woi are casual greetings, not SARA/hate
- Add tests for emoji normalization, slang mapping, badword detection
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Added to safely extract and fetch up to 3 URLs per message (with SSRF protection, 5MB limit, and 8s timeout).
- Implemented recursive extraction to resolve Tenor/Giphy links from their HTML viewers to raw GIF binaries.
- In , fetched images are automatically injected as into the vision LLM context, and truncated webpage text is appended to the message string.
- In llmModerationClient.ts (inner retry), if OpenAI throws a 429 (or 401/403), throw p-retry's AbortError to immediately exit the 3-attempt inner retry loop.
- In aiAnalyzer.ts (outer retry), propagate the AbortError from runModerationAnalysis so the 2-attempt outer retry loop also aborts immediately.
- This ensures that a burst of 20 concurrent tasks hitting rate limits immediately returns the messages to the DB queue (as 'analysis_incomplete') and rapidly increments the individual circuit breaker, pausing processing and preventing a thundering herd instead of making 12 API calls per stuck message.
Infinite recovery loop (#new):
- Inside retryWithBackoff callback in processIndividualFallback, detect
'analysis_incomplete' in the LLM result and throw to trigger backoff.
- Track exhaustedOnIncomplete flag across retries.
- On final exhaustion: write terminal flag 'individual_analysis_exhausted'
to DB so the recovery query (which only looks for 'analysis_incomplete')
never picks this message up again.
- Transient failures (network/parse) are NOT written as exhausted; they
remain as 'analysis_incomplete' and are retried via the CB-throttled
recovery cycle.
Token budget zero-result deadlock (#10):
- If pickBatchWithinBudget returns [] because every candidate message
individually exceeds AI_ANALYSIS_MAX_TARGET_TOKENS, fall back to
messages.slice(0,1) so at least the first message is processed.
- Without this, messages would be permanently stuck as 'pending' because
every recovery tick would fetch them, trim to 0, and exit silently.
- Uses messages.slice(0,1) instead of messages[0]! to avoid the
forbidden noNonNullAssertion lint rule.
Stale state map memory leak (#9):
- startPendingAIAnalysisWorker now prunes conversationErrorCooldown and
conversationProcessing on every recovery interval tick.
- Cooldown entries past their expiry timestamp are deleted.
- Processing entries older than AI_ANALYSIS_PROCESSING_TIMEOUT_MS are
deleted (these represent stale locks from crashed processing runs).
- Prevents unbounded Map growth for long-running bots with many channels.
Batch/individual scheduling collision (#8):
- Build incompleteKeySet (Set<string>) from incompleteKeys before the
batch recovery loop.
- Batch recovery loop skips any key present in incompleteKeySet so a
conversation that has both 'pending' and 'analysis_incomplete' messages
is only targeted by the individual pipeline, not both simultaneously.
- Avoids the DB last-write-wins race where batch and individual pipelines
both update the same message rows concurrently.
#1+#5 - Individual fallback circuit breaker
- Add individualConsecutiveErrors + individualCooldownUntil (30s)
- On success: reset counter; on failure: increment + trip at
AI_ANALYSIS_INDIVIDUAL_CB_THRESHOLD (default 10) consecutive errors
- enqueueIndividualFallbacks checks CB before admitting any work
#1 - Individual fallback concurrency cap
- enqueueIndividualFallbacks enforces AI_ANALYSIS_INDIVIDUAL_MAX_CONCURRENT
(default 20); overflow stays as error/analysis_incomplete in DB and is
recovered by the recovery worker on the next interval
#3 - Unhandled rejection in async setTimeout
- scheduleConversationAnalysis no longer uses async arrow in setTimeout;
all async work is chained with .then()/.catch() explicitly
#4 - Recovery worker ignores individualInFlight
- Add individualInFlightByConversation Map<conversationKey, count>
- processIndividualFallback increments/decrements it in try/finally
- startPendingAIAnalysisWorker skips conversations present in the map
- Recovery worker also processes error/analysis_incomplete messages via
two new messageStore queries: getConversationKeysWithIncompleteAnalysis
and getIncompleteMessagesByConversation
#6 - pickBatchWithinBudget never called
- scheduleConversationAnalysis now calls pickBatchWithinBudget with
AI_ANALYSIS_MAX_TARGET_TOKENS (default 4000) + 50-token per-msg overhead
after fetching messages, before passing to processBatch
#7 - AI_PROCESSING_OVERLAP_MS 30s shorter than max LLM retry window
- Replace hardcoded 30 000 ms constant with configurable
AI_ANALYSIS_PROCESSING_TIMEOUT_MS (default 120 000 ms)
- LLM client: 30s timeout × 3 retries + backoff ≈ 90-100s; 120s is safe
New config keys:
AI_ANALYSIS_PROCESSING_TIMEOUT_MS (default: 120000)
AI_ANALYSIS_INDIVIDUAL_MAX_CONCURRENT (default: 20)
AI_ANALYSIS_INDIVIDUAL_CB_THRESHOLD (default: 10)
AI_ANALYSIS_MAX_TARGET_TOKENS (default: 4000)
New AnalysisQueueStatus fields:
activeIndividualRequests, individualInFlightCount,
individualCircuitBreakerActive
- After a batch LLM call, any result flagged analysis_incomplete is
immediately fanned out to an individual per-message fallback queue
- Batch hard-fail (result.ok=false) and unhandled exceptions now also
route all affected messages to the individual queue instead of waiting
behind the conversation error cooldown
- Individual queue runs fully parallel (fire-and-forget per message),
de-duplicated by a Set<messageId> so no double-processing
- processIndividualFallback runs in the main process (no worker pool IPC
overhead for a single-item call), with retryWithBackoff 2x/2-15s
- AnalysisQueueStatus gains activeIndividualRequests +
individualInFlightCount fields for dashboard observability
MythEclipse