Common Indonesian religious expressions like 'Astaghfirullah', 'Astaga',
'Alhamdulillah', 'Subhanallah', dll were being flagged as vulgar_language
by the LLM. Added explicit rule that these are normal religious/cultural
expressions in Indonesia - not vulgar language - even in all-caps or
with repeated letters.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The rule-based badword detector was injecting [normalized_text] and
[normalization_notes] tags into the LLM prompt that caused false
positive hallucinations - the LLM started associating innocent words
('sapik', 'furina') with furry/sexual_deviation due to misleading
context injected by the normalizer.
Removed:
- indonesianTextNormalizer.ts (full file deletion)
- formatModerationTextEvidenceForPrompt import/usage in llmModerationClient
- formatModerationTextEvidenceForPrompt import/usage in conversationContext
- stale re-exports in index.ts
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Adds explicit rule that Indonesian names/nicknames like 'Sapik' (Syafik),
'Ayang', 'Dek', 'Bang', 'Mas', etc. are NOT furry or sexual_deviation references
- Adds rule prohibiting the LLM from inventing slang meanings for words
it doesn't recognize - default to innocent until proven guilty
- Prevents false positive cascade where LLM confuses names with furry slang
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Nama karakter game/anime populer seperti 'Furina' dari Genshin Impact
sering kena false positive sebagai 'sexual_deviation' karena kemiripan
fonetik dengan kata 'furry'. Menambahkan aturan eksplisit bahwa nama
karakter fiksi normal bukan referensi furry fetish, dgn pengecualian
jika konteks pesan secara eksplisit membahas aspek fetish/seksual.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Update moderation guidelines to prevent false positives on song lyrics, poems, memes, and literary quotes, ensuring political or revolutionary content is not flagged as conflict instigation unless accompanied by explicit incitement.
- Add automatic fallback to streaming mode if the provider rejects non-streaming requests with a 400 error
- Refactor `llmChat` to use an internal execution function to support retry logic with modified parameters
- Update moderation prompt to explicitly allow Japanese pop culture terms (e.g., "moe", "waifu", "wibu") to prevent false positive sexual deviation flags
- Increase OpenAI client timeout from 15s to 60s to handle high-latency models
- Add `test-llm.js` for manual verification of LLM connectivity and response times
Wrap the LLM completion logic in a try-catch block to provide detailed
error logging, including status codes and raw response data, when
API requests fail.
- Add comprehensive error logging for failed LLM API calls.
- Ensure streaming responses are correctly aggregated and returned
even when wrapped in error handling logic.
Update the LLM client to support streaming responses and improve
database migration idempotency.
- Add `stream` support to `llmChat` with dynamic chunk parsing for
compatibility across different LLM providers.
- Remove hardcoded default temperature and top_p to allow for more
flexible parameter passing.
- Update database migrations to use `IF NOT EXISTS` for tables and
indexes to prevent errors during re-runs.
Removes outdated implementation guides and architecture documentation
related to the mascot chatbot and microservices setup.
- Delete `MASCOT_CHATBOT_GUIDE.md`
- Delete `MASCOT_IMPLEMENTATION.md`
- Delete `README_MICROSERVICES.md`
Adds new database schema migrations to support the implementation of
user reputation tracking and channel-specific culture context.
- Creates migration `0005_large_squadron_sinister.sql` containing
`user_reputations` and `channel_cultures` tables.
- Updates Drizzle migration journal to include the new migration step.
- Adds `fix_prompt.py` utility for managing prompt adjustments.
Implements a context-aware moderation system by tracking user behavior
and channel-specific norms to improve AI decision-making accuracy.
- Adds `user_reputations` table to track trust scores, clean streaks,
and infraction history.
- Adds `channel_cultures` table to store AI-generated summaries of
channel-specific norms and slang.
- Implements `userReputationStore` to autonomously update user scores
based on moderation outcomes (clean vs. flagged).
- Implements `cultureLearner` and `channelCultureStore` to manage
evolving channel contexts.
- Enhances LLM prompts to inject user reputation (trust scores,
history) and channel culture summaries, enabling "wisdom-based"
moderation (e.g., giving benefit of the doubt to high-trust users).
- Integrates reputation and culture updates into the existing
`aiAnalyzer` pipeline.
Refactors the AI moderation pipeline to improve concurrency control and
cache efficiency by moving from user-centric to content-centric caching.
- Implements a distributed locking mechanism for media analysis using
`acquireMediaAnalysisLock` to prevent redundant LLM vision calls across
multiple pods.
- Transitions text moderation caching from `user_mod:userId:hash` to a
purely content-based `text_mod:hash` approach to increase hit rates.
- Enhances `getPendingMessagesByConversation` with atomic transactions
and `FOR UPDATE SKIP LOCKED` to safely transition messages from
`pending` to `processing` state.
- Adds `processing` status to the `AIStatus` type and database schema to
track active analysis lifecycles.
- Implements polling logic in `llmModerationClient.ts` to wait for
in-progress media analyses.
Introduces a `processing` state to the AI analysis lifecycle to prevent
duplicate processing of the same messages.
- Implements row-level locking using `FOR UPDATE SKIP LOCKED` in
`messageStore.ts` to ensure atomic message acquisition.
- Adds a `processing` status to the `AIStatus` type and database schema.
- Fixes a TOCTOU race condition in `aiAnalyzer.ts` by synchronizing
the conversation processing lock before async database operations.
- Implements `revertStuckProcessingMessages` to recover messages stuck
in the `processing` state due to worker crashes or timeouts.
- Updates `processBatch` and scheduling logic to correctly manage and
release conversation-level locks.
- Instructs the AI to decode combinations of regional indicator emojis (e.g., 🇬 🇦 🇾) and custom letters spelling out words, rather than dismissing them as 'just a series of emojis'
- Added a specific few-shot example (Contoh 15) to demonstrate flagging this technique when used to spell banned words
- Added explicit zero-tolerance rule for anatomical/sexual vulgarity (e.g. titten, kontol), explicitly forbidding the AI from passing them off as 'casual conversation' or 'jokes'
- Expanded sexual_deviation rule to explicitly cover brief mentions of BL (Boys Love), yaoi, yuri, and LGBT topics, instructing the AI to flag them regardless of casual context
Implements full audit from double_queue_audit.md.
## Critical
fix(backend): reanalyze-batch — per-scope in-flight guard (messages.routes.ts)
Two concurrent admin sessions clicking 'Retry All Errors' simultaneously now
get a 409 REANALYZE_BATCH_IN_PROGRESS for the same guildId:channelId scope.
Prevents the recovery worker from being triggered twice for the same set of
error messages.
fix(discord-gateway): messageUpdate embed resolution skip (messageCapture.ts)
Discord fires messageUpdate when link previews resolve 1-2s after send
even though the message body is unchanged. Compare newContent vs
existingContent before resetting ai_status to pending and re-queueing LLM.
Eliminates a spurious duplicate analysis that could overwrite a valid result.
## High
fix(discord-gateway): scheduleAutoDelete idempotency (aiAnalyzer.ts)
Add autoDeleteInFlight Set. Both processBatch and processIndividualFallback
call scheduleAutoDelete; without the guard, a message that races through
both paths launches two concurrent attemptAutoDeleteFlaggedMessage calls,
producing a duplicate moderation-action log entry and a Discord 10008 error.
The Set is cleaned up in a .finally() block after each attempt completes.
## Medium
fix(frontend): revert optimistic pending state on HTTP failure (useMessages.ts)
Capture the prior MessageRecord inside the setMessages functional updater
(no extra useCallback deps needed). If reanalyzeMessage() throws, restore
the snapshot so the UI reflects the real DB state instead of lying.
fix(discord-gateway): exclude individual_analysis_exhausted from recovery queries
Both getConversationKeysWithIncompleteAnalysis and
getIncompleteMessagesByConversation now add a NOT LIKE guard for
individual_analysis_exhausted. Prevents an infinite recovery loop if a bug
ever writes both flags to the same row.
## Low
fix(discord-gateway): unified timer path in scheduleConversationAnalysis
Remove the separate cooldown-path that created a secondary timer calling
scheduleConversationAnalysis recursively. Replace with a single
clear-and-reset pattern where delayMs = max(cooldownRemainder+500, debounce).
Eliminates the edge case where both timers were live simultaneously.
## Misc
fix(backend): cast req.params.id to String() to satisfy Express typings
Pre-existing tsc error (string | string[]) exposed by our edit.
String() is correct; route params are always scalar strings at runtime.
Three root causes patched:
1. aiAnalyzer.ts — processBatch apiFailedMessages path:
After reverting messages to 'pending', suppress shouldScheduleNext
(was true by default) to prevent scheduleConversationAnalysis from
firing immediately and racing with the recovery worker that will pick
up those same pending messages on its next poll cycle.
Also release the conversationProcessing lock immediately after the
revert so the cooldown timer (not the full processing-timeout) gates
the next attempt.
2. messages.routes.ts — POST /messages/:id/reanalyze:
Add a per-message reanalyzeInFlight Set. Concurrent requests for
the same ID now return HTTP 409 instead of issuing duplicate UPDATEs
and triggering multiple recovery worker activations.
Also narrow the SQL predicate to 'WHERE id = AND ai_status != pending'
so a click that arrives while the recovery worker already picked the
message up is a no-op at the DB level.
