The 32 few-shot examples each re-echoed score/confidence/
recommended_action/categories/policy_version inline (~150 chars ×
32). Those fields carry zero moderation-decision signal — the schema
and their ??-default coercion already live in OUTPUT_INSTRUCTIONS +
moderationResponseParser.ts. Removed 96 redundant key/value pairs.
Kept per-example: message_id, status, flags, severity, evidence,
analysis — the fields that actually teach decisions. Parser derives
the rest via ?? fallback, so real output shape is unchanged.
examples.ts: 21.7K→18.5K chars; FEW_SHOT(mixed) 15.3K→13.4K.
Total mixed system prompt now 33.9K (was 39.3K at audit start,
~14% leaner). tsc + 129 tests + biome green.
Co-Authored-By: Claude Opus 5 (Nous Research)
- prompts/system.ts: merge 3 overlapping framing blocks (Blok Data /
Konteks Pengguna / Framing Konteks vs Target) into 1 tight block —
same coverage, no duplicated "standalone judgment / profile-is-
reference-not-evidence" prose.
- prompts/output.ts: trim duplicated user_history/standalone paragraph
in PERSONALITY & MEMORI (keep concrete per-case lessons).
- prompts/examples.ts: drop 2 exact-duplicate-lesson few-shots (LGBT id=19
dup of id=30; weapons-tech id=33 dup of id=32). All teaching signals
retained via the surviving example of each lesson.
Static system prompt: text 32.7K→29.2K, mixed 39.3K→35.8K chars
(~10% smaller). No moderation rule, zero-tolerance category, or decision
tree altered — accuracy-controlling content untouched. tsc + 129 tests +
biome green.
Co-Authored-By: Claude Opus 5 (Nous Research)
Adds per-message AI moderation analysis time (ai_analysis_duration_ms)
so operators can see how long the LLM took to moderate each message.
Gateway:
- messagesTable: new ai_analysis_duration_ms (bigint) column.
- AIAnalysisUpdate + buildAIAnalysisSet: carry analysisDurationMs through
both single and bulk update paths.
- ai-analysis-worker: measure wall-clock time around runModerationAnalysis
and attach it to every result in the batch.
Backend:
- Mirror schema column; messageMapper maps ai_analysis_duration_ms;
moderation-types + MappedMessage expose it.
Frontend:
- message.ts type gains ai_analysis_duration_ms.
- AiBadge (messages view) shows 'status · 1.2s' when duration is present;
analysis view badge mirrors the same formatting.
DB:
- scripts/add-ai-analysis-duration.sql (idempotent ADD COLUMN IF NOT EXISTS).
No behavior change for moderation logic; null until new gateway build
records values.
Qdrant upserts were failing with 'This operation was aborted' ~32x/2h,
so semantic moderation cache entries were silently dropped. Root cause:
upsertQdrantPoint ran ensureQdrantCollection() on EVERY call — a GET
(and sometimes DELETE+PUT) round-trip — while the request AbortController
had only a 10s timeout. Under moderation load Qdrant is busy (the
gmw_text_moderation collection is not yet HNSW-indexed, so searches are
full-scans), the extra round-trips pushed the upsert past 10s, and the
client aborted it.
- Memoise ensureQdrantCollection() at module scope so the collection is
verified exactly once per process (resetQdrantCollectionCache() for
tests / config reload).
- Bump the upsert request timeout 10s -> 30s so a transiently busy
Qdrant no longer aborts the write.
Qdrant server itself is healthy (<100ms for direct upsert; collection is
green), so no server-side change is needed. Semantic cache should now
populate reliably.
Address every remaining biome lint/format warning across both services
so the codebase ships warning-free:
- textCacheStore: drop unused deleteExpiredQdrantPoints import; hash
image cache key (sha256[:32]) so long/base64 URLs no longer blow the
text_analysis_cache PK B-tree 8191-byte index (was aborting the media
analysis lock INSERT).
- bootstrap: drop unused unhandledRejection promise param.
- moderationOrchestrator: drop unused destructure at L197.
- mediaDownloader / textBatchProcessor / transmitter: replace non-null
assertions with proper null guards (stickerName ?? '', urlImages.get
guard, backpressureQueue.shift guard).
- backend utils: throw lastError ?? fallback instead of lastError!.
- message-capture: remove unused (retentionDb), (moderationActionsDb,
reviewsDb); simplify renderDiscordMentions guard to optional chain.
- transmitter: remove dead write-only field + its assignments.
No behavior change beyond the cache-key hashing (now deterministic
fixed-length) and the intentional null-safety guards.
Two root causes behind 'all image analysis failing':
1. imageResizer still emitted lossless PNG for vision input. A 1024px
Facebook photo balloons to multi-MB PNG base64 that the vision model
silently rejects ('Vision API null response'). Switch to JPEG q85
(no upscaling) — same photo drops to ~100-400KB, model processes fine.
Re-encodes even already-small images so raw originals never bloat the
data URL. Added tests/imageResizer.test.ts covering both cases.
2. acquireMediaAnalysisLock INSERT aborted with 'index row requires N
bytes, maximum size is 8191'. text_analysis_cache.text is the PK in a
B-tree index (8191-byte/row cap); callers pass the raw image URL as the
key, and base64 data URLs / very long URLs blow past the limit, so the
lock INSERT fails and every media analysis is skipped. Hash the URL in
makeImageCacheKey (image:<sha256[:32]>) — fixed-length, deterministic,
well under the limit. All store/get/lock/delete callers already route
through this function so lookup stays consistent.
- config: add AI_LLM_VISION_BASE_URL + AI_LLM_VISION_API_KEY (separate from text router)
- llmClient: llmVision() now calls dedicated vision endpoint when configured
(axios POST to integrate.api.nvidia.com, model nvidia/nemotron-3-nano-omni-30b-a3b-reasoning,
reasoning_budget 16384, non-stream), falls back to router combo otherwise
- keeps text/moderation on omniroute, vision on NVIDIA direct
- Add examples for technical discussions (kinetic energy, drone weapon
engineering, physics simulations) that should be marked clean
- System rule: physics/engineering topics (kinetik, gravitasi, energi,
drone, senjata, drone warfare, CAD, CNC, 3D printing, robotics, aerospace)
are safe when in technical context — flag only if explicit threat
- Riwayat pengguna dengan pelanggaran sebelumnya tidak memengaruhi
penilaian pesan bersih yang terpisah dan tidak mengandung pelanggaran
- Removed getUserRecentInfractions usage in textBatchProcessor.ts and visionAnalyzer.ts
- Removed buildUserHistoryXml import and calls
- Messages are now evaluated standalone, not influenced by past violations in other channels
- Updated moderation prompts with clearer instructions about user_history usage
- Fixes issue where benign messages like 'tubuh manusia vs gravitasi' were incorrectly flagged due to carryover from previous drone weapons discussion
The user history context was causing the LLM to interpret unrelated current messages
as threats because it conflated them with past violations. Now each message is judged
on its own merit with only channel-specific context.
Add debug logging to trace cacheKey + messageId + content length on
every vision cache HIT and MISS, so we can detect if the vision model
returns duplicate analysis for different images (provider issue vs
cache collision). Includes the phash on cache miss (new analysis cached).
Follow-up to 9f7ce7d which fixed makeImageCacheKey to hash full data
URL instead of just first 128 chars (root cause of all images sharing
the same cached 'konten judi' verdict due to hash collision).
Root cause: makeImageCacheKey() only hashed the first 128 chars of the
data URL. Since all resized images use the same MIME prefix
('data:image/png;base64,') + identical base64 header bytes, nearly every
image got the same 16-char hash → 'image:<same-hash>' → all images reused
the first cached vision analysis (often a gambling-detection verdict).
Fix: hash the entire data URL instead of just the prefix. Verified
114 stale 'image:' entries + 745 stale 'phash:' entries purged from prod
DB. tsc --noEmit clean, 133 tests pass.
- Add term_glossary_cache table + migration 0014: resolved definitions are
stored permanently (definitions rarely change); misses stay ephemeral in
Redis/LRU with 1h TTL so transient failures get retried
- Lookup flow: LRU -> Redis -> Postgres (permanent) -> live SearXNG; DB hits
re-warm the fast caches; stale Redis miss sentinels no longer shadow DB
- Rate-limit-aware live lookups: concurrency 2 + stagger, retry once on empty
results, strict definition filter (Wikipedia preferred, rejects
disambiguation/ads/translate-homepages)
- Make SEARXNG_BASE_URL configurable via env (default unchanged)
Root cause (3rd layer after 50371bd + 4f4c435): a vision model run
(2026-08-10) returned 'Maaf, saya tidak melihat gambar apapun yang terlampir...'
and that text was cached as a VALID vision_llm result (image + phash keys,
24h/7d TTL). Every subsequent analysis of the same image (same hash/phash)
hit the poisoned cache, so image analysis looked broken forever even though
9router responded fine — the moderation LLM wrote 'lampiran yang gagal
terbaca' from a cache hit.
Also: mimo via 9router streams reasoning in delta.reasoning +
delta.reasoning_details[].text (content:"") — extractChunkText only read
delta.reasoning_content, so those runs aggregated empty → 'Vision API null
response' (observed 08:54/09:07/09:38).
Fixes:
- llmClient.extractChunkText: fall back to delta.reasoning and
reasoning_details[].text (mimo), on top of reasoning_content (gemma).
- visionAnalyzer: isNoImageSeenText() detects 'no image' style outputs;
such results are NEVER cached, and poisoned entries are purged when hit
(LRU/DB/phash) so re-analysis actually re-runs vision.
- Tests: reasoning/reasoning_details extraction + isNoImageSeenText
(Indonesian + English, no false positives on real descriptions).
Root cause (2nd layer after 50371bd): the analysis worker could pick up an
image message while its attachment upload was still in flight
(upload_status='pending'). downloadAndExtractFrame then fell back to the
Discord CDN URL (cdn.discordapp.com), which often 404s for old/purged links,
and 'if (!res.ok) return' silently dropped the image — no log, no vision
call, empty image map, and the LLM produced a text-only verdict like
'lampiran yang gagal terbaca oleh sistem'.
Fixes:
- ai-analysis-worker: skip targets whose attachment upload is still pending
(both batch + individual paths) — they stay ai_status='pending' and the
next 15s cycle analyzes them after the upload lands.
- mediaDownloader.downloadAndExtractFrame: try uploaded_url first, then
discord_url as fallback; log non-OK responses (status + host) instead of
silently returning; log when all candidate URLs fail.
Root cause: 9router combo 'multimodal' routes to cloudflare-ai/@cf/google/
gemma-4-26b-a4b-it which streams ALL output in delta.reasoning_content
(content:"") and finishes with 'length' at max_tokens. llmClient only read
delta.content, so llmVision returned empty → every image moderation fell back
to text-only analysis ('Meskipun analisis gambar gagal' in every ai_analysis).
Fix: extractChunkText() prefers delta.content then falls back to
delta.reasoning_content (also handles message/text/response fields), with
unit tests for the exact 9router chunk shape. Verified live against a real
DB image: oc/mimo-v2.5-free (new first model in the multimodal combo) returns
a proper description in delta.content.
- <message> targets now carry time (ISO), repetitions (N identical short texts = spam signal), bot and edited flags; escape id/user XML
- rich <user_reputation>: total_infractions, clean_streak, last_offense_days_ago, repeat_offender (7-day window)
- <user_history> with last flagged messages for repeat offenders (wires dead getUserRecentInfractions)
- <user_profile as_of> staleness signal; <location_context topic> from captured channel topic
- prompt framing + output instructions teach the LLM to use the new signals without treating history as proof
- tests: contextEnrichment.test.ts (13) + topic cases in conversationContext.test.ts
When the ONLY violation is offensive_username (message content clean):
- Message is NOT deleted (nickname-only violation bypasses auto-delete)
- Member's server nickname is reset to default username via
setNickname(null) (Discord shows the global username again)
- Action 'reset_nickname' logged to moderation_actions; cooldown
10min per guild:user (LRU) so repeated messages by same member
don't hammer the Discord PATCH
- Config: AUTO_NICKNAME_RESET_ENABLED / AUTO_NICKNAME_RESET_COOLDOWN_MS
- resolveDisplayName(): member.displayName from captured metadata,
falls back to global username
- Applied to context lines, target message blocks, and media message
blocks — LLM sees the name the channel actually sees (nickname can
carry moderation signal itself)
- Conversation context recency gates (GAP_MS/MAX_AGE_MS): drop stale
messages before silence gaps; cold_start anchor + flow descriptor
tells LLM whether conversation is ongoing or restarted
- [location] block: channel name, thread name, nsfw/age flags from
captured metadata (thread names instead of bare IDs)
- Link media -> multimodal: text-batch URL fetches that resolve to
images now run vision analysis (bounded 15s) and switch prompt to
mixed mode; <web_content> gains og:title for page context
- pnpm-workspace.yaml: approve sharp build script (unblocks install)
Audit lanjutan: 6x 'LLM API request failed: Request was aborted' per jam.
Root cause: 9router/omniroute SELALU balas SSE (data: chunks) walau request
tanpa stream:true — SDK OpenAI non-stream menunggu FULL body sebelum parse,
jadi batch moderasi besar yang upstream-nya lambat kena timeout 30-60s dan
di-abort. llmClient sudah punya agregasi streaming (chunks → ChatCompletion).
Fix: stream:true di llmCaller (moderasi batch/individual), llmVision,
cultureLearner, userProfileLearner. Verified: SDK stream test 806ms vs
sebelumnya abort. Caller lain (recovery worker dll) lewat llmCaller sama.
Audit log produksi (sejak deploy13:38) menemukan 3 isu:
1. mediaDownloader.ts spawn /usr/bin/ffprobe + /usr/bin/ffmpeg (path keras) —
ENOENT di Nix karena binary cuma di ffmpeg-headless closure. Pakai
PATH-resolved ('ffprobe'/'ffmpeg') seperti voice-recording module
(ffmpegProcess.ts/transmitter.ts) — 5 media warning hilang.
2. individualFallbackProcessor log error 'Success' di level50 tiap fallback
BERHASIL (logModerationError dengan new Error('Success')) — ganti
logger.info dengan verdict yang sama; error log cuma untuk error asli.
3. moderationResponseParser: strip frasa penutup generik ('Tidak ada
indikasi pelanggaran.') yang masih sering dikeluarkan LLM walau prompt
melarang (277/1486 analisis mengandung frasa, termasuk hari ini).
sanitizeGenericCleanCloser hanya mencocok frasa di AKHIR, teks substantif
tetap utuh. Unit test: 6/6 pass.
QoL lanjutan dari fix60084b3: content pesan mentah masih nampilin
snowflake (<@&roleid>, <@userid>, <:emoji:id>) di log moderasi dan
prompt LLM. Sekarang dirender ke nama yang bisa dibaca:
- Gateway capture: metadata menyimpan mentionedRoles + mentionedUsers
(id+name) dari message.mentions, disimpan ke metadata JSON
- renderDiscordMentions(): <@&id> -> @RoleName, <@id> -> @Username,
<:name:id> -> :name:, fallback @role/@user — dipakai di
conversationContext (konteks LLM) dan moderationBuilders
(getAnalysisContent) sehingga LLM lihat nama role/user beneran,
bukan placeholder generik
- Frontend renderMessageContent() (mirror gateway) dipasang di semua
tempat nampilin content: message-card, message-detail(-view),
search-overlay, search-panel, users/channels section, live-stream,
mod-queue, review list; sticker-only message tetap [Sticker: name],
pesan teks+sticker kini ikut nampilin nama sticker
- tsc --noEmit PASS di gateway & frontend; renderDiscordMentions
diverifikasi manual (6 kasus: role/user/emoji/unknown/plain)
- Deleted the text analysis prompt constants and helpers as they are no longer needed.
- Added batch search functionality for Qdrant to optimize vector searches.
- Implemented methods for deleting expired Qdrant points and invalidating cache based on content hash.
- Updated text batch processor to use new timeout configurations and modified content building for moderation prompts.
- Enhanced text cache store to support new Qdrant integration and improved cache invalidation logic.
- Introduced a new user reputation model with a more nuanced trust scoring system, including penalties and rewards for user behavior.
- Added unit tests for the new trust model to ensure correctness of penalty and trust gain calculations.
- Updated configuration schema to reflect new timeout settings and removed deprecated OpenAI moderation keys.
New qdrantClient.ts (zero-dep fetch REST): ensure collection with cosine
distance (auto-recreate on vector-size change), upsert point w/ verdict
payload, search w/ expires_at filter + score threshold.
textCacheStore: when QDRANT_URL set, embeddings are upserted to Qdrant
(primary) and searched there first; Postgres embedding column remains as
legacy fallback for pre-Qdrant rows. Config: QDRANT_URL/COLLECTION/API_KEY.
QDRANT_URL already in repo .env; added to VPS env + GATEWAY_ENV secret.
OpenAI SDK v6 defaults to encoding_format=base64; llama-nemotron-embed
(Nvidia-backed) returns 400 'do not support base64'. Semantic cache was
silently disabled in prod. encoding_format: 'float' fixes it.
TS compiled this fine, but the JS spec forbids mixing || and ??
without explicit parens; Node threw 'Unexpected token ??' at startup,
crash-looping gmw-discord-gateway (restart counter 250). Wrap the
fallback chain in parens so the expression is valid.
Root cause: ai-analysis-worker read llmResult.explanation and
llmResult.toxicityScore — fields the LLM pipeline never produces
(canonical AnalysisResult uses analysis/score). Every message fell back
to the bare template "Tidak ada indikasi pelanggaran." and the stored
score was always 0.
- Map analysis/score correctly; fallback now quotes the message content
- Prompt: ban generic analysis phrasing, require reply context
- LLM context: include replied-to message content (metadata.reference)
so the model can explain what the user is replying to
- Frontend: show thread/channel names from metadata instead of raw IDs
(message card, detail views, search overlay); detail panel now
displays the ai_analysis text
- Auto-delete log/DM include the descriptive analysis as the reason
Delete fastClassifier.ts (manual regex patterns for phone/email/IP/crypto/
spam/toxicity) and simpleFallback.ts. These hardcoded patterns were the
source of false positives (Discord emoji snowflakes matched phone_number,
URL digits matched phone, etc.) and produced heuristic verdicts whenever
the LLM failed.
New flow: Message → LLM (with conversation context, media evidence, user
reputation) → verdict. On LLM failure the message is marked 'error' and
retried by the recovery worker — no heuristic verdicts, ever.
Discord markdown tokens (custom emoji/mentions/timestamps) are normalized
to readable placeholders ([emoji:name], @user, @role, #channel, [time])
before reaching the LLM via discordTokens.ts.
Custom emoji (<:name:id>), user/role/channel mentions and timestamps embed
long numeric snowflakes that tripped the phone_number / personal_info /
ip_address_sharing patterns — e.g. <:mambotongue:1463255254220148939> was
flagged as phone_number. Strip Discord markdown tokens before pattern
matching and require phone matches to not sit inside a longer digit run.
Build & Deploy / build-and-push (backend) (push) Failing after 35s
Build & Deploy / build-and-push (discord-gateway) (push) Failing after 25s
Build & Deploy / build-and-push (proxy) (push) Failing after 25s
- Remove pnpm workspace, moon repo, and all monorepo tooling
- Delete packages/shared/, embed shared code directly into each service
- Copy packages/shared/src/* -> services/backend/src/shared/ and services/discord-gateway/src/shared/
- Replace all @bete/shared imports with @/shared/ path alias
- Remove @bete/shared workspace dependency from both services
- Update root package.json scripts from --filter to --prefix
- Rewrite Dockerfiles to build each service standalone
- Clean up biome.json, .gitignore, remove root drizzle.config.ts
Build & Deploy / build-and-push (backend) (push) Successful in 25s
Build & Deploy / build-and-push (proxy) (push) Successful in 3m46s
Build & Deploy / build-and-push (discord-gateway) (push) Failing after 4m42s
- Cast llmResult through unknown to handle type mismatch between
shared AnalysisResult and layer-specific local type
- Exclude src/**/archive/** from tsconfig to prevent dead code errors
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Build & Deploy / build-and-push (discord-gateway) (push) Failing after 28s
Build & Deploy / build-and-push (backend) (push) Successful in 1m46s
Build & Deploy / build-and-push (proxy) (push) Successful in 1m37s
\U escapes are not valid in JavaScript/TypeScript regex literals.
Use new RegExp() constructor to avoid TS parser issues.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-28 15:10:47 +07:00
DeveloperandClaude Opus 4.8 (1M context) <noreply@anthropic.com
Full frontend redesign with glassmorphic dark theme, floating top nav,
Live2D mascot, split-pane messages, and Ops Center dashboard.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com
Build & Deploy / build-and-push (discord-gateway) (push) Failing after 2m22s
Build & Deploy / build-and-push (backend) (push) Failing after 3m22s
Build & Deploy / build-and-push (proxy) (push) Successful in 1m36s
Build & Deploy / deploy (push) Skipped
- Consolidate all DB schema definitions into packages/shared as single source of truth
- Migrate backend from raw SQL to Drizzle ORM across all modules
- Extract frontend inline UI into separate component files
- Refactor discord-gateway circuitBreaker into conversationState + moderationState
- Convert messageStore to Proxy singleton pattern
- Add validateBody/validateQuery middleware + Zod schemas for API endpoints
- Modernize Docker builds with multi-stage + pnpm deploy
- Migrate CI/CD from deployment to image-based pipeline
- Remove 60+ unused/dead files (~15K lines)
- Update color scheme from sky-blue to teal-cyan
- Move DB connection management to @bete/shared/database
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Added new dependencies for Next.js and lucide-react in pnpm-workspace.yaml.
- Refactored DashboardPage component to improve readability and error handling.
- Enhanced Header component to display error status with an alert icon.
- Updated MobileTabBar and Sidebar components to use a centralized tabs definition.
- Improved ChannelsView in dashboard-panel to handle channel fetching more cleanly.
- Fixed ActiveSpeaker type to use camelCase for userId.
- Updated MessagesPanel to handle guildId checks more gracefully.
- Adjusted API calls in dashboard and messages to align with backend expectations.
- Refined type definitions across various interfaces for consistency and clarity.
- Fix noImplicitAnyLet: add type to let match variable
- Fix noAssignInExpressions: use matchAll() + for-of instead of while
- Suppress useExhaustiveDependencies in mascot scroll effect
- Suppress useSemanticElements for message card click handler
