Commit Graph
229 Commits
Author SHA1 Message Date
asepharyana 12b23b4632 fix: initialize release variable to prevent potential errors in start/stop methods
Deploy to VPS / deploy (push) Successful in 2m9s
2026-07-26 14:30:36 +07:00
asepharyana 0a6a9fd982 feat: update dependencies and improve dashboard functionality
Deploy to VPS / deploy (push) Failing after 1m43s
- Added new dependencies for Next.js and lucide-react in pnpm-workspace.yaml.
- Refactored DashboardPage component to improve readability and error handling.
- Enhanced Header component to display error status with an alert icon.
- Updated MobileTabBar and Sidebar components to use a centralized tabs definition.
- Improved ChannelsView in dashboard-panel to handle channel fetching more cleanly.
- Fixed ActiveSpeaker type to use camelCase for userId.
- Updated MessagesPanel to handle guildId checks more gracefully.
- Adjusted API calls in dashboard and messages to align with backend expectations.
- Refined type definitions across various interfaces for consistency and clarity.
2026-07-26 14:27:36 +07:00
asepharyana cedce85087 ci: fix lint errors for CI
Deploy to VPS / deploy (push) Successful in 1m47s
- Fix noImplicitAnyLet: add type to let match variable
- Fix noAssignInExpressions: use matchAll() + for-of instead of while
- Suppress useExhaustiveDependencies in mascot scroll effect
- Suppress useSemanticElements for message card click handler
2026-07-26 12:01:51 +07:00
asepharyana 9621317a3c chore: format files for lint
Deploy to VPS / deploy (push) Failing after 8m8s
2026-07-08 01:37:46 +07:00
asepharyana c0edb0fa40 feat: enhance deployment instructions and exclude spam threads from message capture 2026-07-05 06:58:17 +07:00
asepharyana 27e929580e feat: update components and hooks to use get_untracked for improved performance 2026-07-04 03:03:36 +07:00
asepharyanaandworkflow agents ade5d6a7c3 fix: backend and discord-gateway improvements
- Update shared database schema
- Add shared utils
- Refactor backend middleware, auth routes, and dashboard repository
- Improve media analysis client with better error handling
- Fix searxng search URL construction
- Update URL fetcher for robustness

Co-authored-by: workflow agents
2026-07-02 06:02:07 +07:00
asepharyana bf9d018f7a fix: skip AI analysis for channel 1508059937031589949 2026-07-02 04:28:33 +07:00
asepharyana fa82593a38 fix: switch to node:22-slim, install deps from npm, remove vendor 2026-07-02 04:07:18 +07:00
asepharyana 7efaf00c93 Revert "feat: migrate frontend to Astro + expand AI moderation + backend admin/runtime config"
This reverts commit d59b59a7a7.
2026-07-02 03:54:44 +07:00
asepharyana d59b59a7a7 feat: migrate frontend to Astro + expand AI moderation + backend admin/runtime config
Frontend:
- migrate from Vite to Astro (astro.config.mjs, pages/, layouts/)
- add admin panel, settings page, command palette, error boundary
- refactor App.tsx, MascotChatbot, Sidebar, Header, DashboardLayout
- update API client, WebSocket, auth, dashboard features

Backend:
- add admin module and config routes
- refactor middlewares, Redis connection, WebSocket server/bridge
- add runtime config loader

Discord Gateway:
- refactor AI moderation: circuit breaker, concurrency limiter, fallback processor
- add media analysis client, Seaxng search, user profile learner
- add new drizzle migration

Shared:
- extend database schema, add new config fields
2026-07-02 00:02:41 +07:00
MythEclipse d5c22a3959 fix(discord-gateway): handle forwarded message content via messageSnapshots
Two-layer fix for forwarded messages showing as empty/clean:

Layer 1 (messageMetadata.ts): getReferencedMessageContent() now falls
back to message.messageSnapshots Collection when channel.messages.cache
lookup fails. Discord stores forward content in message_snapshots API
field, not in message.content.

Layer 2 (moderationBuilders.ts): buildReferenceXml() now parses msg.
metadata JSON to extract reference.content when DB getMessageById()
fails (cross-server forwards not in local DB).

Previously: forward messages captured with empty parentContent →
LLM saw no reference text → '99% confidence, pesan kosong'.
Now: forward content flows through capture → metadata → analysis.
2026-06-23 23:27:23 +07:00
MythEclipse ed4a506ca7 refactor: split llmModerationClient.ts + add tests + metrics
## Split llmModerationClient.ts (2103 → 3 files)
- **moderationBuilders.ts** (67 lines) — shared: escapeXml, getAnalysisContent, buildReferenceXml
- **mediaAnalysisClient.ts** (656 lines) — vision analysis with multi-layer LRU/DB/phash caching, image/video download, ffmpeg frame extraction, prepareMediaMessage
- **moderationOrchestrator.ts** (998 lines) — callModerationLLM, runTextOnlyBatch, runMediaBatch, runModerationAnalysis, runSimpleTextFallback
- **llmModerationClient.ts** (30 lines) — re-export bridge (backward compat)

No import changes needed — aiAnalysisWorker.ts still imports from llmModerationClient.js.

## Unit tests (backend)
- vitest.config.ts + e2e.test.ts with 9 tests against production:
  - health, metrics, dashboard/stats, recordings, config, auth, guilds, negative (404/400)

## Monitoring metrics
- moderationMetrics.ts in backend health module:
  - LLM call count/duration/tokens
  - Cache hit/miss per layer
  - Media analysis count/download duration
  - Batch size distribution, errors, SearXNG, auto-delete
2026-06-22 20:38:34 +07:00
MythEclipse ad84d0a1bb fix: messages with text+media also go into text batch immediately
- Previously: text+media message went ONLY to media array → text waited for vision
- Now: text part goes to text batch (immediate LLM analysis), media parallel
- DB update is idempotent — second write to same message_id overwrites
- User sees text moderation results instantly, media follows when ready
2026-06-22 17:45:32 +07:00
MythEclipse e8286247d6 fix: parallelize text + media LLM analysis instead of sequential
- text-only and media analysis now run concurrently via Promise.all
- text no longer blocks on media download + vision analysis
- each path independently saves to DB when its own results are ready
- same batch still uses single context fetch + attachment lookup
2026-06-22 17:42:46 +07:00
MythEclipse dfabdc85cd feat: search ALL messages via SearXNG + Redis cache dedup
- Remove shouldSearchContent() trigger gate — search runs on all messages
- extractSearchQueries() now extracts from ANY message, not just trigger-matched
- Redis cache (24h TTL) prevents redundant searches for same query
- initSearxngCache() lazy-connects via config.REDIS_URL
- Cache miss→API, hit→skip — fire-and-forget writes
- Both text batch + media path simplified
2026-06-22 12:41:48 +07:00
MythEclipse 84f87104e7 refactor: remove hardcoded lists from prompt, use SearXNG as single source of truth
- Remove hardcoded hentai title lists (Boku no Pico, Euphoria, etc.) from prompt
- Remove hardcoded SARA examples from prompt (Kitabonia, etc.)
- Prompt now tells LLM to use <web_searches> as evidence instead of hardcoded knowledge
- Evidence priority: searxng > web_content > media_analysis > internal model
- Code-side triggers in searxngSearch.ts still filter which messages to search
- LLM makes final decision based on search results, not static lists
2026-06-22 12:01:55 +07:00
MythEclipse 18ee8a35d2 feat: SearXNG search enrichment for moderation
- extractSearchQueries(): extract specific terms from trigger patterns
  instead of searching entire message (e.g. 'bokuno pico' from 'nonton bokuno pico anime')
- Add hentai studio/publisher triggers (Pink Pineapple, Maryjane, etc.)
- Fix media path SearXNG: use extractSearchQueries instead of quoted-only
- Prompt: add <web_searches> usage instructions so LLM actually uses search data
- Priority: searxng evidence > web_content > internal model knowledge
2026-06-22 10:58:52 +07:00
MythEclipse 641fb47d10 fix: hentai reference detection + severity high auto-delete
- Add explicit hentai title detection rule (Boku no Pico, Euphoria, Bible Black,
  Kuroinu, loli/shota) -> sexual_deviation severity high, recommended_action delete
- LLM flagged correctly but severity 'medium' caused auto-delete skip in fallback path
2026-06-22 10:28:16 +07:00
MythEclipse f502918b27 feat: multimodal video detection + reply/forward/crosspost + batch optimization
- Video frame extraction via ffmpeg (4 key frames per video → vision LLM)
- Video display in FE MessageCard with HTML5 <video> player
- Reply/forward/crosspost indicator in FE + pipeline in DG/BE
- Fix: missing sanitizeAiContent + escapeXml in media path (prompt injection)
- Optimize: text-only batch results saved to DB immediately, no longer wait for media analysis
- BE mapper/schema/repo: add reference fields (is_reply, is_forward, etc.)
2026-06-22 09:46:06 +07:00
MythEclipse dc62b283c0 refactor(ai-moderation): fix all 8 issues per review
- #2: Duplikasi contoh → single-source ALL_EXAMPLES array + formatExamples()
- #3: Hierarki evasion → Level 1 (Wajib Flag) vs Level 2 (Pilih Clean)
- #4: Dead code → remove duplicate PromptMode, cleanup
- #5: Sanitasi AI content → sanitizeAiContent() + CDATA wrapping for
      channel culture & user profile (prevents prompt injection)
- #6: Contoh personality → match actual XML format (<message>,
      <user_profile>, <content> tags instead of plain text)
- #7: Contoh invite link → example #29 added
- #8: 'Deskripsi selesai.' removed from stickerPrompt.ts
- Image analysis balance: teks = gambar (equal weight), no more bias
- Removed LGBT exclusion from SYSTEM_RULES
- Removed all template fallback language, enforce specific descriptions
2026-06-21 22:14:59 +07:00
MythEclipse 04f2862abc feat(ai-moderation): profile learner ai_status filter + channel-aware + simple fallback memory
- userProfileLearner.ts: filter query to only clean messages (eq ai_status='clean')
  to prevent profile contamination from flagged content. Also select channel_id
  to group messages by channel in prompt, enabling channel-aware personality
  summarization (user may behave differently across channels).
- llmModerationClient.ts (runSimpleTextFallback): inject user profile into
  both the classify prompt and the reason prompt, so even the last-resort
  fallback path has personality/memory context instead of being blind.
2026-06-21 20:07:51 +07:00
MythEclipse e5fa2b3f30 feat(ai-moderation): personality-rich user/channel profiling & memory-aware analysis
- moderationPrompt.ts: Add PERSONALITY & MEMORY section instructing LLM to
  use user_profile and channel_culture in analysis, with 3 new few-shot
  examples (normal profile use, deviation detection, don't overfit)
- userProfileLearner.ts: Richer personality summary prompt — asks for
  communication style, passion topics, unique habits/signature traits,
  social dynamics, and writes in warm observer prose
- cultureLearner.ts: Richer channel culture summary — asks for 'vibe',
  inside jokes, social norms, dynamics. Writes in anthropologist style
2026-06-21 19:11:11 +07:00
MythEclipse 21b818a344 feat: add forwarding & reply detection with frontend indicators
- Enhanced messageMetadata to capture referenced message content
- Added reply indicator (↩️ Replying to @username + content snippet)
- Added forward badge (🔁 Forwarded) and crosspost badge (📢 Crossposted)
- Frontend auto-fetches referenced message content when metadata is empty
- Added getMessageById API helper
2026-06-21 18:22:27 +07:00
MythEclipse 2b084b326f a 2026-06-14 05:10:58 +07:00
MythEclipseandClaude 6733ea8f01 fix(db): add 0011_add_voice_transcription to drizzle journal
Migration SQL file existed but was never registered in _journal.json,
so Drizzle skipped it. Every voice recording insert failed with:
  PG code 42703: column "transcription" does not exist

Co-Authored-By: Claude <noreply@anthropic.com>
2026-06-14 04:59:44 +07:00
MythEclipseandClaude a2beda77c4 fix(db): surface PG error code/detail/cause in voice recording insert failures
Drizzle wraps PG errors — the previous log only showed the formatted query
text. Now extracts code, detail, schema, table, constraint, severity from
the error envelope and from .cause to make the actual Postgres error visible.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-06-14 04:40:22 +07:00
MythEclipseandClaude dbc2879e16 fix(voice): resolve Redis subscriber corruption, transmitter races, and DB error logging
Redis root cause: VoiceHandler passed redisPub to transmitter.start(),
which called .subscribe() on it — permanently converting the publish
connection to subscriber mode. Every subsequent command reply and status
update failed.

Fixes:
1. Transmitter now creates its own Redis client via new IORedis()
2. Mutual exclusion gate serialises start/stop to prevent null-deref races
3. PassThrough drain listeners cleaned up to stop MaxListenersExceeded
4. FFmpeg SIGTERM flagged as expected exit (no more false level-50 errors)
5. Voice recording repo captures PG error code/detail for diagnostics

Co-Authored-By: Claude <noreply@anthropic.com>
2026-06-14 04:23:04 +07:00
MythEclipse 32b0b20ca7 chore(services): update components based on recent changes
Changes:
 services/discord-gateway/src/shared/database/voiceRecordingRepo.ts | 18 ++++++++++++++----
 1 file changed, 14 insertions(+), 4 deletions(-)
2026-06-14 03:31:40 +07:00
MythEclipse 93634d129a chore(services): update components based on recent changes
Changes:
 services/discord-gateway/src/modules/command-handler/voice.handler.ts | 10 ++++----
 services/discord-gateway/src/modules/voice-recording/transmitter.ts   | 91 ++++++++++++++++++++++++++++++++++++++++++++++++++++--------------------
 2 files changed, 72 insertions(+), 29 deletions(-)
2026-06-14 03:28:24 +07:00
MythEclipseandClaude c317cfb420 fix(ci): resolve TS build errors across all services
- recordings.routes.ts: cast req.params.id to string (express 5 typed params)
- recordings.service.ts: cast Record<string,unknown>[] via unknown first
- guilds.routes.ts: cast req.params.guildId to string (2 sites)
- transmitter.ts: capture pcmStream ref locally + re-acquire in drain callback

Co-Authored-By: Claude <noreply@anthropic.com>
2026-06-14 01:17:18 +07:00
MythEclipse 92d73fe12d chore(services): update components based on recent changes
Changes:
 .env.example                                                     | 5 +++++
 services/discord-gateway/src/modules/voice-recording/recorder.ts | 2 +-
 2 files changed, 6 insertions(+), 1 deletion(-)
2026-06-14 00:57:08 +07:00
MythEclipse 7789aca620 chore(services): update components based on recent changes
Changes:
 services/discord-gateway/src/app/shutdown.ts | 8 ++++++++
 1 file changed, 8 insertions(+)
2026-06-14 00:45:59 +07:00
MythEclipse ac661b64f4 chore(services): update components based on recent changes
Changes:
 services/discord-gateway/src/app/bootstrap.ts                                    | 24 ++++++++++++++++++++++++
 services/discord-gateway/src/modules/voice-recording/recorder.ts                 | 10 ++++++++++
 services/discord-gateway/src/modules/voice-recording/recorder/speakingHandler.ts |  9 ++++++++-
 3 files changed, 42 insertions(+), 1 deletion(-)
2026-06-14 00:45:31 +07:00
MythEclipse c886dfb184 chore(services): update components based on recent changes
Changes:
 services/discord-gateway/src/modules/voice-pcm-ws/index.ts            |   1 +
 services/discord-gateway/src/modules/voice-pcm-ws/voicePcmWsClient.ts | 151 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 152 insertions(+)
2026-06-14 00:43:50 +07:00
MythEclipse 28baeda5ad chore(services): update components based on recent changes
Changes:
 packages/shared/src/config/index.ts                                          |   7 +++
 packages/shared/src/moderation-types.ts                                      |   1 +
 services/backend/src/modules/recordings/recordings.service.ts                |   3 +-
 services/discord-gateway/drizzle/migrations/0011_add_voice_transcription.sql |   1 +
 services/discord-gateway/src/modules/voice-recording/recorder/segment.ts     |  10 ++++
 services/discord-gateway/src/modules/voice-recording/recorder/uploader.ts    |  15 ++++++
 services/discord-gateway/src/modules/voice-recording/voiceTranscriber.ts     |  51 ++++++++++++++++++
 services/discord-gateway/src/shared/database/schema.ts                       |   1 +
 services/discord-gateway/src/shared/database/voiceRecordingRepo.ts           |  17 ++++++
 services/frontend/src/features/live/components/RecordingsSubPanel.tsx        | 108 +++++++++++++++++++-------------------
2026-06-13 17:08:54 +07:00
MythEclipse d0d2c316cf chore(auto): task completed - unknown 2026-06-13 15:11:06 +07:00
MythEclipse 265251233a chore(auto): task completed - unknown 2026-06-13 15:03:21 +07:00
MythEclipse 9e9ce463cb chore(auto): task completed - unknown 2026-06-13 14:59:00 +07:00
MythEclipseandClaude 3965ea79bc fix(gateway): voice speaking handler, metadata cache, transmitter backpressure
- speakingHandler: check subscriptions.has() BEFORE voiceActiveUser broadcast
- segment.ts: add LRU metadata cache (200 entries, avoids Discord API in hotpath)
- transmitter.ts: add backpressure handling with drain queue/flush
  (PassThrough.write() return value was ignored — potential OOM under load)
- cleanup backpressure queue on transmitter stop

Co-Authored-By: Claude <noreply@anthropic.com>
2026-06-13 14:38:06 +07:00
MythEclipseandClaude 29d4cb87b6 fix(gateway): voice module critical bugs
- shutdown ordering: DB pool closed AFTER voice disconnect (was before)
- muxer: amix (mix) -> concat (sequential) filter for OGG segments
- upload retries: 0->3 with proper backoff 1-10s
- teleUpload retry backoff: minTimeout 0->1000, maxTimeout 0->10000
- mediaSource: 64KB buffer bound on readFirstTwoLines, 1MB on extractMediaInfo
- ffmpegProcess: pipe stderr to structured logger instead of inherit

Co-Authored-By: Claude <noreply@anthropic.com>
2026-06-13 14:35:35 +07:00
MythEclipseandClaude 14c0081f01 feat(discord-gateway): implement voice & push improvements
- Voice disconnect broadcast on stopRecording
- Multi-guild voice support (VoiceController Map<guildId>)
- Session finalization + auto-enqueue muxer job
- Recordings API: duration field, channelId/userId filters
- Transmitter Redis connection reuse (shared conn)
- FFmpeg stderr memory cap (4KB limit)
- 10 new Redis event channels + Redis bridge subscriptions
- New DB tables: message_reactions, message_edits
- Webhook notification module
- Gateway metrics / Prometheus endpoint
- Multi-guild message capture (MONITOR_GUILD_IDS array)
- Thread tracking, presence, channel topic, guild member events
- Edit history snapshot on message update
- Muxer audio post-processing worker

Co-Authored-By: Claude <noreply@anthropic.com>
2026-06-13 13:51:21 +07:00
MythEclipse 111ba0a139 chore(auto): task completed - unknown 2026-06-13 13:36:13 +07:00
MythEclipse 942c33db7e chore(auto): task completed - unknown 2026-06-13 13:35:56 +07:00
MythEclipse eab8505889 chore(auto): task completed - unknown 2026-06-13 13:35:41 +07:00
MythEclipse 2e14b5e4ed chore(auto): task completed - unknown 2026-06-13 13:35:11 +07:00
MythEclipse d569ffa2db chore(auto): task completed - unknown 2026-06-13 13:12:49 +07:00
MythEclipse 4097b23dc3 chore(auto): task completed - unknown 2026-06-13 13:12:41 +07:00
MythEclipse e413cf9fca chore(auto): task completed - unknown 2026-06-13 13:12:29 +07:00
MythEclipse ff006980f8 chore(auto): task completed - unknown 2026-06-13 13:12:01 +07:00