Commit Graph
43 Commits
Author SHA1 Message Date
asepharyana 7efaf00c93 Revert "feat: migrate frontend to Astro + expand AI moderation + backend admin/runtime config"
This reverts commit d59b59a7a7.
2026-07-02 03:54:44 +07:00
asepharyana d59b59a7a7 feat: migrate frontend to Astro + expand AI moderation + backend admin/runtime config
Frontend:
- migrate from Vite to Astro (astro.config.mjs, pages/, layouts/)
- add admin panel, settings page, command palette, error boundary
- refactor App.tsx, MascotChatbot, Sidebar, Header, DashboardLayout
- update API client, WebSocket, auth, dashboard features

Backend:
- add admin module and config routes
- refactor middlewares, Redis connection, WebSocket server/bridge
- add runtime config loader

Discord Gateway:
- refactor AI moderation: circuit breaker, concurrency limiter, fallback processor
- add media analysis client, Seaxng search, user profile learner
- add new drizzle migration

Shared:
- extend database schema, add new config fields
2026-07-02 00:02:41 +07:00
MythEclipseandClaude a2beda77c4 fix(db): surface PG error code/detail/cause in voice recording insert failures
Drizzle wraps PG errors — the previous log only showed the formatted query
text. Now extracts code, detail, schema, table, constraint, severity from
the error envelope and from .cause to make the actual Postgres error visible.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-06-14 04:40:22 +07:00
MythEclipseandClaude dbc2879e16 fix(voice): resolve Redis subscriber corruption, transmitter races, and DB error logging
Redis root cause: VoiceHandler passed redisPub to transmitter.start(),
which called .subscribe() on it — permanently converting the publish
connection to subscriber mode. Every subsequent command reply and status
update failed.

Fixes:
1. Transmitter now creates its own Redis client via new IORedis()
2. Mutual exclusion gate serialises start/stop to prevent null-deref races
3. PassThrough drain listeners cleaned up to stop MaxListenersExceeded
4. FFmpeg SIGTERM flagged as expected exit (no more false level-50 errors)
5. Voice recording repo captures PG error code/detail for diagnostics

Co-Authored-By: Claude <noreply@anthropic.com>
2026-06-14 04:23:04 +07:00
MythEclipse 32b0b20ca7 chore(services): update components based on recent changes
Changes:
 services/discord-gateway/src/shared/database/voiceRecordingRepo.ts | 18 ++++++++++++++----
 1 file changed, 14 insertions(+), 4 deletions(-)
2026-06-14 03:31:40 +07:00
MythEclipse 28baeda5ad chore(services): update components based on recent changes
Changes:
 packages/shared/src/config/index.ts                                          |   7 +++
 packages/shared/src/moderation-types.ts                                      |   1 +
 services/backend/src/modules/recordings/recordings.service.ts                |   3 +-
 services/discord-gateway/drizzle/migrations/0011_add_voice_transcription.sql |   1 +
 services/discord-gateway/src/modules/voice-recording/recorder/segment.ts     |  10 ++++
 services/discord-gateway/src/modules/voice-recording/recorder/uploader.ts    |  15 ++++++
 services/discord-gateway/src/modules/voice-recording/voiceTranscriber.ts     |  51 ++++++++++++++++++
 services/discord-gateway/src/shared/database/schema.ts                       |   1 +
 services/discord-gateway/src/shared/database/voiceRecordingRepo.ts           |  17 ++++++
 services/frontend/src/features/live/components/RecordingsSubPanel.tsx        | 108 +++++++++++++++++++-------------------
2026-06-13 17:08:54 +07:00
MythEclipse 2e14b5e4ed chore(auto): task completed - unknown 2026-06-13 13:35:11 +07:00
MythEclipseandClaude 30f8d7cce3 feat: replace corrections/tuner with dashboard module
Replace the corrections/adaptive-prompt-tuner feature with a new
dashboard module providing server stats and user profile overview.

Backend:
- Add dashboard module (routes, service, repository) with stats + user list + user detail endpoints
- Remove corrections module entirely
- Wire dashboard router in app.ts

Frontend:
- Add dashboard feature (DashboardStats, UserSummaryList, UserProfileDetail components + useDashboard hook)
- Remove tuner feature (CorrectionStats, CorrectionHistory, SubmitCorrection, useCorrections)
- Update API client from corrections → dashboard types/fns
- Rename tab 'tuner' → 'dashboard'
- Update MobileTabBar, Header, Sidebar links

Tests:
- Expand backend placeholder test with dashboard assertions
- Expand discord-gateway placeholder test with config/channel assertions

AI moderation:
- llmModerationClient: improve status/reply detection, expand safety categories, fix timer reset
- userProfileLearner: fix isReply refinement
- userProfileStore: add pending cache check
- messageMetadata: add crosspost type mapping
- migrate.ts: improve partial-index safety in schema push

Co-Authored-By: Claude <noreply@anthropic.com>
2026-06-13 11:24:42 +07:00
MythEclipseandClaude fbc2184c6e feat(ai-moderation): add user profile self-learning system
Add user_profiles table, store, and background learner worker
that summarizes user communication style, topics, and personality.

- New user_profiles table (user_id PK, guild_id, profile_summary, last_analyzed_at)
- userProfileStore.ts — CRUD (get/update) following channelCultureStore pattern
- userProfileLearner.ts — background worker: queries 100 recent msgs per user,
  calls LLM for personality summary, updates every 12h
- Inject <user_profile> XML tag per-message in moderation prompt
- Start worker alongside cultureLearner in aiAnalyzer.ts
- Migration 0008 for user_profiles table

Co-Authored-By: Claude <noreply@anthropic.com>
2026-06-12 20:11:34 +07:00
MythEclipseandClaude Fable 5 906aef1807 fix(database): handle PG15+ public schema CREATE privilege error in seedDrizzleHistory
In PG15+, the CREATE privilege on the public schema is revoked from
non-owner roles by default. The seedDrizzleHistory function's
CREATE TABLE IF NOT EXISTS for __drizzle_migrations fails with 42501,
causing the gateway to crash-loop on startup.

Wrap the CREATE in a try/catch for 42501 — if the table already exists
(created by a prior run), we continue gracefully; otherwise re-throw.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 04:06:11 +07:00
MythEclipse 2e5371f3ee fix(discord-gateway): bypass Drizzle PG15 schema creation restrictions 2026-06-11 03:53:48 +07:00
MythEclipseandClaude Fable 5 378f1b6bdc fix(database): use public schema for drizzle migrations to avoid permission error
PostgreSQL error 42501 when Drizzle tries CREATE SCHEMA IF NOT EXISTS
"drizzle" — the DB user lacks schema creation privileges. Configuring
migrationsSchema: "public" keeps __drizzle_migrations in the existing
public schema, matching what seedDrizzleHistory already expects.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 03:34:16 +07:00
MythEclipseandClaude Opus 4.8 07032ab521 refactor: atomic, DRY, and logging improvements across codebase
- Split llmModerationClient.ts (2170 lines) into 5 focused sub-modules
- Split aiAnalyzer.ts (1282 lines) into 4 modular pipelines
- Split messages.db.ts (826 lines) into 5 domain-specific modules
- Moved shared schema to @bete/shared, eliminated backend duplication
- Added createChildLogger to all voice-recording and AI moderation modules
- Extracted tryCommandThenFallback, normalizeMediaState, DEFAULT_VOICE_STATUS
- Created shared pagination.ts utility, eliminated 5+ cursor-pagination duplications
- Created shared messageMapper.ts for row mapping
- Standardized backend error handling with asyncHandler
- Added frontend createLogger utility and useAsyncAction hook
- Added structured logging to frontend hooks, socket, and API client

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-09 19:46:08 +07:00
MythEclipseandClaude Opus 4.8 b68789fffc refactor: atomic, DRY, and logging improvements
- Shared Redis channel constants as single source of truth (redis-channels.ts)
- commandHandler.ts split into VoiceHandler, MediaHandler, GuildHandler,
  ModerationHandler with handler-registry.ts dispatch
- messageStore.ts (1322 lines) split into domain-specific DB files:
  messages.db.ts, attachments.db.ts, reviews.db.ts,
  moderation-actions.db.ts, retention.db.ts
- recorder.ts startSpeaking callback extracted into speakingHandler.ts,
  streamSetup.ts, segmentFinalizer.ts
- autoDeleteManager.ts split into autoDeleteEligibility.ts,
  autoDeleteNotify.ts, autoDeleteLogger.ts
- Added createChildLogger() logging across 8 service files
- Backend messages.repository.ts migrated from raw SQL to Drizzle ORM
- Fixed biome.json to exclude packages/**/dist/* from lint
- Fixed config.ts GUILD_ID pre-existing type error

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-09 17:34:18 +07:00
MythEclipseandClaude Opus 4.8 3a7b005d95 fix: resolve architecture disconnects and codebase weaknesses
- Add TEXT_CHANNEL_ID and TEXT_GUILD_ID to config schema (fix silent channel monitoring)
- Remove dead files: message-capture/broadcaster.ts, voice-recording/index.ts
- Fix WebSocket voice_command payload to forward from frontend
- Implement moderation:action handler in commandHandler
- Fix useMascotChat to use canonical request() wrapper
- Fix useAudioPlayback userId hash collision (use string not parseInt)
- Add catch blocks to useMediaControl.skip/stop
- Add typed broadcast functions (messageAnalyzed, voicePcmData, voiceActiveUser)
- Apply Biome formatting and lint fixes

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-09 16:36:23 +07:00
MythEclipseandClaude Opus 4.8 3614d32701 fix: resolve architecture disconnects and codebase weaknesses
- Standardize MessageRecord types — single source of truth from @bete/shared
- Clean up config: remove unused GUILD_ID/TEXT_GUILD_ID/TEXT_CHANNEL_ID, fix WEBSERVER_PORT default (3001), remove default admin password
- Move mascot_chat_messages table to Drizzle schema with proper migration
- Remove runtime DDL (CREATE TABLE IF NOT EXISTS) from mascot-chat repository
- Remove phantom analytics/ module from documentation
- Add better-sqlite3 dependency to root devDependencies
- Replace 'as any' casts with proper type assertions across AI moderation
- Add error logging to silent catch blocks in LLM client
- Apply Biome formatting and import organization

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-09 13:07:01 +07:00
MythEclipse 67d66bb5dd refactor(shared): centralize configuration and types
Migrate configuration validation and core moderation types from individual services to the `@bete/shared` package to ensure consistency across the monorepo.

- Move `AppConfig` and moderation-related interfaces to `packages/shared`.
- Replace service-specific Zod schemas with the centralized shared configuration.
- Refactor `services/backend` and `services/discord-gateway` to consume shared config and types.
- Remove redundant type definitions and local configuration logic in services.
- Update `packages/shared` exports to include new `config` and `moderation-types` modules.
- Clean up unused files and deprecated utility functions in `packages/shared`.
2026-06-09 11:56:03 +07:00
MythEclipseandClaude Opus 4.8 4becf0d6f1 refactor: comprehensive codebase cleanup and architecture hardening
- Sprint 1 (Quick Wins): Remove dead analytics modules, fix 4 unresolved
  imports, replace 3 console.warn with logger, remove mock-crc import
- Sprint 2 (Architecture): Create MascotChatRepository, AnalysisRepository,
  3 Zod schemas (mascot-chat, analysis, voice), deduplicate error classes,
  move 3 SQL queries from routes to repository
- Sprint 3 (Complexity): Replace 7 any types with proper interfaces,
  extract 6 helpers from prepareMediaMessage (CC 85 -> ~15)
- Sprint 4 (Config): Remove 22 dead env vars from .env, add 30 missing
  vars to .env.example, standardize naming

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-09 10:16:04 +07:00
MythEclipseandClaude Opus 4.8 13a75a5101 fix(voice): restore voice features and apply critical optimizations
Voice Feature Restoration:
- Implemented full Redis pub/sub pipeline for real-time voice data
- Added VOICE_PCM and VOICE_ACTIVE_USER Redis channels
- Implemented EventBroadcaster.voicePcmData() and voiceActiveUser() methods
- Extended backend redis-bridge to subscribe to voice channels
- Updated backend WebSocket server for binary PCM broadcast
- Replaced globalThis PcmBroadcaster pattern with proper EventBroadcaster DI
- Fixed root cause: PcmBroadcaster functions were never initialized

Bug Fixes:
- Fixed prism-media version conflict (2.0.0-alpha.0 → 1.3.5)
- Fixed type inconsistency in commandHandler.ts (AudioPlayerStatus → string)

Critical Optimizations:
- P1.1: Fixed unbounded memory growth in aiAnalyzer (added LRU caching, max 10K entries)
- P1.2: Converted sync file I/O to async in audio hot paths (recorder, sessionRecording)
- P1.3: Replaced process.exit(1) with proper error handling (bootstrap, aiAnalysisWorker)

Code Quality:
- Removed unused logger field in EventBroadcaster
- Replaced console.* with structured logger.* calls (player, decoder)
- Fixed typos and removed commented debug code
- Added DatabaseError class for better error handling

Files Modified: 18 (discord-gateway: 14, backend: 3, root: 1)
Architecture: Discord → EventBroadcaster → Redis → Backend WebSocket → Frontend

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-08 19:14:34 +07:00
MythEclipse 2643a3c278 refactor(discord-gateway): optimize redis usage and audio stream handling
- refactor(command-handler): replace per-call Redis connection creation with a persistent publisher connection to reduce overhead
- refactor(recorder): switch from manual audio stream subscription to direct event listeners on the existing stream
- feat(recorder): implement exponential backoff for voice connection retries
- chore(config): update default DECODER_COOLDOWN_MS to 30000ms
2026-06-08 18:09:17 +07:00
MythEclipse 71a8a9e6e1 fix(ai-moderation): patch structural bypass vulnerabilities in NLP pipeline
- Implement Pre-computation Normalization for Polyglot Obfuscation.

- Inject Ontological Graph for literal translation evasion (e.g., 'kostum hewan').

- Enforce Entropy-Triggered Routing to deny softmax fallback exploitation.

- Format discord-gateway codebase.
2026-06-06 15:38:36 +07:00
MythEclipse 6a3d0eb17c feat(discord-gateway): replace base64 sticker cache column with image URL storage 2026-06-06 14:10:55 +07:00
MythEclipse 25c86dcc30 fix(ai-moderation): raise retry backoffs and cooldowns, make 429 retryable, reduce prompt false positives 2026-06-05 21:47:02 +07:00
MythEclipse 2f3d7e1d61 feat(ai-moderation): introduce user reputation and channel culture context
Implements a context-aware moderation system by tracking user behavior
and channel-specific norms to improve AI decision-making accuracy.

- Adds `user_reputations` table to track trust scores, clean streaks,
  and infraction history.
- Adds `channel_cultures` table to store AI-generated summaries of
  channel-specific norms and slang.
- Implements `userReputationStore` to autonomously update user scores
  based on moderation outcomes (clean vs. flagged).
- Implements `cultureLearner` and `channelCultureStore` to manage
  evolving channel contexts.
- Enhances LLM prompts to inject user reputation (trust scores,
  history) and channel culture summaries, enabling "wisdom-based"
  moderation (e.g., giving benefit of the doubt to high-trust users).
- Integrates reputation and culture updates into the existing
  `aiAnalyzer` pipeline.
2026-06-05 18:04:57 +07:00
MythEclipse 09f6e80ddd fix(ai-moderation): resolve race conditions and implement processing state
Introduces a `processing` state to the AI analysis lifecycle to prevent
duplicate processing of the same messages.

- Implements row-level locking using `FOR UPDATE SKIP LOCKED` in
  `messageStore.ts` to ensure atomic message acquisition.
- Adds a `processing` status to the `AIStatus` type and database schema.
- Fixes a TOCTOU race condition in `aiAnalyzer.ts` by synchronizing
  the conversation processing lock before async database operations.
- Implements `revertStuckProcessingMessages` to recover messages stuck
  in the `processing` state due to worker crashes or timeouts.
- Updates `processBatch` and scheduling logic to correctly manage and
  release conversation-level locks.
2026-06-05 16:36:14 +07:00
MythEclipse d1d4510ce8 chore(format): auto-format files with biome 2026-06-04 18:52:56 +07:00
MythEclipse ce8a42f5fd feat(ai-moderation): eliminate LLM-based badword detection in favor of pure rule-based matching 2026-06-04 17:33:08 +07:00
MythEclipse 5d9713b162 chore(config): update TELE_UPLOAD_URL domain across all services 2026-06-04 17:12:02 +07:00
MythEclipse d81aaf2b4a feat(db): add migration to drop unused AI columns and dynamically resolve initial migration tag 2026-06-02 23:55:12 +07:00
MythEclipse f10fa9aa88 feat(discord-gateway): export correctedModerationsTable from database schema 2026-06-02 23:05:37 +07:00
MythEclipse eb16c1f504 ``` 2026-06-02 23:05:33 +07:00
MythEclipse 0cd4f197e9 No staged changes found — the staging area is empty. Stage some files first with git add, then I can help craft the commit message. 2026-06-02 23:05:06 +07:00
MythEclipse f163ead3cf feat(discord-gateway): add local badword pre-filter, accurate token estimation, Piscina maxThreads config, and PromptMode support 2026-06-02 22:50:23 +07:00
MythEclipse 1e1fbb4583 fix(discord-gateway): require LLM API key only when AI analysis is enabled 2026-06-02 21:46:15 +07:00
MythEclipse 19028cf244 refactor: migrate discord-gateway from winston to @bete/shared/logger and utils
- Replace all local logger imports (../../shared/logger/logger.js) with @bete/shared/logger across 26 files
- Remove winston dependency, add pino to discord-gateway package.json
- Delete shared/logger/logger.ts (winston-based, 132 lines) and serialization.ts (109 lines)
- Replace local retryWithBackoff imports with @bete/shared/utils across 6 files
- Delete shared/utils/retry.ts (42 lines)
- Add CustomLogger type alias to @bete/shared/logger for backwards compatibility
- Remove logger param from all retryWithBackoff calls and uploadToTele interfaces
- Frontend: convert entity type files to re-exports from shared/api/client.ts
- Full monorepo typecheck clean (4/4 packages)

35 files changed, 42 insertions(+), 488 deletions(-)
2026-06-02 21:23:12 +07:00
MythEclipse b23ccf0837 chore: session auto-commit 2026-06-02 17:12:42 +07:00
MythEclipse 498ab73d62 chore: auto-commit task - 2026-06-02 17:08:42 +07:00
MythEclipse 406a6cbf79 refactor: migrate sticker cache to PostgreSQL, remove versioning
- Replace file-based sticker cache (.dat + index.json) with PostgreSQL sticker_cache table
- Remove model_version column and all versioning logic (git branch detection, GitHub API, CACHE_MODEL_VERSION)
- Strip VISION_MODEL_VERSION, logCacheEvent calls, and version filtering from SQL queries
- Remove STICKER_CACHE_DIR and STICKER_CACHE_MAX_SIZE_MB config variables
- Delete dead migration add_model_version_to_cache.sql

textCacheStore.ts reduced 482→204 lines (-57%)
stickerCache.ts reduced 210→144 lines (-31%)
Total: 11 files changed, 233 insertions, 591 deletions
2026-06-02 14:05:51 +07:00
MythEclipseandClaude Opus 4.8 e584a7941e fix(text-cache): apply model version migration on existing dbs
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-02 13:05:05 +07:00
MythEclipse 385fc0bbbf feat: add response logger for moderation/vision audit trail + cache model versioning
- Add responseLogger.ts with comprehensive logging: vision analysis, moderation
  analysis batches, cache hit/miss events, errors, retries, false positives,
  model version changes, cache invalidation
- Integrate responseLogger calls into llmModerationClient.ts (cache events,
  moderation results, errors, retries)
- Add model_version column to text_analysis_cache schema with composite
  index for efficient filtering
- Add migration SQL for model_version column
- Document CACHE_MODEL_VERSION in .env.example
2026-06-02 12:23:52 +07:00
MythEclipseandClaude Opus 4.8 64643bb066 perf: hapus semua cooldown/backoff delay — retry tanpa bottleneck
Semua retry sekarang instant (retries=0, minTimeout=0, maxTimeout=0):
- retry.ts (kedua) — default minTimeout/maxTimeout 0, factor 1
- config.ts (kedua) — DECODER_COOLDOWN_MS=0, AI_ANALYSIS_ERROR_COOLDOWN_MS=0
- llmModerationClient.ts (kedua) — retries 0 tanpa delay
- aiAnalyzer.ts (kedua) — retries 0, AI_ANALYSIS_ERROR_COOLDOWN jadi 0
- indonesianTextNormalizer.ts (kedua) — semua rate limit cooldown 0
- recorder.ts (kedua), teleUpload (3 files), uploader (2 files) — retry instant
- attachmentUploader (kedua) — retries 0
- syncRoutes.ts — BACKLOG_SYNC_COOLDOWN 0

20 files changed. Semua cooldown bottleneck dihapus.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-02 11:29:52 +07:00
MythEclipseandClaude Opus 4.6 58afa98132 fix(discord-gateway): skip migration if schema tables already exist
Instead of failing on CREATE TABLE when tables already exist, check
information_schema for all 10 schema tables. If all exist, skip
migration gracefully. This handles deployments where DB was created
by a previous deployment with different migration files.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-06-01 23:09:04 +07:00
MythEclipseandClaude Opus 4.8 c48a0c5e3b refactor: split monolith into 3 microservices (frontend, backend, discord-gateway)
- Extract services into services/{frontend,backend,discord-gateway}
- Create packages/shared/ for shared logger, errors, utils, types
- Setup Modular MVC pattern in backend (controller→service→repository)
- Setup event-driven architecture in discord-gateway with Redis pub/sub
- Move Docker files to infra/docker/ with per-service Dockerfiles
- Update docker-compose.yml to use Traefik-only routing (no port exposes)
- Update GitHub Actions deploy workflow for multi-service matrix build
- Fix all import paths and resolve type errors across all services
- All 3 services pass tsc --noEmit clean

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-01 21:44:29 +07:00