From c57ee12da1bdf2a30c3a2d8199b5a4d5e5a76625 Mon Sep 17 00:00:00 2001 From: asepharyana Date: Thu, 24 Sep 2026 15:09:13 +0700 Subject: [PATCH] docs(gateway): consolidate README/ARCHITECTURE, drop stale MODULE_STRUCTURE MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit README.md was the extraction-era document (referenced winston, mock-crc.ts, llmModerationClient.ts, indonesianTextNormalizer.ts — all long gone) and duplicated ARCHITECTURE.md. Rewritten as a short run-the-service guide; layout/design lives only in ARCHITECTURE.md. MODULE_STRUCTURE.md deleted: it was a stale duplicate of ARCHITECTURE.md, referenced by nothing but itself. ARCHITECTURE.md updated to the post-refactor reality: app/ lifecycle split (bootstrap/lifecycle/process-guards/metrics-collector), ai-moderation recovery-worker + cache-prune, per-module index.ts facades, one-way dependency rule, corrected init/shutdown/observability sections. --- services/discord-gateway/ARCHITECTURE.md | 93 +++-- services/discord-gateway/MODULE_STRUCTURE.md | 73 ---- services/discord-gateway/README.md | 356 +++---------------- 3 files changed, 115 insertions(+), 407 deletions(-) delete mode 100644 services/discord-gateway/MODULE_STRUCTURE.md diff --git a/services/discord-gateway/ARCHITECTURE.md b/services/discord-gateway/ARCHITECTURE.md index 7cb6571d..5cc1552c 100644 --- a/services/discord-gateway/ARCHITECTURE.md +++ b/services/discord-gateway/ARCHITECTURE.md @@ -5,10 +5,9 @@ messages/attachments/reactions/threads/presence, runs LLM-based AI moderation, and publishes everything to Redis pub/sub for the backend to consume. The backend serves the HTTP/WS API to the frontend. -> NOTE: this doc is the source of truth for the module layout. The older -> `MODULE_STRUCTURE.md` was stale (referenced `winston`, `mock-crc.ts`, -> `indonesianTextNormalizer.ts`, and `aiAnalysisWorker.ts`/`llmModerationClient.ts` -> which were renamed/merged). If they disagree, this file wins. +> NOTE: this doc is the source of truth for the module layout. The old +> `MODULE_STRUCTURE.md` was a stale duplicate and has been removed. `README.md` +> only covers how to run the service. ## Top-level layout @@ -16,33 +15,41 @@ consume. The backend serves the HTTP/WS API to the frontend. services/discord-gateway/ ├── src/ │ ├── index.ts # Entry point → initializeDiscordGateway() -│ ├── app/ -│ │ ├── bootstrap.ts # Wires client, DB, Redis, workers, schedulers -│ │ ├── shutdown.ts # Graceful shutdown (SIGINT/SIGTERM + transient errors) +│ ├── app/ # Process lifecycle +│ │ ├── bootstrap.ts # Startup order: config → DB → services → metrics → login +│ │ ├── lifecycle.ts # Everything wired on the Discord 'ready' hook +│ │ ├── process-guards.ts # SIGINT/SIGTERM + uncaught-error policy +│ │ ├── metrics-collector.ts # AI pipeline Prometheus gauges +│ │ ├── shutdown.ts # Graceful shutdown sequence │ │ └── retention.ts # Expired-record cleanup scheduler -│ ├── shared/ +│ ├── shared/ # Infrastructure — never imports from modules/ │ │ ├── config/ # Zod-validated env (index.ts = schema+loader) │ │ ├── database/ # Drizzle ORM + pg Pool + migrations │ │ │ ├── init.ts drizzle.ts pool.ts migrate.ts migrateCli.ts │ │ │ └── schema/ # messages, cache, meta, analytics │ │ ├── logger/ # pino wrapper + createChildLogger() -│ │ ├── errors/ # AppError / ConfigError ... +│ │ ├── errors/ # AppError / ConfigError ... + errorMessage() +│ │ │ # + isTransientStreamError() │ │ ├── utils/ # retry, pagination │ │ ├── discord/clientOptions.ts # discord.js-selfbot-v13 client options │ │ ├── uploader.ts # Shared attachment upload helper -│ │ ├── redis-channels.ts # Redis channel-name constants +│ │ ├── redis-channels.ts # Redis channel + command constants │ │ └── moderation-types.ts # Shared AI analysis domain types -│ └── modules/ +│ └── modules/ # Feature modules, each with an index.ts facade │ ├── message-capture/ # Discord event listeners + DB store │ ├── ai-moderation/ # LLM moderation pipeline (see below) │ ├── attachment-upload/ # Download + (sharp) resize + upload -│ ├── event-broadcaster/ # RedisEventPublisher + EventBroadcaster +│ ├── event-broadcaster/ # RedisEventPublisher + EventBroadcaster │ ├── command-handler/ # Redis-subscribed backend→gateway commands │ ├── reaction-tracking/ thread-tracking/ user-presence/ -│ ├── channel-topic/ guild-member-events/ +│ ├── channel-topic/ guild-member-events/ monitor/ │ └── gateway-metrics/ # Prometheus /metrics endpoint (port 4016) ``` +Dependency direction is one-way: `index.ts` → `app/` → `modules/` → `shared/`. +Code outside a module imports its `index.ts` facade, never an internal file; +deep imports stay valid inside the module itself. + ## AI moderation pipeline (`ai-moderation/`) LLM-only judge — no regex/heuristic classification. One orchestrator call @@ -54,8 +61,15 @@ concurrency semaphores. The text lane frees its lock and saves+broadcasts the moment text analysis finishes — it never waits on a slow vision/media batch of the same conversation, and vice versa. -- `aiAnalyzer.ts` — public API: `queueMessageAnalysis`, `getAnalysisQueueStatus`, - `startPendingAIAnalysisWorker` (recovery worker + cache-prune). +- `aiAnalyzer.ts` — public API: `queueMessageAnalysis`, `queueConversationAnalysis`, + `getAnalysisQueueStatus`, `startPendingAIAnalysisWorker`. Short-circuits + age-restricted and skip-list messages before any LLM work. +- `recovery-worker.ts` — periodic sweep for stranded `pending` messages + (re-scheduled per lane) and `error`/`analysis_incomplete` messages + (individual fallback queue); prunes stale lane locks, per-conversation CB + counters and individual in-flight markers. +- `cache-prune.ts` — throttled (6h) expired-verdict sweep across Postgres and + Qdrant, driven from the recovery interval. - `batchScheduler.ts` — per-conversation per-LANE debounce → `processBatch` (lane-aware). `splitMessagesByLane` / `laneOfMessage` live in `analysisLanes.ts` (pure, unit-testable). @@ -123,28 +137,51 @@ See `src/shared/redis-channels.ts` for the canonical names. ## Initialization flow +`bootstrap.ts` runs these steps in order (each is a named function): + 1. Validate env (Zod). Refuse to start if `AI_ANALYSIS_ENABLED` but no key. -2. `AUTO_MIGRATE_ON_STARTUP` → run pending Drizzle migrations. -3. `initializeDatabase()` (pg Pool, min 0). -4. Create discord.js-selfbot-v13 client; register listeners on `ready`. -5. Start `gmw-discord-gateway` metrics server (port `METRICS_PORT`, default 4016). -6. `client.login(token)`. + → `assertConfigIsUsable()` +2. Build long-lived services: Discord client, `RedisEventPublisher` + + `EventBroadcaster`, `CommandHandler`; install the shutdown handler. +3. Connect infrastructure → `connectDatabase()`: + `AUTO_MIGRATE_ON_STARTUP` runs pending Drizzle migrations, then + `initializeDatabase()` (pg Pool, min 0). +4. `registerClientDebugLogging()` — only client debug lines carrying signal. +5. Install process guards (`registerProcessGuards`). +6. Register pipeline gauges + start the metrics server (port `METRICS_PORT`, + default 4016). +7. `client.login(token)`. + +On the Discord `ready` event, `lifecycle.ts` runs `startGatewayLifecycle()`: + +1. Inject the event broadcaster into message-capture and moderation-actions + (before any listener can fire). +2. Register Discord listeners: message-capture, reaction, thread, presence, + channel-topic, guild-member. +3. Start background work: AI analysis worker + recovery worker, command + handler, retention cleanup, weekly digest. ## Graceful shutdown -`SIGINT`/`SIGTERM` (and uncaught transient stream errors: EPIPE / ECONNRESET / -ERR_STREAM_DESTROYED / ERR_STREAM_WRITE_AFTER_END are treated as non-fatal): -stop metrics → close event broadcaster (Redis) → close command handler → -close DB → destroy client → exit. +`process-guards.ts` owns the policy. `SIGINT`/`SIGTERM` and non-transient +uncaught exceptions/rejections run `shutdown.ts`; transient stream errors +(EPIPE / ECONNRESET / ERR_STREAM_DESTROYED / ERR_STREAM_WRITE_AFTER_END, see +`isTransientStreamError()`) are logged and IGNORED so the bot stays online. + +Shutdown order: stop metrics → close event broadcaster (Redis) → close command +handler → close DB → destroy client → exit. ## Observability Prometheus scrapes `127.0.0.1:4016/metrics` (`bete_*` prefix). Collectors run per-scrape and expose: process memory/uptime, and (when AI analysis is on) live -pipeline gauges — `ai_analysis_queued_conversations`, -`ai_analysis_active_batch_requests`, `ai_analysis_active_individual_requests`, -`ai_analysis_individual_in_flight`, `ai_analysis_individual_circuit_breaker_active`, -`ai_analysis_worker_threads`, `ai_analysis_worker_threads_active`. +pipeline gauges registered by `app/metrics-collector.ts` — +`ai_analysis_queued_conversations`, `ai_analysis_active_batch_requests`, +`ai_analysis_active_text_requests`, `ai_analysis_active_media_requests`, +`ai_analysis_active_individual_requests`, `ai_analysis_individual_in_flight`, +`ai_analysis_individual_circuit_breaker_active`, +`ai_analysis_worker_threads_{text,media}`, +`ai_analysis_worker_threads_active_{text,media}`. ## Key invariants (do not break) diff --git a/services/discord-gateway/MODULE_STRUCTURE.md b/services/discord-gateway/MODULE_STRUCTURE.md deleted file mode 100644 index 471219b4..00000000 --- a/services/discord-gateway/MODULE_STRUCTURE.md +++ /dev/null @@ -1,73 +0,0 @@ -# Discord Gateway Service — Module Structure - -> Kept as a compact module map. For the authoritative layout, design -> decisions, and invariants, see `ARCHITECTURE.md`. This file was rewritten -> on 2026-08-16 to fix stale references (`winston` → pino, -> `mock-crc.ts`/`indonesianTextNormalizer.ts` removed, -> `aiAnalysisWorker.ts` → `ai-analysis-worker.ts`, -> `llmModerationClient.ts` → `llmClient.ts`). - -## Top-level - -``` -services/discord-gateway/ -├── src/ -│ ├── index.ts # Entry point -│ ├── app/ # bootstrap, shutdown, retention -│ ├── shared/ # config, database, logger, errors, utils, discord, uploader -│ └── modules/ -│ ├── message-capture/ # Discord listeners + DB store + metadata -│ ├── ai-moderation/ # LLM moderation pipeline (largest module) -│ ├── attachment-upload/ # Download + sharp resize + upload -│ ├── event-broadcaster/ # RedisEventPublisher + EventBroadcaster -│ ├── command-handler/ # Backend→gateway Redis commands -│ ├── reaction-tracking/ thread-tracking/ user-presence/ -│ ├── channel-topic/ guild-member-events/ -│ └── gateway-metrics/ # Prometheus /metrics (port 4016) -├── tests/ # Vitest suites (129 tests) -├── drizzle/ # Drizzle migration SQL + journal -├── ARCHITECTURE.md README.md package.json tsconfig.json vitest.config.ts -``` - -## Module responsibilities (summary) - -### message-capture -Captures `messageCreate`/`messageUpdate`/`messageDelete`, extracts metadata, -stores to Postgres, publishes to Redis. Controller–Service–Repository split: -`messageCapture.ts` (listener) → `messageStore.ts` (DB) + `messageMetadata.ts` -(service). - -### ai-moderation -LLM-only moderation. Entry: `aiAnalyzer.ts` (`queueMessageAnalysis`, -`startPendingAIAnalysisWorker`, `getAnalysisQueueStatus`). Scheduling: -`batchScheduler.ts` → `batchProcessor.ts` (batch lock + circuit breaker) → -`individualFallbackProcessor.ts` (per-message retry). Heavy work runs in the -Piscina pool via `ai-analysis-worker.ts` (jobs `batch` / `individual`). -Orchestration/caching: `moderationOrchestrator.ts` (exact hash → batched -semantic Qdrant → LLM), `textBatchProcessor.ts` / `mediaBatchProcessor.ts` -(one LLM call per sub-batch), `llmClient.ts` (central streaming client), -`embeddingClient.ts` + `qdrantClient.ts` (semantic cache), plus -`channelCultureStore.ts` / `userProfileStore.ts`. - -### attachment-upload -`attachmentUploader.ts` (download → upload to storage) + `imageResizer.ts` -(sharp resize). Emits `discord:attachment:*`. - -### event-broadcaster -`RedisEventPublisher` (ioredis publish) + `EventBroadcaster` (typed methods). -Channel names in `src/shared/redis-channels.ts`. - -### gateway-metrics -`metrics.ts` Prometheus HTTP server on `METRICS_PORT` (4016). Collectors run -per scrape; live pipeline gauges registered in `bootstrap.ts`. - -## Shared infrastructure -- **config** — Zod schema in `shared/config/index.ts` (single source of truth). -- **database** — Drizzle ORM over `pg`; pool `min:0` (`shared/config`). -- **logger** — `pino` wrapper, `createChildLogger()` for context loggers. -- **errors** — `AppError` hierarchy (`ConfigError`, …). - -## Notes -- No HTTP server (other than the metrics endpoint). Pure event-driven. -- `MODULE_STRUCTURE.md` is intentionally a sketch; `ARCHITECTURE.md` is the - detailed reference. When they diverge, `ARCHITECTURE.md` wins. diff --git a/services/discord-gateway/README.md b/services/discord-gateway/README.md index 9e88d7fa..c8608674 100644 --- a/services/discord-gateway/README.md +++ b/services/discord-gateway/README.md @@ -1,319 +1,63 @@ -# Discord Gateway Service - Extraction Complete +# Discord Gateway -## Overview +Event-driven selfbot service: captures Discord events, runs LLM moderation, +publishes everything to Redis for the backend to consume. -Successfully extracted Discord Gateway service with **Modular MVC + Event-Driven Architecture** using Redis pub/sub for inter-service communication. +> Architecture, invariants and the AI pipeline are documented in +> **`ARCHITECTURE.md`** — that file is the source of truth. This README only +> covers how to run it. -## Directory Structure +## Commands -``` -services/discord-gateway/ -├── src/ -│ ├── app/ -│ │ ├── bootstrap.ts # Service initialization (Discord client, DB, Redis) -│ │ └── shutdown.ts # Graceful shutdown handler -│ ├── shared/ # Shared infrastructure layer -│ │ ├── config/ -│ │ │ └── config.ts # Zod-validated environment config -│ │ ├── database/ -│ │ │ ├── schema.ts # Drizzle ORM schema -│ │ │ ├── drizzle.ts # PostgreSQL connection -│ │ │ ├── migrate.ts # Migration runner -│ │ ├── errors/ -│ │ │ └── errors.ts # Custom error classes -│ │ ├── logger/ -│ │ │ ├── logger.ts # Winston logger wrapper -│ │ │ └── serialization.ts # Log serialization -│ │ ├── utils/ -│ │ │ └── retry.ts # Retry with exponential backoff -│ │ └── discord/ -│ │ └── clientOptions.ts # Discord.js client config -│ ├── modules/ # Feature modules (Modular MVC) -│ │ ├── message-capture/ # Controller-Service-Repository -│ │ │ ├── messageCapture.ts # Controller: Discord event listeners -│ │ │ ├── messageStore.ts # Repository: DB operations -│ │ │ ├── messageMetadata.ts # Service: Metadata extraction -│ │ │ ├── types.ts # Domain types -│ │ │ └── index.ts # Module exports -│ │ ├── ai-moderation/ # Controller-Service-Repository -│ │ │ ├── aiAnalyzer.ts # Controller: Analysis orchestration -│ │ │ ├── llmModerationClient.ts # Service: LLM API client -│ │ │ ├── aiAnalysisWorker.ts # Service: Worker pool -│ │ │ ├── indonesianTextNormalizer.ts # Service: Text normalization -│ │ │ ├── moderationPrompt.ts # Service: Prompt generation -│ │ │ └── index.ts # Module exports -│ │ ├── attachment-upload/ # Controller-Service-Repository -│ │ │ ├── attachmentUploader.ts # Service: Upload orchestration -│ │ │ ├── imageResizer.ts # Service: Image resizing -│ │ │ └── index.ts # Module exports -│ │ └── event-broadcaster/ # Event-driven layer -│ │ ├── eventBroadcaster.ts # Service: Redis pub/sub publisher -│ │ ├── eventTypes.ts # Domain: Event type definitions -│ │ └── index.ts # Module exports -│ ├── mock-crc.ts # CRC polyfill for discord.js -│ └── index.ts # Service entry point -├── ARCHITECTURE.md # Detailed architecture documentation -├── package.json # Service dependencies -└── tsconfig.json # TypeScript configuration (inherited) +```bash +pnpm install +pnpm typecheck # tsc --noEmit +pnpm lint # biome check --diagnostic-level=error . +pnpm test # vitest run (138 tests) +pnpm build # tsc — CI/prod builds run this inside nix, which also + # runs scripts/fix-imports.mjs to rewrite @/ aliases and + # extensionless imports for Node ESM +pnpm dev # tsx watch src/index.ts +pnpm start # node dist/index.js ``` -## Architecture Patterns +Deployment is CI-only: `nix build .#discord-gateway` → Attic cache → systemd +restart on the VPS. Do not build/hand-copy the artifact. -### 1. Modular MVC Structure -Each feature module follows **Controller-Service-Repository** pattern: - -**Message Capture Module**: -- **Controller** (`messageCapture.ts`): Listens to Discord events (messageCreate, messageUpdate, messageDelete) -- **Service** (`messageMetadata.ts`): Extracts and normalizes message metadata -- **Repository** (`messageStore.ts`): Database CRUD operations - -**AI Moderation Module**: -- **Controller** (`aiAnalyzer.ts`): Orchestrates analysis workflow -- **Service** (`llmModerationClient.ts`): LLM API integration -- **Service** (`aiAnalysisWorker.ts`): Worker pool management -- **Service** (`indonesianTextNormalizer.ts`): Text preprocessing - -**Attachment Upload Module**: -- **Service** (`attachmentUploader.ts`): Upload orchestration -- **Service** (`imageResizer.ts`): Image processing - -### 2. Event-Driven Architecture -**Redis Pub/Sub** replaces WebSocket broadcaster: +## Layout ``` -Discord Events → Discord Gateway Service → Redis Pub/Sub → Backend Service - ↓ - Event Channels: - - discord:message:created - - discord:message:updated - - discord:message:deleted - - discord:message:analyzed - - discord:attachment:created - - discord:attachment:uploaded - - discord:analysis:queue_status +src/ +├── index.ts # entry → initializeDiscordGateway() +├── app/ # process lifecycle +│ ├── bootstrap.ts # startup order: config → DB → services → metrics → login +│ ├── lifecycle.ts # everything wired on the Discord 'ready' hook +│ ├── process-guards.ts # SIGINT/SIGTERM + uncaught error policy +│ ├── metrics-collector.ts # AI pipeline Prometheus gauges +│ ├── shutdown.ts # graceful shutdown sequence +│ └── retention.ts # expired-record cleanup scheduler +├── shared/ # infrastructure — never imports from modules/ +│ ├── config/ database/ logger/ errors/ utils/ +│ ├── discord/clientOptions.ts +│ ├── redis-channels.ts # canonical Redis channel + command constants +│ └── moderation-types.ts # domain types shared across services +└── modules/ # feature modules (each exposes an index.ts facade) + ├── ai-moderation/ # LLM moderation pipeline (largest module) + ├── message-capture/ # Discord listeners + message/attachment DB + ├── attachment-upload/ # download → resize → upload + ├── event-broadcaster/ # Redis pub/sub publisher + ├── command-handler/ # backend → gateway commands over Redis + ├── gateway-metrics/ # Prometheus /metrics (port 4016) + ├── monitor/ # weekly digest scheduler + └── reaction-tracking/ thread-tracking/ user-presence/ + channel-topic/ guild-member-events/ ``` -### 3. Shared Infrastructure Layer -Centralized, reusable components: -- **Config**: Zod-validated environment variables -- **Logger**: Winston logger with context support -- **Database**: Drizzle ORM with PostgreSQL -- **Errors**: Custom error classes with codes and HTTP status codes -- **Utils**: Retry logic with exponential backoff -- **Discord**: Client configuration and options +Dependency direction is one-way: `index.ts` → `app/` → `modules/` → `shared/`. +Callers outside a module import its `index.ts` facade, never an internal file. -### 4. No HTTP Server -- **Event-driven only**: No Express, WebSocket, or HTTP routes -- **Redis pub/sub**: All inter-service communication via Redis -- **Backend service**: Consumes events and serves HTTP API -- **Frontend**: Continues to use Backend HTTP API +## Testing -## Key Features - -### Message Capture -1. Discord emits `messageCreate`, `messageUpdate`, `messageDelete` events -2. `messageCapture.ts` listener receives and validates event -3. Extract metadata: user, channel, content, timestamp, attachments -4. `messageStore.ts` inserts into PostgreSQL -5. `eventBroadcaster.messageCreated()` publishes to Redis -6. Backend service subscribes and processes - -### AI Moderation -1. `aiAnalyzer.ts` queues messages for analysis -2. `llmModerationClient.ts` calls LLM API with context -3. `indonesianTextNormalizer.ts` preprocesses text -4. Results stored in database -5. `eventBroadcaster.messageAnalyzed()` publishes results -6. Backend service receives and updates UI - -### Attachment Upload -1. `messageCapture.ts` detects attachments -2. `attachmentUploader.ts` downloads from Discord -3. `imageResizer.ts` resizes images if needed -4. Upload to external storage with retry logic -5. `eventBroadcaster.attachmentUploaded()` publishes -6. Backend service stores metadata - -## Initialization Flow - -``` -1. Load environment config (Zod validation) - ↓ -2. Initialize PostgreSQL connection - ↓ -3. Run pending database migrations - ↓ -4. Create Discord client with optimized cache - ↓ -5. Initialize Redis event broadcaster - ↓ -6. Register Discord event listeners - - messageCapture (message events) - - aiAnalyzer (analysis worker) - ↓ -7. Login to Discord - ↓ -8. Listen for graceful shutdown signals -``` - -## Graceful Shutdown - -On SIGINT/SIGTERM/uncaughtException/unhandledRejection: -1. Close PostgreSQL connection -2. Close Redis connection -3. Destroy Discord client -4. Exit process (code 0 for clean, 1 for error) - -## Dependencies - -**Core Discord**: -- `discord.js-selfbot-v13` — Discord client (selfbot variant) - -**Media Processing**: -- `sharp` — Image resizing - -**Data & Config**: -- `drizzle-orm` — Type-safe ORM -- `pg` — PostgreSQL driver -- `zod` — Config validation -- `ioredis` — Redis client - -**Logging & Utilities**: -- `winston` — Structured logging -- `p-retry` — Retry with backoff -- `p-limit` — Concurrency limiting -- `piscina` — Worker pool - -## No Breaking Changes - -- Original `src/` remains untouched -- Discord Gateway is a **new service** in `services/discord-gateway/` -- Can run alongside existing monolith during transition -- Backend service will consume Redis events -- Frontend continues to use Backend HTTP API - -## Next Steps - -1. **Create Backend service** (`services/backend/`) - - HTTP API endpoints - - Redis event subscribers - - Database models - - WebSocket broadcaster - -2. **Update Frontend** (`frontend/`) - - Connect to Backend HTTP API - - Subscribe to WebSocket events - -3. **Nix & CI/CD** - - flake.nix package for Discord Gateway - - systemd services (gmw-backend, gmw-discord-gateway) - - GitHub Actions for build/deploy (nix copy → systemctl restart) - -4. **Documentation** - - API documentation - - Event schema documentation - - Deployment guide - -## Files Created - -**Total: 43 files** - -### Shared Infrastructure (9 files) -- `src/shared/config/config.ts` -- `src/shared/database/` (5 files) -- `@bete/shared/errors` (shared package) -- `src/shared/logger/logger.ts` -- `src/shared/logger/serialization.ts` -- `src/shared/utils/retry.ts` -- `src/shared/discord/clientOptions.ts` - -### Modules (28 files) -- `src/modules/message-capture/` (5 files) -- `src/modules/ai-moderation/` (6 files) -- `src/modules/attachment-upload/` (3 files) -- `src/modules/event-broadcaster/` (3 files) - -### App & Entry (4 files) -- `src/app/bootstrap.ts` -- `src/app/shutdown.ts` -- `src/index.ts` -- `src/mock-crc.ts` - -### Configuration (2 files) -- `package.json` -- `ARCHITECTURE.md` - -## Verification Checklist - -✅ Directory structure created -✅ Shared infrastructure migrated -✅ Message capture module migrated -✅ AI moderation module migrated -✅ Attachment upload module migrated -✅ Event broadcaster module created (Redis pub/sub) -✅ Bootstrap and entry point created -✅ Package.json with dependencies -✅ No HTTP server code (Express, WebSocket removed) -✅ Event-driven architecture implemented -✅ Graceful shutdown handler -✅ Module index files for clean exports -✅ Architecture documentation - -## Event Flow Diagram - -``` -┌─────────────────────────────────────────────────────────────────┐ -│ Discord Gateway Service │ -├─────────────────────────────────────────────────────────────────┤ -│ │ -│ ┌────────────────────────────┐ ┌────────────────────────────┐ │ -│ │ Message Capture │ │ AI Moderation │ │ -│ │ (Controller) │ │ (Controller) │ │ -│ └──────────────┬─────────────┘ └──────────────┬─────────────┘ │ -│ │ │ │ -│ ├───────────────────────────────┤ │ -│ │ │ │ -│ ▼ ▼ │ -│ ┌───────────────────────────────────────────────────────────┐ │ -│ │ Event Broadcaster (Redis Pub/Sub) │ │ -│ │ - discord:message:created │ │ -│ │ - discord:message:updated │ │ -│ │ - discord:message:deleted │ │ -│ │ - discord:message:analyzed │ │ -│ │ - discord:attachment:created │ │ -│ │ - discord:attachment:uploaded │ │ -│ │ - discord:analysis:queue_status │ │ -│ └───────────────────────────────────────────────────────────┘ │ -│ │ │ -└────────────────────────────────┼────────────────────────────────┘ - │ - │ Redis Pub/Sub - │ - ▼ -┌─────────────────────────────────────────────────────────────────┐ -│ Backend Service │ -│ (Subscribes to events, serves HTTP API, manages WebSocket) │ -└─────────────────────────────────────────────────────────────────┘ - │ - │ HTTP API - │ - ▼ -┌─────────────────────────────────────────────────────────────────┐ -│ Frontend Application │ -│ (React SPA, real-time updates via WebSocket) │ -└─────────────────────────────────────────────────────────────────┘ -``` - -## Summary - -The Discord Gateway service has been successfully extracted with: -- **Modular MVC architecture** for clean separation of concerns -- **Event-driven design** using Redis pub/sub for inter-service communication -- **Shared infrastructure layer** for reusable components -- **No HTTP server** — pure event-driven service -- **Graceful shutdown** handling -- **Type-safe configuration** with Zod validation -- **Structured logging** with Winston -- **PostgreSQL integration** with Drizzle ORM - -The service is ready for integration with the Backend service, which will consume Redis events and serve the HTTP API to the Frontend. +Vitest, tests in `tests/`. Config supplies dummy env vars so the suite runs +without live Postgres/Redis/Qdrant; external services are mocked. `llmE2e.test.ts` +is skipped by default and needs real credentials (`pnpm test:e2e:live`).