Compare commits

...

13 Commits

Author SHA1 Message Date
MythEclipse
95cb8b837a feat(thread-discovery): implement separate endpoint for fetching threads and update channel loading logic 2026-05-13 21:22:05 +07:00
MythEclipse
0b8111de81 feat(moderation): implement backlog message synchronization and enhance message metadata handling 2026-05-13 21:04:45 +07:00
MythEclipse
d55b56c897 feat(moderation): enhance message capture and storage with thread support
- Added functions to retrieve message location, sticker metadata, and display content in messageCapture.ts.
- Updated captureMessage function to store thread information and sticker metadata in the database.
- Modified messageStore.ts to support querying messages and attachments by thread ID.
- Updated types.ts to include thread_id in AttachmentRecord.
- Altered database schema in muxer-queue.ts to add thread_id column to attachments.
- Introduced ChannelSummary interface and listWatchableChannels method in voiceController.ts to fetch watchable channels.
- Added API endpoint in webserver.ts to retrieve channels for a given guild.
2026-05-13 20:52:37 +07:00
MythEclipse
c7d8353403 chore: update devDependencies and add bun-types to tsconfig 2026-05-13 19:49:56 +07:00
MythEclipse
9f09f5ef28 refactor: replace Database type with SqliteDatabase in moderation modules 2026-05-13 19:47:44 +07:00
MythEclipse
471e3bac82 docs: update CLAUDE.md with complete moderation watcher documentation 2026-05-13 19:35:42 +07:00
MythEclipse
6d353c1753 feat: add moderation dashboard UI
- Create responsive dashboard with dark theme
- Implement three tabs: Text Messages, Images, Voice
- Add channel/thread filtering
- Real-time WebSocket updates with polling fallback
- Display message metadata (author, timestamp, edits, deletions)
- Show image previews with upload status and URLs
2026-05-13 19:34:39 +07:00
MythEclipse
692962408f feat: add REST API and WebSocket events for moderation
- Add /api/messages endpoint for querying text and image data
- Add WebSocket broadcast functions for real-time updates
- Support message_created, message_updated, message_deleted events
- Support attachment_uploaded event for real-time image updates
2026-05-13 19:34:15 +07:00
MythEclipse
738f5cfbd6 feat: integrate message capture into bot startup
- Register message capture handlers on bot ready
- Export getDatabase function from muxer-queue
- Initialize database schema for messages and attachments
2026-05-13 19:34:15 +07:00
MythEclipse
b13dfb2ece config: add moderation watcher configuration variables
- Add MONITOR_GUILD_ID for target server monitoring
- Add PICSER_UPLOAD_URL for attachment upload endpoint
- Add attachment upload timeout and size limit settings
- Add retry attempts configuration for upload failures
2026-05-13 19:34:14 +07:00
MythEclipse
017efb0b86 feat: implement moderation message capture system
- Add message store with database operations (insert, update, query)
- Implement attachment uploader with picser integration
- Add Discord event listeners for message create/update/delete
- Support attachment upload with retry logic and error handling
- Add comprehensive unit tests for message store and uploader
2026-05-13 19:34:14 +07:00
MythEclipse
579fcb4684 docs: add moderation watcher expansion design spec 2026-05-13 19:29:01 +07:00
MythEclipse
220c3b93d2 feat: implement Opus decoder runtime checks and add tests for decoder functionality 2026-05-13 18:56:44 +07:00
24 changed files with 3595 additions and 472 deletions

View File

@@ -1,7 +1,5 @@
# Discord Bot Configuration
DISCORD_TOKEN=your_bot_token_here
VOICE_CHANNEL_ID=your_voice_channel_id_here
GUILD_ID=your_guild_id_here
# Recording Configuration
RECORDINGS_DIR=./recordings
@@ -31,5 +29,12 @@ RECONNECT_TIMEOUT_MS=5000
LOG_LEVEL=info
NODE_ENV=development
# Moderation Configuration
MONITOR_GUILD_ID=your_guild_id_here
PICSER_UPLOAD_URL=https://picser.asepharyana.tech/api/upload
ATTACHMENT_UPLOAD_TIMEOUT_MS=30000
ATTACHMENT_MAX_SIZE_MB=100
ATTACHMENT_RETRY_ATTEMPTS=3
BACKLOG_SYNC_HOURS=24
BACKLOG_SYNC_BATCH_SIZE=100

1
.gitignore vendored
View File

@@ -2,3 +2,4 @@ node_modules
recordings
.env
dist/
.muxer-queue.**

513
CLAUDE.md Normal file
View File

@@ -0,0 +1,513 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Project Overview
**Discord Moderation Watcher Bot** — A comprehensive monitoring bot that captures voice, text messages, and images from Discord servers. Records audio from voice channels, captures all text messages (new/edited/deleted) from channels and threads, and uploads attachments to external storage. All data stored in SQLite with real-time dashboard.
Built with **Bun** + **discord.js-selfbot-v13** + **@discordjs/voice** + **Express** + **WebSocket**.
## Architecture
### High-Level Flow
1. **Bot Entry** (`src/index.ts`) — Initializes Discord client, registers event listeners, starts webserver
2. **Message Capture** (`src/moderation/messageCapture.ts`) — Listens to Discord events (messageCreate, messageUpdate, messageDelete)
3. **Message Store** (`src/moderation/messageStore.ts`) — Database operations for messages and attachments
4. **Attachment Uploader** (`src/moderation/attachmentUploader.ts`) — Downloads from Discord, uploads to picser, stores URLs
5. **Voice Controller** (`src/voiceController.ts`) — Manages voice channel connections
6. **Recorder** (`src/recorder.ts`) — Records voice audio to OGG segments
7. **Web Server** (`src/webserver.ts`) — Express + WebSocket for REST API and real-time updates
8. **Dashboard** (`public/dashboard.html`) — Web UI with three tabs (Text, Images, Voice)
### Key Modules
**Moderation Subsystem** (`src/moderation/`):
- `types.ts` — TypeScript types for messages, attachments, voice segments
- `messageCapture.ts` — Discord event listeners (messageCreate, messageUpdate, messageDelete)
- `messageStore.ts` — Database CRUD operations (insert, update, query)
- `attachmentUploader.ts` — Picser integration with retry logic and error handling
**Database Schema** (SQLite):
- `messages` table — text messages with edit/delete tracking, user metadata, timestamps
- `attachments` table — attachment metadata, Discord URLs, picser URLs, upload status
- Indexes on channel_id, user_id, created_at for fast queries
**Voice Recording** (existing, unchanged):
- `recorder.ts` — Joins voice channel, subscribes to user audio streams
- `recorder/audioStream.ts` — Opus packet subscription
- `recorder/decoder.ts` — Opus decoder with runtime checks
- `recorder/segment.ts` — OGG file rotation (5s segments)
**Web Interface**:
- REST API: `/api/messages?channel=<id>&type=text|image`
- WebSocket: real-time events (message_created, message_updated, message_deleted, attachment_uploaded)
- Dashboard: three tabs (Text Messages, Images, Voice) with channel filtering
### Recording Structure
```
recordings/
├── <user-id>/
│ ├── <user-id>-<session-start>-0.ogg
│ ├── <user-id>-<session-start>-0.json
│ └── ...
messages (SQLite):
├── id, guild_id, channel_id, thread_id
├── user_id, username, avatar_url
├── content, edited_content
├── created_at, edited_at, deleted_at
└── type (text|edited|deleted)
attachments (SQLite):
├── id, message_id, guild_id, channel_id, user_id
├── filename, size, type (MIME)
├── discord_url, uploaded_url (picser raw_commit)
├── upload_status (pending|uploaded|failed)
└── created_at, uploaded_at
```
## Development Commands
```bash
# Install dependencies
bun install
# Development (auto-restart on file changes)
bun run dev
# Production
bun run start
# Type checking
bun run typecheck
# Linting (Biome)
bun run lint
# Format code (Biome)
bun run format
# Run tests
bun run test
# Build TypeScript
bun run build
```
## Configuration
All config via `.env` (see `.env.example`). Key variables:
**Discord & Monitoring:**
- `DISCORD_TOKEN` — Bot token (required)
- `MONITOR_GUILD_ID` — Target server to monitor (required for moderation)
- `GUILD_ID` — Legacy voice channel guild (optional)
- `VOICE_CHANNEL_ID` — Legacy voice channel ID (optional)
**Recording:**
- `RECORDINGS_DIR` — Where to save audio files (default: `./recordings`)
- `RECORDING_SEGMENT_MS` — OGG segment duration (default: 5000ms)
**Decoder:**
- `DECODER_ROTATE_MS` — Opus decoder rotation interval (default: 5000ms)
- `DECODER_COOLDOWN_MS` — Cooldown after decoder error (default: 30000ms)
**Attachments:**
- `PICSER_UPLOAD_URL` — Picser upload endpoint (default: https://picser.asepharyana.tech/api/upload)
- `ATTACHMENT_UPLOAD_TIMEOUT_MS` — Upload timeout (default: 30000ms)
- `ATTACHMENT_MAX_SIZE_MB` — Max file size (default: 100MB)
- `ATTACHMENT_RETRY_ATTEMPTS` — Retry count (default: 3)
**Web Server:**
- `WEBSERVER_PORT` — HTTP/WebSocket port (default: 3000)
**Connection:**
- `VOICE_CONNECTION_TIMEOUT_MS` — Voice join timeout (default: 15000ms)
- `RECONNECT_TIMEOUT_MS` — Reconnect timeout (default: 5000ms)
- `AUDIO_STREAM_SILENCE_DURATION_MS` — Silence threshold (default: 3000ms)
**Logging:**
- `LOG_LEVEL` — Pino log level (default: info)
- `VERBOSE` — Enable debug logging (default: false)
- `NODE_ENV` — Environment (development|production|test)
## Testing
Tests use **Vitest** in `tests/` directory. Run with `bun run test`.
**Test Coverage:**
- `tests/moderation/messageStore.test.ts` — Message store CRUD operations
- `tests/moderation/attachmentUploader.test.ts` — Picser response parsing
- `tests/config.test.ts` — Configuration validation
- `tests/decoder.test.ts` — Opus decoder runtime detection
## Code Style
- **Formatter**: Biome (2-space indent)
- **Linter**: Biome with custom rules (warn on non-null assertions, noExplicitAny)
- **Language**: TypeScript with strict mode
- **Logging**: Use `createChildLogger(context)` for scoped logs
- **Errors**: Throw custom AppError subclasses with code + statusCode
- **Database**: Use prepared statements, never string interpolation
## Key Patterns
### Message Capture Lifecycle
1. Discord event fires (messageCreate, messageUpdate, messageDelete)
2. Check if guild matches MONITOR_GUILD_ID
3. Extract message metadata (user, channel, content, timestamp)
4. Insert into messages table
5. Broadcast WebSocket event to connected clients
6. If attachments exist:
- Insert into attachments table with status='pending'
- Start async upload to picser (non-blocking)
- On success: update uploaded_url, status='uploaded'
- On failure: store error, status='failed'
### Attachment Upload Flow
1. Download from Discord URL (with timeout)
2. Validate file size against ATTACHMENT_MAX_SIZE_MB
3. Upload to picser with retry logic (exponential backoff)
4. Parse response, extract raw_commit URL
5. Update database with uploaded_url and status
6. Broadcast attachment_uploaded event
### WebSocket Protocol
**Inbound (browser → bot):**
- Binary: Raw PCM buffers (24kHz mono s16le) for voice transmission
**Outbound (bot → browser):**
- Binary: 4-byte user ID hash + PCM chunk (voice)
- JSON: `{ type: "user_state", users: [...] }` (active speakers)
- JSON: `{ type: "message_created", data: {...} }` (new text message)
- JSON: `{ type: "message_updated", data: {...} }` (edited message)
- JSON: `{ type: "message_deleted", data: {...} }` (deleted message)
- JSON: `{ type: "attachment_uploaded", data: {...} }` (image uploaded)
### Graceful Shutdown
Handles SIGINT/SIGTERM/uncaughtException/unhandledRejection:
1. Stop voice connection
2. Pause player
3. Destroy Discord client
4. Exit process
## Dashboard Usage
**Access:** `http://localhost:3000/dashboard.html`
**Features:**
- Three tabs: Text Messages | Images | Voice
- Channel/thread filter dropdown
- Real-time WebSocket updates
- Polling fallback if WebSocket disconnects
- Message display with metadata (author, timestamp, edits, deletions)
- Image grid with previews and upload status
- Voice segment list (future enhancement)
**Keyboard/UI:**
- Click tab to switch content type
- Select channel to filter
- Click image to view full size
- WebSocket status indicator (green = connected)
## Common Tasks
### Add a new config variable
1. Add to `configSchema` in `src/config.ts` with Zod validation
2. Add to `.env.example` with description
3. Use via `config.VARIABLE_NAME`
### Add a new REST endpoint
1. Add route in `src/webserver.ts` (Express)
2. Use database functions from `src/moderation/messageStore.ts`
3. Wrap in try-catch, pass errors to Express error handler
4. Return JSON response
### Add a new WebSocket event
1. Define broadcast function in `src/webserver.ts` (attach to globalThis)
2. Call from event handler (e.g., messageCapture.ts)
3. Send JSON with `{ type, data, timestamp }`
4. Handle in dashboard JavaScript
### Debug message capture
- Set `VERBOSE=true` in `.env` for detailed logging
- Check `/health` endpoint for active users/connections
- Monitor `/metrics` endpoint (Prometheus format)
- Check `recordings/<user-id>/` for voice segments
- Query SQLite directly: `sqlite3 .muxer-queue.db "SELECT * FROM messages LIMIT 10;"`
### Debug attachment uploads
- Check `upload_status` in attachments table
- View `upload_error` field for failure reasons
- Monitor logs for "Attachment upload" messages
- Verify picser endpoint is accessible
- Check file size against ATTACHMENT_MAX_SIZE_MB
## Dependencies
**Core:**
- **discord.js-selfbot-v13** — Discord client (selfbot variant)
- **@discordjs/voice** — Voice connection management
- **@discordjs/opus** — Native Opus codec (optional, required for web PCM)
- **prism-media** — Audio encoding/decoding (Opus, OGG)
**Web:**
- **express** — HTTP server
- **ws** — WebSocket server
- **helmet** — Security headers
**Data:**
- **better-sqlite3** — SQLite database
- **zod** — Config validation
**Logging & Monitoring:**
- **pino** — Structured logging
- **pino-http** — HTTP request logging
- **prom-client** — Prometheus metrics
**Utilities:**
- **p-retry** — Retry logic with backoff
- **class-transformer** — Object transformation
- **class-validator** — Data validation
**Dev:**
- **Biome** — Linting/formatting
- **Vitest** — Testing framework
- **TypeScript** — Type checking
## Notes
- Bot uses selfbot variant (user account) rather than standard bot token — check Discord ToS
- Opus decoding disabled on Bun without native opus to avoid crashes
- OGG segments include metadata JSON for each segment (user info, timestamps, duration)
- WebSocket broadcasts PCM in real-time; browser can transmit audio back to Discord
- Graceful shutdown ensures clean disconnection and resource cleanup
- All database operations use prepared statements to prevent SQL injection
- Attachment uploads are non-blocking (async) to avoid blocking message capture
- Message capture continues even if attachment upload fails
- Dashboard uses textContent for XSS prevention (not innerHTML)
## Future Enhancements
- Reaction tracking
- Message search/full-text search
- Moderation actions (flag, delete, mute)
- Export/archive functionality
- Retention policies (auto-delete old data)
- Voice segment metadata in dashboard
- User activity analytics
- Audit log export
## Architecture
### High-Level Flow
1. **Bot Entry** (`src/index.ts`) — Initializes Discord client, sets up graceful shutdown, starts webserver
2. **Voice Controller** (`src/voiceController.ts`) — Manages guild/channel selection and connection lifecycle
3. **Recorder** (`src/recorder.ts`) — Joins voice channel, subscribes to user audio streams, handles Opus decoding and segment rotation
4. **Web Server** (`src/webserver.ts`) — Express + WebSocket server for:
- REST API: guild/channel listing, connect/disconnect
- WebSocket: real-time PCM broadcast to browser, browser-to-Discord audio transmission
5. **Muxer Queue** (`src/muxer-queue.ts`) — SQLite-backed job queue for post-processing audio segments (future use)
### Key Modules
- **Recorder subsystem** (`src/recorder/`):
- `audioStream.ts` — Subscribes to Discord audio receiver, emits Opus packets
- `decoder.ts` — Opus decoder with runtime checks (Bun vs Node), cooldown/rotation logic for web PCM broadcast
- `segment.ts` — Manages OGG file rotation (5s default segments per user)
- `metadata.ts` — Collects user/role info, creates segment metadata JSON
- **Voice Connection** — Uses `@discordjs/voice` receiver to subscribe to speaking users; each user gets their own stream
- **Audio Pipeline**:
- Discord → Opus packets → PacketFilter → OGG segments (disk) + OpusDecoder → PCM (web broadcast)
- Browser → 24kHz mono PCM → upsample to 48kHz stereo → Opus encoder → OGG → Discord player
- **Metrics** (`src/metrics.ts`) — Prometheus metrics for audio levels, recordings, connections, WebSocket clients
- **Logging** (`src/logger.ts`) — Pino logger with pretty-print in dev, JSON in prod
- **Config** (`src/config.ts`) — Zod-validated environment variables with sensible defaults
- **Error Handling** (`src/errors.ts`) — Custom error classes (AppError, ConfigError, AudioError, VoiceConnectionError, ValidationError)
### Recording Structure
```
recordings/
├── <user-id>/
│ ├── <user-id>-<session-start>-0.ogg
│ ├── <user-id>-<session-start>-0.json
│ ├── <user-id>-<session-start>-1.ogg
│ ├── <user-id>-<session-start>-1.json
│ └── ...
```
Each segment is 5s (configurable). Metadata JSON includes user info, roles, timestamps, duration.
### Database
- **Muxer Queue** (`.muxer-queue.db`) — SQLite with WAL mode, tracks pending/processing/completed/failed jobs for audio post-processing
## Development Commands
```bash
# Install dependencies
bun install
# Development (auto-restart on file changes)
bun run dev
# Production
bun run start
# Type checking
bun run typecheck
# Linting (Biome)
bun run lint
# Format code (Biome)
bun run format
# Run tests
bun run test
# Build TypeScript
bun run build
```
## Configuration
All config via `.env` (see `.env.example`). Key variables:
- `DISCORD_TOKEN` — Bot token (required)
- `RECORDINGS_DIR` — Where to save audio files (default: `./recordings`)
- `RECORDING_SEGMENT_MS` — OGG segment duration (default: 5000ms)
- `DECODER_ROTATE_MS` — Opus decoder rotation interval (default: 5000ms)
- `DECODER_COOLDOWN_MS` — Cooldown after decoder error (default: 30000ms)
- `WEBSERVER_PORT` — HTTP/WebSocket port (default: 3000)
- `VOICE_CONNECTION_TIMEOUT_MS` — Voice join timeout (default: 15000ms)
- `AUDIO_STREAM_SILENCE_DURATION_MS` — Silence threshold before ending stream (default: 3000ms)
- `LOG_LEVEL` — Pino log level (default: info)
- `VERBOSE` — Enable debug logging (default: false)
## Testing
Tests use **Vitest** in `tests/` directory. Run with `bun run test`.
Example: `tests/decoder.test.ts` tests Opus decoder runtime detection (Bun vs Node, native opus availability).
## Code Style
- **Formatter**: Biome (2-space indent)
- **Linter**: Biome with custom rules (warn on non-null assertions, noExplicitAny)
- **Language**: TypeScript with strict mode
- **Logging**: Use `createChildLogger(context)` for scoped logs
- **Errors**: Throw custom AppError subclasses with code + statusCode
## Key Patterns
### Voice Connection Lifecycle
1. `VoiceController.connect(guildId, channelId)` → calls `startRecording()`
2. `startRecording()` joins channel, sets up receiver, subscribes to speaking users
3. On user speak: create stream, segment manager, decoder; pipe to OGG + web broadcast
4. On silence (3s): close stream, save metadata JSON
5. `VoiceController.disconnect()` → calls `stopRecording()` → destroys connection
### Audio Decoding (Web Broadcast)
- OpusDecoder wraps prism decoder with error recovery
- Rotates decoder every 5s to prevent memory leaks
- Cools down for 30s after error before retrying
- Downsamples 48kHz stereo → 24kHz mono for web transmission
### WebSocket Protocol
- **Inbound** (browser → bot): Raw PCM buffers (24kHz mono s16le)
- **Outbound** (bot → browser):
- Binary: 4-byte user ID hash + PCM chunk
- JSON: `{ type: "user_state", users: [...] }` on connect/user activity change
### Graceful Shutdown
Handles SIGINT/SIGTERM/uncaughtException/unhandledRejection:
1. Stop voice connection
2. Pause player
3. Destroy Discord client
4. Exit process
## Future Expansion (Text/Image Monitoring)
Current scope: voice only. Planned additions:
- Text channel message capture
- Image/attachment logging
- Per-channel/per-user filtering
- Moderation action triggers
These will likely require:
- Additional event listeners in recorder
- Extended metadata schema
- New storage/indexing strategy
- Webhook/alert system
## Common Tasks
### Add a new config variable
1. Add to `configSchema` in `src/config.ts` with Zod validation
2. Add to `.env.example`
3. Use via `config.VARIABLE_NAME`
### Add a new REST endpoint
1. Add route in `src/webserver.ts` (Express)
2. Use `VoiceController` methods or create new ones
3. Wrap in try-catch, pass errors to Express error handler
### Add metrics
1. Define gauge/counter/histogram in `src/metrics.ts`
2. Update in relevant code paths
3. Metrics exposed at `/metrics` endpoint (Prometheus format)
### Debug audio issues
- Set `VERBOSE=true` in `.env` for detailed logging
- Check `/health` endpoint for active users/connections
- Monitor audio levels via `/metrics` (audio_level_db gauge)
- Check segment files in `recordings/<user-id>/` directory
## Dependencies
- **discord.js-selfbot-v13** — Discord client (selfbot variant for user account access)
- **@discordjs/voice** — Voice connection management
- **@discordjs/opus** — Native Opus codec (optional, required for web PCM decode)
- **prism-media** — Audio encoding/decoding (Opus, OGG)
- **express** — HTTP server
- **ws** — WebSocket server
- **better-sqlite3** — SQLite database (muxer queue)
- **pino** — Structured logging
- **prom-client** — Prometheus metrics
- **zod** — Config validation
- **Biome** — Linting/formatting
- **Vitest** — Testing framework
## Notes
- Bot uses selfbot variant (user account) rather than standard bot token — check Discord ToS
- Opus decoding disabled on Bun without native opus to avoid crashes
- OGG segments include metadata JSON for each segment (user info, timestamps, duration)
- WebSocket broadcasts PCM in real-time; browser can transmit audio back to Discord
- Graceful shutdown ensures clean disconnection and resource cleanup

View File

@@ -0,0 +1,64 @@
# Backlog Sync Rich Metadata Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Fetch prior Discord messages up to 24 hours on startup, persist rich Discord-client-like metadata, and render rich message content in homepage tabs.
**Architecture:** Add `messageMetadata.ts` for reusable extraction, `backlogSync.ts` for bounded startup history fetch, reuse existing store/uploader. UI reads metadata JSON and renders stickers, embeds, attachments/replies/thread badges.
**Tech Stack:** Bun, TypeScript, discord.js-selfbot-v13, bun:sqlite, Express/WebSocket, vanilla HTML/CSS/JS.
---
### Task 1: Extract rich message metadata
**Files:**
- Create: `src/moderation/messageMetadata.ts`
- Modify: `src/moderation/messageCapture.ts`
- [ ] Create helper functions: `getMessageLocation`, `getStickerMetadata`, `getEmbedMetadata`, `getAttachmentMetadata`, `getMessageMetadata`, `getDisplayContent`.
- [ ] Replace duplicate capture helper logic with imports from `messageMetadata.ts`.
- [ ] Verify: `bun run typecheck`.
### Task 2: Make message inserts idempotent
**Files:**
- Modify: `src/moderation/messageStore.ts`
- [ ] Change message insert to `INSERT OR IGNORE` so backlog sync and live events do not conflict.
- [ ] Change attachment insert to `INSERT OR IGNORE`.
- [ ] Verify: `bun run typecheck && bun run test`.
### Task 3: Add backlog sync
**Files:**
- Create: `src/moderation/backlogSync.ts`
- Modify: `src/index.ts`
- Modify: `src/config.ts`
- Modify: `.env.example`
- [ ] Add config: `BACKLOG_SYNC_HOURS=24`, `BACKLOG_SYNC_BATCH_SIZE=100`.
- [ ] Fetch text/thread channels from monitored guild on ready.
- [ ] For each channel/thread, page `channel.messages.fetch({ limit, before })` until older than cutoff.
- [ ] Store messages with rich metadata and attachments.
- [ ] Start sync after registering live capture; run async and log progress.
- [ ] Verify: `bun run typecheck && bun run test`.
### Task 4: Render richer UI
**Files:**
- Modify: `public/index.html`
- [ ] Render metadata embeds as embed cards.
- [ ] Render attachments as inline previews/links in Text tab.
- [ ] Render reply and thread badges.
- [ ] Keep sticker rendering.
- [ ] Verify static JS syntax by typecheck/tests where applicable.
### Task 5: Final verification
**Files:** all touched files
- [ ] Run `bun run typecheck`.
- [ ] Run `bun run test`.
- [ ] Verify short DB init with `bun -e 'import("./src/muxer-queue.ts").then((m)=>{const db=m.getDatabase(); db.close(); console.log("sqlite ok")})'`.

View File

@@ -0,0 +1,38 @@
# Separate Thread Discovery Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Keep channel listing fast and move expensive active/archived thread discovery to a separate endpoint loaded asynchronously by the homepage.
**Architecture:** `VoiceController` exposes cache-only channels and network-backed threads separately. `webserver.ts` adds `/api/guilds/:guildId/threads`. `public/index.html` loads channels first, then appends thread options after thread endpoint returns.
**Tech Stack:** TypeScript, discord.js-selfbot-v13, Express, vanilla JS.
---
### Task 1: Add thread discovery method
**Files:**
- Modify: `src/voiceController.ts`
- [ ] Add `listThreads(guildId)` that fetches active and archived threads per parent text channel.
- [ ] Keep `listWatchableChannels` cache-only.
- [ ] Verify typecheck.
### Task 2: Add thread API endpoint
**Files:**
- Modify: `src/webserver.ts`
- [ ] Add `GET /api/guilds/:guildId/threads`.
- [ ] Return thread summaries.
- [ ] Verify typecheck.
### Task 3: Update homepage dropdown loading
**Files:**
- Modify: `public/index.html`
- [ ] `loadChannels` fetches `/channels` first and renders immediately.
- [ ] Then fetches `/threads` async and appends thread options.
- [ ] Verify typecheck/tests.

View File

@@ -0,0 +1,240 @@
# Moderation Watcher Expansion Design
**Date:** 2026-05-13
**Status:** Design Phase
**Scope:** Expand Discord bot from voice-only recorder to full moderation watcher capturing text, images, and voice
## Overview
Transform the existing voice recorder bot into a unified moderation watcher that captures:
- **Voice:** Audio from voice channels (existing)
- **Text:** Messages (new/edited/deleted) from all channels and threads
- **Images:** Attachments uploaded to all channels and threads
All data stored in SQLite database. Attachments uploaded to external picser service. Unified dashboard with separate tabs for each content type, filterable by channel/thread.
## Requirements
### Functional
1. **Text Message Capture**
- Capture new messages: content, author, channel, timestamp
- Capture edited messages: original + edited content, edit timestamp
- Capture deleted messages: content, author, deletion timestamp
- Store in database with full metadata
2. **Image/Attachment Capture**
- Detect attachments in messages
- Upload to `https://picser.asepharyana.tech/api/upload`
- Store `raw_commit` URL in database
- Store attachment metadata: filename, size, type, upload timestamp
3. **Voice Recording** (existing, no changes)
- Continue recording voice segments as-is
- Segments already stored in database via muxer queue
4. **Dashboard API**
- `/api/messages?channel=<id>&type=text|image|voice` — Query messages by type and channel
- `/api/channels` — List all monitored channels
- Real-time WebSocket updates: `message_created`, `message_updated`, `message_deleted`, `attachment_uploaded`
5. **Dashboard UI**
- Three tabs: Voice | Text | Images
- Channel/thread filter dropdown
- Display messages/attachments with metadata (author, timestamp, content)
- Real-time updates via WebSocket, polling fallback
### Non-Functional
- **Target Server:** Configured via `MONITOR_GUILD_ID` environment variable
- **Database:** Single SQLite (`.muxer-queue.db`), extended schema
- **Attachment Upload:** Async, non-blocking; store URL when ready
- **Real-time:** WebSocket for live updates, REST polling as fallback
- **Performance:** Index on channel_id, user_id, created_at for fast queries
## Architecture
### Database Schema
**New Tables:**
```sql
-- Text messages
CREATE TABLE messages (
id TEXT PRIMARY KEY,
guild_id TEXT NOT NULL,
channel_id TEXT NOT NULL,
thread_id TEXT,
user_id TEXT NOT NULL,
username TEXT NOT NULL,
avatar_url TEXT,
content TEXT NOT NULL,
edited_content TEXT,
created_at INTEGER NOT NULL,
edited_at INTEGER,
deleted_at INTEGER,
type TEXT NOT NULL DEFAULT 'text', -- 'text', 'edited', 'deleted'
metadata TEXT -- JSON: roles, etc.
);
CREATE INDEX idx_messages_channel ON messages(channel_id);
CREATE INDEX idx_messages_user ON messages(user_id);
CREATE INDEX idx_messages_created ON messages(created_at DESC);
CREATE INDEX idx_messages_thread ON messages(thread_id);
-- Attachments
CREATE TABLE attachments (
id TEXT PRIMARY KEY,
message_id TEXT NOT NULL,
guild_id TEXT NOT NULL,
channel_id TEXT NOT NULL,
user_id TEXT NOT NULL,
filename TEXT NOT NULL,
size INTEGER NOT NULL,
type TEXT NOT NULL, -- MIME type
discord_url TEXT NOT NULL,
uploaded_url TEXT, -- picser raw_commit URL
upload_status TEXT NOT NULL DEFAULT 'pending', -- 'pending', 'uploaded', 'failed'
upload_error TEXT,
created_at INTEGER NOT NULL,
uploaded_at INTEGER,
FOREIGN KEY (message_id) REFERENCES messages(id)
);
CREATE INDEX idx_attachments_channel ON attachments(channel_id);
CREATE INDEX idx_attachments_message ON attachments(message_id);
CREATE INDEX idx_attachments_status ON attachments(upload_status);
```
### Event Handlers
**New Discord Event Listeners:**
1. `messageCreate` — Insert into `messages` table
2. `messageUpdate` — Update `messages` table with edited content + timestamp
3. `messageDelete` — Mark message as deleted (soft delete with `deleted_at`)
4. `messageReactionAdd` — (Optional: track reactions)
**Attachment Processing:**
- On `messageCreate`: Extract attachments, insert into `attachments` table with `upload_status='pending'`
- Async job: Download from Discord URL, upload to picser, update `uploaded_url` and `upload_status`
- If upload fails: Set `upload_status='failed'`, store error message
### API Endpoints
**REST:**
```
GET /api/messages?channel=<id>&type=text|image|voice&limit=50&offset=0
→ Returns paginated messages/attachments
GET /api/channels
→ Returns list of all channels in monitored guild
GET /api/attachments?channel=<id>&limit=50
→ Returns attachments with upload status
```
**WebSocket Events (outbound):**
```json
{
"type": "message_created",
"data": { "id", "channel_id", "user_id", "username", "content", "created_at" }
}
{
"type": "message_updated",
"data": { "id", "edited_content", "edited_at" }
}
{
"type": "message_deleted",
"data": { "id", "deleted_at" }
}
{
"type": "attachment_uploaded",
"data": { "id", "message_id", "filename", "uploaded_url", "created_at" }
}
```
### File Structure
```
src/
├── moderation/
│ ├── messageCapture.ts -- Discord event listeners
│ ├── attachmentUploader.ts -- Upload to picser, manage queue
│ ├── messageStore.ts -- Database operations
│ └── types.ts -- Message/Attachment types
├── webserver.ts -- Add /api/messages, /api/channels endpoints
├── index.ts -- Register message event listeners
└── config.ts -- Add MONITOR_GUILD_ID
```
### Configuration
**New Environment Variables:**
```env
MONITOR_GUILD_ID=<guild-id> # Target server to monitor
PICSER_UPLOAD_URL=https://picser.asepharyana.tech/api/upload
ATTACHMENT_UPLOAD_TIMEOUT_MS=30000 # Upload timeout
ATTACHMENT_MAX_SIZE_MB=100 # Max file size to upload
```
## Implementation Phases
### Phase 1: Database & Core Capture
- Extend SQLite schema (messages, attachments tables)
- Implement message capture handlers (create/edit/delete)
- Add message store functions (insert, update, query)
### Phase 2: Attachment Upload
- Implement picser uploader with retry logic
- Add attachment processing queue
- Store URLs in database
### Phase 3: API & WebSocket
- Add REST endpoints for querying messages/attachments
- Add WebSocket events for real-time updates
- Implement channel listing
### Phase 4: Dashboard UI
- Build frontend with Voice | Text | Images tabs
- Implement channel filter
- Add real-time WebSocket listener + polling fallback
## Error Handling
- **Upload failures:** Retry with exponential backoff, store error in `upload_error` field
- **Database errors:** Log and continue (don't crash bot)
- **Missing attachments:** Handle Discord URL expiry gracefully
- **WebSocket disconnects:** Clients reconnect and poll for missed messages
## Testing
- Unit tests for message store functions (insert, update, query)
- Integration tests for attachment uploader (mock picser API)
- E2E tests for Discord event capture (mock Discord client)
## Success Criteria
- ✅ All text messages captured (new/edited/deleted)
- ✅ All attachments uploaded to picser with URLs stored
- ✅ Dashboard displays all three content types in separate tabs
- ✅ Channel filter works correctly
- ✅ Real-time WebSocket updates working
- ✅ Polling fallback works if WebSocket disconnects
- ✅ No data loss on bot restart
- ✅ Graceful handling of upload failures
## Future Enhancements
- Reaction tracking
- Message search/full-text search
- Moderation actions (flag, delete, mute)
- Export/archive functionality
- Retention policies (auto-delete old data)

View File

@@ -37,10 +37,10 @@
"devDependencies": {
"@biomejs/biome": "latest",
"@types/better-sqlite3": "^7.6.13",
"@types/bun": "latest",
"@types/express": "^5.0.6",
"@types/fluent-ffmpeg": "^2.1.28",
"@types/ws": "^8.18.1",
"bun-types": "latest",
"pino-pretty": "^10.3.1",
"vitest": "latest"
}

528
public/dashboard.html Normal file
View File

@@ -0,0 +1,528 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Moderation Dashboard</title>
<style>
* {
margin: 0;
padding: 0;
box-sizing: border-box;
}
body {
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, sans-serif;
background: #0f0f0f;
color: #e0e0e0;
padding: 20px;
}
.container {
max-width: 1400px;
margin: 0 auto;
}
header {
margin-bottom: 30px;
border-bottom: 2px solid #333;
padding-bottom: 20px;
}
h1 {
font-size: 28px;
margin-bottom: 10px;
color: #fff;
}
.status {
display: flex;
gap: 20px;
font-size: 14px;
color: #999;
}
.status-item {
display: flex;
align-items: center;
gap: 8px;
}
.status-dot {
width: 8px;
height: 8px;
border-radius: 50%;
background: #666;
}
.status-dot.connected {
background: #4ade80;
}
.controls {
display: flex;
gap: 15px;
margin-bottom: 20px;
align-items: center;
}
select {
padding: 8px 12px;
background: #1a1a1a;
border: 1px solid #333;
color: #e0e0e0;
border-radius: 4px;
font-size: 14px;
cursor: pointer;
}
select:hover {
border-color: #555;
}
.tabs {
display: flex;
gap: 10px;
margin-bottom: 20px;
border-bottom: 1px solid #333;
}
.tab {
padding: 12px 20px;
background: none;
border: none;
color: #999;
cursor: pointer;
font-size: 14px;
font-weight: 500;
border-bottom: 2px solid transparent;
transition: all 0.2s;
}
.tab:hover {
color: #e0e0e0;
}
.tab.active {
color: #fff;
border-bottom-color: #4ade80;
}
.content {
display: none;
}
.content.active {
display: block;
}
.message-list {
display: flex;
flex-direction: column;
gap: 12px;
}
.message-item {
background: #1a1a1a;
border: 1px solid #333;
border-radius: 6px;
padding: 15px;
transition: all 0.2s;
}
.message-item:hover {
border-color: #555;
background: #222;
}
.message-header {
display: flex;
align-items: center;
gap: 12px;
margin-bottom: 10px;
}
.avatar {
width: 32px;
height: 32px;
border-radius: 50%;
background: #333;
flex-shrink: 0;
}
.avatar img {
width: 100%;
height: 100%;
border-radius: 50%;
}
.user-info {
flex: 1;
}
.username {
font-weight: 600;
color: #fff;
font-size: 14px;
}
.timestamp {
font-size: 12px;
color: #666;
}
.message-content {
color: #e0e0e0;
font-size: 14px;
line-height: 1.5;
margin-bottom: 10px;
word-break: break-word;
}
.message-meta {
display: flex;
gap: 15px;
font-size: 12px;
color: #666;
}
.badge {
display: inline-block;
padding: 2px 8px;
background: #333;
border-radius: 3px;
font-size: 11px;
color: #999;
}
.badge.edited {
background: #4a3a00;
color: #ffd700;
}
.badge.deleted {
background: #3a0000;
color: #ff6b6b;
}
.image-grid {
display: grid;
grid-template-columns: repeat(auto-fill, minmax(200px, 1fr));
gap: 15px;
}
.image-item {
background: #1a1a1a;
border: 1px solid #333;
border-radius: 6px;
overflow: hidden;
transition: all 0.2s;
}
.image-item:hover {
border-color: #555;
}
.image-preview {
width: 100%;
height: 150px;
background: #0a0a0a;
display: flex;
align-items: center;
justify-content: center;
overflow: hidden;
}
.image-preview img {
width: 100%;
height: 100%;
object-fit: cover;
}
.image-info {
padding: 12px;
border-top: 1px solid #333;
}
.image-filename {
font-size: 12px;
color: #e0e0e0;
margin-bottom: 8px;
word-break: break-all;
}
.image-meta {
display: flex;
justify-content: space-between;
font-size: 11px;
color: #666;
}
.image-url {
color: #4ade80;
cursor: pointer;
text-decoration: none;
}
.image-url:hover {
text-decoration: underline;
}
.empty {
text-align: center;
padding: 40px;
color: #666;
}
.error {
background: #3a0000;
border: 1px solid #660000;
color: #ff6b6b;
padding: 12px;
border-radius: 4px;
margin-bottom: 20px;
}
</style>
</head>
<body>
<div class="container">
<header>
<h1>🛡️ Moderation Dashboard</h1>
<div class="status">
<div class="status-item">
<div class="status-dot" id="wsStatus"></div>
<span id="wsStatusText">Connecting...</span>
</div>
</div>
</header>
<div class="controls">
<label for="channelFilter">Filter by Channel:</label>
<select id="channelFilter">
<option value="">All Channels</option>
</select>
</div>
<div class="tabs">
<button class="tab active" data-tab="text">💬 Text Messages</button>
<button class="tab" data-tab="image">🖼️ Images</button>
<button class="tab" data-tab="voice">🎙️ Voice</button>
</div>
<div id="error" class="error" style="display: none;"></div>
<div id="text" class="content active">
<div class="message-list" id="textList"></div>
</div>
<div id="image" class="content">
<div class="image-grid" id="imageGrid"></div>
</div>
<div id="voice" class="content">
<div class="message-list" id="voiceList"></div>
</div>
</div>
<script>
const API_BASE = window.location.origin;
let ws = null;
let selectedChannel = '';
let messageCache = { text: [], image: [], voice: [] };
function connectWebSocket() {
const protocol = window.location.protocol === 'https:' ? 'wss:' : 'ws:';
ws = new WebSocket(`${protocol}//${window.location.host}/ws`);
ws.onopen = () => {
updateWSStatus(true);
console.log('WebSocket connected');
};
ws.onmessage = (event) => {
try {
if (typeof event.data === 'string') {
const message = JSON.parse(event.data);
handleWebSocketMessage(message);
}
} catch (e) {
console.error('Failed to parse WebSocket message:', e);
}
};
ws.onerror = (error) => {
console.error('WebSocket error:', error);
updateWSStatus(false);
};
ws.onclose = () => {
updateWSStatus(false);
setTimeout(connectWebSocket, 3000);
};
}
function updateWSStatus(connected) {
const dot = document.getElementById('wsStatus');
const text = document.getElementById('wsStatusText');
if (connected) {
dot.classList.add('connected');
text.textContent = 'Connected';
} else {
dot.classList.remove('connected');
text.textContent = 'Disconnected';
}
}
function handleWebSocketMessage(message) {
const { type, data } = message;
if (type === 'message_created') {
messageCache.text.unshift(data);
renderMessages();
} else if (type === 'message_updated') {
const msg = messageCache.text.find(m => m.id === data.id);
if (msg) {
msg.edited_content = data.edited_content;
msg.edited_at = data.edited_at;
}
renderMessages();
} else if (type === 'message_deleted') {
messageCache.text = messageCache.text.filter(m => m.id !== data.id);
renderMessages();
} else if (type === 'attachment_uploaded') {
messageCache.image.unshift(data);
renderImages();
}
}
async function fetchMessages() {
try {
const params = new URLSearchParams();
if (selectedChannel) params.append('channel', selectedChannel);
params.append('type', 'text');
params.append('limit', '50');
const response = await fetch(`${API_BASE}/api/messages?${params}`);
if (!response.ok) throw new Error(`HTTP ${response.status}`);
const result = await response.json();
messageCache.text = result.data || [];
renderMessages();
} catch (error) {
showError(`Failed to fetch messages: ${error.message}`);
}
}
async function fetchImages() {
try {
const params = new URLSearchParams();
if (selectedChannel) params.append('channel', selectedChannel);
params.append('type', 'image');
params.append('limit', '50');
const response = await fetch(`${API_BASE}/api/messages?${params}`);
if (!response.ok) throw new Error(`HTTP ${response.status}`);
const result = await response.json();
messageCache.image = result.data || [];
renderImages();
} catch (error) {
showError(`Failed to fetch images: ${error.message}`);
}
}
function renderMessages() {
const list = document.getElementById('textList');
if (messageCache.text.length === 0) {
list.innerHTML = '<div class="empty">No messages</div>';
return;
}
list.innerHTML = messageCache.text
.map(msg => `
<div class="message-item">
<div class="message-header">
<div class="avatar">
${msg.avatar_url ? `<img src="${msg.avatar_url}" alt="${msg.username}">` : ''}
</div>
<div class="user-info">
<div class="username">${escapeHtml(msg.username)}</div>
<div class="timestamp">${new Date(msg.created_at).toLocaleString()}</div>
</div>
</div>
<div class="message-content">${escapeHtml(msg.content)}</div>
<div class="message-meta">
${msg.type === 'edited' ? '<span class="badge edited">Edited</span>' : ''}
${msg.type === 'deleted' ? '<span class="badge deleted">Deleted</span>' : ''}
${msg.edited_at ? `<span class="badge">Edited at ${new Date(msg.edited_at).toLocaleString()}</span>` : ''}
</div>
</div>
`)
.join('');
}
function renderImages() {
const grid = document.getElementById('imageGrid');
if (messageCache.image.length === 0) {
grid.innerHTML = '<div class="empty">No images</div>';
return;
}
grid.innerHTML = messageCache.image
.map(img => `
<div class="image-item">
<div class="image-preview">
${img.uploaded_url ? `<img src="${img.uploaded_url}" alt="${img.filename}">` : '<span>Uploading...</span>'}
</div>
<div class="image-info">
<div class="image-filename">${escapeHtml(img.filename)}</div>
<div class="image-meta">
<span>${(img.size / 1024).toFixed(1)}KB</span>
${img.uploaded_url ? `<a href="${img.uploaded_url}" target="_blank" class="image-url">View</a>` : '<span>Pending</span>'}
</div>
</div>
</div>
`)
.join('');
}
function showError(message) {
const errorDiv = document.getElementById('error');
errorDiv.textContent = message;
errorDiv.style.display = 'block';
setTimeout(() => {
errorDiv.style.display = 'none';
}, 5000);
}
function escapeHtml(text) {
const div = document.createElement('div');
div.textContent = text;
return div.innerHTML;
}
document.querySelectorAll('.tab').forEach(tab => {
tab.addEventListener('click', () => {
document.querySelectorAll('.tab').forEach(t => t.classList.remove('active'));
document.querySelectorAll('.content').forEach(c => c.classList.remove('active'));
tab.classList.add('active');
const tabName = tab.dataset.tab;
document.getElementById(tabName).classList.add('active');
if (tabName === 'text') fetchMessages();
else if (tabName === 'image') fetchImages();
});
});
document.getElementById('channelFilter').addEventListener('change', (e) => {
selectedChannel = e.target.value;
fetchMessages();
fetchImages();
});
connectWebSocket();
fetchMessages();
</script>
</body>
</html>

File diff suppressed because it is too large Load Diff

View File

@@ -27,6 +27,13 @@ const configSchema = z.object({
NODE_ENV: z
.enum(["development", "production", "test"])
.default("development"),
MONITOR_GUILD_ID: z.string().min(1).optional(),
PICSER_UPLOAD_URL: z.string().url().default("https://picser.asepharyana.tech/api/upload"),
ATTACHMENT_UPLOAD_TIMEOUT_MS: z.coerce.number().positive().default(30000),
ATTACHMENT_MAX_SIZE_MB: z.coerce.number().positive().default(100),
ATTACHMENT_RETRY_ATTEMPTS: z.coerce.number().positive().default(3),
BACKLOG_SYNC_HOURS: z.coerce.number().positive().default(24),
BACKLOG_SYNC_BATCH_SIZE: z.coerce.number().int().positive().max(100).default(100),
});
export type AppConfig = z.infer<typeof configSchema>;

View File

@@ -8,16 +8,18 @@ import { createChildLogger } from "./logger";
import { discordPlayer } from "./player";
import { VoiceController } from "./voiceController";
import { startWebserver } from "./webserver";
import { registerMessageCapture } from "./moderation/messageCapture";
import { syncBacklogMessages } from "./moderation/backlogSync";
import { getDatabase } from "./muxer-queue";
const logger = createChildLogger("bot");
const token = config.DISCORD_TOKEN;
// Inisialisasi selfbot client
const client = new Client();
const voiceController = new VoiceController(client);
const db = getDatabase();
// Track shutdown state
let isShuttingDown = false;
async function gracefulShutdown(signal: string) {
@@ -30,15 +32,12 @@ async function gracefulShutdown(signal: string) {
logger.info({ signal }, "Graceful shutdown initiated");
try {
// Step 1: Stop voice connection
logger.info("Stopping voice connection...");
await voiceController.disconnect();
// Step 2: Pause player
logger.info("Pausing player...");
discordPlayer.pause();
// Step 3: Destroy client
logger.info("Destroying Discord client...");
try {
client.destroy();
@@ -56,6 +55,10 @@ async function gracefulShutdown(signal: string) {
client.on("ready", async () => {
logger.info({ user: client.user?.tag }, "Bot logged in");
registerMessageCapture(client, db);
syncBacklogMessages(client, db).catch((error) => {
logger.warn({ error }, "Backlog sync failed");
});
startWebserver(config.WEBSERVER_PORT, client, voiceController);
});
@@ -63,7 +66,6 @@ client.on("error", (err) => {
logger.error({ error: err }, "Client error");
});
// Graceful shutdown handlers
process.on("SIGINT", () => {
gracefulShutdown("SIGINT");
});
@@ -72,16 +74,15 @@ process.on("SIGTERM", () => {
gracefulShutdown("SIGTERM");
});
// Handle uncaught exceptions
process.on("uncaughtException", (err) => {
logger.error({ error: err }, "Uncaught exception");
gracefulShutdown("uncaughtException");
});
// Handle unhandled promise rejections
process.on("unhandledRejection", (reason, promise) => {
logger.error({ reason, promise }, "Unhandled rejection");
gracefulShutdown("unhandledRejection");
});
client.login(token);

View File

@@ -0,0 +1,136 @@
import { createChildLogger } from "../logger";
import { config } from "../config";
import { retryWithBackoff } from "../retry";
import type { SqliteDatabase } from "../muxer-queue";
import { updateAttachmentAsUploaded, updateAttachmentAsFailedUpload } from "./messageStore";
const logger = createChildLogger("attachment-uploader");
export interface PicserUploadResponse {
success: boolean;
filename: string;
urls: {
raw_commit?: string;
[key: string]: string | undefined;
};
size: number;
type: string;
}
export interface ParsedUploadResponse {
success: boolean;
url: string;
filename: string;
size: number;
type: string;
}
export function parseUploadResponse(response: PicserUploadResponse): ParsedUploadResponse {
if (!response.success) {
throw new Error("Upload failed: success=false");
}
const rawCommitUrl = response.urls.raw_commit;
if (!rawCommitUrl) {
throw new Error("Upload response missing raw_commit URL");
}
return {
success: true,
url: rawCommitUrl,
filename: response.filename,
size: response.size,
type: response.type,
};
}
export async function uploadAttachmentToPicser(
fileBuffer: Buffer,
filename: string,
): Promise<ParsedUploadResponse> {
const formData = new FormData();
const blob = new Blob([new Uint8Array(fileBuffer)], { type: "application/octet-stream" });
formData.append("file", blob, filename);
try {
const response = await retryWithBackoff(
async () => {
const res = await fetch(config.PICSER_UPLOAD_URL, {
method: "POST",
body: formData,
signal: AbortSignal.timeout(config.ATTACHMENT_UPLOAD_TIMEOUT_MS),
});
if (!res.ok) {
throw new Error(`Upload failed with status ${res.status}`);
}
return res.json() as Promise<PicserUploadResponse>;
},
{
retries: config.ATTACHMENT_RETRY_ATTEMPTS,
minTimeout: 1000,
maxTimeout: 5000,
logger,
},
);
const parsed = parseUploadResponse(response);
logger.info({ filename, url: parsed.url }, "Attachment uploaded successfully");
return parsed;
} catch (error) {
logger.error(
{ filename, error: error instanceof Error ? error.message : String(error) },
"Failed to upload attachment",
);
throw error;
}
}
export async function downloadDiscordAttachment(url: string): Promise<Buffer> {
try {
const response = await fetch(url, {
signal: AbortSignal.timeout(config.ATTACHMENT_UPLOAD_TIMEOUT_MS),
});
if (!response.ok) {
throw new Error(`Download failed with status ${response.status}`);
}
const buffer = await response.arrayBuffer();
return Buffer.from(buffer);
} catch (error) {
logger.error(
{ url, error: error instanceof Error ? error.message : String(error) },
"Failed to download Discord attachment",
);
throw error;
}
}
export async function processAttachmentUpload(
db: SqliteDatabase,
attachmentId: string,
discordUrl: string,
filename: string,
): Promise<void> {
try {
logger.info({ attachmentId, filename }, "Starting attachment upload");
const buffer = await downloadDiscordAttachment(discordUrl);
const sizeMb = buffer.length / (1024 * 1024);
if (sizeMb > config.ATTACHMENT_MAX_SIZE_MB) {
throw new Error(`File size ${sizeMb.toFixed(2)}MB exceeds limit of ${config.ATTACHMENT_MAX_SIZE_MB}MB`);
}
const result = await uploadAttachmentToPicser(buffer, filename);
updateAttachmentAsUploaded(db, attachmentId, result.url, Date.now());
logger.info({ attachmentId, uploadedUrl: result.url }, "Attachment upload completed");
} catch (error) {
const errorMsg = error instanceof Error ? error.message : String(error);
updateAttachmentAsFailedUpload(db, attachmentId, errorMsg);
logger.error({ attachmentId, error: errorMsg }, "Attachment upload failed");
}
}

View File

@@ -0,0 +1,120 @@
import type { Client, Message } from "discord.js-selfbot-v13";
import { config } from "../config";
import { createChildLogger } from "../logger";
import type { SqliteDatabase } from "../muxer-queue";
import { captureMessage } from "./messageCapture";
const logger = createChildLogger("backlog-sync");
function isWatchableChannel(channel: { type?: string; messages?: unknown }): boolean {
return Boolean(
channel.messages &&
["GUILD_TEXT", "GUILD_PUBLIC_THREAD", "GUILD_PRIVATE_THREAD"].includes(
channel.type ?? "",
),
);
}
async function collectWatchableChannels(guild: any): Promise<any[]> {
const channels: any[] = [];
for (const channel of guild.channels.cache.values()) {
if (isWatchableChannel(channel)) {
channels.push(channel);
}
if (channel.threads?.fetch) {
for (const archived of [false, true]) {
const fetched = await channel.threads
.fetch({ archived, limit: 100 })
.catch(() => null);
if (!fetched?.threads) continue;
for (const thread of fetched.threads.values()) {
if (isWatchableChannel(thread)) channels.push(thread);
}
}
}
}
return Array.from(new Map(channels.map((channel) => [channel.id, channel])).values());
}
async function syncChannelMessages(
db: SqliteDatabase,
channel: any,
cutoffTime: number,
): Promise<number> {
let before: string | undefined;
let synced = 0;
let shouldContinue = true;
while (shouldContinue) {
const batch = await channel.messages.fetch({
limit: config.BACKLOG_SYNC_BATCH_SIZE,
...(before ? { before } : {}),
});
if (batch.size === 0) break;
const messages = Array.from(batch.values()) as Message[];
for (const message of messages) {
if (message.author?.bot) continue;
if (message.createdTimestamp < cutoffTime) {
shouldContinue = false;
continue;
}
await captureMessage(db, message, "text");
synced++;
}
before = messages[messages.length - 1]?.id;
if (!before || batch.size < config.BACKLOG_SYNC_BATCH_SIZE) break;
}
return synced;
}
export async function syncBacklogMessages(
client: Client,
db: SqliteDatabase,
): Promise<void> {
if (!config.MONITOR_GUILD_ID) {
logger.warn("MONITOR_GUILD_ID not configured, skipping backlog sync");
return;
}
const guild = client.guilds.cache.get(config.MONITOR_GUILD_ID);
if (!guild) {
logger.warn({ guildId: config.MONITOR_GUILD_ID }, "Monitor guild not found, skipping backlog sync");
return;
}
const cutoffTime = Date.now() - config.BACKLOG_SYNC_HOURS * 60 * 60 * 1000;
await guild.channels.fetch().catch(() => null);
const channels = await collectWatchableChannels(guild);
let total = 0;
logger.info(
{ guildId: guild.id, channels: channels.length, hours: config.BACKLOG_SYNC_HOURS },
"Starting message backlog sync",
);
for (const channel of channels) {
try {
const count = await syncChannelMessages(db, channel as any, cutoffTime);
total += count;
logger.info({ channelId: channel.id, count }, "Backlog channel sync completed");
} catch (error) {
logger.warn(
{
channelId: channel.id,
error: error instanceof Error ? error.message : String(error),
},
"Backlog channel sync failed",
);
}
}
logger.info({ total }, "Message backlog sync completed");
}

View File

@@ -0,0 +1,176 @@
import type { Client, Message } from "discord.js-selfbot-v13";
import { createChildLogger } from "../logger";
import type { SqliteDatabase } from "../muxer-queue";
import { config } from "../config";
import { insertMessage, insertAttachment } from "./messageStore";
import { processAttachmentUpload } from "./attachmentUploader";
import { getDisplayContent, getMessageLocation, getMessageMetadata } from "./messageMetadata";
import type { MessageRecord, AttachmentRecord } from "./types";
const logger = createChildLogger("message-capture");
export async function captureMessage(
db: SqliteDatabase,
message: Message,
type: "text" | "edited" | "deleted",
): Promise<void> {
const location = getMessageLocation(message);
const metadata = getMessageMetadata(message);
const messageRecord: MessageRecord = {
id: message.id,
guild_id: message.guildId!,
channel_id: location.channelId,
thread_id: location.threadId,
user_id: message.author!.id,
username: message.author!.username,
avatar_url: message.author!.avatarURL() || null,
content: getDisplayContent(message),
edited_content: null,
created_at: message.createdTimestamp,
edited_at: null,
deleted_at: null,
type,
metadata: JSON.stringify(metadata),
};
insertMessage(db, messageRecord);
const broadcaster = globalThis as any;
if (broadcaster.broadcastMessageCreated) {
broadcaster.broadcastMessageCreated({
...messageRecord,
type: "text",
});
}
if (message.attachments.size > 0) {
for (const [, attachment] of message.attachments) {
const attachmentRecord: AttachmentRecord = {
id: attachment.id,
message_id: message.id,
guild_id: message.guildId!,
channel_id: location.channelId,
thread_id: location.threadId,
user_id: message.author!.id,
filename: attachment.name || "unknown",
size: attachment.size,
type: attachment.contentType || "application/octet-stream",
discord_url: attachment.url,
uploaded_url: null,
upload_status: "pending",
upload_error: null,
created_at: Date.now(),
uploaded_at: null,
};
insertAttachment(db, attachmentRecord);
processAttachmentUpload(db, attachment.id, attachment.url, attachment.name || "unknown")
.then(() => {
if (broadcaster.broadcastAttachmentUploaded) {
broadcaster.broadcastAttachmentUploaded({
id: attachment.id,
message_id: message.id,
filename: attachment.name || "unknown",
channel_id: location.channelId,
created_at: Date.now(),
});
}
})
.catch((error) => {
logger.error(
{ attachmentId: attachment.id, error: error instanceof Error ? error.message : String(error) },
"Background attachment upload failed",
);
});
}
}
logger.info(
{
messageId: message.id,
channelId: message.channelId,
attachmentCount: message.attachments.size,
},
"Message captured",
);
}
export function registerMessageCapture(client: Client, db: SqliteDatabase): void {
client.on("messageCreate", async (message) => {
if (!message.guildId || message.guildId !== config.MONITOR_GUILD_ID) return;
if (message.author?.bot) return;
try {
await captureMessage(db, message, "text");
} catch (error) {
logger.error(
{ messageId: message.id, error: error instanceof Error ? error.message : String(error) },
"Failed to capture message",
);
}
});
client.on("messageUpdate", async (_oldMessage, newMessage) => {
if (!newMessage.guildId || newMessage.guildId !== config.MONITOR_GUILD_ID) return;
if (newMessage.author?.bot) return;
try {
const { updateMessageAsEdited } = await import("./messageStore");
const existing = db
.prepare("SELECT id FROM messages WHERE id = ?")
.get(newMessage.id) as { id: string } | undefined;
if (existing) {
const editedAt = Date.now();
updateMessageAsEdited(db, newMessage.id, getDisplayContent(newMessage as Message), editedAt);
const broadcaster = globalThis as any;
if (broadcaster.broadcastMessageUpdated) {
broadcaster.broadcastMessageUpdated({
id: newMessage.id,
edited_content: getDisplayContent(newMessage as Message),
edited_at: editedAt,
});
}
} else if (newMessage.author) {
await captureMessage(db, newMessage as Message, "text");
}
} catch (error) {
logger.error(
{ messageId: newMessage.id, error: error instanceof Error ? error.message : String(error) },
"Failed to capture message update",
);
}
});
client.on("messageDelete", async (message) => {
if (!message.guildId || message.guildId !== config.MONITOR_GUILD_ID) return;
if (!message.author) return;
try {
const { updateMessageAsDeleted } = await import("./messageStore");
const deletedAt = Date.now();
updateMessageAsDeleted(db, message.id, deletedAt);
const broadcaster = globalThis as any;
if (broadcaster.broadcastMessageDeleted) {
broadcaster.broadcastMessageDeleted({
id: message.id,
deleted_at: deletedAt,
});
}
logger.info({ messageId: message.id }, "Message deletion captured");
} catch (error) {
logger.error(
{ messageId: message.id, error: error instanceof Error ? error.message : String(error) },
"Failed to capture message deletion",
);
}
});
logger.info("Message capture handlers registered");
}

View File

@@ -0,0 +1,167 @@
import type { Message, TextChannel, ThreadChannel } from "discord.js-selfbot-v13";
export interface MessageLocation {
channelId: string;
threadId: string | null;
threadName: string | null;
channelName: string | null;
}
export interface RichMessageMetadata {
stickers: Array<{ id: string; name: string; url: string; format: string | null }>;
embeds: Array<{
title: string | null;
description: string | null;
url: string | null;
color: number | null;
image: string | null;
thumbnail: string | null;
author: { name: string | null; url: string | null; iconURL: string | null } | null;
footer: { text: string | null; iconURL: string | null } | null;
fields: Array<{ name: string; value: string; inline: boolean }>;
}>;
attachments: Array<{
id: string;
name: string;
url: string;
contentType: string | null;
size: number;
}>;
author: {
id: string;
username: string;
tag: string | null;
avatarURL: string | null;
bot: boolean;
};
member: {
displayName: string | null;
roles: Array<{ id: string; name: string }>;
joinedTimestamp: number | null;
} | null;
channel: MessageLocation;
reference: {
messageId: string | null;
channelId: string | null;
guildId: string | null;
} | null;
}
export function getMessageLocation(message: Message): MessageLocation {
const channel = message.channel as TextChannel | ThreadChannel;
if (!channel.isThread?.()) {
return {
channelId: message.channelId,
threadId: null,
threadName: null,
channelName: "name" in channel ? channel.name : null,
};
}
return {
channelId: channel.parentId ?? message.channelId,
threadId: channel.id,
threadName: channel.name,
channelName: channel.parent?.name ?? null,
};
}
export function getStickerMetadata(message: Message): RichMessageMetadata["stickers"] {
return Array.from(message.stickers.values()).map((sticker) => ({
id: sticker.id,
name: sticker.name,
url: sticker.url,
format: sticker.format ?? null,
}));
}
export function getAttachmentMetadata(message: Message): RichMessageMetadata["attachments"] {
return Array.from(message.attachments.values()).map((attachment) => ({
id: attachment.id,
name: attachment.name || "unknown",
url: attachment.url,
contentType: attachment.contentType ?? null,
size: attachment.size,
}));
}
export function getEmbedMetadata(message: Message): RichMessageMetadata["embeds"] {
return message.embeds.map((embed) => ({
title: embed.title ?? null,
description: embed.description ?? null,
url: embed.url ?? null,
color: embed.color ?? null,
image: embed.image?.url ?? null,
thumbnail: embed.thumbnail?.url ?? null,
author: embed.author
? {
name: embed.author.name ?? null,
url: embed.author.url ?? null,
iconURL: embed.author.iconURL ?? null,
}
: null,
footer: embed.footer
? {
text: embed.footer.text ?? null,
iconURL: embed.footer.iconURL ?? null,
}
: null,
fields: embed.fields.map((field) => ({
name: field.name,
value: field.value,
inline: Boolean(field.inline),
})),
}));
}
export function getMessageMetadata(message: Message): RichMessageMetadata {
const member = message.member;
return {
stickers: getStickerMetadata(message),
embeds: getEmbedMetadata(message),
attachments: getAttachmentMetadata(message),
author: {
id: message.author.id,
username: message.author.username,
tag: "tag" in message.author ? message.author.tag : null,
avatarURL: message.author.avatarURL() ?? null,
bot: Boolean(message.author.bot),
},
member: member
? {
displayName: member.displayName ?? null,
roles: member.roles.cache.map((role) => ({ id: role.id, name: role.name })),
joinedTimestamp: member.joinedTimestamp ?? null,
}
: null,
channel: getMessageLocation(message),
reference: message.reference
? {
messageId: message.reference.messageId ?? null,
channelId: message.reference.channelId ?? null,
guildId: message.reference.guildId ?? null,
}
: null,
};
}
export function getDisplayContent(message: Message): string {
if (message.content.trim().length > 0) return message.content;
const stickers = getStickerMetadata(message);
if (stickers.length > 0) {
return stickers.map((sticker) => `[Sticker: ${sticker.name}]`).join(" ");
}
const attachments = getAttachmentMetadata(message);
if (attachments.length > 0) {
return attachments.map((attachment) => `[Attachment: ${attachment.name}]`).join(" ");
}
const embeds = getEmbedMetadata(message);
if (embeds.length > 0) {
return embeds.map((embed) => embed.title || embed.description || "[Embed]").join(" ");
}
return "";
}

View File

@@ -0,0 +1,222 @@
import { createChildLogger } from "../logger";
import type { SqliteDatabase } from "../muxer-queue";
import type { MessageRecord, AttachmentRecord } from "./types";
const logger = createChildLogger("message-store");
export function insertMessage(db: SqliteDatabase, message: MessageRecord): void {
try {
const stmt = db.prepare(`
INSERT OR IGNORE INTO messages (
id, guild_id, channel_id, thread_id, user_id, username, avatar_url,
content, edited_content, created_at, edited_at, deleted_at, type, metadata
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
`);
stmt.run(
message.id,
message.guild_id,
message.channel_id,
message.thread_id,
message.user_id,
message.username,
message.avatar_url,
message.content,
message.edited_content,
message.created_at,
message.edited_at,
message.deleted_at,
message.type,
message.metadata,
);
logger.debug({ messageId: message.id, channelId: message.channel_id }, "Message inserted");
} catch (error) {
logger.error(
{ messageId: message.id, error: error instanceof Error ? error.message : String(error) },
"Failed to insert message",
);
throw error;
}
}
export function updateMessageAsEdited(
db: SqliteDatabase,
messageId: string,
editedContent: string,
editedAt: number,
): void {
try {
const stmt = db.prepare(`
UPDATE messages
SET edited_content = ?, edited_at = ?, type = 'edited'
WHERE id = ?
`);
stmt.run(editedContent, editedAt, messageId);
logger.debug({ messageId }, "Message marked as edited");
} catch (error) {
logger.error(
{ messageId, error: error instanceof Error ? error.message : String(error) },
"Failed to update message as edited",
);
throw error;
}
}
export function updateMessageAsDeleted(
db: SqliteDatabase,
messageId: string,
deletedAt: number,
): void {
try {
const stmt = db.prepare(`
UPDATE messages
SET deleted_at = ?, type = 'deleted'
WHERE id = ?
`);
stmt.run(deletedAt, messageId);
logger.debug({ messageId }, "Message marked as deleted");
} catch (error) {
logger.error(
{ messageId, error: error instanceof Error ? error.message : String(error) },
"Failed to update message as deleted",
);
throw error;
}
}
export function getMessagesByChannel(
db: SqliteDatabase,
channelId: string,
limit: number = 50,
offset: number = 0,
): MessageRecord[] {
try {
const stmt = db.prepare(`
SELECT * FROM messages
WHERE channel_id = ? OR thread_id = ?
ORDER BY created_at DESC
LIMIT ? OFFSET ?
`);
const rows = stmt.all(channelId, channelId, limit, offset) as MessageRecord[];
return rows;
} catch (error) {
logger.error(
{ channelId, error: error instanceof Error ? error.message : String(error) },
"Failed to get messages by channel",
);
throw error;
}
}
export function insertAttachment(db: SqliteDatabase, attachment: AttachmentRecord): void {
try {
const stmt = db.prepare(`
INSERT OR IGNORE INTO attachments (
id, message_id, guild_id, channel_id, thread_id, user_id, filename, size, type,
discord_url, uploaded_url, upload_status, upload_error, created_at, uploaded_at
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
`);
stmt.run(
attachment.id,
attachment.message_id,
attachment.guild_id,
attachment.channel_id,
attachment.thread_id,
attachment.user_id,
attachment.filename,
attachment.size,
attachment.type,
attachment.discord_url,
attachment.uploaded_url,
attachment.upload_status,
attachment.upload_error,
attachment.created_at,
attachment.uploaded_at,
);
logger.debug({ attachmentId: attachment.id, messageId: attachment.message_id }, "Attachment inserted");
} catch (error) {
logger.error(
{ attachmentId: attachment.id, error: error instanceof Error ? error.message : String(error) },
"Failed to insert attachment",
);
throw error;
}
}
export function getAttachmentsByChannel(
db: SqliteDatabase,
channelId: string,
limit: number = 50,
offset: number = 0,
): AttachmentRecord[] {
try {
const stmt = db.prepare(`
SELECT * FROM attachments
WHERE channel_id = ? OR thread_id = ?
ORDER BY created_at DESC
LIMIT ? OFFSET ?
`);
const rows = stmt.all(channelId, channelId, limit, offset) as AttachmentRecord[];
return rows;
} catch (error) {
logger.error(
{ channelId, error: error instanceof Error ? error.message : String(error) },
"Failed to get attachments by channel",
);
throw error;
}
}
export function updateAttachmentAsUploaded(
db: SqliteDatabase,
attachmentId: string,
uploadedUrl: string,
uploadedAt: number,
): void {
try {
const stmt = db.prepare(`
UPDATE attachments
SET uploaded_url = ?, upload_status = 'uploaded', uploaded_at = ?
WHERE id = ?
`);
stmt.run(uploadedUrl, uploadedAt, attachmentId);
logger.debug({ attachmentId, uploadedUrl }, "Attachment marked as uploaded");
} catch (error) {
logger.error(
{ attachmentId, error: error instanceof Error ? error.message : String(error) },
"Failed to update attachment as uploaded",
);
throw error;
}
}
export function updateAttachmentAsFailedUpload(
db: SqliteDatabase,
attachmentId: string,
error: string,
): void {
try {
const stmt = db.prepare(`
UPDATE attachments
SET upload_status = 'failed', upload_error = ?
WHERE id = ?
`);
stmt.run(error, attachmentId);
logger.debug({ attachmentId, error }, "Attachment marked as failed upload");
} catch (error) {
logger.error(
{ attachmentId, error: error instanceof Error ? error.message : String(error) },
"Failed to update attachment as failed",
);
throw error;
}
}

56
src/moderation/types.ts Normal file
View File

@@ -0,0 +1,56 @@
export interface MessageRecord {
id: string;
guild_id: string;
channel_id: string;
thread_id: string | null;
user_id: string;
username: string;
avatar_url: string | null;
content: string;
edited_content: string | null;
created_at: number;
edited_at: number | null;
deleted_at: number | null;
type: "text" | "edited" | "deleted";
metadata: string | null;
}
export interface AttachmentRecord {
id: string;
message_id: string;
guild_id: string;
channel_id: string;
thread_id: string | null;
user_id: string;
filename: string;
size: number;
type: string;
discord_url: string;
uploaded_url: string | null;
upload_status: "pending" | "uploaded" | "failed";
upload_error: string | null;
created_at: number;
uploaded_at: number | null;
}
export interface VoiceSegmentRecord {
id: string;
user_id: string;
session_id: string;
guild_id: string;
channel_id: string;
filename: string;
duration_ms: number;
created_at: number;
}
export interface DashboardMessage {
id: string;
channel_id: string;
user_id: string;
username: string;
avatar_url: string | null;
content: string;
created_at: number;
type: "text" | "image" | "voice";
}

View File

@@ -1,9 +1,21 @@
import path from "node:path";
import Database from "better-sqlite3";
import { Database } from "bun:sqlite";
import { createChildLogger } from "./logger";
const logger = createChildLogger("muxer-queue");
export interface SqliteStatement {
run: (...params: unknown[]) => { changes: number };
all: (...params: unknown[]) => unknown[];
get: (...params: unknown[]) => unknown;
}
export interface SqliteDatabase {
prepare: (sql: string) => SqliteStatement;
exec: (sql: string) => void;
close: () => void;
}
export interface MuxerJobData {
userId: string;
sessionId: string;
@@ -23,13 +35,14 @@ interface StoredJob {
}
const dbPath = path.join(process.cwd(), ".muxer-queue.db");
let db: Database.Database | null = null;
let db: SqliteDatabase | null = null;
function initializeDatabase(): Database.Database {
const database = new Database(dbPath);
database.pragma("journal_mode = WAL");
function initializeDatabase(): SqliteDatabase {
const database = new Database(dbPath) as SqliteDatabase;
database.exec(`
PRAGMA journal_mode = WAL;
CREATE TABLE IF NOT EXISTS muxer_jobs (
id TEXT PRIMARY KEY,
data TEXT NOT NULL,
@@ -43,18 +56,71 @@ function initializeDatabase(): Database.Database {
CREATE INDEX IF NOT EXISTS idx_status ON muxer_jobs(status);
CREATE INDEX IF NOT EXISTS idx_createdAt ON muxer_jobs(createdAt);
CREATE TABLE IF NOT EXISTS messages (
id TEXT PRIMARY KEY,
guild_id TEXT NOT NULL,
channel_id TEXT NOT NULL,
thread_id TEXT,
user_id TEXT NOT NULL,
username TEXT NOT NULL,
avatar_url TEXT,
content TEXT NOT NULL,
edited_content TEXT,
created_at INTEGER NOT NULL,
edited_at INTEGER,
deleted_at INTEGER,
type TEXT NOT NULL DEFAULT 'text',
metadata TEXT
);
CREATE INDEX IF NOT EXISTS idx_messages_channel ON messages(channel_id);
CREATE INDEX IF NOT EXISTS idx_messages_user ON messages(user_id);
CREATE INDEX IF NOT EXISTS idx_messages_created ON messages(created_at DESC);
CREATE INDEX IF NOT EXISTS idx_messages_thread ON messages(thread_id);
CREATE TABLE IF NOT EXISTS attachments (
id TEXT PRIMARY KEY,
message_id TEXT NOT NULL,
guild_id TEXT NOT NULL,
channel_id TEXT NOT NULL,
thread_id TEXT,
user_id TEXT NOT NULL,
filename TEXT NOT NULL,
size INTEGER NOT NULL,
type TEXT NOT NULL,
discord_url TEXT NOT NULL,
uploaded_url TEXT,
upload_status TEXT NOT NULL DEFAULT 'pending',
upload_error TEXT,
created_at INTEGER NOT NULL,
uploaded_at INTEGER,
FOREIGN KEY (message_id) REFERENCES messages(id)
);
CREATE INDEX IF NOT EXISTS idx_attachments_channel ON attachments(channel_id);
CREATE INDEX IF NOT EXISTS idx_attachments_message ON attachments(message_id);
CREATE INDEX IF NOT EXISTS idx_attachments_status ON attachments(upload_status);
`);
try {
database.exec("ALTER TABLE attachments ADD COLUMN thread_id TEXT");
} catch {
// Column already exists on databases initialized after the moderation schema was added.
}
return database;
}
function getDatabase(): Database.Database {
function getDatabase(): SqliteDatabase {
if (!db) {
db = initializeDatabase();
}
return db;
}
export { getDatabase };
export async function enqueueMuxerJob(data: MuxerJobData): Promise<string> {
try {
const database = getDatabase();

View File

@@ -1,6 +1,34 @@
import { createRequire } from "node:module";
import prism from "prism-media";
import { config } from "../config";
const require = createRequire(import.meta.url);
interface OpusDecoderRuntime {
isBun: boolean;
canLoadNativeOpus: boolean;
}
export function shouldEnableDefaultOpusDecoder(
runtime: OpusDecoderRuntime,
): boolean {
return !runtime.isBun || runtime.canLoadNativeOpus;
}
function canLoadNativeOpus(): boolean {
try {
require("@discordjs/opus");
return true;
} catch {
return false;
}
}
const defaultDecoderEnabled = shouldEnableDefaultOpusDecoder({
isBun: Boolean(process.versions.bun),
canLoadNativeOpus: canLoadNativeOpus(),
});
export interface OpusDecoderOptions {
cooldownMs: number;
rotateMs: number;
@@ -23,8 +51,14 @@ export class OpusDecoder {
this.onData = options.onData;
this.createDecoderFn =
options.createDecoder ??
(() =>
new prism.opus.Decoder({
(() => {
if (!defaultDecoderEnabled) {
throw new Error(
"Native @discordjs/opus is unavailable under Bun; web PCM decode disabled to avoid opusscript aborts",
);
}
return new prism.opus.Decoder({
frameSize: config.OPUS_FRAME_SIZE,
channels: config.AUDIO_CHANNELS as 1 | 2,
rate: config.AUDIO_SAMPLE_RATE as
@@ -33,7 +67,8 @@ export class OpusDecoder {
| 16000
| 24000
| 48000,
}));
});
});
}
rotateIfNeeded(): void {

View File

@@ -25,6 +25,12 @@ export interface VoiceChannelSummary {
name: string;
}
export interface ChannelSummary {
id: string;
name: string;
type: string;
}
export class VoiceController {
private activeGuildId: string | null = null;
private activeChannelId: string | null = null;
@@ -63,6 +69,49 @@ export class VoiceController {
.sort((a, b) => a.name.localeCompare(b.name));
}
async listWatchableChannels(guildId: string): Promise<ChannelSummary[]> {
const guild = this.getGuild(guildId);
await guild.channels.fetch().catch(() => null);
return guild.channels.cache
.filter((channel) => channel.type === "GUILD_TEXT")
.map((channel) => ({
id: channel.id,
name: channel.name,
type: channel.type,
}))
.sort((a, b) => a.name.localeCompare(b.name));
}
async listThreads(guildId: string): Promise<ChannelSummary[]> {
const guild = this.getGuild(guildId);
await guild.channels.fetch().catch(() => null);
const threads: ChannelSummary[] = [];
for (const channel of guild.channels.cache.values()) {
const threadParent = channel as typeof channel & {
threads?: { fetch: (options: { archived: boolean; limit: number }) => Promise<any> };
};
if (!threadParent.threads?.fetch) continue;
for (const archived of [false, true]) {
const fetched = await threadParent.threads.fetch({ archived, limit: 100 }).catch(() => null);
if (!fetched?.threads) continue;
for (const thread of fetched.threads.values()) {
threads.push({
id: thread.id,
name: `${channel.name} / ${thread.name}`,
type: thread.type,
});
}
}
}
return Array.from(new Map(threads.map((thread) => [thread.id, thread])).values())
.sort((a, b) => a.name.localeCompare(b.name));
}
async connect(guildId: string, channelId: string): Promise<VoiceStatus> {
if (!this.client.isReady()) {
throw new AppError(

View File

@@ -11,6 +11,8 @@ import { createChildLogger, logger } from "./logger";
import { getMetrics, uptimeGauge } from "./metrics";
import { discordPlayer } from "./player";
import type { VoiceController } from "./voiceController";
import { getDatabase } from "./muxer-queue";
import { getMessagesByChannel, getAttachmentsByChannel } from "./moderation/messageStore";
const wsLogger = createChildLogger("webserver");
@@ -57,8 +59,12 @@ export function startWebserver(
const wss = new WebSocketServer({ server, path: wsPath });
wsLogger.info({ port, wsPath }, "WebSocket server listening");
// Security headers
app.use(helmet());
// Security headers. CSP disabled because the current static UI uses inline scripts/styles.
app.use(
helmet({
contentSecurityPolicy: false,
}),
);
// HTTP request logging
app.use(pinoHttp({ logger }));
@@ -100,6 +106,22 @@ export function startWebserver(
}
});
app.get("/api/guilds/:guildId/channels", async (req, res, next) => {
try {
res.json(await voiceController.listWatchableChannels(req.params.guildId));
} catch (error) {
next(error);
}
});
app.get("/api/guilds/:guildId/threads", async (req, res, next) => {
try {
res.json(await voiceController.listThreads(req.params.guildId));
} catch (error) {
next(error);
}
});
app.post("/api/connect", async (req, res, next) => {
try {
const { guildId, channelId } = req.body as {
@@ -129,6 +151,44 @@ export function startWebserver(
}
});
// Moderation API endpoints
app.get("/api/messages", async (req, res, next) => {
try {
const db = getDatabase();
const { channel, type, limit = "50", offset = "0" } = req.query as {
channel?: string;
type?: string;
limit?: string;
offset?: string;
};
if (!channel) {
throw new AppError("channel query parameter is required", "MISSING_CHANNEL", 400);
}
const limitNum = Math.min(parseInt(limit) || 50, 100);
const offsetNum = parseInt(offset) || 0;
if (type === "image") {
const attachments = getAttachmentsByChannel(db, channel, limitNum, offsetNum);
res.json({
type: "image",
data: attachments,
count: attachments.length,
});
} else {
const messages = getMessagesByChannel(db, channel, limitNum, offsetNum);
res.json({
type: "text",
data: messages,
count: messages.length,
});
}
} catch (error) {
next(error);
}
});
// Inbound: Discord PCM → tagged chunks → browser
(global as any).broadcastPcmToWeb = (chunk: Buffer, userId: string) => {
let hash = 0;
@@ -165,6 +225,33 @@ export function startWebserver(
});
}
function broadcastMessageEvent(type: string, data: any) {
const payload = JSON.stringify({
type,
data,
timestamp: Date.now(),
});
wsClients.forEach((client) => {
if (client.readyState === 1) client.send(payload);
});
}
(global as any).broadcastMessageCreated = (data: any) => {
broadcastMessageEvent("message_created", data);
};
(global as any).broadcastMessageUpdated = (data: any) => {
broadcastMessageEvent("message_updated", data);
};
(global as any).broadcastMessageDeleted = (data: any) => {
broadcastMessageEvent("message_deleted", data);
};
(global as any).broadcastAttachmentUploaded = (data: any) => {
broadcastMessageEvent("attachment_uploaded", data);
};
// --- Outbound: browser PCM (24kHz mono) → Opus → Discord ---
const RATE = 48000;
const CHANNELS = 2;

33
tests/decoder.test.ts Normal file
View File

@@ -0,0 +1,33 @@
import process from "node:process";
import { beforeEach, describe, expect, it, vi } from "vitest";
beforeEach(() => {
process.env = {
...process.env,
DISCORD_TOKEN: "token",
NODE_ENV: "test",
};
vi.resetModules();
});
describe("shouldEnableDefaultOpusDecoder", () => {
it("disables default decoder on Bun when native opus is unavailable", async () => {
const { shouldEnableDefaultOpusDecoder } = await import(
"../src/recorder/decoder"
);
expect(
shouldEnableDefaultOpusDecoder({ isBun: true, canLoadNativeOpus: false }),
).toBe(false);
});
it("enables default decoder when native opus is available", async () => {
const { shouldEnableDefaultOpusDecoder } = await import(
"../src/recorder/decoder"
);
expect(
shouldEnableDefaultOpusDecoder({ isBun: true, canLoadNativeOpus: true }),
).toBe(true);
});
});

View File

@@ -0,0 +1,46 @@
import { describe, it, expect, beforeEach } from "vitest";
beforeEach(() => {
process.env = {
...process.env,
DISCORD_TOKEN: "test-token",
MONITOR_GUILD_ID: "test-guild",
NODE_ENV: "test",
};
});
describe("attachmentUploader", () => {
it("parses picser upload response correctly", async () => {
const { parseUploadResponse } = await import("../../src/moderation/attachmentUploader");
const response = {
success: true,
filename: "uploads/abc123.jpg",
urls: {
raw_commit: "https://raw.githubusercontent.com/user/repo/commit/uploads/abc123.jpg",
},
size: 102400,
type: "image/jpeg",
};
const result = parseUploadResponse(response);
expect(result.success).toBe(true);
expect(result.url).toBe("https://raw.githubusercontent.com/user/repo/commit/uploads/abc123.jpg");
expect(result.filename).toBe("uploads/abc123.jpg");
});
it("handles upload response with missing raw_commit", async () => {
const { parseUploadResponse } = await import("../../src/moderation/attachmentUploader");
const response = {
success: true,
filename: "uploads/abc123.jpg",
urls: {},
size: 102400,
type: "image/jpeg",
};
expect(() => parseUploadResponse(response)).toThrow();
});
});

View File

@@ -6,6 +6,7 @@
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"types": ["bun-types"],
"outDir": "dist",
"rootDir": "src",
"experimentalDecorators": true,