diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..51ac5731 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,179 @@ +# GMW — Agent Development Guide + +> **Rule #1: No code without a spec.** +> Setiap perubahan signifikan dimulai dari spec di `docs/specs/`. +> Baca spec → verifikasi facts → implement → verify → commit. +> Gunakan `todo.md` (TODO list terdokumentasi) untuk melacak progress. + +## Project overview + +GMW (Go Mod Watch) adalah Discord bot + dashboard untuk AI-powered moderation. +Monorepo berisi 3 service utama: + +| Service | Path | Port | Tech | +|---|---|---|---| +| **discord-gateway** | `services/discord-gateway/` | 4016 (metrics) | discord.js-selfbot-v13, Piscina, Drizzle ORM, pino | +| **backend** | `services/backend/` | 4001 | Express, oRPC, Drizzle ORM, Redis pub/sub, Vitest | +| **frontend** | `services/frontend/` | 4017 (standalone) | Next.js 16 App Router, React 19, Tailwind v4, SWR | + +## Data flow + +``` +Discord → discord-gateway → Redis pub/sub → backend (:4001) ←→ frontend Next.js SSR + ↑ REST /api/* + └ WS /ws +``` + +- Gateway = event-driven (no HTTP, except Prometheus :4016/metrics) +- Backend = HTTP + WebSocket server, serves the frontend +- Frontend = SSR (RSC) + client hydration, proxied by nginx :4009 + +## Konvensi dokumentasi: `docs/` + +Semua dokumen kerja ada di `docs/`, bukan di `.hermes/`: + +``` +docs/ +├── README.md # Panduan workflow (spec-driven + todo) +├── spec-template.md # Template spec standar +├── todo-template.md # Template todo list +└── specs/ # Semua spec & implementation plan + ├── YYYY-MM-DD_-spec.md # Spec (apa & mengapa) + └── YYYY-MM-DD_.md # Plan/fix (spec + plan dalam satu file) +``` + +Service-specific docs: `services//docs/specs/`. + +## Workflow: Spec-Driven Development + +### Checklist wajib untuk setiap perubahan signifikan + +1. **Tulis spec** → `docs/specs/YYYY-MM-DD_-spec.md` +2. **Tulis `todo.md`** → daftar task konkret yang bisa diceklis +3. **Verifikasi facts** → baca kode aktual, konfirmasi referensi file:line +4. **Keputusan desain** → pilih approach, dokumentasikan alternatif yang ditolak +5. **Implementasi** → ikuti spec + todo step-by-step +6. **Verifikasi** → jalankan semua verification steps dari spec +7. **Commit** → reference spec di commit message + +### Todo list (`todo.md`) + +Setiap tugas berjalan WAJIB punya `todo.md`. Format: + +```markdown +# Todo — + +## Task +- [ ] Tulis spec +- [ ] Verifikasi facts (baca kode: file:line) +- [ ] Implementasi tahap 1: ... +- [ ] Implementasi tahap 2: ... +- [ ] Verifikasi: pnpm typecheck / lint / build / test +- [ ] Commit + push +``` + +Aturan: +- Task harus **konkret & verifiable** — bukan "fix bug", tapi "ubah X di file Y". +- Ceklis `[x]` saat selesai, jangan menunggu batch di akhir. +- Gunakan `docs/todo-template.md` sebagai template. +- Simpan `todo.md` di `docs/` untuk tugas lintas-service, atau di + `services//docs/` untuk tugas satu service. + +### Spec template + +Salin `docs/spec-template.md` untuk setiap spec baru. Sections: + +1. **Problem** — apa yang rusak/missing, dengan evidence +2. **Root cause** — analisis teknis (bukan symptom) +3. **Behavior target** — perilaku setelah fix, daftar verifiable +4. **Verified facts** — fakta dari pembacaan kode, citation ke file:line +5. **Keputusan desain** — pilihan + rationale + alternatif ditolak +6. **Perubahan file** — semua file yang disentuh, per service +7. **Schema/type changes** — perubahan tipe/DB +8. **Verification** — command executable + expected outcome + +### Kapan perlu spec + todo.md + +Perlu: fitur baru, bug fix non-trivial, refactor behavior-changing, perubahan +DB schema, perubahan API contract, perubahan arsitektur. + +Tidak perlu: typo fix, dep bump, format/lint auto-fix, test-only, README update. + +Baca selengkapnya di `docs/README.md`. + +## Coding conventions + +### General + +- **TypeScript strict mode** — semua service +- **Biome** — formatting + linting (`pnpm format`, `pnpm lint`) +- **Bun** — package manager (`bun install`, `bun run`) +- **Bisa bilingual** — code comments & specs boleh Indonesia/English + +### Per-service conventions + +#### discord-gateway +- **Event-driven** — no HTTP server (except metrics). Listeners → Redis pub/sub. +- **Module pattern**: `src/modules//` — each module encapsulates own logic. +- **Piscina pools**: text pool (4 threads) + media pool (2 threads), each with own pg Pool. +- **Logger**: `createChildLogger('module-name')` — never raw `console`. +- **Config**: Zod-validated env in `shared/config/index.ts` — single source of truth. +- **DB**: Drizzle ORM. Migrations in `drizzle/migrations/`. +- **Invariant**: LLM is the only judge. Never reintroduce regex content classification. +- **Invariant**: Discord tokens sanitized before reaching LLM. +- Lihat `services/discord-gateway/AGENTS.md` untuk detail. + +#### backend +- **Modular MVC**: `modules//` — schema → repository → service → controller → routes. +- **No cross-module repo imports** — each module owns its data. +- **Data flows up only**: Repository → Service → Controller. +- **Error hierarchy**: `AppError` subclasses with code + statusCode. +- **Config**: Zod-validated env in `shared/config/index.ts`. +- **API**: oRPC for type-safe procedures + standard Express routes. +- Lihat `services/backend/AGENTS.md` untuk detail. + +#### frontend +- **SSR-first**: `page.tsx` (server component) → fetch via `src/lib/api/server.ts` → pass to `view.tsx` (client). +- **No auth**: all endpoints public. +- **Never hardcode host**: same-origin or `GMW_BACKEND_URL` only. +- **WebSocket**: `src/lib/ws/` — auto-reconnecting, typed events. +- **Local dev**: `NEXT_PUBLIC_API_URL`, `NEXT_PUBLIC_WS_URL`, `GMW_BACKEND_URL`. +- Lihat `services/frontend/AGENTS.md` untuk detail Next.js rules. + +## Build & Verify + +```bash +# Per service (run from service root) +pnpm typecheck # TypeScript strict +pnpm lint # Biome check +pnpm build # Compile +pnpm test # Vitest (gateway & backend only) +pnpm format # Biome auto-format +``` + +## Commit conventions + +``` +(): + + + +Ref: docs/specs/.md +``` + +Types: `feat`, `fix`, `refactor`, `test`, `docs`, `chore`, `build`, `ci` + +## Deployment + +CI/CD: GitHub Actions → build → deploy to production server via Nix flakes. +- Gateway: `nixos-rebuild` or `systemctl restart gmw-discord-gateway` +- Backend: `nixos-rebuild` or `systemctl restart gmw-backend` +- Frontend: Next.js standalone, proxied by nginx :4009 + +## Remember + +- Spec dulu, code belakangan. +- Todo list (`todo.md`) wajib untuk tugas yang berjalan — ceklis tiap selesai. +- Verified facts harus dari pembacaan kode aktual, bukan asumsi. +- Setiap change harus verifiable — tulis command di spec. +- One spec = one focused change. Don't mix unrelated features. \ No newline at end of file diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000..43c994c2 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1 @@ +@AGENTS.md diff --git a/ROADMAP.md b/ROADMAP.md new file mode 100644 index 00000000..f3f41ae0 --- /dev/null +++ b/ROADMAP.md @@ -0,0 +1,9 @@ +# Roadmap + +What is built, what is next, and what has been deliberately declined. + +--- + +## Next + +_Empty._ diff --git a/TODO.md b/TODO.md new file mode 100644 index 00000000..25628afa --- /dev/null +++ b/TODO.md @@ -0,0 +1,23 @@ +# TODO + +Next up. One item, one outcome, verifiable when done. + +Longer-term direction lives in [ROADMAP.md](ROADMAP.md). + +--- + +## Now + +_Empty._ + +--- + +## Next + +_Empty._ + +--- + +## Maintenance + +_Empty._ diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 00000000..3a9dd54a --- /dev/null +++ b/docs/README.md @@ -0,0 +1,131 @@ +# docs/ — Spec-Driven Development Hub + +Direktori ini adalah pusat dari workflow **spec-driven development** di GMW. +Semua perubahan signifikan dimulai dari satu spec, **sebelum kode ditulis**, +dan setiap tugas yang berjalan dilacak lewat `todo.md`. + +## Struktur + +``` +docs/ +├── README.md # Dokumen ini +├── spec-template.md # Template standar untuk semua spec +├── todo-template.md # Template todo list +└── specs/ # Semua spec dan implementation plan + ├── YYYY-MM-DD_-spec.md # Spec (apa & mengapa) + └── YYYY-MM-DD_.md # Plan/fix (bisa langsung spec+plan) +``` + +### Naming convention + +``` +YYYY-MM-DD_-spec.md # Spec baru untuk fitur/fix +YYYY-MM-DD_.md # Plan yang sudah include spec di dalamnya +``` + +Contoh: +- `2026-08-30_recordings-v2-features-spec.md` — spec untuk Recordings v2 +- `2026-08-24-attachment-delay-fix.md` — spec + plan untuk attachment delay fix + +### Service-specific docs + +``` +services//docs/specs/ # Spec yang spesifik untuk 1 service +``` + +## Workflow: Spec-Driven Development + +### Prinsip utama + +> **No code without a spec.** Semua perubahan signifikan harus punya spec +> terlebih dahulu. Spec adalah kontrak: apa yang akan dibangun, mengapa, +> dan bagaimana memverifikasinya. + +### Todo list (`todo.md`) — WAJIB + +Setiap tugas berjalan (implementasi fitur/fix) harus punya `todo.md`: + +```markdown +# Todo — + +## Task +- [ ] Tulis spec +- [ ] Verifikasi facts (baca kode: file:line) +- [ ] Implementasi tahap 1: ... +- [ ] Implementasi tahap 2: ... +- [ ] Verifikasi: pnpm typecheck / lint / build / test +- [ ] Commit + push +``` + +Aturan `todo.md`: +- Task **konkret & verifiable** — "ubah X di file Y", bukan "fix bug". +- Satu task `[in_progress]` pada satu waktu; ceklis `[x]` segera setelah selesai. +- Gunakan `docs/todo-template.md` sebagai template. +- Simpan `todo.md` di `docs/` (tugas lintas-service) atau di + `services//docs/` (tugas satu service). + +### Kapan perlu spec + todo + +| Perlu spec + todo | Tidak perlu | +|---|---| +| Fitur baru | Typo fix | +| Bug fix non-trivial | Dependency bump (dependabot) | +| Refactor yang mengubah behavior | Format/lint auto-fix | +| Perubahan DB schema | Test-only change | +| Perubahan API contract | README/doc update | +| Perubahan arsitektur | Variable rename | + +### Workflow step-by-step + +1. **Tulis spec** — Salin `docs/spec-template.md`, isi semua section yang relevan. + - **Verified facts**: baca kode yang terpengaruh, catat temuan dengan file:line. + - **Root cause**: analisis sebab (jangan hanya describe symptom). + - **Decisions**: pilih pendekatan, jelaskan alternatif yang ditolak. + - **Verification**: command executable, bukan "should work". + - Simpan sebagai `docs/specs/YYYY-MM-DD_-spec.md`. + +2. **Tulis `todo.md`** — Pecah spec jadi task konkret yang bisa diceklis + (gunakan `docs/todo-template.md`), dengan urutan implementasi yang logis. + +3. **Review spec** — Baca ulang spec sendiri. Cek: + - Apakah verified facts benar-benar verified (bukan asumsi)? + - Apakah file changes lengkap (tidak ada yang terlewat)? + - Apakah verification steps executable? + - Jika spec untuk user request: pastikan user setuju dengan approach. + +4. **Implement** — Ikuti spec + todo step-by-step. Ceklis `[x]` setiap task + yang selesai. Jika menemukan sesuatu yang berubah dari asumsi spec, + **update spec dulu**, baru implement. + +5. **Verify** — Jalankan semua verification steps di spec. Catat hasilnya. + +6. **Commit** — Reference spec di commit message: + ``` + feat(module): deskripsi singkat + + Ref: docs/specs/YYYY-MM-DD_-spec.md + ``` + +### Tips menulis spec yang baik + +- **Evidence-based**: setiap klaim harus ada sumbernya (file:line, log, error). +- **Actionable**: orang lain (atau AI agent) harus bisa implementasi dari spec saja. +- **Verifiable**: setiap requirement harus bisa di-test secara eksplisit. +- **Scoped**: satu spec = satu perubahan terfokus. Jangan campur 3 fitur dalam 1 spec. +- **Bahasa campuran**: narrative boleh Indonesia/English, istilah teknis pakai English. + +### Role of AI agents + +AI coding agents (Shiro Neko / Hermes) **harus**: +- Membaca spec sebelum menulis kode +- Membuat & meng-update `todo.md` untuk setiap tugas berjalan +- Verifikasi fakta dari kode aktual (bukan dari asumsi training data) +- Update spec jika temuan baru mengubah approach +- Jalankan verification steps setelah implementasi +- Reference spec di commit message + +AI agents **tidak boleh**: +- Langsung implement tanpa membaca/menulis spec +- Mengabaikan verified facts yang bertentangan dengan asumsi +- Skip verification steps +- Mengerjakan tugas tanpa todo list yang jelas \ No newline at end of file diff --git a/docs/spec-template.md b/docs/spec-template.md new file mode 100644 index 00000000..6838bccb --- /dev/null +++ b/docs/spec-template.md @@ -0,0 +1,80 @@ +# Spec: + +Status: **DRAFT** | **APPROVED** | **IN PROGRESS** | **DONE** +Date: YYYY-MM-DD +Author: +Related: +Todo: + +## Problem + + + +## Root cause + + + +## Behavior target + + + +## Verified facts + +>Fakta-fakta teknis yang **sudah diverifikasi** dari pembacaan kode, log produksi, +>atau eksperimen langsung. Setiap fakta harus citation ke file:line. +>Jika belum diverifikasi, tulis `UNVERIFIED` dan rencana verifikasi. + +- `path/to/file.ts:42` — +- `path/to/file.ts:88` — +- Kolom DB `table.column` — +- Config `ENV_VAR` default — + +## Keputusan desain + + + +1. ****: + - Alternatif yang ditolak: + +## Perubahan file + +>Daftar **semua file** yang perlu diubah/ditambah/dihapus, dikelompokkan per service. +>Untuk setiap file: sebutkan **apa yang berubah** (bukan copy-paste kode). + +### Gateway (`services/discord-gateway/`) +- `src/modules//.ts` — + +### Backend (`services/backend/`) +- `src/modules//.ts` — + +### Frontend (`services/frontend/`) +- `src//.ts|tsx` — + +### Database / Config +- + +## Schema/type changes + + + +## Verification + +>**Harus spesifik dan executable.** Jangan tulis "works correctly". +>Tulis command yang bisa dijalankan dan expected outcome. + +- **Typecheck**: ` pnpm typecheck` — clean +- **Lint**: ` pnpm lint` — no errors +- **Build**: ` pnpm build` — compiles +- **Test**: ` pnpm test` — all pass (+ test baru jika ada) +- **Smoke test**: +- **DB**: +- **Deploy**: CI green → deploy → cek + +## Notes (opsional) + + \ No newline at end of file diff --git a/docs/specs/2026-08-31_video-receive-dave-streamwatch-spec.md b/docs/specs/2026-08-31_video-receive-dave-streamwatch-spec.md index 5ea585eb..14aae0aa 100644 --- a/docs/specs/2026-08-31_video-receive-dave-streamwatch-spec.md +++ b/docs/specs/2026-08-31_video-receive-dave-streamwatch-spec.md @@ -3,8 +3,8 @@ Status: **P1–P3 DONE + 4th CRITICAL FIX deployed (f1a7b0c2); DAVE Ready + MLS handshake CONFIRMED live; P4 = waiting on active streamer to confirm video-burst→mp4** Date: 2026-08-31 Author: Hermes -Related: `.hermes/plans/2026-08-31_video-receive-eager-selfbot-connection-spec.md` (superseded by this) - `.hermes/plans/2026-08-31_video-receive-phaseC-spec.md` (Phase C build, selfbot path — dead) +Related: `docs/specs/2026-08-31_video-receive-eager-selfbot-connection-spec.md` (superseded by this) + `docs/specs/2026-08-31_video-receive-phaseC-spec.md` (Phase C build, selfbot path — dead) ## Problem / Ground truth (established from live logs 2026-08-31) GMW must record OTHER members' screen-share + camera video in a voice channel it diff --git a/docs/specs/2026-08-31_video-receive-eager-selfbot-connection-spec.md b/docs/specs/2026-08-31_video-receive-eager-selfbot-connection-spec.md index 864147b4..87f5bfba 100644 --- a/docs/specs/2026-08-31_video-receive-eager-selfbot-connection-spec.md +++ b/docs/specs/2026-08-31_video-receive-eager-selfbot-connection-spec.md @@ -3,7 +3,7 @@ Status: PLANNED Date: 2026-08-31 Author: Hermes -Related: `.hermes/plans/2026-08-31_video-receive-phaseC-spec.md` (Phase C build, made Option A this fix) +Related: `docs/specs/2026-08-31_video-receive-phaseC-spec.md` (Phase C build, made Option A this fix) ## Symptom (from live logs, 2026-08-31 ~12:34) A user was actively screen-sharing + on camera in the recorded voice channel. diff --git a/docs/specs/2026-08-31_video-receive-phaseC-spec.md b/docs/specs/2026-08-31_video-receive-phaseC-spec.md index f7acf566..e8d55159 100644 --- a/docs/specs/2026-08-31_video-receive-phaseC-spec.md +++ b/docs/specs/2026-08-31_video-receive-phaseC-spec.md @@ -3,7 +3,7 @@ Status: PLANNED (not built) Date: 2026-08-31 Author: Hermes -Related: `.hermes/plans/2026-08-30_video-record-receive-spec.md` (Phase A/B — raw UDP hook, superseded for receive) +Related: `docs/specs/2026-08-30_video-record-receive-spec.md` (Phase A/B — raw UDP hook, superseded for receive) ## TL;DR — what changed vs Phase A/B diff --git a/docs/specs/2026-09-02_selfbot-manual-video-watch-spec.md b/docs/specs/2026-09-02_selfbot-manual-video-watch-spec.md index f7efc84f..9838a1d0 100644 --- a/docs/specs/2026-09-02_selfbot-manual-video-watch-spec.md +++ b/docs/specs/2026-09-02_selfbot-manual-video-watch-spec.md @@ -3,7 +3,7 @@ Status: PLANNED (not yet built) Date: 2026-09-02 Author: Hermes -Related: `.hermes/plans/2026-08-31_video-receive-phaseC-spec.md` (auto-receive, superseded +Related: `docs/specs/2026-08-31_video-receive-phaseC-spec.md` (auto-receive, superseded for selfbot), `gmw-ops/references/selfbot-presence-detection-limits.md`, `gmw-ops/references/discord-voice-fork-video-receive.md` diff --git a/docs/todo-template.md b/docs/todo-template.md new file mode 100644 index 00000000..a01f8865 --- /dev/null +++ b/docs/todo-template.md @@ -0,0 +1,27 @@ +# Todo — + +Status: **IN PROGRESS** | **BLOCKED** | **DONE** +Spec: -spec.md`> +Date: YYYY-MM-DD + +## Task + +> Urutan implementasi sesuai spec. Task harus konkret & verifiable +> ("ubah X di file Y"), bukan "fix bug". Satu task dikerjakan pada satu +> waktu; ceklis `[x]` segera setelah selesai. + +- [ ] Tulis spec +- [ ] Verifikasi facts (baca kode: `file:line`) +- [ ] Implementasi tahap 1: +- [ ] Implementasi tahap 2: +- [ ] Implementasi tahap 3: +- [ ] Typecheck: `pnpm typecheck` — clean +- [ ] Lint: `pnpm lint` — no errors +- [ ] Test: `pnpm test` — all pass +- [ ] Build: `pnpm build` — compiles +- [ ] Daftar ulang hasil verifikasi di spec +- [ ] Commit + push (reference spec di commit message) + +## Notes + + \ No newline at end of file diff --git a/services/backend/AGENTS.md b/services/backend/AGENTS.md new file mode 100644 index 00000000..688ef744 --- /dev/null +++ b/services/backend/AGENTS.md @@ -0,0 +1,117 @@ +# Backend Service — Agent Guide + +> **Read `../../AGENTS.md` first.** This file adds backend-specific conventions. + +Backend service: Express HTTP server + WebSocket, serves the GMW dashboard API. + +## Quick reference + +```bash +pnpm typecheck # tsc --noEmit +pnpm lint # biome check --diagnostic-level=error . +pnpm build # tsc +pnpm test # vitest run +pnpm format # biome format --write . +``` + +## Architecture (Modular MVC) + +``` +src/ +├── shared/ # Infrastructure (no business logic) +│ ├── config/index.ts # Zod-validated env +│ ├── database/ # Drizzle ORM + pg Pool +│ ├── errors/index.ts # AppError hierarchy +│ ├── logger/index.ts # pino + createChildLogger() +│ ├── middlewares/index.ts # errorHandler, asyncHandler, rateLimit +│ └── utils/ # Pagination, messageMapper +├── modules/ # Feature modules +│ └── / +│ ├── .schema.ts # Zod validation schemas +│ ├── .repository.ts # DB operations only +│ ├── .service.ts # Business logic +│ ├── .controller.ts # HTTP handlers +│ └── routes/index.ts # Express router +├── http/ # app.ts (factory) + server.ts (startup) +├── ws/ # WebSocket server + Redis bridge +└── index.ts # Entry point +``` + +## Dependency rules + +- Controller → Service → Repository → Database +- No cross-module repo imports (each module owns its data) +- No HTTP in Service layer (no req/res) +- No DB in Controller layer +- Any layer → Config, Logger, Errors + +## API: oRPC + Express + +- **oRPC** (`src/orpc/router.ts` + `src/orpc/ws.ts`): type-safe procedures for frontend +- **Express routes** (`src/modules/*/routes/`): REST endpoints under `/api/` +- **WebSocket** (`src/ws/`): Redis bridge → broadcast to connected browsers +- Never invent endpoints. Match what the frontend calls (`src/lib/api/`). + +## Key modules + +| Module | Purpose | DB tables | +|---|---|---| +| messages | Store & query Discord messages | `messages`, `ai_moderations`, `ai_moderation_flags` | +| moderation | Moderation actions & metrics | `ai_moderations`, `moderation_actions` | +| media | Media file management | `media_attachments` | +| voice | Live speakers + recordings | `voice_recordings` | +| recordings | Recordings API | `voice_recordings` | +| dashboard | Stats aggregation | Various (read-only) | +| knowledge | Semantic search | Qdrant vector DB | +| chatbot | AI chatbot with tools | `chatbot_history` | +| health | Health checks + metrics | Various | +| analysis | Text analysis cache | `text_analysis_cache` | +| ui-state | Persist UI preferences | `ui_state` | + +## Config (env vars) + +All validated via Zod in `shared/config/index.ts`. Key vars: + +- `WEBSERVER_PORT` (default 4001) +- `DATABASE_URL` or individual `DATABASE_HOST/PORT/NAME/USER/PASSWORD` +- `REDIS_URL` — for pub/sub with gateway +- `MONITOR_GUILD_ID` — primary Discord guild +- `ADMIN_PASSWORD` — admin endpoints + +## Data contract with frontend + +The frontend fetches via `src/lib/api/server.ts` (SSR, server-side) and +`src/lib/api/client.ts` (browser, same-origin through proxy). + +**Do not change response shapes without updating both sides.** Check +`services/frontend/src/lib/types/` for frontend type definitions. + +## Redis channels (inbound from gateway) + +``` +discord:message:{created,updated,deleted,analyzed} +discord:attachment:{created,uploaded} +discord:voice:{started,stopped,uploaded,active_user,pcm,analyzed} +discord:analysis:queue_status +discord:reaction:{added,removed} +discord:thread:{created,deleted,updated} +discord:channel_topic:updated +discord:presence:updated +discord:guild_member:{added,removed} +``` + +Canonical names: `src/shared/redis-channels.ts`. + +## Testing + +- Vitest for unit tests +- Mock database and external services +- Test files: `src/modules//*.test.ts` or `tests/*.test.ts` +- Run: `pnpm test` + +## Common pitfalls + +- **oRPC vs REST**: check both routers when adding an endpoint +- **Redis channel mismatch**: gateway publishes → backend subscribes. Channel + names must match exactly (see `redis-channels.ts` in BOTH services) +- **DB pool**: backend uses one pool (main thread). Gateway has per-piscina-thread pools. diff --git a/services/discord-gateway/AGENTS.md b/services/discord-gateway/AGENTS.md new file mode 100644 index 00000000..876be841 --- /dev/null +++ b/services/discord-gateway/AGENTS.md @@ -0,0 +1,153 @@ +# Discord Gateway — Agent Guide + +> **Read `../../AGENTS.md` first.** This file adds gateway-specific conventions. + +Event-driven microservice: captures Discord events, runs AI moderation, publishes to Redis. + +## Quick reference + +```bash +pnpm typecheck # tsc --noEmit +pnpm lint # biome check --diagnostic-level=error . +pnpm build # tsc +pnpm test # vitest run +pnpm format # biome format --write . +``` + +## Architecture (Event-driven) + +``` +src/ +├── index.ts # Entry point → initializeDiscordGateway() +├── app/ +│ ├── bootstrap.ts # Wires client, DB, Redis, workers, schedulers +│ ├── shutdown.ts # Graceful shutdown +│ └── retention.ts # Expired-record cleanup +├── shared/ +│ ├── config/index.ts # Zod-validated env (SINGLE source of truth) +│ ├── database/ # Drizzle ORM + pg Pool + migrations +│ ├── logger/index.ts # pino + createChildLogger() +│ ├── errors/index.ts # AppError hierarchy +│ ├── utils/ # retry, pagination +│ ├── discord/clientOptions.ts # discord.js-selfbot-v13 options +│ ├── uploader.ts # Attachment upload helper +│ ├── redis-channels.ts # Redis channel-name constants +│ └── moderation-types.ts # Shared AI analysis types +├── modules/ +│ ├── ai-moderation/ # LLM moderation pipeline (largest module) +│ ├── message-capture/ # Discord event listeners + DB store +│ ├── voice-recording/ # Voice connect + Opus→OGG recording +│ │ └── recorder/ # decoder, segment, session, uploader, oggCrc +│ ├── voice-pcm-ws/ # Real-time PCM → backend WebSocket +│ ├── attachment-upload/ # Download + sharp resize + upload +│ ├── event-broadcaster/ # Redis pub/sub publisher +│ ├── command-handler/ # Backend→gateway Redis commands +│ ├── reaction-tracking/ # Reaction events +│ ├── thread-tracking/ # Thread events +│ ├── user-presence/ # Presence/status events +│ ├── channel-topic/ # Channel topic events +│ ├── guild-member-events/ # Member join/leave +│ └── gateway-metrics/ # Prometheus /metrics (port 4016) +└── tests/ # Vitest suites +``` + +## Key invariants (DO NOT BREAK) + +1. **LLM is the only judge.** Failed LLM → `status:"error"` + recovery retry. + **Never** reintroduce regex/heuristic content classification. +2. **Discord tokens sanitized** before reaching LLM (`discordTokens.ts`). +3. **Semantic cache is batched** — one embed call + one Qdrant batch search. +4. **Streaming is mandatory** against the omniroute base URL. + +## AI moderation pipeline + +``` +aiAnalyzer.ts → batchScheduler.ts → batchProcessor.ts → individualFallbackProcessor.ts + ↓ ↓ ↓ ↓ +moderationOrchestrator.ts → (hash cache → Qdrant → LLM) + ↓ ↓ ↓ +textBatchProcessor.ts mediaBatchProcessor.ts llmClient.ts + embeddingClient.ts + qdrantClient.ts +``` + +- Entry: `aiAnalyzer.ts` (`queueMessageAnalysis`, `startPendingAIAnalysisWorker`) +- Concurrency: LLM semaphore (`AI_LLM_MAX_CONCURRENT`, default 5) +- Piscina: text pool (4 threads) + media pool (2 threads) +- **Each worker thread has its own pg Pool** (min 0, grows to `POSTGRES_POOL_MAX`) + +## Voice recording pipeline + +``` +receiver.speaking "start" → speakingHandler(userId) + → collectUserMetadata → receiver.subscribe → PacketFilter → oggPacketStream + → SegmentManager.open → OggLogicalBitstream → .ogg file + → data: rotateIfNeeded + decoder.write + → end: finalizeSegment → upload + transcribe +``` + +Key files: +- `voiceController.ts` — connect/disconnect/list +- `recorder.ts` — orchestration +- `recorder/segment.ts` — segment rotation +- `recorder/sessionRecording.ts` — session management +- `recorder/uploader.ts` — upload to storage +- `voiceTranscriber.ts` — Whisper transcription (if enabled) + +## Module: message-capture + +- `messageCapture.ts` — Discord event listeners (messageCreate/Update/Delete) +- `messageStore.ts` — DB operations +- `messageMetadata.ts` — metadata extraction +- `messagesDb.ts` / `messagesCrud.ts` — DB schema operations +- `archiveEmbedder.ts` — Qdrant embedding (respect age-restricted guard) +- `retentionDb.ts` / `reviewsDb.ts` / `attachmentsDb.ts` — auxiliary tables + +## Redis channels (outbound to backend) + +See `src/shared/redis-channels.ts` for canonical names. Examples: +``` +discord:message:created, discord:voice:active_user, discord:attachment:uploaded +``` + +## Config (env vars) + +All validated via Zod in `shared/config/index.ts`. Critical: + +- `DISCORD_TOKEN` — selfbot token +- `MONITOR_GUILD_ID` — primary guild +- `DATABASE_URL` — PostgreSQL +- `REDIS_URL` — pub/sub to backend +- `AI_ANALYSIS_ENABLED` — master toggle for AI moderation +- `AI_LLM_BASE_URL` / `AI_LLM_API_KEY` — LLM router +- `AI_VOICE_TRANSCRIPTION_ENABLED` — toggle Whisper transcription +- `PISCINA_MAX_THREADS` / `PISCINA_MEDIA_MAX_THREADS` — worker pool sizing + +## Concurrency model + +- Main thread: event loop + LLM semaphore +- Text Piscina pool: `PISCINA_MAX_THREADS` (default 4) +- Media Piscina pool: `PISCINA_MEDIA_MAX_THREADS` (default 2) +- Batch routed to media pool if ANY message has attachment/sticker/embed +- Each worker thread initializes own pg Pool (min 0, grows to `POSTGRES_POOL_MAX`) +- MemoryMax: 1G (service systemd limit) + +## Testing + +- Vitest. Test files: `tests/.test.ts` +- Run: `pnpm test` +- Key test areas: batch operations, cache guards, image/video handling, context enrichment +- Unit tests for pure helpers (classifier, normalize, parse), integration for pipeline stages + +## Common pitfalls + +- **Piscina pool isolation**: worker threads are NOT the main thread. Cannot + share state via module-level variables. Use DB or Redis for cross-thread state. +- **AfterSilence race**: `@discordjs/voice` AfterSilence can fail to emit "end" + on disconnect. Always have a watchdog/timeout. +- **Cache eviction**: LRU caches (user metadata, term glossary) evict at max size. + Don't assume cache hit after eviction. +- **Circuit breaker**: per-conversation CB opens after repeated failures. + Check `circuitBreaker.ts` state when debugging "missing analysis". +- **Gateway≠Backend schema**: both have `redis-channels.ts` and `moderation-types.ts`. + Keep them in sync manually. diff --git a/services/frontend/AGENTS.md b/services/frontend/AGENTS.md index 646f98ab..441fd751 100644 --- a/services/frontend/AGENTS.md +++ b/services/frontend/AGENTS.md @@ -1,3 +1,5 @@ +@../../AGENTS.md + # Bete Frontend — Project Overview Next.js 16 (App Router), React 19, TypeScript strict, Tailwind v4, shadcn/ui, base-ui.