Commit Graph
675 Commits
Author SHA1 Message Date
asepharyana 842610b1af fix(ai-moderation): bedah delay attachment ~330s -> target <20s
Root cause (trace msg 1541417073245290638):
- Race-guard upload-pending balik results:[] diperlakukan sbg SUKSES
  -> row yatam 'processing' sampai cleanup 300s mengembalikan
- Vision gagal 3x utk GIF besar (SSE truncation) tanpa fallback

Fix:
- Sinyal eksplisit uploadPending dari worker race guard
- Classifier murni classifyIndividualWorkerResult(): upload_pending ->
  requeue pending + reschedule segera (250ms), bukan error palsu;
  empty-results ok:true kini error transien (bug silent-success mati)
- llmVision fallback stream:false sekali saat SSE truncation
- Safety-net cleanup stuck processing 300s -> 120s
2026-08-24 22:05:28 +07:00
asepharyana fca96396b9 style(ai-moderation): sort import moderationOrchestrator (biome organizeImports) 2026-08-24 20:20:26 +07:00
asepharyana ccf3fa260e perf(ai-moderation): dual-key write-back — clean verdict ikut di-cache global
Analisis pertama tetap berkonteks (chat history) demi akurasi, tapi
verdict clean non-actionable (conf>=0.85) juga ditulis di bare key
tanpa konteks. Repeat teks sama di channel lain -> exact cache HIT,
bukan LLM call baru. Guard sama dgn read path; dedupe LRU per proses;
bare row tanpa embedding (tier semantic sudah global).
2026-08-24 20:18:14 +07:00
asepharyana 1accfd9390 perf(ai-moderation): naikkan cache hit dgn guard akurasi
- Fase-1 exact-cache lookup: N query serial -> SATU query ANY($1::text[])
- Global reuse utk bare key legacy, HANYA verdict non-actionable
  (clean/flagless/action=none, conf>=0.85, umur<=72h) — flagged/warn
  tetap context-scoped
- Semantic cache dua-band: clean band 0.92 default, actionable tetap
  0.97; di antara band -> LLM (fail-open ke akurasi)
- hit_count kini di-increment (bulk UPDATE per batch) -> hit-rate terukur
- Cache hasil wikipediaSearch di Redis (6h, hanya hasil non-kosong)
- Memoize fetchUrlSafely utk type=text (LRU 30m + in-flight dedupe)
- makeImageCacheKey strip query CDN Discord (?ex/is/hm, format/width)
  -> attachment sama = satu key vision, skip re-download+re-vision

Spec: .hermes/plans/2026-08-24-ai-analysis-cache-optimization.md
Tests: +33 (cacheGuards, discordImageKeyNormalize, cacheBatchLookup)
2026-08-24 18:51:23 +07:00
asepharyana 440ec41da8 polish(frontend): perkuat bracket & marker segitiga game-menu (nudge anim) 2026-08-24 17:52:50 +07:00
asepharyana 1c8c0ca081 fix(frontend): normalisasi trailing slash pada isActivePath agar nav aktif terdeteksi 2026-08-24 17:44:42 +07:00
asepharyana 5f42c17caa feat(frontend): tema monokrom + sidebar animasi game-menu + gate ringan mobile 2026-08-24 17:35:51 +07:00
asepharyana eda5c752b7 revert(frontend): kembalikan shell usable — hapus total eksperimen constellation (three/d3-force dihapus) 2026-08-24 16:32:16 +07:00
asepharyana 25d5097edb fix(frontend): definite-height overlay chain so absolute panels anchor to viewport 2026-08-24 16:15:55 +07:00
asepharyana d3e3b4764a fix(frontend): let empty overlay areas click through to the constellation sky 2026-08-24 16:02:57 +07:00
asepharyana 33a557c761 fix(frontend): soften constellation glow for light theme 2026-08-24 15:48:15 +07:00
asepharyana 32de2819df feat(frontend): a11y constellation mirror — sr/keyboard-accessible node list 2026-08-24 15:47:19 +07:00
asepharyana a21d252e9b feat(frontend): eased camera fly-to on scene transitions (reduced-motion aware) 2026-08-24 15:46:08 +07:00
asepharyana 84a766db0c feat(frontend): voice stage-orbit, media queue-spiral & recordings timeline-ring scenes 2026-08-24 15:44:10 +07:00
asepharyana 0581dc3485 feat(frontend): messages orbital-belt & moderation verdict-hub scenes 2026-08-24 15:35:51 +07:00
asepharyana 3d6c07bd91 feat(frontend): glossary satellite ring + analysis search console scenes 2026-08-24 15:29:21 +07:00
asepharyana 60ae1fb5c3 feat(frontend): dashboard & channels constellation scenes — publish bridge + floating overlays 2026-08-24 15:22:27 +07:00
asepharyana 1fafebb16d refactor(frontend): remove classic dashboard chrome (topbar/navrail/mobilenav) 2026-08-24 15:07:40 +07:00
asepharyana a9e09c38e9 feat(frontend): constellation stage + floating chrome replace classic shell 2026-08-24 15:04:31 +07:00
asepharyana 3d4236e8df feat(frontend): constellation lib scaffold — graph/layout/camera pure modules (tested) 2026-08-24 14:46:38 +07:00
asepharyana 16becd5340 perf(ai): kontiguitas batch budget + max_tokens dinamis + urutan kronologis RETURNING
- pickBatchWithinBudget: stop di overflow pertama (break), bukan skip —
  batch tetap prefix kronologis tanpa gap analisis di tengah timeline.
  Diekstrak ke batchBudget.ts (pure, estimator di-inject) + regression test.
- callModerationLLM: param opsional maxTokens; text/media caller menghitung
  ceiling dari estimasi prompt (floor 2048, cap 16384) — batch kecil tak
  lagi reserve window completion 16k.
- getPending/IncompleteMessagesByConversation: sort hasil UPDATE..RETURNING
  by created_at ASC — Postgres tak menjamin urutan, konsumen (anchor konteks
  messages[0], prefix batch) bergantung pada urutan kronologis.
2026-08-22 17:19:41 +07:00
asepharyana 1397380fe9 fix(ai): pertahankan status warn di cache moderasi + bersihkan prompt stale
- normalizeStoredStatus(): exact-hash & semantic (Qdrant/PG) cache reader
  sebelumnya menipiskan 'warn' jadi 'flagged'/'clean' (type narrowing
  legacy clean|flagged) — merusak gating auto-delete & label dashboard.
  Kini status tersimpan dipertahankan penuh (clean/warn/flagged).
- prompts: hapus referensi <user_history> yang tak pernah di-inject,
  SearXNG -> Wikipedia (sudah migrasi), referensi section yang tak ada,
  typo 'secifik', dan baris list rusak '|-'.
- moderationBuilders: buang dead code buildUserProfilesBlock/
  buildUserProfileRef/UserProfileEntry/buildUserHistoryXml (tanpa caller
  produksi sejak context minimization) + test-nya.
- test baru: tests/storedStatusNormalization.test.ts (regresi warn).
2026-08-22 16:17:55 +07:00
asepharyana 4ffc99b3fe fix(media): yt-dlp format fallback chain untuk direct-file URL
upload.asepharyana.my.id redirect ke file mp3 tunggal; generic extractor
yt-dlp expose format ID '0' sehingga '-f bestaudio' gagal 'Requested
format is not available'. Chain bestaudio[ext=m4a]/bestaudio/best tetap
dapat m4a di YouTube dan jatuh ke 'best' untuk direct file.
2026-08-22 14:41:53 +07:00
asepharyana 81ce5188ea fix(frontend): pindah aria-label mic meter ke wrapper role=status 2026-08-22 12:55:52 +07:00
asepharyana 4e0c21d86c feat(frontend): perbagus voice & audio playback UX
- Recordings: custom RecordingAudioPlayer (play/pause, buffering spinner,
  click-to-seek, time label, eq bars, single-playback antar kartu) +
  highlight kartu now-playing
- Media: thumbnail di disc hero + queue row, equalizer saat playing,
  badge 'up next', label Paused vs Now playing
- MiniPlayer global di AppFrame (fixed bottom-right, hidden on /media)
  menggantikan use-media-player.tsx dead provider (dihapus)
- Voice: mic level meter live (AnalyserNode RMS) + slider mic/listen volume
2026-08-22 12:53:18 +07:00
asepharyana df69b3f05d perf: optimasi rule moderasi — hapus redundansi di SYSTEM_RULES + OUTPUT_INSTRUCTIONS
Konsolidasi rule redundan yang banyak duplikat:

rules.ts:
- LGBT zero-tolerance: 3× (rule + dual-mode + pohon) → 1× di §LARANGAN BERAT, pohon cukup referensi
- Israel/Palestina/Yahudi: 2× (rule + pohon) → 1× di §LARANGAN BERAT, pohon referensi
- SARA agama: 6 sub-rules + ATURAN KRITIS → 1 paragraf konsolidat di §LARANGAN BERAT
- Pohon keputusan: 12 baris re-deskripsi panjang → 12 baris singkat dengan cross-reference ke §
- Evasi: 4 sumber (anti-evasion + foreign vulgar + zero-tolerance + acak/fragmentasi) → 1× + hierarki
- Aturan gambar: 7 baris tersecut → 7 bullet padat

output.ts:
- 3 larangan 'JANGAN PERNAH' untuk analysis generik → 1 larangan padat
- 6 contoh baik/buruk → format ✓/✗ kompak per kategori
- 7 CRITICAL bullet → 1 paragraf + 2 bullet

Token savings: ~206 tokens/call (rules.ts: 85, output.ts: 121)
All 117 tests pass. tsc clean.
2026-08-20 23:01:32 +07:00
asepharyana eee332412f chore: hapus fitur materi (learning materials + RAG chat)
Hapus fitur materi seluruhnya dari GMW monorepo:

Backend:
- Hapus module materi/ (index.ts, materi.repository.ts, materi.schema.ts,
  materi.service.ts, ragClient.ts)
- Hapus materiRouter dari orpc/router.ts (imports, const, appRouter entry)
- Hapus pgMateriDocumentsTable + types dari shared/database/schema.ts

Frontend:
- Hapus route pages app/(dashboard)/materi/ (page, [id], chat, new)
- Hapus lib/api/materi.ts (oRPC client wrappers)
- Hapus lib/types/materi.ts + export dari index.ts
- Hapus nav item "Materi" dari lib/navigation.ts + unused BookOpen import

Scripts:
- Hapus scripts/add-materi-documents.sql
- Tambah scripts/drop-materi-documents.sql (ops DB cleanup)

Verification:
- Backend: npx tsc --noEmit — clean (exit 0)
- Frontend: npx tsc --noEmit — clean (exit 0)
- Grep: zero materi code references remaining (hanya di drop-materi-documents.sql)

RAG dependencies (messages/embed.ts, messages/qdrant.ts) tetap karena juga
dipakai oleh messages.service.ts.
2026-08-20 22:50:11 +07:00
mytheclipsebotreview f750f39b50 fix(materi): type-cast tags as string[] to satisfy tsc (CI gate)
tsc reported 'Property some does not exist on type {}' on doc.tags
because Drizzle jsonb inference returns a generic object. Cast explicitly.
This unblocks the GMW GitHub Actions deploy (test job).
2026-08-20 19:18:24 +07:00
mytheclipsebotreview f1d90b6097 fix(materi): align tags column type to jsonb (schema ↔ migration mismatch)
The Drizzle schema used pgText('tags').array() which emits a Postgres
text[] column, but the migration defines tags as jsonb. The mismatch caused
INSERT/LIST on materi_documents to throw INTERNAL_SERVER_ERROR (500) because
Drizzle sent a text[] where the column expected jsonb.

- schema.ts: tags → pgJsonb('tags').notNull().default('[]')
- migration: dropped+recreated to match schema (jsonb, ms epoch defaults,
  owner_user_id default 'anonymous')
2026-08-20 19:08:32 +07:00
mytheclipsebotreview 7f4196124d fix(ci): lint and a11y fixes unblocks GHA deploy
- materi/new/page.tsx: add htmlFor+id pairs for all 5 form labels (a11y)
- biome --write --unsafe: fix useTemplate, useLiteralKeys, import sort
  across materi module files (backend + frontend)
- These pre-existing lint errors from 0aa893a blocked the deploy pipeline
2026-08-20 18:45:09 +07:00
mytheclipsebotreview 5658726ea5 fix(frontend): sort messages by created_at to fix WS race condition
Two independent WS handlers (useMessagesWsSync for message_created/
updated/analyzed, and useMessagesStream for message_snapshot) both
prepend live messages to the SWR list without enforcing order.
When frames arrive out-of-order (common with batched WS delivery),
the message feed gets scrambled.

Fix: add sortMessages() helper that sorts newest-first by created_at
(the list's stored order before .reverse() for display) and apply it
in every patchLists/mutate updater: message_created, message_updated,
message_analyzed, message_snapshot, and useLoadMore page appends.

Function declaration is hoisted so useLoadMore (defined above the
helper) can use it.
2026-08-20 18:16:43 +07:00
mytheclipsebotreview f5d5690401 fix: double-.js extension in @/ alias resolution (fix-imports.mjs)
The fix-imports.mjs script blindly appended '.js' to every @/ alias
import, even when the source specifier already carried a .js
extension (e.g. '@/shared/config/index.js'). This produced
'index.js.js' in the emitted dist/, causing ERR_MODULE_NOT_FOUND
at startup.

This was latent: only triggered once digestScheduler.ts (which
uses @/shared/config/index.js with explicit extension) was built.
The user-reputation removal (2a8f6d9) was also blocked by this
bug — stale binary kept crashing with 'user_reputations' query
errors because it was never redeployed.

Fix: only append .js when the @/ specifier has no existing
extension. Applied to both gateway and backend scripts.
2026-08-20 18:08:30 +07:00
asepharyana 0aa893ab7d feat: add Materi section + AI agent RAG to GMW business flow
Backend:
- New materi module: schema (materi_documents table), repository, service
- ragClient: semantic + keyword search over materi docs, plus Discord
  archive via Qdrant, then LLM answer generation (RAG)
- Wire materiRouter into appRouter (list/detail/create/update/delete/chat)

Frontend:
- New types (MateriDocument, CreateMateriInput, RAG chat shapes)
- API client (SSR HTTP RPCLink + browser WS RPCLink)
- Routes: /materi list, /materi/[id] detail, /materi/new form,
  /materi/chat RAG chat UI
- Sidebar nav item 'Materi'

Migration: scripts/add-materi-documents.sql (CREATE TABLE IF NOT EXISTS)
2026-08-20 15:57:04 +07:00
asepharyana 6f20b0f146 docs: clarify termGlossary uses Wikipedia (not SearXNG) for definition lookups
SearXNG was already replaced by Wikipedia REST/Action APIs (wikipediaClient.ts).
Update comments to reflect the current implementation: term glossary now
resolves definitions via Wikipedia → Redis → Postgres cache chain, with no
SearXNG dependency.
2026-08-20 15:32:20 +07:00
asepharyana 80248d4b7a feat: add 'screenshare' to MediaMode union for screen-share audio recording
Prepares the media type system to distinguish screenshare audio SSRCs
from mic voice SSRCs once the hookScreenShareAudio capture logic is
wired in.
2026-08-20 15:29:22 +07:00
asepharyana 20e991062c fix: screen-share audio capture — hook VoiceReceiver.onUdpMessage to discover unregistered SSRCs
Discord GoLive sends screen-share audio on a separate SSRC from the
user's microphone. In @discordjs/voice v0.19, VoiceReceiver.onUdpMessage
silently drops packets for SSRCs not in ssrcMap (which is only populated
from VOICE_STATE_UPDATE/VOICE_SERVER_UPDATE). This caused screen-share
audio to never trigger receiver.speaking and never reach the speakingHandler.

Fix: hookScreenShareAudio() wraps onUdpMessage to:
1. Detect incoming RTP packets with unknown SSRCs (OPRUS payload type 120)
2. Infer the owning userId by proximity to known audioSSRC
3. Clone the user's VoiceUserData into ssrcMap under the new SSRC
4. Let the original handler decrypt and forward to the subscription stream
5. Listen on ssrcMap 'create'/'update' events for video SSRC changes

Also removes the broken initial approach (polling ssrcMap which never
contains screen-share SSRCs).
2026-08-20 15:28:55 +07:00
asepharyana b784d6d796 feat(gateway): weekly moderation digest via WEBHOOK_URLS (#15)
Automated public weekly summary: top categories/domains/channels + coverage rate, posted to configured webhook. Uses getDatabase() direct query (no oRPC HTTP dependency), guards one-fire-per-week on restart.
2026-08-18 20:51:12 +07:00
asepharyana 00e8d68ce5 feat(gmw): public features #7-14 — scam domains, top channels, hourly heatmap, category drill-down, coverage stats, channel culture glossary, term KB, edit history
ALSO fixes: dashboard.repository still JOINed dropped user_reputations table (listUsers/getUserDetail crash).
2026-08-18 20:45:12 +07:00
asepharyana 2a8f6d9062 refactor(gateway): remove user reputation feature entirely
Drop trust-score/infraction system: delete userReputationStore, remove call sites in fallback/batch processors, drop formatReputationAttrs, drop user_reputations table (migration 0016), delete trust-model test, update docs.
2026-08-18 18:27:15 +07:00
asepharyana 9b3134d767 feat(gmw): public features #2-#6 — live moderation feed, toxic topic trends, channel timeline, CSV export, activity heatmap
- Live Moderation Feed: gateway publishes discord:moderation:action (Redis) → backend WS emits moderation_action → public web shows realtime stream.
- Toxic Topic Trends: backend moderation.trends aggregates categories/severity/action_type (read-only) → SVG bar + donut.
- Channel Timeline: messages view gets Feed/Timeline toggle with date-grouped separators.
- CSV Export: client-side downloadCsv for moderation actions (no backend write scope).
- Activity Heatmap: backend messages.activity (per-hour volume by channel) → pure-SVG grid.

User reputation deliberately excluded — no such feature exists in the codebase.
All read-only / public-facing / fully automatic per project rules.
2026-08-18 17:43:02 +07:00
asepharyana 36363fa3db fix(gateway): skip bot-only channel 1318544753821880362 from capture
Add to BOT_EXCLUDED_CHANNEL_IDS default alongside 1206269771340058694
so bot messages in that channel are no longer captured/analyzed/embedded.
2026-08-18 16:13:25 +07:00
asepharyana d133cc3271 style(gateway): sort imports in archiveEmbedder (biome) 2026-08-18 15:53:18 +07:00
asepharyana 5a70a685b4 fix(gateway): correct @/ alias import style (no .js) in archiveEmbedder
Gateway @/ alias imports use no .js extension (relative imports
keep .js). The .js suffix on @/ paths caused double-extension
ERR_MODULE_NOT_FOUND (embeddingClient.js.js) at runtime.
2026-08-18 15:44:18 +07:00
asepharyana 100b62800c fix(backend): drop .js extension on @/ alias imports (embed/qdrant)
Backend uses extensionless @/ alias imports; the double .js caused
ERR_MODULE_NOT_FOUND at runtime (index.js.js).
2026-08-18 15:19:05 +07:00
asepharyana 1ae19074ee feat(gmw): moderation explainability + semantic message search
- Persist structured verdict (flags/severity/confidence/evidence) on
  moderation_actions so the public web can show WHY a message was moderated.
- Add a persistent Qdrant archive collection (gmw_message_archive); embed
  every captured message at capture time (fire-and-forget, best-effort).
- Public semantic search over the archive (backend oRPC + FE toggle on the
  messages view). Both features are read-only/public and fully automatic.

Migration: 0015_add_moderation_explainability.sql
2026-08-18 15:11:01 +07:00
asepharyana d68f6b653a perf(ai-moderation): compact system prompt + memoize build + hoist vision pass
- Memoize buildSystemPrompt by (mode|channelCulture); identical signatures
  now reuse the ~5k-token core instead of rebuilding per sub-batch call
  (textBatchProcessor rebuilt it inside the loop; a 200-msg batch re-sent
  the full system prompt ~4x). Correction tail stays per-attempt (uncached).
- Hoist URL-image -> vision evidence out of the per-sub-batch loop in
  textBatchProcessor: it depends only on fetched images + full target set,
  so compute once per whole batch, not per sub-batch.
- Compact system instructions: collapse 3x-duplicated 'evaluate by content
  alone' statements into one standalone rule; trim output.ts channel-culture
  + context framing already covered by rules.ts/system.ts; drop duplicate
  programming-error-log few-shot (id 17, covered by rules AMAN list).
- Fix misleading config default: AI_LLM_BASE_URL default -> omniroute
  (gateway already runs omniroute via BWS; 9router was dead/misleading).

typecheck + lint + build green.
2026-08-18 11:49:39 +07:00
asepharyana 29baba3a72 fix(backend): sort vitest import in stream-many test (CI biome gate)
Reorder `describe, it, expect` → `describe, expect, it` to satisfy the
Biome organizeImports check that gates the all-services CI pipeline.
2026-08-18 11:03:24 +07:00
asepharyana 217ecc1aa1 feat(frontend): content micro-animations — stagger, button press, toast polish
- animate-stagger utility + staggerDelay() helper; lists now rise in sequence
  (recordings grid, moderation rows, message list, media queue, dashboard tiles).
- Button gets a subtle active:scale-[0.97] press feedback.
- Toaster: toast-in slide-up, tone-accent border, rounded hover-close target.
- All motion is reduced-motion aware (killed under prefers-reduced-motion).
2026-08-18 11:01:00 +07:00
asepharyana 0cb0b82fb1 feat(messages): stream history one-message-per-WS-frame instead of 50-row batch
- backend: add streamMany generator (paginated, yields one record at a time)
  + messagesService.streamMessages + WS 'stream_messages' handler emitting
  'message_snapshot' per message, 'message_snapshot_end' with nextCursor
- frontend: useMessagesStream hook accumulates snapshots into SWR list,
  SSR getMessages seeds first paint, WsHook gains sendText
- add stream-many.test.ts locking the one-at-a-time + cursor contract
2026-08-18 10:21:08 +07:00
asepharyana 95f2903067 feat(frontend): micro-interactions — page-enter transition + MetricTile lift
- Add animate-fade-up utility (reduced-motion aware) and a PageTransition
  wrapper; every dashboard view now rises + settles on mount/route change.
- MetricTile gains a subtle hover lift (-translate-y) + ring-focus glow.
- Theme toggle and command palette (⌘K) already existed and persist; no-op.
2026-08-18 09:50:52 +07:00