3 Commits

Include renames

Author	SHA1	Message	Date
asepharyana	7ef86c81ca	feat(ai): audit + harden embedding pipeline ... - Normalize text before embedding (strip mentions/URLs/emoji/markdown/control chars, lowercase, truncate) on both write and query sides so vectors aren't diluted and tokens aren't wasted - embeddingClient: retry embeddings (maxRetries 2), validate batch dimension consistency, preserve index alignment for empty-normalized texts - archiveEmbedder: store normalized text in archive payload, skip empty-normalized content - backend: normalize search queries, make archive search similarity threshold configurable (AI_LLM_EMBEDDING_ARCHIVE_MIN_SIMILARITY, default 0.6)	2026-09-01 18:01:47 +07:00
Developer	7ab9a7fd2d	fix(automod): force float encoding for embeddings — Nvidia models reject base64 ... OpenAI SDK v6 defaults to encoding_format=base64; llama-nemotron-embed (Nvidia-backed) returns 400 'do not support base64'. Semantic cache was silently disabled in prod. encoding_format: 'float' fixes it.	2026-07-31 20:14:18 +07:00
Developer	1249ae81d8	perf(automod): compress prompts ~40% + semantic cache via AI_LLM_EMBEDDING_MODEL ... Prompt overhaul (token-frugal, same quality): - rules.ts 28KB -> 10.3KB: every normative rule kept (safe lists, SARA 6 kategori, LGBT/Israel zero tolerance, anti-evasion, decision tree, evasi hierarchy, image rules) with duplicated phrasing removed - examples.ts 24.7KB -> 20KB: all 31 teaching examples kept; analysis strings shortened, redundant categories/policy_version dropped from example outputs (both optional in the response schema) - output.ts 13.8KB -> 6.8KB: compressed schema + personality + format rules; CRITICAL bans on generic analysis and reply-context requirement retained - system.ts: MEDIA_INSTRUCTIONS compressed, key rules kept Semantic moderation cache (AI_LLM_EMBEDDING_MODEL): - New embeddingClient.ts: OpenAI-compatible embeddings + cosine similarity; degrades gracefully when model/key unset - textCacheStore: stores embedding JSON per verdict, findSimilarTextModeration reuses near-duplicate verdicts (min 0.97 cosine, processing locks skipped) - moderationOrchestrator: after exact-hash miss, embed text-only targets and reuse stored verdict for near-duplicates -> skips expensive chat completion for spam variants; fresh verdicts written back with embedding - Config: AI_LLM_EMBEDDING_MODEL / MIN_SIMILARITY (0.97) / MAX_CANDIDATES (30) - Migration 0012: ADD COLUMN embedding to text_analysis_cache (idempotent) - .env.example documents the new vars	2026-07-31 19:37:53 +07:00