feat: expand AI moderation with structured analysis, review workflow, and guardrails

- Add structured AI moderation fields (categories, severity, confidence,
  recommended_action, policy_version, evidence) to messages table
- Add moderation_reviews, moderation_actions, and retention_policies tables
- Upgrade LLM response parsing to support structured metadata with backwards
  compatibility for legacy responses
- Implement public AI evaluation review UI with decision controls
  (approve, false positive + reanalyze, escalate)
- Add auto-delete guardrails requiring high confidence, severity, and
  allowed categories; log all attempts to moderation_actions
- Add retention manager scaffolding for messages/attachments/voice
- Add action executor for moderation actions (mute, warn, kick, ban)
- Add review routes: GET/POST/PATCH /api/reviews, GET/POST/PATCH /api/actions
- Preserve auth separation: voice/media/recordings gated, review public

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
MythEclipse
2026-05-30 01:02:51 +07:00
co-authored by Claude Opus 4.6
parent b938420eb3
commit c894e5cd75
15 changed files with 2104 additions and 27 deletions
+12
View File
@@ -307,6 +307,12 @@ async function processIndividualFallback(
score: r.score,
raw: JSON.stringify(analysisResult.raw),
analysis: r.analysis,
categories: r.categories,
severity: r.severity,
confidence: r.confidence,
recommendedAction: r.recommendedAction,
policyVersion: r.policyVersion,
evidence: r.evidence,
analyzedAt: Date.now(),
error: null,
},
@@ -359,6 +365,12 @@ async function processIndividualFallback(
raw: null,
analysis:
"Individual fallback exhausted all retries: LLM consistently dropped this message even in single-target mode",
categories: ["individual_analysis_exhausted"],
severity: "none",
confidence: 0,
recommendedAction: "review",
policyVersion: "default-2026-05-30",
evidence: [],
analyzedAt: Date.now(),
error: lastError,
},