feat: expand AI moderation with structured analysis, review workflow, and guardrails
- Add structured AI moderation fields (categories, severity, confidence, recommended_action, policy_version, evidence) to messages table - Add moderation_reviews, moderation_actions, and retention_policies tables - Upgrade LLM response parsing to support structured metadata with backwards compatibility for legacy responses - Implement public AI evaluation review UI with decision controls (approve, false positive + reanalyze, escalate) - Add auto-delete guardrails requiring high confidence, severity, and allowed categories; log all attempts to moderation_actions - Add retention manager scaffolding for messages/attachments/voice - Add action executor for moderation actions (mute, warn, kick, ban) - Add review routes: GET/POST/PATCH /api/reviews, GET/POST/PATCH /api/actions - Preserve auth separation: voice/media/recordings gated, review public Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.6
parent
b938420eb3
commit
c894e5cd75
@@ -307,6 +307,12 @@ async function processIndividualFallback(
|
||||
score: r.score,
|
||||
raw: JSON.stringify(analysisResult.raw),
|
||||
analysis: r.analysis,
|
||||
categories: r.categories,
|
||||
severity: r.severity,
|
||||
confidence: r.confidence,
|
||||
recommendedAction: r.recommendedAction,
|
||||
policyVersion: r.policyVersion,
|
||||
evidence: r.evidence,
|
||||
analyzedAt: Date.now(),
|
||||
error: null,
|
||||
},
|
||||
@@ -359,6 +365,12 @@ async function processIndividualFallback(
|
||||
raw: null,
|
||||
analysis:
|
||||
"Individual fallback exhausted all retries: LLM consistently dropped this message even in single-target mode",
|
||||
categories: ["individual_analysis_exhausted"],
|
||||
severity: "none",
|
||||
confidence: 0,
|
||||
recommendedAction: "review",
|
||||
policyVersion: "default-2026-05-30",
|
||||
evidence: [],
|
||||
analyzedAt: Date.now(),
|
||||
error: lastError,
|
||||
},
|
||||
|
||||
Reference in New Issue
Block a user