fix(automod): flow real LLM analysis + descriptive fallback
Build & Deploy (Nix) / build-and-deploy (backend) (push) Successful in 3m2s
Build & Deploy (Nix) / build-and-deploy (discord-gateway) (push) Successful in 2m25s
Build & Deploy (Nix) / build-and-deploy (proxy) (push) Successful in 2m40s

Root cause: ai-analysis-worker read llmResult.explanation and
llmResult.toxicityScore — fields the LLM pipeline never produces
(canonical AnalysisResult uses analysis/score). Every message fell back
to the bare template "Tidak ada indikasi pelanggaran." and the stored
score was always 0.

- Map analysis/score correctly; fallback now quotes the message content
- Prompt: ban generic analysis phrasing, require reply context
- LLM context: include replied-to message content (metadata.reference)
  so the model can explain what the user is replying to
- Frontend: show thread/channel names from metadata instead of raw IDs
  (message card, detail views, search overlay); detail panel now
  displays the ai_analysis text
- Auto-delete log/DM include the descriptive analysis as the reason
This commit is contained in:
Developer
2026-07-31 19:11:13 +07:00
parent 0bd4369ae9
commit 60084b3cc3
11 changed files with 142 additions and 34 deletions
@@ -58,11 +58,8 @@ export interface AnalysisResult {
| "review"
| "delete"
| "escalate";
toxicityScore: number;
harmScore: number;
jailbreakScore: number;
safetyScore: number;
explanation: string;
score: number;
analysis: string;
correctedFlags?: string[];
}
@@ -182,7 +179,7 @@ export default async function workerRouter(
* Runs the LLM moderation analysis on a single message.
*
* The LLM verdict IS the result — confidence, severity, flags and
* explanation all come from the model. On failure the message is marked
* analysis all come from the model. On failure the message is marked
* "error" (explicit, retryable) instead of receiving a heuristic verdict.
*/
async function runLLMAnalysis(
@@ -214,15 +211,10 @@ async function runLLMAnalysis(
severity: llmResult.severity ?? "none",
confidence: normalizeConfidence(llmResult.confidence),
recommendedAction: llmResult.recommendedAction ?? "none",
toxicityScore: llmResult.toxicityScore ?? 0,
harmScore: llmResult.harmScore ?? 0,
jailbreakScore: llmResult.jailbreakScore ?? 0,
safetyScore: llmResult.safetyScore ?? 0,
explanation:
llmResult.explanation?.trim() ||
(llmResult.status === "clean"
? "Tidak ada indikasi pelanggaran."
: "Pesan terindikasi melanggar kebijakan (analisis AI)."),
score: llmResult.score ?? 0,
analysis:
llmResult.analysis?.trim() ||
buildFallbackAnalysis(message, llmResult.status ?? "clean"),
};
} catch (error) {
const errorMsg = error instanceof Error ? error.message : String(error);
@@ -242,6 +234,28 @@ function normalizeConfidence(raw: number | undefined | null): number {
return 0.7;
}
/**
* Content-aware fallback when the LLM returns an empty `analysis` field.
* Quotes the actual message content so the log still explains WHAT was said
* instead of a bare template like "Tidak ada indikasi pelanggaran."
*/
function buildFallbackAnalysis(
message: MessageRecord,
status: string,
): string {
const raw = (message.edited_content ?? message.content ?? "").trim();
const snippet = raw.length > 120 ? `${raw.slice(0, 120).trimEnd()}` : raw;
if (status === "clean") {
return snippet
? `Tidak ada indikasi pelanggaran. Pesan: "${snippet}" dinilai wajar dalam konteks percakapan.`
: "Tidak ada indikasi pelanggaran. Isi pesan dinilai wajar dalam konteks percakapan.";
}
return snippet
? `Pesan terindikasi melanggar kebijakan: "${snippet}".`
: "Pesan terindikasi melanggar kebijakan (analisis AI).";
}
function buildFallbackResult(
messageId: string,
reason: string,
@@ -254,11 +268,8 @@ function buildFallbackResult(
severity: "none",
confidence: 0,
recommendedAction: "review",
toxicityScore: 0,
harmScore: 0,
jailbreakScore: 0,
safetyScore: 0,
explanation: reason,
score: 0,
analysis: reason,
};
}
@@ -319,14 +330,14 @@ async function processBatch(job: {
result: {
status: result.status,
flags: JSON.stringify(result.flags),
score: result.toxicityScore,
analysis: result.explanation,
score: result.score,
analysis: result.analysis,
categories: result.categories,
severity: result.severity,
confidence: result.confidence,
recommendedAction: result.recommendedAction,
analyzedAt: Date.now(),
error: result.status === "error" ? result.explanation : null,
error: result.status === "error" ? result.analysis : null,
},
}));
@@ -33,6 +33,7 @@ export async function logDeletionToChannel(
const severity = message.ai_severity ?? "none";
const categories =
message.ai_categories ?? message.ai_moderation_flags ?? "—";
const reason = message.ai_analysis ?? "—";
const snippet = (message.edited_content ?? message.content).substring(
0,
200,
@@ -42,6 +43,7 @@ export async function logDeletionToChannel(
`**Status:** ${message.ai_status}\n` +
`**Severitas:** ${severity}\n` +
`**Kategori:** ${categories}\n` +
`**Alasan:** ${reason}\n` +
`**Isi:** ${snippet}\n` +
`**Waktu:** <t:${Math.floor(Date.now() / 1000)}:R>`,
);
@@ -20,8 +20,14 @@ export async function sendDeletionNotification(
try {
const targetUser = await client.users.fetch(message.user_id);
if (targetUser) {
// Prefer the descriptive LLM analysis so the user understands WHY;
// fall back to category/flag labels when it is unavailable.
const analysis = (message.ai_analysis ?? "").trim();
const reason: string =
message.ai_categories ?? message.ai_moderation_flags ?? "(unknown)";
(analysis.length > 240 ? `${analysis.slice(0, 240)}` : analysis) ||
message.ai_categories ??
message.ai_moderation_flags ??
"(unknown)";
await targetUser.send(
`Pesan Anda di **${guildName}** telah dihapus oleh sistem moderasi otomatis.\n` +
`Alasan: ${reason}\n` +
@@ -42,12 +42,44 @@ export function estimateTokens(text: string): number {
}
/**
* Formats reference info for a message (reply/forward/crosspost)
* Formats reference info for a message (reply/forward/crosspost).
*
* The replied-to content is stored in the message metadata
* (`metadata.reference.content` / `repliedUsername`) at capture time, so
* include it here — the LLM can then explain WHAT the user is replying to
* instead of only seeing a raw message ID it cannot resolve.
*/
function formatReferenceInfo(msg: MessageRecord): string {
const parts: string[] = [];
let repliedContent: string | null = null;
let repliedUsername: string | null = null;
try {
const meta = JSON.parse(msg.metadata ?? "") as {
reference?: {
content?: string | null;
repliedUsername?: string | null;
} | null;
};
repliedContent = meta?.reference?.content ?? null;
repliedUsername = meta?.reference?.repliedUsername ?? null;
} catch {
// metadata malformed — fall back to ID-only reference
}
const repliedText = (repliedContent ?? "").trim();
const repliedSnippet = repliedText
? sanitizeDiscordTokens(
repliedText.length > 200
? `${repliedText.slice(0, 200)}`
: repliedText,
)
: null;
if (msg.is_reply && msg.reference_message_id) {
parts.push(`[reply_to: ${msg.reference_message_id}]`);
const who = repliedUsername ? ` oleh ${repliedUsername}` : "";
const what = repliedSnippet ? `: "${repliedSnippet}"` : "";
parts.push(`[reply_to: ${msg.reference_message_id}${who}${what}]`);
if (msg.reference_channel_id) {
parts.push(`(reply_channel: ${msg.reference_channel_id})`);
}
@@ -153,9 +153,11 @@ Contoh buruk: "Pengirim bercanda tentang agama." (JANGAN menggunakan kata "berca
CRITICAL:
- JANGAN PERNAH menulis "Pesan hanya berisi..." atau "Pesan tidak mengandung..." sebagai analysis.
- JANGAN PERNAH menulis "Tidak ada indikasi pelanggaran" atau frasa generik serupa sebagai analysis — wajib sebutkan TOPIK/ISI pesan secara spesifik apa yang sedang dibicarakan pengirim.
- JANGAN PERNAH menulis template generik seperti "Pengirim mengirimkan sebuah file GIF tanpa pelanggaran". Kamu WAJIB mendeskripsikan isi visualnya secara spesifik berdasarkan Media analysis.
- JANGAN PERNAH menyebutkan nama / username pengguna secara langsung. Selalu gunakan kata "Pengirim" atau "Pengguna".
- Selalu sebutkan ISI KONTEN secara spesifik — apa yang dibicarakan, apa yang terlihat di gambar.
- Jika pesan adalah BALASAN (reply) ke pesan lain, jelaskan konteks balasannya: apa yang sedang dibicarakan, siapa yang dibalas (tanpa nama, cukup peran/isi pesan yang dibalas), dan bagaimana tanggapan pengirim terhadapnya.
- Gunakan informasi dari Media analysis untuk mendeskripsikan gambar.
- Analisis harus MEMBERI KONTEKS, bukan hanya menyatakan status.
- GUNAKAN <user_profile> untuk personalisasi analysis — jadikan analysis terasa seperti sistem "mengenal" pengguna.