fix(automod): flow real LLM analysis + descriptive fallback
Build & Deploy (Nix) / build-and-deploy (backend) (push) Successful in 3m2s
Build & Deploy (Nix) / build-and-deploy (discord-gateway) (push) Successful in 2m25s
Build & Deploy (Nix) / build-and-deploy (proxy) (push) Successful in 2m40s
Build & Deploy (Nix) / build-and-deploy (backend) (push) Successful in 3m2s
Build & Deploy (Nix) / build-and-deploy (discord-gateway) (push) Successful in 2m25s
Build & Deploy (Nix) / build-and-deploy (proxy) (push) Successful in 2m40s
Root cause: ai-analysis-worker read llmResult.explanation and llmResult.toxicityScore — fields the LLM pipeline never produces (canonical AnalysisResult uses analysis/score). Every message fell back to the bare template "Tidak ada indikasi pelanggaran." and the stored score was always 0. - Map analysis/score correctly; fallback now quotes the message content - Prompt: ban generic analysis phrasing, require reply context - LLM context: include replied-to message content (metadata.reference) so the model can explain what the user is replying to - Frontend: show thread/channel names from metadata instead of raw IDs (message card, detail views, search overlay); detail panel now displays the ai_analysis text - Auto-delete log/DM include the descriptive analysis as the reason
This commit is contained in:
@@ -58,11 +58,8 @@ export interface AnalysisResult {
|
||||
| "review"
|
||||
| "delete"
|
||||
| "escalate";
|
||||
toxicityScore: number;
|
||||
harmScore: number;
|
||||
jailbreakScore: number;
|
||||
safetyScore: number;
|
||||
explanation: string;
|
||||
score: number;
|
||||
analysis: string;
|
||||
correctedFlags?: string[];
|
||||
}
|
||||
|
||||
@@ -182,7 +179,7 @@ export default async function workerRouter(
|
||||
* Runs the LLM moderation analysis on a single message.
|
||||
*
|
||||
* The LLM verdict IS the result — confidence, severity, flags and
|
||||
* explanation all come from the model. On failure the message is marked
|
||||
* analysis all come from the model. On failure the message is marked
|
||||
* "error" (explicit, retryable) instead of receiving a heuristic verdict.
|
||||
*/
|
||||
async function runLLMAnalysis(
|
||||
@@ -214,15 +211,10 @@ async function runLLMAnalysis(
|
||||
severity: llmResult.severity ?? "none",
|
||||
confidence: normalizeConfidence(llmResult.confidence),
|
||||
recommendedAction: llmResult.recommendedAction ?? "none",
|
||||
toxicityScore: llmResult.toxicityScore ?? 0,
|
||||
harmScore: llmResult.harmScore ?? 0,
|
||||
jailbreakScore: llmResult.jailbreakScore ?? 0,
|
||||
safetyScore: llmResult.safetyScore ?? 0,
|
||||
explanation:
|
||||
llmResult.explanation?.trim() ||
|
||||
(llmResult.status === "clean"
|
||||
? "Tidak ada indikasi pelanggaran."
|
||||
: "Pesan terindikasi melanggar kebijakan (analisis AI)."),
|
||||
score: llmResult.score ?? 0,
|
||||
analysis:
|
||||
llmResult.analysis?.trim() ||
|
||||
buildFallbackAnalysis(message, llmResult.status ?? "clean"),
|
||||
};
|
||||
} catch (error) {
|
||||
const errorMsg = error instanceof Error ? error.message : String(error);
|
||||
@@ -242,6 +234,28 @@ function normalizeConfidence(raw: number | undefined | null): number {
|
||||
return 0.7;
|
||||
}
|
||||
|
||||
/**
|
||||
* Content-aware fallback when the LLM returns an empty `analysis` field.
|
||||
* Quotes the actual message content so the log still explains WHAT was said
|
||||
* instead of a bare template like "Tidak ada indikasi pelanggaran."
|
||||
*/
|
||||
function buildFallbackAnalysis(
|
||||
message: MessageRecord,
|
||||
status: string,
|
||||
): string {
|
||||
const raw = (message.edited_content ?? message.content ?? "").trim();
|
||||
const snippet = raw.length > 120 ? `${raw.slice(0, 120).trimEnd()}…` : raw;
|
||||
|
||||
if (status === "clean") {
|
||||
return snippet
|
||||
? `Tidak ada indikasi pelanggaran. Pesan: "${snippet}" dinilai wajar dalam konteks percakapan.`
|
||||
: "Tidak ada indikasi pelanggaran. Isi pesan dinilai wajar dalam konteks percakapan.";
|
||||
}
|
||||
return snippet
|
||||
? `Pesan terindikasi melanggar kebijakan: "${snippet}".`
|
||||
: "Pesan terindikasi melanggar kebijakan (analisis AI).";
|
||||
}
|
||||
|
||||
function buildFallbackResult(
|
||||
messageId: string,
|
||||
reason: string,
|
||||
@@ -254,11 +268,8 @@ function buildFallbackResult(
|
||||
severity: "none",
|
||||
confidence: 0,
|
||||
recommendedAction: "review",
|
||||
toxicityScore: 0,
|
||||
harmScore: 0,
|
||||
jailbreakScore: 0,
|
||||
safetyScore: 0,
|
||||
explanation: reason,
|
||||
score: 0,
|
||||
analysis: reason,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -319,14 +330,14 @@ async function processBatch(job: {
|
||||
result: {
|
||||
status: result.status,
|
||||
flags: JSON.stringify(result.flags),
|
||||
score: result.toxicityScore,
|
||||
analysis: result.explanation,
|
||||
score: result.score,
|
||||
analysis: result.analysis,
|
||||
categories: result.categories,
|
||||
severity: result.severity,
|
||||
confidence: result.confidence,
|
||||
recommendedAction: result.recommendedAction,
|
||||
analyzedAt: Date.now(),
|
||||
error: result.status === "error" ? result.explanation : null,
|
||||
error: result.status === "error" ? result.analysis : null,
|
||||
},
|
||||
}));
|
||||
|
||||
|
||||
@@ -33,6 +33,7 @@ export async function logDeletionToChannel(
|
||||
const severity = message.ai_severity ?? "none";
|
||||
const categories =
|
||||
message.ai_categories ?? message.ai_moderation_flags ?? "—";
|
||||
const reason = message.ai_analysis ?? "—";
|
||||
const snippet = (message.edited_content ?? message.content).substring(
|
||||
0,
|
||||
200,
|
||||
@@ -42,6 +43,7 @@ export async function logDeletionToChannel(
|
||||
`**Status:** ${message.ai_status}\n` +
|
||||
`**Severitas:** ${severity}\n` +
|
||||
`**Kategori:** ${categories}\n` +
|
||||
`**Alasan:** ${reason}\n` +
|
||||
`**Isi:** ${snippet}\n` +
|
||||
`**Waktu:** <t:${Math.floor(Date.now() / 1000)}:R>`,
|
||||
);
|
||||
|
||||
@@ -20,8 +20,14 @@ export async function sendDeletionNotification(
|
||||
try {
|
||||
const targetUser = await client.users.fetch(message.user_id);
|
||||
if (targetUser) {
|
||||
// Prefer the descriptive LLM analysis so the user understands WHY;
|
||||
// fall back to category/flag labels when it is unavailable.
|
||||
const analysis = (message.ai_analysis ?? "").trim();
|
||||
const reason: string =
|
||||
message.ai_categories ?? message.ai_moderation_flags ?? "(unknown)";
|
||||
(analysis.length > 240 ? `${analysis.slice(0, 240)}…` : analysis) ||
|
||||
message.ai_categories ??
|
||||
message.ai_moderation_flags ??
|
||||
"(unknown)";
|
||||
await targetUser.send(
|
||||
`Pesan Anda di **${guildName}** telah dihapus oleh sistem moderasi otomatis.\n` +
|
||||
`Alasan: ${reason}\n` +
|
||||
|
||||
@@ -42,12 +42,44 @@ export function estimateTokens(text: string): number {
|
||||
}
|
||||
|
||||
/**
|
||||
* Formats reference info for a message (reply/forward/crosspost)
|
||||
* Formats reference info for a message (reply/forward/crosspost).
|
||||
*
|
||||
* The replied-to content is stored in the message metadata
|
||||
* (`metadata.reference.content` / `repliedUsername`) at capture time, so
|
||||
* include it here — the LLM can then explain WHAT the user is replying to
|
||||
* instead of only seeing a raw message ID it cannot resolve.
|
||||
*/
|
||||
function formatReferenceInfo(msg: MessageRecord): string {
|
||||
const parts: string[] = [];
|
||||
|
||||
let repliedContent: string | null = null;
|
||||
let repliedUsername: string | null = null;
|
||||
try {
|
||||
const meta = JSON.parse(msg.metadata ?? "") as {
|
||||
reference?: {
|
||||
content?: string | null;
|
||||
repliedUsername?: string | null;
|
||||
} | null;
|
||||
};
|
||||
repliedContent = meta?.reference?.content ?? null;
|
||||
repliedUsername = meta?.reference?.repliedUsername ?? null;
|
||||
} catch {
|
||||
// metadata malformed — fall back to ID-only reference
|
||||
}
|
||||
|
||||
const repliedText = (repliedContent ?? "").trim();
|
||||
const repliedSnippet = repliedText
|
||||
? sanitizeDiscordTokens(
|
||||
repliedText.length > 200
|
||||
? `${repliedText.slice(0, 200)}…`
|
||||
: repliedText,
|
||||
)
|
||||
: null;
|
||||
|
||||
if (msg.is_reply && msg.reference_message_id) {
|
||||
parts.push(`[reply_to: ${msg.reference_message_id}]`);
|
||||
const who = repliedUsername ? ` oleh ${repliedUsername}` : "";
|
||||
const what = repliedSnippet ? `: "${repliedSnippet}"` : "";
|
||||
parts.push(`[reply_to: ${msg.reference_message_id}${who}${what}]`);
|
||||
if (msg.reference_channel_id) {
|
||||
parts.push(`(reply_channel: ${msg.reference_channel_id})`);
|
||||
}
|
||||
|
||||
@@ -153,9 +153,11 @@ Contoh buruk: "Pengirim bercanda tentang agama." (JANGAN menggunakan kata "berca
|
||||
|
||||
CRITICAL:
|
||||
- JANGAN PERNAH menulis "Pesan hanya berisi..." atau "Pesan tidak mengandung..." sebagai analysis.
|
||||
- JANGAN PERNAH menulis "Tidak ada indikasi pelanggaran" atau frasa generik serupa sebagai analysis — wajib sebutkan TOPIK/ISI pesan secara spesifik apa yang sedang dibicarakan pengirim.
|
||||
- JANGAN PERNAH menulis template generik seperti "Pengirim mengirimkan sebuah file GIF tanpa pelanggaran". Kamu WAJIB mendeskripsikan isi visualnya secara spesifik berdasarkan Media analysis.
|
||||
- JANGAN PERNAH menyebutkan nama / username pengguna secara langsung. Selalu gunakan kata "Pengirim" atau "Pengguna".
|
||||
- Selalu sebutkan ISI KONTEN secara spesifik — apa yang dibicarakan, apa yang terlihat di gambar.
|
||||
- Jika pesan adalah BALASAN (reply) ke pesan lain, jelaskan konteks balasannya: apa yang sedang dibicarakan, siapa yang dibalas (tanpa nama, cukup peran/isi pesan yang dibalas), dan bagaimana tanggapan pengirim terhadapnya.
|
||||
- Gunakan informasi dari Media analysis untuk mendeskripsikan gambar.
|
||||
- Analisis harus MEMBERI KONTEKS, bukan hanya menyatakan status.
|
||||
- GUNAKAN <user_profile> untuk personalisasi analysis — jadikan analysis terasa seperti sistem "mengenal" pengguna.
|
||||
|
||||
Reference in New Issue
Block a user