docs: add design spec for robust moderation parsing and test fixes

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-18 04:36:00 +07:00
parent f94009767b
commit 329cb464e8
1 changed files with 56 additions and 0 deletions
@@ -0,0 +1,56 @@
 # Design Spec: Robust Moderation & Test Improvements
 **Date**: 2026-05-18  
 **Topic**: Robust LLM Moderation Parsing, Capping Multimodal Attachments, and Fixing Dev/Streaming Tests
 ---
 ## 1. Goal & Context
 The project contains an LLM-based content moderation system that analyzes Discord messages and their image attachments. Real-world utilization revealed several issues:
 1. **Multimodal API Limits**: High numbers of image attachments in the target or surrounding context messages exceed API limits (e.g. Nemotron/Omni models cap at 8 images), triggering an HTTP 400 error.
 2. **LLM Output Variance**: LLM responses containing reasoning processes, conversational preambles, or markdown wrappers fail to parse under the current naive brace-matching algorithm, yielding `No JSON object found` or `Response missing 'results' array`.
 3. **Snowflake Precision Loss**: Snowflake IDs returned by the LLM sometimes suffer from floating-point rounding or formatting issues, preventing them from matching the original string-based target IDs.
 4. **Dev/Streaming Test Failures**: Failing tests in `ytdlp.test.ts` and `playTranscode.test.ts` due to mismatched parameters and type assertions.
 ---
 ## 2. Architecture & Detailed Design
 ### A. Multimodal Attachment Filtering & Prioritization
 In `src/moderation/llmModerationClient.ts`:
 * Extract all image attachments.
 * Sort and prioritize attachments:
  * Targets first: Attachments belonging to messages in the active `targets` list.
  * Context second: Attachments belonging to context messages, sorted by `created_at` descending (most recent first).
 * Slice the resulting array to a maximum of **8 elements** to ensure we never hit model limits.
 * If the list is empty, proceed with the existing transparent 1x1 dummy PNG fallback.
 ### B. Resilient JSON Extraction
 Implement `extractJson` inside `src/moderation/llmModerationClient.ts`:
 1. **Markdown Blocks**: Scan for code blocks using `/```(?:json)?\s*([\s\S]*?)\s*```/g`. Try to parse the first match yielding an object.
 2. **Exhaustive Span Search**: If markdown parsing fails, locate the indices of all `{` and `}` characters in the string. Try all matching pairs, starting from the largest span to the smallest.
 3. **Error Reporting**: If no candidate substring parses as an object, throw `No JSON object found in response`.
 ### C. Message ID Fuzzy Mapping
 * Map `message_id` back to target IDs by stringifying and checking exact match.
 * If not matched and the ID ends with `"00"` or contains `"e+"` (indicating exponential format or floating point precision loss), search `targetIds` for a prefix match (first 10 characters) and restore the original ID.
 ### D. Streaming & Dev Test Fixes
 * **`tests/media/ytdlp.test.ts`**: Update the assertion to expect `--format best[protocol^=http]/best` to match the actual production code.
 * **`tests/streaming/playTranscode.test.ts`**: Safely check if the input `readable` is an object and has the `.on` function before calling `readable.on("data", ...)`.
 ---
 ## 3. Test Plan & Expanded Coverage
 We will implement dedicated unit tests in `tests/moderation/llmModerationClient.test.ts`:
 1. **Image Capping & Prioritization**: Ensure image attachments are sorted correctly and capped at 8.
 2. **Complex Conversational Content**: Verify extraction from messages wrapped in markdown, with leading/trailing text, and multiple code blocks.
 3. **Reasoning Blocks**: Verify extraction when reasoning blocks contain separate `{` and `}` symbols.
 4. **Precision Loss Scenarios**: Verify automatic correction of floating-point string representations of Snowflake IDs.
 ---
 ## 4. Success Criteria
 * All tests pass successfully (`pnpm run test` exits with `0`).
 * System remains highly resilient to formatting variance in LLM responses.
 * No 400 Bad Request errors occur due to exceeding the maximum image attachment limit.