From ff90f9e95c293893ab4f1bc8155ac5f30654f44f Mon Sep 17 00:00:00 2001 From: MythEclipse Date: Fri, 15 May 2026 08:32:13 +0700 Subject: [PATCH] feat: add selfbot performance and feature optimization design document --- ...elfbot-performance-feature-optimization.md | 663 ++++++++++++++++++ ...5-15-selfbot-performance-feature-design.md | 37 + 2 files changed, 700 insertions(+) create mode 100644 docs/superpowers/plans/2026-05-15-selfbot-performance-feature-optimization.md create mode 100644 docs/superpowers/specs/2026-05-15-selfbot-performance-feature-design.md diff --git a/docs/superpowers/plans/2026-05-15-selfbot-performance-feature-optimization.md b/docs/superpowers/plans/2026-05-15-selfbot-performance-feature-optimization.md new file mode 100644 index 0000000..f0ba0f6 --- /dev/null +++ b/docs/superpowers/plans/2026-05-15-selfbot-performance-feature-optimization.md @@ -0,0 +1,663 @@ +# Selfbot Performance Feature Optimization Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Optimize the app's selfbot client runtime and vendor internals for lower memory pressure, safer REST retries, reduced voice cleanup leaks, faster gateway queue processing, and lightweight observability. + +**Architecture:** Start with app-level client options because they are low-risk and immediately reduce cache pressure. Then patch vendor internals in isolated areas: REST manager/request handling, voice packet cleanup, and WebSocket shard queueing. Keep public imports and runtime APIs compatible with `discord.js-selfbot-v13` consumers. + +**Tech Stack:** Node.js, TypeScript, CommonJS vendor package, discord.js-selfbot-v13 workspace dependency, Undici, Vitest, Biome, TypeScript. + +--- + +## File Structure + +- Modify `src/index.ts`: instantiate `Client` with low-memory cache/sweeper/REST options. +- Modify `vendor/discord.js-selfbot-v13/src/rest/RESTManager.js`: own per-client dispatcher state and super-properties cache helpers. +- Modify `vendor/discord.js-selfbot-v13/src/rest/APIRequest.js`: use per-client dispatcher and cached `x-super-properties` header. +- Modify `vendor/discord.js-selfbot-v13/src/rest/RequestHandler.js`: add backoff/jitter helper and debug telemetry for retry attempts. +- Modify `vendor/discord.js-selfbot-v13/src/client/voice/receiver/PacketHandler.js`: clear speaking timers and reduce RTP parse allocations. +- Modify `vendor/discord.js-selfbot-v13/src/client/websocket/WebSocketShard.js`: use cursor-backed gateway queue with compatible priority insertion and destroy cleanup. +- Create `tests/vendor/selfbotClientOptions.test.ts`: verify app client options factory if extracted. +- Create `tests/vendor/requestHandlerBackoff.test.ts`: verify retry delay calculation is bounded and grows. +- Create `tests/vendor/websocketQueue.test.ts`: verify FIFO, priority, and destroy queue reset semantics for the new queue helpers if exported/testable. + +## Task 1: Extract and Test Low-Memory Client Options + +**Files:** +- Create: `src/discordClientOptions.ts` +- Modify: `src/index.ts:4-25` +- Test: `tests/vendor/selfbotClientOptions.test.ts` + +- [ ] **Step 1: Write the failing test** + +Create `tests/vendor/selfbotClientOptions.test.ts`: + +```ts +import { describe, expect, it } from "vitest"; +import { createDiscordClientOptions } from "../../src/discordClientOptions"; + +describe("createDiscordClientOptions", () => { + it("uses low-memory message cache and active sweepers", () => { + const options = createDiscordClientOptions(); + + expect(options.restRequestTimeout).toBe(15_000); + expect(options.retryLimit).toBe(2); + expect(options.restGlobalRateLimit).toBe(45); + expect(options.sweepers).toEqual({ + messages: { interval: 300, lifetime: 600 }, + threads: { interval: 3600, lifetime: 14400 }, + }); + + expect(options.partials).toEqual(["USER", "CHANNEL", "GUILD_MEMBER", "MESSAGE"]); + }); +}); +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `pnpm exec vitest run tests/vendor/selfbotClientOptions.test.ts` + +Expected: FAIL with module not found for `src/discordClientOptions`. + +- [ ] **Step 3: Add the client options factory** + +Create `src/discordClientOptions.ts`: + +```ts +import { Options } from "discord.js-selfbot-v13"; + +export function createDiscordClientOptions() { + return { + makeCache: Options.cacheWithLimits({ + ...Options.defaultMakeCacheSettings, + MessageManager: 25, + ReactionManager: 0, + ReactionUserManager: 0, + PresenceManager: 0, + }), + partials: ["USER", "CHANNEL", "GUILD_MEMBER", "MESSAGE"], + sweepers: { + messages: { interval: 300, lifetime: 600 }, + threads: { interval: 3600, lifetime: 14400 }, + }, + restRequestTimeout: 15_000, + retryLimit: 2, + restGlobalRateLimit: 45, + }; +} +``` + +- [ ] **Step 4: Use the factory in the app entry point** + +Modify `src/index.ts`: + +```ts +import { Client } from "discord.js-selfbot-v13"; +import { config } from "./config"; +import { closeDatabase, initializeDatabase } from "./database/drizzle"; +import { createDiscordClientOptions } from "./discordClientOptions"; +``` + +Replace: + +```ts +const client = new Client(); +``` + +with: + +```ts +const client = new Client(createDiscordClientOptions()); +``` + +- [ ] **Step 5: Run the focused test** + +Run: `pnpm exec vitest run tests/vendor/selfbotClientOptions.test.ts` + +Expected: PASS. + +- [ ] **Step 6: Run typecheck** + +Run: `pnpm run typecheck` + +Expected: PASS. If TypeScript cannot type `Options` from the vendor package, add a local return type only if necessary; do not weaken the factory to `any`. + +- [ ] **Step 7: Commit** + +Run: + +```bash +git add src/discordClientOptions.ts src/index.ts tests/vendor/selfbotClientOptions.test.ts +git commit -m "perf: tune selfbot client runtime options" +``` + +## Task 2: Add Per-Client REST Dispatcher and Cached Super Properties + +**Files:** +- Modify: `vendor/discord.js-selfbot-v13/src/rest/RESTManager.js:1-69` +- Modify: `vendor/discord.js-selfbot-v13/src/rest/APIRequest.js:1-166` + +- [ ] **Step 1: Add REST manager state** + +Modify `vendor/discord.js-selfbot-v13/src/rest/RESTManager.js` imports: + +```js +const { Collection } = require('@discordjs/collection'); +const makeFetchCookie = require('fetch-cookie'); +const { CookieJar } = require('tough-cookie'); +const { buildConnector, Client: UndiciClient, ProxyAgent, fetch: fetchOriginal } = require('undici'); +const APIRequest = require('./APIRequest'); +const routeBuilder = require('./APIRouter'); +const RequestHandler = require('./RequestHandler'); +const { Error } = require('../errors'); +const { ciphers } = require('../util/Constants'); +const { Endpoints } = require('../util/Constants'); +const Util = require('../util/Util'); +``` + +- [ ] **Step 2: Add per-client dispatcher fields and helper methods** + +Inside `RESTManager` constructor after `this.fetch = ...`, add: + +```js +this.dispatcher = null; +this.superPropertiesSource = null; +this.superPropertiesHeader = null; +``` + +Add methods before `request(method, url, options = {})`: + +```js +getDispatcher() { + if (this.dispatcher) return this.dispatcher; + + const proxy = Util.checkUndiciProxyAgent(this.client.options.http.agent); + if (proxy) { + this.dispatcher = new ProxyAgent({ + ...proxy, + ciphers: ciphers.join(':'), + }); + } else { + this.dispatcher = new UndiciClient('https://discord.com', { + connect: buildConnector({ ciphers: ciphers.join(':') }), + }); + } + + return this.dispatcher; +} + +getSuperPropertiesHeader() { + const source = JSON.stringify(this.client.options.ws.properties); + if (source !== this.superPropertiesSource) { + this.superPropertiesSource = source; + this.superPropertiesHeader = Buffer.from(source, 'ascii').toString('base64'); + } + return this.superPropertiesHeader; +} +``` + +- [ ] **Step 3: Remove module-global dispatcher from APIRequest** + +Modify `vendor/discord.js-selfbot-v13/src/rest/APIRequest.js` imports to: + +```js +const Buffer = require('node:buffer').Buffer; +const { setTimeout } = require('node:timers'); +const { FormData } = require('undici'); +``` + +Remove: + +```js +const { FormData, buildConnector, Client, ProxyAgent } = require('undici'); +const { ciphers } = require('../util/Constants'); +const Util = require('../util/Util'); + +let agent = null; +``` + +- [ ] **Step 4: Use REST manager dispatcher and cached header** + +In `APIRequest.make`, delete the `if (!agent) { ... }` block. + +Replace the `x-super-properties` header construction with: + +```js +'x-super-properties': this.rest.getSuperPropertiesHeader(), +``` + +Replace fetch dispatcher: + +```js +dispatcher: agent, +``` + +with: + +```js +dispatcher: this.rest.getDispatcher(), +``` + +- [ ] **Step 5: Run vendor lint through root lint** + +Run: `pnpm run lint` + +Expected: PASS or existing unrelated lint failures. If failures are in edited vendor files, fix them. + +- [ ] **Step 6: Commit** + +Run: + +```bash +git add vendor/discord.js-selfbot-v13/src/rest/RESTManager.js vendor/discord.js-selfbot-v13/src/rest/APIRequest.js +git commit -m "perf: cache selfbot rest dispatcher metadata" +``` + +## Task 3: Add REST Retry Backoff With Jitter + +**Files:** +- Modify: `vendor/discord.js-selfbot-v13/src/rest/RequestHandler.js:1-505` +- Test: `tests/vendor/requestHandlerBackoff.test.ts` + +- [ ] **Step 1: Export a pure backoff helper for tests** + +Add near the top of `vendor/discord.js-selfbot-v13/src/rest/RequestHandler.js` after `calculateReset`: + +```js +function calculateRetryDelay(retryCount, random = Math.random) { + const base = 250; + const max = 5_000; + const exponential = Math.min(max, base * 2 ** Math.max(0, retryCount - 1)); + return exponential + Math.floor(random() * base); +} +``` + +At the bottom, replace: + +```js +module.exports = RequestHandler; +``` + +with: + +```js +module.exports = RequestHandler; +module.exports.calculateRetryDelay = calculateRetryDelay; +``` + +- [ ] **Step 2: Write the focused helper test** + +Create `tests/vendor/requestHandlerBackoff.test.ts`: + +```ts +import { describe, expect, it } from "vitest"; + +const { calculateRetryDelay } = await import( + "../../vendor/discord.js-selfbot-v13/src/rest/RequestHandler.js" +); + +describe("calculateRetryDelay", () => { + it("increases exponentially and applies bounded jitter", () => { + expect(calculateRetryDelay(1, () => 0)).toBe(250); + expect(calculateRetryDelay(2, () => 0)).toBe(500); + expect(calculateRetryDelay(3, () => 0)).toBe(1000); + expect(calculateRetryDelay(10, () => 0)).toBe(5000); + expect(calculateRetryDelay(1, () => 0.999)).toBe(499); + }); +}); +``` + +- [ ] **Step 3: Run the helper test** + +Run: `pnpm exec vitest run tests/vendor/requestHandlerBackoff.test.ts` + +Expected: PASS. + +- [ ] **Step 4: Apply backoff to network errors** + +In `RequestHandler.execute`, replace the catch block after `request.make(...)` with: + +```js + } catch (error) { + if (request.retries === this.manager.client.options.retryLimit) { + throw new HTTPError( + error.message, + error.constructor.name, + error.status, + request, + ); + } + + request.retries++; + const delay = calculateRetryDelay(request.retries); + this.manager.client.emit( + DEBUG, + `[Request Handler] Retrying failed request after ${delay}ms.\n Method : ${request.method}\n Path : ${request.path}\n Route : ${request.route}\n Retry : ${request.retries}`, + ); + await sleep(delay); + return this.execute(request); + } +``` + +- [ ] **Step 5: Apply backoff to 5xx responses** + +In the 5xx block, replace: + +```js + request.retries++; + return this.execute(request); +``` + +with: + +```js + request.retries++; + const delay = calculateRetryDelay(request.retries); + this.manager.client.emit( + DEBUG, + `[Request Handler] Retrying server error after ${delay}ms.\n Method : ${request.method}\n Path : ${request.path}\n Route : ${request.route}\n Status : ${res.status}\n Retry : ${request.retries}`, + ); + await sleep(delay); + return this.execute(request); +``` + +- [ ] **Step 6: Run focused and full tests** + +Run: `pnpm exec vitest run tests/vendor/requestHandlerBackoff.test.ts` + +Expected: PASS. + +Run: `pnpm run test` + +Expected: PASS. + +- [ ] **Step 7: Commit** + +Run: + +```bash +git add vendor/discord.js-selfbot-v13/src/rest/RequestHandler.js tests/vendor/requestHandlerBackoff.test.ts +git commit -m "perf: back off selfbot rest retries" +``` + +## Task 4: Clean Voice Receiver Timers and Reduce RTP Buffer Work + +**Files:** +- Modify: `vendor/discord.js-selfbot-v13/src/client/voice/receiver/PacketHandler.js:1-280` + +- [ ] **Step 1: Patch AES decrypt concat allocation** + +In `parseBuffer`, replace: + +```js + packet = Buffer.concat([ + decipheriv.update(encrypted), + decipheriv.final(), + ]); +``` + +with: + +```js + const updated = decipheriv.update(encrypted); + const final = decipheriv.final(); + packet = final.length === 0 ? updated : Buffer.concat([updated, final]); +``` + +- [ ] **Step 2: Patch XChaCha auth tag concat allocation** + +Replace: + +```js + Buffer.concat([encrypted, authTag]), +``` + +with: + +```js + buffer.subarray(headerSize, buffer.length - UNPADDED_NONCE_LENGTH), +``` + +- [ ] **Step 3: Add speaking timeout cleanup** + +In `destroyAllStream()`, after clearing video streams, add: + +```js + for (const timeout of this.speakingTimeouts.values()) { + clearTimeout(timeout); + } + const clearedSpeakingTimeouts = this.speakingTimeouts.size; + this.speakingTimeouts.clear(); + this.emit('debug', { + message: 'Destroyed voice receiver streams', + audioStreams: this.streams.size, + videoStreams: this.videoStreams.size, + speakingTimeouts: clearedSpeakingTimeouts, + }); +``` + +Then adjust ordering so the counts are captured before `streams.clear()` and `videoStreams.clear()`: + +```js + destroyAllStream() { + const audioStreams = this.streams.size; + const videoStreams = this.videoStreams.size; + for (const stream of this.streams.values()) { + stream.stream.destroy(); + } + this.streams.clear(); + for (const stream of this.videoStreams.values()) { + stream.destroy(); + } + this.videoStreams.clear(); + for (const timeout of this.speakingTimeouts.values()) { + clearTimeout(timeout); + } + const speakingTimeouts = this.speakingTimeouts.size; + this.speakingTimeouts.clear(); + this.emit('debug', { + message: 'Destroyed voice receiver streams', + audioStreams, + videoStreams, + speakingTimeouts, + }); + } +``` + +- [ ] **Step 4: Run lint** + +Run: `pnpm run lint` + +Expected: PASS or only unrelated existing failures. Fix edited-file failures. + +- [ ] **Step 5: Run tests** + +Run: `pnpm run test` + +Expected: PASS. + +- [ ] **Step 6: Commit** + +Run: + +```bash +git add vendor/discord.js-selfbot-v13/src/client/voice/receiver/PacketHandler.js +git commit -m "perf: clean up selfbot voice receiver state" +``` + +## Task 5: Replace Gateway Queue Shift With Cursor Queue + +**Files:** +- Modify: `vendor/discord.js-selfbot-v13/src/client/websocket/WebSocketShard.js:108-120,818-954` + +- [ ] **Step 1: Add queue cursor metadata** + +In the `ratelimit` object, change: + +```js +queue: [], +``` + +To: + +```js +queue: [], +queueOffset: 0, +``` + +- [ ] **Step 2: Update priority insertion** + +Replace `send(data, important = false)` with: + +```js + send(data, important = false) { + if (important) { + if (this.ratelimit.queueOffset === 0) { + this.ratelimit.queue.unshift(data); + } else { + this.ratelimit.queue[--this.ratelimit.queueOffset] = data; + } + } else { + this.ratelimit.queue.push(data); + } + this.processQueue(); + } +``` + +- [ ] **Step 3: Update queue processing** + +Replace `processQueue()` with: + +```js + processQueue() { + if (this.ratelimit.remaining === 0) return; + if (this.ratelimit.queueOffset >= this.ratelimit.queue.length) return; + if (this.ratelimit.remaining === this.ratelimit.total) { + this.ratelimit.timer = setTimeout(() => { + this.ratelimit.remaining = this.ratelimit.total; + this.processQueue(); + }, this.ratelimit.time).unref(); + } + while (this.ratelimit.remaining > 0) { + const item = this.ratelimit.queue[this.ratelimit.queueOffset++]; + if (!item) { + this._compactQueue(); + return; + } + this._send(item); + this.ratelimit.remaining--; + } + this._compactQueue(); + } +``` + +- [ ] **Step 4: Add queue compaction helper** + +Add before `destroy(...)`: + +```js + _compactQueue() { + if (this.ratelimit.queueOffset === 0) return; + if (this.ratelimit.queueOffset >= this.ratelimit.queue.length) { + this.ratelimit.queue.length = 0; + this.ratelimit.queueOffset = 0; + return; + } + if (this.ratelimit.queueOffset > 512) { + this.ratelimit.queue = this.ratelimit.queue.slice(this.ratelimit.queueOffset); + this.ratelimit.queueOffset = 0; + } + } +``` + +- [ ] **Step 5: Reset cursor on destroy** + +In `destroy`, after: + +```js +this.ratelimit.queue.length = 0; +``` + +Add: + +```js +this.ratelimit.queueOffset = 0; +``` + +- [ ] **Step 6: Run lint and tests** + +Run: `pnpm run lint` + +Expected: PASS or only unrelated existing failures. Fix edited-file failures. + +Run: `pnpm run test` + +Expected: PASS. + +- [ ] **Step 7: Commit** + +Run: + +```bash +git add vendor/discord.js-selfbot-v13/src/client/websocket/WebSocketShard.js +git commit -m "perf: optimize selfbot gateway send queue" +``` + +## Task 6: Final Verification and Manual Runtime Notes + +**Files:** +- Modify only if verification exposes issues. + +- [ ] **Step 1: Run full lint** + +Run: `pnpm run lint` + +Expected: PASS. + +- [ ] **Step 2: Run full typecheck** + +Run: `pnpm run typecheck` + +Expected: PASS. + +- [ ] **Step 3: Run full tests** + +Run: `pnpm run test` + +Expected: PASS. + +- [ ] **Step 4: Run build** + +Run: `pnpm run build` + +Expected: PASS. + +- [ ] **Step 5: Inspect git diff** + +Run: `git diff --stat HEAD~5..HEAD` if each task was committed, or `git diff --stat` if not. + +Expected: changes limited to app client options, vendor REST, vendor voice, vendor WebSocket, tests, and this plan/spec. + +- [ ] **Step 6: Record manual Discord runtime limitation** + +If no Discord token/runtime environment is available, final response must state: + +```text +Automated verification passed. I could not perform live Discord runtime verification in this environment. Manual checks still needed: login, message capture, backlog sync, voice connect, voice record, disconnect, reconnect. +``` + +- [ ] **Step 7: Commit verification fixes only if needed** + +If Step 1-4 required fixes, commit only those fixes: + +```bash +git add +git commit -m "fix: stabilize selfbot optimization verification" +``` + +## Self-Review + +- Spec coverage: app runtime config is Task 1; REST dispatcher/header/backoff is Tasks 2-3; voice cleanup/allocation is Task 4; gateway queue is Task 5; verification/manual runtime note is Task 6. +- Placeholder scan: no TBD/TODO/fill-in steps remain; each code step includes concrete snippets and paths. +- Type consistency: `createDiscordClientOptions`, `calculateRetryDelay`, `getDispatcher`, `getSuperPropertiesHeader`, `_compactQueue`, and `queueOffset` are introduced before use and named consistently. diff --git a/docs/superpowers/specs/2026-05-15-selfbot-performance-feature-design.md b/docs/superpowers/specs/2026-05-15-selfbot-performance-feature-design.md new file mode 100644 index 0000000..11a0506 --- /dev/null +++ b/docs/superpowers/specs/2026-05-15-selfbot-performance-feature-design.md @@ -0,0 +1,37 @@ +# Selfbot Performance and Feature Optimization Design + +## Goal + +Improve `vendor/discord.js-selfbot-v13` and the app's Discord client setup for lower memory use, more stable REST behavior, lower voice hot-path allocation, and better observability while preserving the existing public API used by the bot. + +## Scope + +This is an aggressive optimization pass. It includes app-level client configuration plus internal vendor patches in REST, voice, and gateway queue handling. Changes must remain compatible with existing imports from `discord.js-selfbot-v13` and the current moderation/voice flows. + +## App Runtime Configuration + +`src/index.ts` will instantiate `Client` with explicit low-memory options instead of using `new Client()` defaults. Message cache will be reduced or disabled because captured messages are persisted to the database. Sweepers will remove old message/thread cache entries. REST retry/timeouts will remain conservative to avoid bursty backlog sync behavior. + +## Vendor REST Improvements + +`src/rest/APIRequest.js` currently uses a module-level Undici dispatcher and rebuilds expensive headers per request. The dispatcher will become per REST manager/client so proxy or client-specific settings cannot leak across clients. The `x-super-properties` header will be cached and reused while client properties remain unchanged. `RequestHandler` will add exponential backoff with jitter for network aborts and 5xx retries. + +## Vendor Voice Improvements + +`PacketHandler` will clean all speaking timeouts during stream destruction. Voice stream cleanup will clear audio and video stream maps reliably. RTP/decrypt hot-path allocation will be reduced where possible without changing emitted packet payloads or stream API behavior. + +## Vendor Gateway Queue Improvements + +`WebSocketShard` will replace repeated `Array.shift()` dequeue with a cursor-backed queue to avoid O(n) work under high gateway send volume. `send(data, important)` behavior will remain compatible, including priority insertion. + +## Observability + +Vendor internals will emit or debug useful operational data where it does not create noisy logs by default: REST retry/backoff attempts, voice stream cleanup counts, and gateway queue size/rate-limit state. The app can wire these later if needed. + +## Error Handling + +REST retry backoff must not bypass existing rate limit handling. Captcha and MFA retry paths keep their current behavior. Voice cleanup must ignore already-closed streams and never throw during disconnect. Gateway queue changes must clear queued state on destroy exactly as before. + +## Testing + +Run `pnpm run lint`, `pnpm run typecheck`, and `pnpm run test`. If runtime Discord login/voice testing cannot be performed in this environment, report that limitation explicitly and identify the manual test path: login, message capture, backlog sync, voice connect, voice record, disconnect/reconnect.