Optimize the S3 GET path for chunked/multipart objects and reduce Telegram
API round-trips:
- object-stream: fetch object parts concurrently (bounded, in-order fan-in)
instead of serializing N sequential Telegram CDN fetches. Response latency
is now ~the slowest part fetch, not the sum of all part fetches.
- bot-pool.getFileInfo: cache file_id -> file_path in the existing in-memory
cache so repeated S3 GET/HEAD of the same object skip the Telegram API call
(file-controller had its own cache wrapper; the S3 path did not).
- chunked-storage + s3-controller: resolve multipart/chunked part CDN URLs
concurrently via Promise.all instead of sequentially.
- s3-controller multipart: await writer.end() before re-reading the temp part
file to avoid a flush race.
Adds object-stream-parallel.test.ts covering in-order fan-in, byte ranges,
and single-part responses even when the slowest part resolves out of order.
- Removed UploadBatcher (src/infrastructure/telegram/upload-batcher.ts + DI):
pending uploads no longer lost on crash, files sent directly to Telegram
- Changed upload-controller to use Bun.file().stream() instead of createReadStream
- Made PER_BOT_CONCURRENCY configurable via TELEGRAM_BOT_CONCURRENCY env
- Fixed 18 test files with updated import paths and mock shapes
- Updated package.json test script: telegramQueue.test.ts → bot-pool.test.ts
- Build, lint, and test suite all pass
- Add MAX_OUTER_RETRIES constant and transientAttempts counter for outer-loop retry
- Restore getFileInfo transient retry logging with bot identity and fileId
- Create test/bot-pool.test.ts with 4 tests for core BotPool behavior
- Add empty-bots guard in selectBot() returning null
- Add safety net comment and improved logging for outer 429 catch
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each bot has its own PQueue (concurrency=1). Uploads are assigned to
the least-loaded available bot. On 429, the bot is marked rate-limited
and the upload retries on the next available bot.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CRITICAL:
- Content-MD5 no longer loads entire file via arrayBuffer() — MD5 computed
incrementally in streamBodyToTemp alongside SHA-256 (fixes OOM for GB files)
HIGH:
- Add 120s timeout to Telegraf API calls via Promise.race in executeWithBotRetry
(prevents queue slot exhaustion from hung Telegram connections)
- Add queue size limit (1000 pending max) — reject new tasks when full
- Add graceful shutdown drain — waitForQueue with 30s timeout before exit
- Fix temp file leak when findFileByBucketAndKey throws (wrap in try-catch)
- Fix createReadStream fd leak — destroy stream on forwardToStorage error
- writer.end() wrapped in silent try-catch to prevent error swallowing
- writer.end() result ignored, writerFailed flag prevents double-end
MEDIUM:
- Remove 'retry after' from isTransientError patterns to stop double-retry
layering (was causing up to 96 bot attempts per chunk)
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Replace unsafe 'Function' type in test with ITelegramService interface
- Biome auto-fix formatting and import sorting across 8 files
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Remove rate limiting from all S3 endpoints (used by Docker registry
for concurrent blob pushes — 429 would abort the entire push).
- Add retry with exponential backoff in botPool.forwardToStorage for
transient Telegram errors (network timeouts, 5xx, socket issues).
- Add retry with exponential backoff in botPool.getFileInfo per bot.
- Introduce isTransientError() pattern matcher covering ~20 transient
error signatures.
- Fix temp file leak in handlePutObject when storeFileFromTemp throws.
- Fix pre-existing missing botPool namespace on getFileInfo call in
handleGetMultipartObject.
- Fix route handler return type in PUT handler.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Fix infrastructure imports: chunked-storage uses new path for shared/utils and interfaces/s3
- Fix health-controller: imports from infrastructure/persistence/drizzle instead of old db/
- Fix routes/index.ts: imports from new interfaces/s3 and middleware paths
- Marked Telegram-specific types in shared/utils/file.ts as future extraction
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
asepharyana
Claude