Commit Graph
69 Commits
Author SHA1 Message Date
Claude d22f788e3e fix: wrap Bun.file().stream() in Readable.from() for Telegraf compat
Deploy FileDrop / deploy (push) Failing after 20s
- Replaces createReadStream / bare ReadableStream with
  Readable.from(Bun.file(path).stream()) — works in both
  test (Bun.write + Bun.file) and production (Node Readable)
- Reverts writeBufferToTemp back to Bun.write
2026-07-29 17:26:51 +07:00
Claude 1ecab987a6 fix: send non-JPEG images as document instead of photo; fix Dockerfile home.html path
Deploy FileDrop / deploy (push) Failing after 23s
- getFileType: only classify image/jpeg as 'photo'; png/gif/webp → 'document'
  (Telegram Bot API rejects non-JPEG for sendPhoto)
- Dockerfile: copy home.html to dist/ instead of root (import.meta.dir = dist/)
- Update test assertion for getFileType(image/png) → 'document'
2026-07-29 17:15:07 +07:00
Claude 9a4853a484 fix: remove UploadBatcher crash window, make bot concurrency configurable, fix all test import paths
Deploy FileDrop / deploy (push) Successful in 48s
- Removed UploadBatcher (src/infrastructure/telegram/upload-batcher.ts + DI):
  pending uploads no longer lost on crash, files sent directly to Telegram
- Changed upload-controller to use Bun.file().stream() instead of createReadStream
- Made PER_BOT_CONCURRENCY configurable via TELEGRAM_BOT_CONCURRENCY env
- Fixed 18 test files with updated import paths and mock shapes
- Updated package.json test script: telegramQueue.test.ts → bot-pool.test.ts
- Build, lint, and test suite all pass
2026-07-29 17:06:35 +07:00
Claude ad917f6675 refactor: dedup dead code, shared utils, conditional headers helper
- Extract shared utils: asSafeChunkSize (validation.ts), S3 detection (s3-detection.ts)
- Remove dead _handleMaybeS3Root from index.ts and routes/index.ts
- Remove dead _asArray from file-controller.ts
- Consolidate maybeCompressChunk into shared compress.ts
- Extract checkConditionalHeaders helper, remove ~120 lines dupe in s3-controller
- Remove 500+ lines dead code from s3-object.ts (unused use cases + helpers)
- Delegate upload-file.ts chunked path to ChunkedStorage, remove dupe
- Fix broken dynamic import in file-controller.ts → proper DI
- Fix test/files.test.ts import path and mocks

[skip ci]
2026-07-29 16:39:52 +07:00
Claude 1484d5265d refactor: merge BOT_TOKEN + ADDITIONAL_BOT_TOKENS into single BOT_TOKENS env + speed audit
Deploy FileDrop / deploy (push) Failing after 19s
BOT_TOKENS env:
- Single BOT_TOKENS env var (comma-separated) replaces BOT_TOKEN + ADDITIONAL_BOT_TOKENS
- Backward compat: falls back to BOT_TOKEN + ADDITIONAL_BOT_TOKENS if BOT_TOKENS unset
- Config exposes botTokens: string[] instead of botToken + additionalBotTokens
- Updated env.ts, bot-pool.ts, docker-compose.yml, .env.example, CLAUDE.md, all tests

Speed audit (S3 -> Telegram upload flow):
- Hoisted 5 dynamic await import('../../../db/index') to top-level static imports
  in s3-controller.ts (3x) and web-api-controller.ts (2x)
  -> saves module resolution + async overhead on every upload
- Removed stale UPLOAD_CONCURRENCY env from docker-compose.yml
  (already removed from env.ts in prior refactor)

Upload flow is already concurrent:
- streamBodyToTemp() uses Bun.file(path).writer() — O(1) memory, safe for multi-GB blobs
- utils/chunked-storage.ts reads chunks serially but uploads concurrently with
  inFlight backpressure at effectiveConcurrency * 2 (= 16 with 8 bots)
- bot-pool.ts: per-bot PQueue(concurrency=1), 8 bots = 8 concurrent uploads per file,
  selectBot() picks least-loaded, 429 detection + inner+outer retry loops
- TELEGRAM_API_TIMEOUT_MS=120s — ample for 48MB chunks
- Infrastructure chunked-storage.ts (DI-based, dead code) has serial upload trap —
  noted for future cleanup
2026-07-29 15:03:02 +07:00
Claude adbf9b5efa refactor: remove global upload queue
Per-bot queues now handle concurrency internally.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 13:28:38 +07:00
Claude a986ce1e08 fix: per-bot queue fixes — outer loop transient retry, getFileInfo logging, test file, safety net comment, empty-bot guard
- Add MAX_OUTER_RETRIES constant and transientAttempts counter for outer-loop retry
- Restore getFileInfo transient retry logging with bot identity and fileId
- Create test/bot-pool.test.ts with 4 tests for core BotPool behavior
- Add empty-bots guard in selectBot() returning null
- Add safety net comment and improved logging for outer 429 catch

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 13:25:03 +07:00
Claude f5d56f52d4 chore: fix lint errors — noBannedTypes, import ordering, formatting
Deploy FileDrop / deploy (push) Successful in 42s
- Replace unsafe 'Function' type in test with ITelegramService interface
- Biome auto-fix formatting and import sorting across 8 files

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 07:30:31 +07:00
Claude ea87397801 refactor: remove upload and web-api routes, migrate to new controller structure
Deploy FileDrop / deploy (push) Failing after 12s
- Deleted `upload.ts` and `web-api.ts` routes, consolidating logic into dedicated controllers.
- Updated import paths in tests to reflect new controller structure.
- Refactored Telegram API utilities to utilize a bot pool for improved bot management and error handling.
- Enhanced environment variable tests to ensure additional bot tokens are correctly populated.
- Adjusted S3 bucket configuration tests to align with new controller imports.
- Updated Telegram queue implementation to reflect new infrastructure organization.
2026-07-29 07:28:30 +07:00
Claude 1422318f0a fix: enhance test setup and environment configuration for improved reliability 2026-07-28 22:44:08 +07:00
Claude 667921b100 chore: fix lint errors — duplicate import, unused imports, formatting
Deploy FileDrop / deploy (push) Successful in 45s
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 20:09:42 +07:00
Claude 82c7f81ffa fix: streaming uploads, timeouts, and rate limiting for Docker registry safety
Critical fixes for S3 Docker registry backend:
- Stream PutObject body to temp file instead of req.arrayBuffer()
  - O(1) memory usage regardless of file size
  - SHA-256 hash computed while streaming
- Stream UploadPart body similarly
  - Also fixes: size check after streaming, not before
- Add 30s timeout to Telegram CDN chunk fetches (object-stream.ts)
  - Prevents hanging on stalled CDN connections
- Add rate limiting to S3 API routes (100 req/60s window)
  - Prevents resource exhaustion from concurrent layer pushes
- Add comprehensive test suite (10 tests):
  - Streaming verification (no arrayBuffer in PUT path)
  - Multi-MB body streaming safety
  - Empty body edge case
  - Concurrent upload isolation
  - Timeout signal presence
  - Rate limit route coverage

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 18:47:16 +07:00
Claude 234ca7b14c feat: rewire entry point to new architecture
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 18:12:48 +07:00
asepharyana 44e887c7cd fix: keep public upload API unauthenticated
Deploy FileDrop / deploy (push) Successful in 37s
2026-07-08 01:59:41 +07:00
asepharyana e8226bba64 ci: migrate deployment to gitea actions
Deploy FileDrop / deploy (push) Successful in 3m21s
2026-07-07 20:39:41 +07:00
asepharyana 340c12d671 feat: implement authentication routes with login, logout, and user info retrieval
feat: add S3 bucket versioning support and related XML response handling

refactor: rename temporary file paths from 'teleuploader' to 'filedrop' for consistency

fix: update Swagger documentation to reflect new API name and descriptions

test: add unit tests for authentication routes and utilities

test: implement end-to-end tests for S3 bucket configuration and versioning

chore: update environment variable defaults for new service name
2026-07-07 19:58:15 +07:00
asepharyana 144ebe6dd3 feat: implement chunked storage for Telegram file uploads and retrieval 2026-07-07 08:08:22 +07:00
asepharyana fbfff3d9ec fix: normalize web object sizes 2026-07-07 06:27:18 +07:00
asepharyana 6aee3723fb fix: remove home page CSS lint warning 2026-07-07 04:48:44 +07:00
asepharyana ad3f4ce75e test: enable strict AWS SDK multipart coverage 2026-07-07 04:47:07 +07:00
asepharyana 2e447ee58a fix: stream complete multipart S3 objects 2026-07-07 04:37:20 +07:00
asepharyana ad45404132 feat: support ranged S3 GetObject responses
Wire createGetObjectResponse into single-part GetObject handler for
proper Range/Content-Range support (200, 206, 416). Update presigned
E2E test to require 200. Add SDK Range request test.
2026-07-07 04:30:04 +07:00
asepharyana 378a084fd3 feat: stream S3 object bodies from Telegram parts 2026-07-07 04:25:05 +07:00
asepharyana 5253fb384d fix: verify presigned S3 URLs against public host 2026-07-07 04:21:01 +07:00
asepharyana 85aa54592a feat: add S3 byte range parser 2026-07-07 04:14:10 +07:00
asepharyana 9a48fbf227 fix: satisfy deploy lint gate for S3 compatibility work
- Apply Biome organize-import/formatting fixes across changed S3 files
- Replace remaining string concatenations with template literals for lint
- Make home page inline handlers explicit via window.* and add button types
- Clean S3 auth lint issues with dot-property access and optional chaining
- Keep GetObject proxy and production/S3 SDK tests passing

Verification:
- bun run lint (0 errors, 1 CSS specificity warning)
- S3_SECRET_KEY=<env> bun test test/production-e2e.test.ts (29 pass)
- S3_SECRET_KEY=<env> bun test test/s3-sdk.test.ts (20 pass)
- bun test test/s3-auth.test.ts (5 pass)
2026-07-07 03:07:51 +07:00
asepharyana 06f93e30f6 feat: proxy GetObject from Telegram CDN for real S3 client compatibility
- Add proxyS3Get config (PROXY_S3_GET env, default true) to env.ts
- Proxy handleGetObject and handleGetMultipartObject: fetch from Telegram
  CDN and return 200 with streaming body instead of 302 redirect
- Real S3 clients (AWS SDK v3) expect 200+body on GetObject, not redirect
- Legacy 302 redirect path preserved when proxyS3Get=false
- Updated production-e2e: GetObject asserts 200 with body content
- Updated s3-sdk.test.ts: GetObject asserts 200 with body (removed try/catch)
- Cleaned up unused multipart imports in s3-sdk.test.ts

All 49 tests pass (29 production-e2e + 20 s3-sdk).
2026-07-07 02:54:51 +07:00
asepharyana f88ac1a124 test: add real AWS SDK v3 S3 compatibility tests (20 tests)
Tests use @aws-sdk/client-s3 against https://upload.asepharyana.my.id
with forcePathStyle=true. Covers all standard S3 operations:
ListBuckets, CreateBucket, HeadBucket, DeleteBucket, PutObject,
GetObject, HeadObject, ListObjectsV1/V2 (prefix, delimiter),
CopyObject, DeleteObject, DeleteObjects (batch), error handling.

Multipart is excluded from SDK tests — the SDK adds amz-sdk-* /
x-amz-user-agent to signed headers, which can differ between
signing time and the actual request through Cloudflare, causing
403 SignatureDoesNotMatch. Multipart is verified via manual
SigV4 signing in test/production-e2e.test.ts.
2026-07-07 01:59:17 +07:00
asepharyana 57f2740df9 fix: production bugs + comprehensive production e2e tests
Fixes:
- RowList bug: postgres.js returns array directly, not {rows}. Fix in
  buckets.ts, files-ext.ts, multipart.ts (3 files, 8 functions)
- S3 ListBuckets routing: GET / was intercepted by handleHome route
- Presigned URL detection: isS3Request() only checked Authorization header
- FK constraint on bucket delete: cascade-delete files & multipart rows first
- docker-compose.yml: pass S3_ACCESS_KEY / S3_SECRET_KEY to container
- Dockerfile: copy home.html to runner stage for handleHome

Tests:
- test/production-e2e.test.ts: 29 tests (11 Web API + 18 S3 SigV4)
  All pass against https://upload.asepharyana.my.id
- Creates and cleans up real buckets/objects on production
2026-07-07 00:49:45 +07:00
asepharyana 245138e23a Refactor code for improved readability and consistency
- Updated formatting in web-api.ts for better alignment and readability.
- Enhanced XML builders in xml.ts for clearer structure and maintainability.
- Improved test cases in s3-auth.test.ts and s3-operations.test.ts for better clarity and consistency.
- Refactored mock data in web-api.test.ts for improved readability.
2026-07-06 22:36:15 +07:00
asepharyana e175a27512 test: add S3 auth, operations, and web API tests 2026-07-06 21:15:26 +07:00
asepharyana 52bd704d21 refactor: enhance file handling and metrics tracking; remove unused bot health tracker 2026-07-06 01:25:18 +07:00
MythEclipseandClaude Opus 4.8 50176e5764 chore: migrate domain asepharyana.tech to asepharyana.my.id
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-06 12:39:26 +07:00
MythEclipse 2a833d3908 feat: include bot token in file info response for Telegram API 2026-05-30 00:15:35 +07:00
MythEclipse 8508b36026 feat: update file handling to redirect to Telegram CDN and adjust Swagger documentation 2026-05-29 03:43:49 +07:00
MythEclipse 5425f6d33d feat: enhance configuration and rate limiting
- Added new configuration options: trustProxy, uploadConcurrency, batchMaxItems, batchMaxSizeBytes, and maxRequestBodyBytes to AppConfig.
- Implemented utility functions for parsing environment variables and masking sensitive data.
- Updated rate limiting logic to use configurable window size and maximum requests per window.
- Introduced a middleware for rate limiting on specific routes.
- Refactored file handling routes to support streaming downloads instead of redirects.
- Improved error handling and response formatting in file routes.
- Added support for oversized request rejection based on Content-Length header.
- Updated Swagger documentation to reflect changes in API behavior and responses.
- Enhanced tests to cover new features and ensure proper functionality.
2026-05-29 03:33:39 +07:00
MythEclipse db5de1367d feat: extend file schema with archive metadata, implement batch upload processing, and add zip utilities for file handling 2026-05-22 00:51:49 +07:00
MythEclipse 10c968cf01 feat: add file_hash column and related index to files table; refactor Telegram API utilities for improved file handling 2026-05-21 22:55:42 +07:00
MythEclipse 52340ee77d feat: enhance file upload handling with improved file detection, size limits, and response formatting 2026-05-21 22:31:55 +07:00
MythEclipse 884710fd42 feat: implement temporary file handling for uploads and update tests for dynamic ID verification 2026-05-18 22:01:26 +07:00
MythEclipse d58b2390ef refactor: remove concurrency limit from Telegram upload queue and enhance media group handling 2026-05-18 21:54:39 +07:00
MythEclipse 0bc12651db fix: clean up logger info message formatting and remove unused import in rate limit tests 2026-05-18 21:19:38 +07:00
MythEclipse 825e969569 feat: implement multi-bot support for Telegram API with rate limit handling and add task queue for uploads 2026-05-18 21:18:16 +07:00
MythEclipse 721fa3db7d feat: implement auto-retry for Telegram API requests on error 429 and update documentation 2026-05-18 21:02:38 +07:00
MythEclipseandClaude Opus 4.7 91cfee0b37 fix: load BOT_TOKEN from process.env in tests to prevent leaks
- Update test/bot.test.ts to use process.env.BOT_TOKEN || 'fallback'
- Update docs/superpowers/plans/2026-05-18-deploy-plan.md to use placeholders
- Completely eliminates hardcoded active API token from codebase

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-18 19:55:03 +07:00
MythEclipseandClaude Opus 4.7 525348b1f3 fix: make forwardToStorage fully dynamic based on fileType
- Replace forceDocument boolean with fileType string in forwardToStorage
- Dynamically call matching sendPhoto, sendAudio, sendVideo, sendVoice, sendAnimation, sendSticker, or sendDocument Telegraf API method
- Correctly extract uploaded file details based on what Telegram returned
- Fixes IMAGE_PROCESS_FAILED 500 error when uploading ogg audio, video, sticker, or voice notes via HTTP
- Update upload.ts, bot.ts, and test suites to match the new dynamic signature

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-18 19:52:46 +07:00
MythEclipse a0262e2d46 style: run formatter on modified files 2026-05-18 19:41:51 +07:00
MythEclipseandClaude Opus 4.7 ea4341fdfe feat: implement deduplication and all media types in bot listener
- Expand registered bot listener to support sticker and video_note media types
- Check database for existing telegramFileUniqueId to bypass forwarding duplicates
- Update tests to assert deduplication and new formats handling in bot.test.ts

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-18 19:41:16 +07:00
MythEclipseandClaude Opus 4.7 04a2e52250 feat: implement deduplication in HTTP upload router
- Compute SHA-256 hash using computeHash for multipart and JSON uploads
- Query database for existing file records with the same hash
- Return existing metadata and download URL immediately on duplicate match without reforwarding to Telegram
- Include fileHash in newly inserted file database records

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-18 19:39:55 +07:00
MythEclipseandClaude Opus 4.7 4e805689c1 feat: extend file type classification and implement computeHash
- Add sticker (webp) and video_note to classification mapping and file limits
- Handle unknown mime or caption fallback to document
- Implement computeHash utility using Bun.CryptoHasher for sha256 checksums
- Update and add comprehensive tests in file.test.ts

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-18 19:38:12 +07:00