Compare commits

...
95 Commits
Author SHA1 Message Date
asepharyana 88e2a3faad ci: add Nix GC cleanup job on VPS after deploy 2026-08-04 13:57:45 +07:00
aseph 75b6d0a527 ci: use free GHA Nix cache (disable FlakeHub cache, not subscribed) 2026-08-03 16:44:14 +07:00
asepharyana 0c2ee3b3cf ci: enable FlakeHub Cache (id-token: write + use-flakehub) 2026-08-03 16:20:34 +07:00
asepharyana 12b6a133a7 ci: use biome lint gate instead of flaky bun test suite (bun 1.3.14 module resolution bug) 2026-08-03 13:41:32 +07:00
asepharyana d2e2b450f6 ci: add test gate before Nix deploy 2026-08-03 13:37:36 +07:00
asepharyana 17d877e2c6 docs: sync remaining .md to 4000s infra 2026-08-02 16:46:31 +07:00
asepharyana 62021397e0 chore: sync port references and docs to 4000s infra 2026-08-02 16:19:30 +07:00
asepharyana ad7747bfd4 chore: update port to 4000 in tests and compose 2026-08-02 14:30:58 +07:00
asepharyana 66fb92e5af test(e2e): make 1GB suite opt-in (RUN_LARGE_E2E=1) + fail-fast source probe
The 1GB suite downloads a 1GB fixture from Hetzner and uploads it to the
live deployment. Bun's Response-body write hangs against this source, so
the suite burned the full hook timeout on every routine run. Now:
- skipped by default (opt-in via RUN_LARGE_E2E=1)
- pre-flight Range probe (8s AbortSignal) fails fast when source is down
- 10min hook timeout retained for when the suite is actually opted in
2026-08-01 21:02:10 +07:00
asepharyana 673c934f37 test(e2e): 10min timeout for 1GB beforeAll download hook
Bun's default hook timeout (5s) killed the Hetzner 1GB download in beforeAll.
2026-08-01 20:43:00 +07:00
asepharyana 90d6c7dd6f fix(web): export showAuthScreen to window so topbar Login works
The Login button (onclick=window.showAuthScreen()) was dead — showAuthScreen
was defined but never exported via Object.assign. Also drop dead code from
the e2e suite (unused createReadStream import, unused s3StreamRequest).
2026-08-01 20:34:22 +07:00
asepharyana 2653e23283 test(e2e): auth headers on web API uploads + 1GB large-file suite
- Web API upload/delete tests now send ADMIN_API_TOKEN (POST/DELETE require auth)
- New 1GB suite: S3 single PUT, S3 multipart (40MB parts), ListObjects verify,
  POST /api/upload; skips without S3_SECRET_KEY + ADMIN_API_TOKEN
- Fix: Bun.write accepts the Response directly (dl.stream() doesn't exist)
2026-08-01 20:33:10 +07:00
asepharyana 91ec588a88 feat(web): public read-only file browser — GET API public, writes require admin
- /api/v1/* GET (list buckets/objects, download) no longer requires auth
- POST/DELETE/PUT stay behind requireAuth (upload, create/delete bucket, copy, delete object)
- FE drops blocking login screen: visitors browse + download freely
- Admin-only UI (create bucket, upload dropzone, delete, S3 creds) hidden in read-only mode
- Login button in topbar to unlock admin actions
2026-08-01 20:31:40 +07:00
asepharyana 864d41d8fc fix(nix): restrict flake to x86_64-linux (nixpkgs 26.11 dropped darwin) 2026-08-01 18:03:38 +07:00
asepharyana c14be68ff8 ci: publish flake to FlakeHub (rolling) 2026-08-01 17:58:16 +07:00
asepharyana 811a68821d ci: migrate CI to GitHub Actions (deploy nix + mirror ke Gitea backup)
Mirror to Gitea / mirror (push) Successful in 22s
Build & Deploy (Nix) / build-and-deploy (push) Failing after 35m18s
2026-08-01 16:38:58 +07:00
asepharyana bf616f6790 fix: root page 500 — resolve home.html for dev & prod layouts
Build & Deploy (Nix) / build-and-deploy (push) Successful in 54s
handleHome looked up `${import.meta.dir}/home.html` which exists in neither
layout: dev (src/interfaces/http/controllers/) nor prod bundle
($out/share/teleuploader/dist/ — flake copies home.html beside dist/).

Add resolveHomeHtml(): walk up from import.meta.dir (bounded) to find
home.html. Works for dev (src/home.html, 4 levels up) and prod
(../home.html, 1 level up). Fails fast with a clear error instead of a
bare ENOENT 500. Tests cover both layouts + not-found fallback.
2026-08-01 13:15:10 +07:00
asepharyana 24dfb1c6b1 fix: fail-fast guard TELEGRAM_CHUNK_SIZE_BYTES <= 19MB (Telegram getFile limit 20MB)
Build & Deploy (Nix) / build-and-deploy (push) Successful in 56s
Chunk parts > 19MB are stored to Telegram but getFile cannot resolve files over 20MB ('Bad Request: file is too big'), making every part undownloadable (prod bug 2026-08-01: 48MB chunk -> download 500).

- src/env.ts: reject TELEGRAM_CHUNK_SIZE_BYTES > 19922944 at startup (log error + throw), default changed 20MB -> 19MB
- src/shared/utils/validation.ts: TELEGRAM_CHUNK_SIZE_MAX_BYTES constant; asSafeChunkSize now enforces the max at runtime (covers S3 multipart parts too)
- test/env.test.ts: unit tests + subprocess fail-fast tests (48MB rejected, 19MB accepted)
- test/helpers/setup-env.ts: pin safe chunk size so a stale .env can't break the suite
- .env.example + CLAUDE.md: document the 20MB getFile limit
2026-08-01 13:06:42 +07:00
Claude b9a3fcd828 ci: nix build in CI + profile-based deploy (auto-update systemd)
Build & Deploy (Nix) / build-and-deploy (push) Successful in 49s
2026-07-30 16:27:35 +07:00
Claude f00b944e56 fix: nix build di CI — init none + flakes enabled
Build & Deploy (Nix) / build-and-deploy (push) Successful in 49s
2026-07-30 16:23:15 +07:00
Claude 73b0630cbd v1.1.0 + ci: build di CI, nix copy ke VPS (ringan, gak build di target)
Build & Deploy (Nix) / build-and-deploy (push) Successful in 1m0s
2026-07-30 16:13:59 +07:00
Claude d1771720bd ci: build di VPS target via SSH (seperti yang sudah terverifikasi)
Build & Deploy (Nix) / build-and-deploy (push) Successful in 12s
2026-07-30 16:08:13 +07:00
Claude 7dd2a4f0c7 fix: build di CI, nix copy ke VPS target (ringan)
Build & Deploy (Nix) / build-and-deploy (push) Failing after 56s
2026-07-30 16:05:20 +07:00
Claude fbc1f1822a fix: deploy via SSH, build + deploy on VPS directly
Build & Deploy (Nix) / build-and-deploy (push) Successful in 15s
The CI runner container can't access the VPS host's systemd and Nix
directly. Instead of installing Nix in the container and trying to
access the host, SSH directly to the VPS to build and deploy.

This approach:
1. SSHs to the VPS using the VPS_SSH_KEY_VALUE secret
2. Pulls the latest code on the VPS
3. Builds with Nix directly on the VPS
4. Updates nix-env profile and restarts systemd service

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 15:43:14 +07:00
Claude c768b29e9d fix: deploy via Docker socket with host filesystem access
Build & Deploy (Nix) / build-and-deploy (push) Failing after 53s
Docker socket /var/run/docker.sock is available in the runner
container. Use docker run --pid=host --privileged to access
the VPS host filesystem via chroot to execute nix-env and systemctl.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 15:39:38 +07:00
Claude 59535d66b8 fix: add environment inspect step before deploy
Build & Deploy (Nix) / build-and-deploy (push) Failing after 46s
Diagnosing deploy failure: systemctl unavailable inside runner container.
Adding inspect step to understand available mounts and access mechanisms.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 15:35:47 +07:00
Claude 319d2980c5 fix: use full path for nix-env in deploy step
Build & Deploy (Nix) / build-and-deploy (push) Failing after 47s
The deploy step runs with 'sudo' which resets PATH, so nix-env is
not found. Use explicit path to the nix-env binary.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 15:33:29 +07:00
Claude 4acfa3d2e9 fix: use Determinate Systems nix installer without --no-daemon
Build & Deploy (Nix) / build-and-deploy (push) Failing after 52s
Re-instate Determinate Systems installer with correct flags:
- install --no-confirm (not --no-daemon which it doesn't support)
- Nix installs to /nix/var/nix/profiles/default/bin
- Source daemon profile in build step

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 15:32:09 +07:00
Claude 5bd3d286ce fix: use Nix official installer with --no-daemon
Build & Deploy (Nix) / build-and-deploy (push) Failing after 11s
The Determinate Systems installer doesn't support --no-daemon.
Switch to the official Nix installer which has a well-documented
--no-daemon flag suitable for container/CI use.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 15:30:59 +07:00
Claude bfafcc0a20 fix: install Nix in CI before building
Build & Deploy (Nix) / build-and-deploy (push) Failing after 7s
The Gitea Actions runner image (docker.gitea.com/runner-images:ubuntu-latest)
does not have Nix pre-installed, causing 'nix: command not found' on the
build step.

Add an 'Install Nix' step using the Determinate Systems installer
(--no-daemon mode since the runner runs in a container).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 15:29:46 +07:00
Claude c26ca806d0 fix: update CI to Nix build + systemd deploy (no Docker)
Build & Deploy (Nix) / build-and-deploy (push) Failing after 5s
2026-07-30 15:08:23 +07:00
Claude bc3d1f2e3f chore: trigger CI
Deploy FileDrop / deploy (push) Successful in 1m35s
2026-07-30 14:37:10 +07:00
Claude f95022142d feat: add Nix flake for TeleUploader build
Deploy FileDrop / deploy (push) Waiting to run
- flake.nix with Bun build from nixpkgs (1.3.x)
- flake.lock pinned inputs
- .gitignore result symlink from nix build

Build: nix build .#teleuploader --impure --option sandbox false
Until CI/CD with binary cache is set up
2026-07-30 13:00:53 +07:00
Claude bd766792b1 a
Deploy FileDrop / deploy (push) Failing after 14m44s
2026-07-29 19:25:31 +07:00
Claude d22f788e3e fix: wrap Bun.file().stream() in Readable.from() for Telegraf compat
Deploy FileDrop / deploy (push) Failing after 20s
- Replaces createReadStream / bare ReadableStream with
  Readable.from(Bun.file(path).stream()) — works in both
  test (Bun.write + Bun.file) and production (Node Readable)
- Reverts writeBufferToTemp back to Bun.write
2026-07-29 17:26:51 +07:00
Claude 1ecab987a6 fix: send non-JPEG images as document instead of photo; fix Dockerfile home.html path
Deploy FileDrop / deploy (push) Failing after 23s
- getFileType: only classify image/jpeg as 'photo'; png/gif/webp → 'document'
  (Telegram Bot API rejects non-JPEG for sendPhoto)
- Dockerfile: copy home.html to dist/ instead of root (import.meta.dir = dist/)
- Update test assertion for getFileType(image/png) → 'document'
2026-07-29 17:15:07 +07:00
Claude 9a4853a484 fix: remove UploadBatcher crash window, make bot concurrency configurable, fix all test import paths
Deploy FileDrop / deploy (push) Successful in 48s
- Removed UploadBatcher (src/infrastructure/telegram/upload-batcher.ts + DI):
  pending uploads no longer lost on crash, files sent directly to Telegram
- Changed upload-controller to use Bun.file().stream() instead of createReadStream
- Made PER_BOT_CONCURRENCY configurable via TELEGRAM_BOT_CONCURRENCY env
- Fixed 18 test files with updated import paths and mock shapes
- Updated package.json test script: telegramQueue.test.ts → bot-pool.test.ts
- Build, lint, and test suite all pass
2026-07-29 17:06:35 +07:00
Claude ad917f6675 refactor: dedup dead code, shared utils, conditional headers helper
- Extract shared utils: asSafeChunkSize (validation.ts), S3 detection (s3-detection.ts)
- Remove dead _handleMaybeS3Root from index.ts and routes/index.ts
- Remove dead _asArray from file-controller.ts
- Consolidate maybeCompressChunk into shared compress.ts
- Extract checkConditionalHeaders helper, remove ~120 lines dupe in s3-controller
- Remove 500+ lines dead code from s3-object.ts (unused use cases + helpers)
- Delegate upload-file.ts chunked path to ChunkedStorage, remove dupe
- Fix broken dynamic import in file-controller.ts → proper DI
- Fix test/files.test.ts import path and mocks

[skip ci]
2026-07-29 16:39:52 +07:00
Claude fab91ad69c feat: dedup wave 3 — constant + streamToTemp utility
Deploy FileDrop / deploy (push) Successful in 44s
- ♻️ randomUUID() → nanoid() di s3-object.ts (eliminasi node:crypto)
-  DEFAULT_FILE_TYPE constant, ganti 8× hardcoded 'document'
-  streamToTemp() shared utility (src/shared/utils/temp-stream.ts)
- ♻️ 3× streaming-to-temp pattern di s3-controller, upload-controller,
  web-api-controller → pake streamToTemp()
- Lint  Build 
2026-07-29 16:10:49 +07:00
Claude 3501d547c0 feat: dedup wave 2 — file factory + V1/V2 mapping + bot handler
Deploy FileDrop / deploy (push) Successful in 42s
-  buildNewFile() factory (src/domain/entities/file-factory.ts)
  eliminates ~380 lines of archive null / isDeleted / field defaults
- ♻️ 27 creation blocks → buildNewFile() across 8 files
- ♻️ S3 V1/V2 object listing mapping → mapFileToListEntry()
- ♻️ Bot handler size validation → checkFileSize() from shared utils
- 🔎 computeHash audit: all 10 instances are streaming (no dedup)
- Lint  Build  Push 
2026-07-29 16:02:01 +07:00
Claude 332853f398 chore: dedup dead code + compress utility
Deploy FileDrop / deploy (push) Successful in 43s
- Hapus src/shared/utils/retry.ts (0 imports, dead)
- Hapus src/shared/errors/index.ts (7 classes, 0 imports)
- Hapus drizzle type exports dari schema.ts (tidak dipake)
- Hapus src/config/index.ts, redirect 8 imports langsung ke env.ts
- Ekstrak maybeCompressChunk ke shared/utils/compress.ts
- Hapus duplikasi gzipSync di upload-file.ts + chunked-storage.ts
- Lint clean, build clean
2026-07-29 15:51:39 +07:00
Claude 4c216b1d9f refactor: full DDD + Clean Architecture refactor
Deploy FileDrop / deploy (push) Successful in 48s
- Hapus src/utils/ (17 files) + src/db/ (8 files) dead code
- Absorb 7 re-export stubs → real impl di lokasi DDD
- Buat src/infrastructure/di.ts (DI container)
- Rewrite 5 controllers pakai repository/DI
- Fix shared/utils imports, env.ts, routes, index.ts
- Update package.json build path migrate
- Lint clean, build clean
2026-07-29 15:41:50 +07:00
Claude 3580001b8a fix: add backward compat env vars to docker-compose for Gitea CI
Deploy FileDrop / deploy (push) Successful in 27s
2026-07-29 15:08:49 +07:00
Claude 6d3696d261 fix: handler.ts uses config.botTokens[0] instead of removed config.botToken
Deploy FileDrop / deploy (push) Successful in 27s
2026-07-29 15:08:23 +07:00
Claude 1484d5265d refactor: merge BOT_TOKEN + ADDITIONAL_BOT_TOKENS into single BOT_TOKENS env + speed audit
Deploy FileDrop / deploy (push) Failing after 19s
BOT_TOKENS env:
- Single BOT_TOKENS env var (comma-separated) replaces BOT_TOKEN + ADDITIONAL_BOT_TOKENS
- Backward compat: falls back to BOT_TOKEN + ADDITIONAL_BOT_TOKENS if BOT_TOKENS unset
- Config exposes botTokens: string[] instead of botToken + additionalBotTokens
- Updated env.ts, bot-pool.ts, docker-compose.yml, .env.example, CLAUDE.md, all tests

Speed audit (S3 -> Telegram upload flow):
- Hoisted 5 dynamic await import('../../../db/index') to top-level static imports
  in s3-controller.ts (3x) and web-api-controller.ts (2x)
  -> saves module resolution + async overhead on every upload
- Removed stale UPLOAD_CONCURRENCY env from docker-compose.yml
  (already removed from env.ts in prior refactor)

Upload flow is already concurrent:
- streamBodyToTemp() uses Bun.file(path).writer() — O(1) memory, safe for multi-GB blobs
- utils/chunked-storage.ts reads chunks serially but uploads concurrently with
  inFlight backpressure at effectiveConcurrency * 2 (= 16 with 8 bots)
- bot-pool.ts: per-bot PQueue(concurrency=1), 8 bots = 8 concurrent uploads per file,
  selectBot() picks least-loaded, 429 detection + inner+outer retry loops
- TELEGRAM_API_TIMEOUT_MS=120s — ample for 48MB chunks
- Infrastructure chunked-storage.ts (DI-based, dead code) has serial upload trap —
  noted for future cleanup
2026-07-29 15:03:02 +07:00
Claude 245ea169ad chore: add 2 bot tokens, update PRODUCTION_ENV for 6 bots
Deploy FileDrop / deploy (push) Successful in 52s
- Now 6 total bots: 1 main + 5 additional
- Removed UPLOAD_CONCURRENCY from PRODUCTION_ENV

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 14:48:50 +07:00
Claude 2cf6aa3275 refactor: implement per-bot queue architecture
Deploy FileDrop / deploy (push) Successful in 45s
- Each bot has its own PQueue with concurrency=1
- selectBot() assigns uploads to least-loaded available bot
- 429 rate limits are tracked per-bot with cooldown timers
- Failed uploads retry on next available bot
- Removed global upload-queue.ts and uploadConcurrency config
- Updated ITelegramService interface
2026-07-29 13:36:16 +07:00
Claude d0de4de2d5 refactor: remove uploadConcurrency from config
Effective concurrency derived from bot pool size. Chunked-storage
backpressure now uses botPool.getEffectiveConcurrency().

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 13:30:49 +07:00
Claude adbf9b5efa refactor: remove global upload queue
Per-bot queues now handle concurrency internally.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 13:28:38 +07:00
Claude a986ce1e08 fix: per-bot queue fixes — outer loop transient retry, getFileInfo logging, test file, safety net comment, empty-bot guard
- Add MAX_OUTER_RETRIES constant and transientAttempts counter for outer-loop retry
- Restore getFileInfo transient retry logging with bot identity and fileId
- Create test/bot-pool.test.ts with 4 tests for core BotPool behavior
- Add empty-bots guard in selectBot() returning null
- Add safety net comment and improved logging for outer 429 catch

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 13:25:03 +07:00
Claude 5617d0ff35 refactor: per-bot queue with selectBot() and rate-limit tracking
Each bot has its own PQueue (concurrency=1). Uploads are assigned to
the least-loaded available bot. On 429, the bot is marked rate-limited
and the upload retries on the next available bot.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 13:17:49 +07:00
Claude d7d6ae0f0d refactor: remove enqueueUpload from ITelegramService and BotPool
Per-bot queue replaces global upload queue — BotPool handles
queueing internally.

- Remove enqueueUpload method signature from ITelegramService interface
- Remove enqueueUpload method from BotPool class
- Remove import of enqueueUpload from upload-queue module
- Refactor forwardToStorage to call executeWithBotRetry directly
  instead of wrapping via enqueueUpload
- Fix trailing blank lines flagged by Biome formatter

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 13:14:26 +07:00
Claude ea31c4c591 docs: add per-bot queue design spec
Per-bot queue architecture for Telegram upload rate-limit safety.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 13:09:34 +07:00
Claude 996b7d06ef chore: add more bot tokens for better rate limit distribution
Deploy FileDrop / deploy (push) Successful in 41s
Previous: 4 bots → now: 6 bots
Each bot handles fewer concurrent uploads, reducing 429 rate limits
and avoiding thundering-herd sleep patterns.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 13:02:00 +07:00
Claude 3f541121f5 fix: yield microtask after Promise.race in backpressure check
Deploy FileDrop / deploy (push) Successful in 48s
Adds a setTimeout(0) microtask yield after Promise.race to ensure
the .finally() handler that removes promises from the inFlight
set has executed before the next backpressure check.

Also ensure parts array is sorted by partNumber after concurrent
uploads complete, since promises resolve in arbitrary order.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 12:44:14 +07:00
Claude 1a38b32fbc feat: concurrent chunk uploads within single file
Deploy FileDrop / deploy (push) Successful in 41s
Previously uploadFileInTelegramChunks awaited each chunk's upload
before reading the next, making all chunks sequential within a file.
Now chunks are uploaded concurrently using a managed Set of in-flight
promises with backpressure limiting (2x uploadConcurrency).

This means a single 1GB Docker layer split into 48MB chunks will
have up to 32 chunks uploading simultaneously, not one at a time.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 12:28:05 +07:00
Claude b034c3131d chore: trigger deploy for TELEGRAM_CHUNK_SIZE_BYTES=48MB
Deploy FileDrop / deploy (push) Successful in 52s
2026-07-29 12:23:44 +07:00
Claude d8b8a1381d chore: increase Telegram chunk size to 48MB (max ~49MB)
Deploy FileDrop / deploy (push) Successful in 44s
Tested actual Telegram Bot API limit:
- 49MB 
- 50MB  (413 Request Entity Too Large)
Set TELEGRAM_CHUNK_SIZE_BYTES=50331648 (48MB) for safety

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 11:57:46 +07:00
Claude b0c5327bb3 fix: add content_type column to multipart_uploads migration
Deploy FileDrop / deploy (push) Successful in 50s
The createMultipartUpload function inserts content_type but the
database column was missing, causing 500 errors on every Gitea
Docker registry push (which uses multipart uploads for blob storage).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 10:09:24 +07:00
Claude e3e7430ce0 fix: zero-downtime deploy with --wait flag + fallback
Deploy FileDrop / deploy (push) Canceled after 0s
2026-07-29 09:57:47 +07:00
Claude d8b1878a75 fix: add ETag and headers to 304 Not Modified responses
Deploy FileDrop / deploy (push) Failing after 14m18s
AWS SDK requires ETag header in 304 responses. Without it, the SDK
throws UnknownError despite receiving a valid 304 status code.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 09:23:16 +07:00
Claude e50e297e79 fix: add ETag and headers to 304 Not Modified responses
Deploy FileDrop / deploy (push) Failing after 15s
AWS SDK requires ETag header in 304 responses. Without it, the SDK
throws UnknownError despite receiving a valid 304 status code.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 09:19:57 +07:00
Claude 888af45a7f fix: normalizeUri SigV4 trailing slash + cleanup debug logs
Deploy FileDrop / deploy (push) Successful in 35s
2026-07-29 09:16:07 +07:00
Claude cd9852a5ec fix: preserve trailing slashes in normalizeUri for SigV4
Deploy FileDrop / deploy (push) Successful in 36s
The empty-segment skip in normalizeUri (introduced in round 1 fix)
was stripping trailing slashes from canonical URIs, e.g. /bucket/
became /bucket. The AWS SDK signs with the trailing slash intact, so
signatures never matched for any S3 operation with a body.

The fix: only skip '.' segments (dot-segment removal per RFC 3986),
preserve all other segments including empty ones from trailing
slashes and double slashes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 09:14:18 +07:00
Claude c18b9d265b fix: preserve trailing slash in SigV4 canonical URI
Deploy FileDrop / deploy (push) Successful in 35s
AWS SDK includes trailing slash in the canonical URI for bucket
operations (e.g. PUT /bucket-name/). My earlier 'fix' that stripped
trailing slashes broke SigV4 signature verification. The trailing
slash is intentional per AWS SigV4 — only dot-segments are removed,
not trailing slashes.

Re-verified with @smithy/signature-v4: path /bucket-name/ produces
the client signature, while /bucket-name does not match.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 09:08:17 +07:00
Claude 66c4247e64 debug: add SigV4 mismatch logging with expected vs received signature
Deploy FileDrop / deploy (push) Successful in 38s
2026-07-29 09:03:51 +07:00
Claude 9bc2f22589 fix: strip trailing slash in SigV4 canonical URI
Deploy FileDrop / deploy (push) Successful in 36s
AWS SigV4 canonical URI must not have trailing slash (except root '/').
Bun can receive paths with trailing slash from SDK, causing signature
mismatch for all bucket operations (CreateBucket, HeadBucket, etc.)

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 08:56:08 +07:00
Claude f9484738db debug: add SigV4 canonical request logging for troubleshooting
Deploy FileDrop / deploy (push) Successful in 36s
2026-07-29 08:54:10 +07:00
Claude 2882247ad3 fix: add catch-all S3 route for path-style requests (/{bucket}/{key})
Deploy FileDrop / deploy (push) Successful in 36s
Bun's '/' route only matches root path '/'. S3 SDK clients using
forcePathStyle:true send ALL requests to /{bucket}/{key} which never
matched any route → 404. Added '/*' catch-all that checks for S3
auth headers before dispatching.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 08:50:02 +07:00
Claude d8da2044b2 fix: S3→Telegram upload pipeline — OOM, queue limits, shutdown drain, timeouts
Deploy FileDrop / deploy (push) Successful in 46s
CRITICAL:
- Content-MD5 no longer loads entire file via arrayBuffer() — MD5 computed
  incrementally in streamBodyToTemp alongside SHA-256 (fixes OOM for GB files)

HIGH:
- Add 120s timeout to Telegraf API calls via Promise.race in executeWithBotRetry
  (prevents queue slot exhaustion from hung Telegram connections)
- Add queue size limit (1000 pending max) — reject new tasks when full
- Add graceful shutdown drain — waitForQueue with 30s timeout before exit
- Fix temp file leak when findFileByBucketAndKey throws (wrap in try-catch)
- Fix createReadStream fd leak — destroy stream on forwardToStorage error
- writer.end() wrapped in silent try-catch to prevent error swallowing
- writer.end() result ignored, writerFailed flag prevents double-end

MEDIUM:
- Remove 'retry after' from isTransientError patterns to stop double-retry
  layering (was causing up to 96 bot attempts per chunk)

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 08:40:50 +07:00
Claude de7d276245 fix: round 2 S3 audit — CRITICAL SigV4 payload hash bug, timeouts, Content-MD5/Length validation
Deploy FileDrop / deploy (push) Successful in 43s
CRITICAL:
- SigV4 canonical request used sha256Hex('') instead of x-amz-content-sha256
  header value — every PUT/POST with body would fail 403. Now uses the
  signed header value for canonical request, verifyBodyHash after streaming
  for integrity.

HIGH:
- Add 30s AbortSignal.timeout to all Telegram CDN fetches in object-stream.ts
  (previously could hang indefinitely, exhausting connection pool)

MEDIUM:
- Content-MD5 validation: compute and compare when header is present
- Content-Length validation: reject if actual body size != header
- max-keys=0 clamping: enforce minimum of 1 per S3 spec

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 08:22:45 +07:00
Claude e1e228430f chore: add pre-commit hook with husky — wajib lint pass sebelum commit
Deploy FileDrop / deploy (push) Failing after 10s
- Install husky v9, init .husky/pre-commit
- Hook runs 'bun run lint' and rejects commit on failure
- 'bun install' auto-activates hooks via prepare script

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 08:15:06 +07:00
Claude af160e0f33 fix: audit S3 protocol — 15+ security & correctness fixes
Deploy FileDrop / deploy (push) Successful in 43s
HIGH severity fixes:
- H1: Bot token leak via 302 redirect — always proxy S3 GETs
- H2: PUT TOCTOU race — add unique partial index (bucket_id, s3_key) WHERE NOT deleted
- H3: GET/HEAD ignore conditional headers (If-Match, If-None-Match, etc.)
- H4: Body payload hash not verified — add verifyBodyHash() post-stream check
- H5: Header-based auth has no expiry check — add 15-min clock skew window
- H7: Multipart abort does not delete parts — DELETE before UPDATE status
- H8: CompleteMultipartUpload skips part number & etag verification
- H9: XML regex fails on keys containing < — use non-greedy [\s\S]*?
- H10: Path-style vs virtual-hosted key decode mismatch

MEDIUM severity fixes:
- M1: Add Date header fallback for x-amz-date
- M2/M3: Validate service/termination in credential scope
- M4: Temp file leak when forwardToStorage throws in handleUploadPart
- M5: Multipart key consistency check (s3Key matches URL)
- M7: Use stored content-type from multipart initiate
- M9: Copy conditional headers skip when fileHash is null
- M11: Add 1000-key limit on DeleteObjects
- M13: Stricter bucket name validation (no .., no IP format)
- M14: NaN partNumber bypasses validation

LOW fixes:
- normalizeUri: dot-segment removal per RFC 3986
- localeCompare -> byte-order comparison in canonical query string
- Validate host in signed headers
- Server: AmazonS3 header on all responses
- x-amz-id-2 separate from x-amz-request-id
- IPv6 handling in stripPort
- Quiet element whitespace tolerance in XML parser
- content-type: application/xml on empty 2xx responses
- Duplicate interfaces/s3/ -> re-exports from utils/s3/

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 08:02:28 +07:00
Claude f5d56f52d4 chore: fix lint errors — noBannedTypes, import ordering, formatting
Deploy FileDrop / deploy (push) Successful in 42s
- Replace unsafe 'Function' type in test with ITelegramService interface
- Biome auto-fix formatting and import sorting across 8 files

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 07:30:31 +07:00
Claude ea87397801 refactor: remove upload and web-api routes, migrate to new controller structure
Deploy FileDrop / deploy (push) Failing after 12s
- Deleted `upload.ts` and `web-api.ts` routes, consolidating logic into dedicated controllers.
- Updated import paths in tests to reflect new controller structure.
- Refactored Telegram API utilities to utilize a bot pool for improved bot management and error handling.
- Enhanced environment variable tests to ensure additional bot tokens are correctly populated.
- Adjusted S3 bucket configuration tests to align with new controller imports.
- Updated Telegram queue implementation to reflect new infrastructure organization.
2026-07-29 07:28:30 +07:00
Claude 73adb5f58e fix: make S3 resilient for Docker registry — no rate limit, retry on transient Telegram errors
Deploy FileDrop / deploy (push) Failing after 15s
- Remove rate limiting from all S3 endpoints (used by Docker registry
  for concurrent blob pushes — 429 would abort the entire push).
- Add retry with exponential backoff in botPool.forwardToStorage for
  transient Telegram errors (network timeouts, 5xx, socket issues).
- Add retry with exponential backoff in botPool.getFileInfo per bot.
- Introduce isTransientError() pattern matcher covering ~20 transient
  error signatures.
- Fix temp file leak in handlePutObject when storeFileFromTemp throws.
- Fix pre-existing missing botPool namespace on getFileInfo call in
  handleGetMultipartObject.
- Fix route handler return type in PUT handler.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 07:25:47 +07:00
Claude 1422318f0a fix: enhance test setup and environment configuration for improved reliability 2026-07-28 22:44:08 +07:00
Claude 667921b100 chore: fix lint errors — duplicate import, unused imports, formatting
Deploy FileDrop / deploy (push) Successful in 45s
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 20:09:42 +07:00
Claude 002492626b fix: secure SigV4, temp leaks, OOM risk, duplicate migration, and cache issues
Deploy FileDrop / deploy (push) Failing after 15s
Security fixes:
- SigV4 signature comparison now uses crypto.timingSafeEqual (timing attack fix)
  - AccessKey, region, and HMAC signature all timing-safe
- Presigned URL expiry capped at 7 days (AWS spec compliance)
- Removed duplicate migration import (dead code)

Memory & leak fixes:
- Temp file leak in createZip(): cleanup temp file on error in both utils/ and shared/utils/
- OOM risk in web-api/v1 upload: stream File to temp instead of arrayBuffer()
- Removed duplicate migration import at startup

Performance fixes:
- Removed file.arrayBuffer() -> Bun.write() pattern in web-api-controller (stream + hash)

Test improvements:
- All fixes verified: 74/75 tests pass (1 pre-existing env config test)
- S3 auth tests: 7/7 pass after timing-safe fix
2026-07-28 19:33:05 +07:00
Claude 82c7f81ffa fix: streaming uploads, timeouts, and rate limiting for Docker registry safety
Critical fixes for S3 Docker registry backend:
- Stream PutObject body to temp file instead of req.arrayBuffer()
  - O(1) memory usage regardless of file size
  - SHA-256 hash computed while streaming
- Stream UploadPart body similarly
  - Also fixes: size check after streaming, not before
- Add 30s timeout to Telegram CDN chunk fetches (object-stream.ts)
  - Prevents hanging on stalled CDN connections
- Add rate limiting to S3 API routes (100 req/60s window)
  - Prevents resource exhaustion from concurrent layer pushes
- Add comprehensive test suite (10 tests):
  - Streaming verification (no arrayBuffer in PUT path)
  - Multi-MB body streaming safety
  - Empty body edge case
  - Concurrent upload isolation
  - Timeout signal presence
  - Rate limit route coverage

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 18:47:16 +07:00
Claude da7d7c2396 chore: clean up leftover directories from agent portability
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 18:39:27 +07:00
Claude a9a1938ffe Merge branch 'worktree-ddd-clean-architecture-restructure'
# Conflicts:
#	src/infrastructure/cache/index.ts
#	src/interfaces/http/middleware/auth.ts
#	src/interfaces/http/middleware/rate-limit.ts
#	src/shared/logger/index.ts
2026-07-28 18:38:49 +07:00
Claude ee10cb494e fix: resolve code review issues - import paths and structure cleanup
- Fix infrastructure imports: chunked-storage uses new path for shared/utils and interfaces/s3
- Fix health-controller: imports from infrastructure/persistence/drizzle instead of old db/
- Fix routes/index.ts: imports from new interfaces/s3 and middleware paths
- Marked Telegram-specific types in shared/utils/file.ts as future extraction

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 18:21:57 +07:00
Claude 4568644922 fix: correct import paths and add missing protocol files for DDD structure
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 18:19:45 +07:00
Claude 234ca7b14c feat: rewire entry point to new architecture
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 18:12:48 +07:00
Claude e40dfd8084 feat: create HTTP middleware layer
Create auth.ts and rate-limit.ts middleware files in the interfaces layer
as part of the DDD/clean architecture restructure. Also add a config
re-export at src/interfaces/config/index.ts so the middleware can access
configuration through the interfaces layer boundary.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 18:06:38 +07:00
Claude ea48e8fc03 feat: create application use cases (bucket, s3-object, multipart)
- manage-bucket.ts: extract bucket CRUD logic with validation
  (createListBuckets, createGetBucket, createCreateBucket,
   createDeleteBucket, createBucketExists)
- s3-object.ts: extract S3 object operations
  (createGetObject, createHeadObject, createPutObject,
   createCopyObject, createDeleteObject, createDeleteObjects,
   createListObjects, createFindObject)
- multipart-upload.ts: extract S3 multipart upload logic
  (createInitiateMultipartUpload, createUploadPart,
   createCompleteMultipartUpload, createAbortMultipartUpload,
   createListMultipartUploads, createListParts)

All use cases follow the existing factory function pattern with
dependency injection via repository/telegram service interfaces.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 18:04:01 +07:00
Claude e2de8c4245 feat: create application use cases (upload, get-file, auth)
Create upload-file.ts use case with factory pattern supporting dedup, file type detection, size validation, and chunked/single storage strategies.
Create get-file.ts use case supporting redirect, chunked, and archive-entry retrieval strategies.
Create authenticate.ts use case with login, logout, and me operations.
All use cases use dependency injection and return typed DTOs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 17:59:49 +07:00
Claude 5e29589f1a feat: create application DTOs
Add data-transfer-object interfaces for the application layer:
upload, file, bucket, S3, and auth domains.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 17:56:49 +07:00
Claude b15219335d feat: create infrastructure cache layer
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 17:49:17 +07:00
Claude ee3167fbfb feat: create domain port interfaces with JSDoc
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 17:48:24 +07:00
Claude e7657453d5 feat: create shared utilities layer with JSDoc
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 17:43:18 +07:00
Claude 5b5d4b0ba8 feat: create shared errors and logger layer
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 17:42:06 +07:00
Claude af8949de02 docs: add DDD/clean architecture restructure implementation plan
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 17:31:11 +07:00
Claude d8f02b79b1 docs: add DDD/clean architecture restructure design spec
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 17:29:20 +07:00
134 changed files with 12096 additions and 4202 deletions
+12
View File
@@ -0,0 +1,12 @@
{
"sessionStartHook": "echo '📁 TeleUploader — Telegram file uploader backend' && echo ' Bun project — use bun, not node/npm/yarn' && echo ' Tests: bun test --preload ./test/helpers/setup-env.ts <file>'",
"permissions": {
"allow": [
"Bun(bun test *)",
"Bun(bunx *)",
"Bun(bun run *)",
"Bun(bun build *)",
"Bun(bun install)"
]
}
}
+7 -7
View File
@@ -1,14 +1,13 @@
BOT_TOKEN=isi_token_bot_telegram BOT_TOKENS=isi_token_bot_1,isi_token_bot_2,isi_token_bot_3
ADDITIONAL_BOT_TOKENS=token_cadangan_1,token_cadangan_2,token_cadangan_3
STORAGE_CHANNEL_ID=-1001234567890 STORAGE_CHANNEL_ID=-1001234567890
BASE_URL=https://tele.asepharyana.my.id BASE_URL=https://upload.asepharyana.my.id
DATABASE_URL=postgresql://user:password@localhost:5432/telegram_uploader DATABASE_URL=postgresql://asephs:***@100.121.180.82:6432/uploader
PORT=3000 PORT=4000
NODE_ENV=production NODE_ENV=production
LOG_LEVEL=info LOG_LEVEL=info
RATE_LIMIT_WINDOW_MS=60000 RATE_LIMIT_WINDOW_MS=60000
RATE_LIMIT_MAX_REQUESTS=30 RATE_LIMIT_MAX_REQUESTS=30
# TRUST_PROXY=true # Uncomment when behind reverse proxy (Traefik, Nginx) for correct client IP detection # TRUST_PROXY=true # Uncomment when behind reverse proxy (Caddy, Nginx) for correct client IP detection
# S3-compatible API credentials # S3-compatible API credentials
# S3_ACCESS_KEY=filedrop-admin # S3_ACCESS_KEY=filedrop-admin
@@ -17,7 +16,8 @@ RATE_LIMIT_MAX_REQUESTS=30
# S3_VHOST_DOMAINS=upload.asepharyana.my.id,upload.asepharyana.web.id # S3_VHOST_DOMAINS=upload.asepharyana.my.id,upload.asepharyana.web.id
# Telegram-safe internal chunking for large stored files # Telegram-safe internal chunking for large stored files
# TELEGRAM_CHUNK_SIZE_BYTES=20971520 # Telegram getFile download limit is 20 MB; guard rejects > 19922944 (19 MB)
# TELEGRAM_CHUNK_SIZE_BYTES=19922944
# COMPRESS_CHUNKED_UPLOADS=true # COMPRESS_CHUNKED_UPLOADS=true
# CHUNK_COMPRESSION_MIN_SIZE_BYTES=4096 # CHUNK_COMPRESSION_MIN_SIZE_BYTES=4096
-48
View File
@@ -1,48 +0,0 @@
name: Deploy FileDrop
on:
push:
branches:
- main
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Check out repository
uses: actions/checkout@v4
- name: Set up Bun
uses: oven-sh/setup-bun@v2
with:
bun-version: latest
- name: Install dependencies
run: bun install --frozen-lockfile
- name: Lint
run: bun run lint
- name: Build
run: bun run build
- name: Deploy to VPS
shell: bash
env:
VPS_HOST: ${{ secrets.VPS_HOST }}
VPS_USER: ${{ secrets.VPS_USER }}
VPS_SSH_KEY_VALUE: ${{ secrets.VPS_SSH_KEY }}
PRODUCTION_ENV: ${{ secrets.PRODUCTION_ENV }}
ADMIN_PASSWORD: ${{ secrets.ADMIN_PASSWORD }}
run: |
set -eu
key_file="${RUNNER_TEMP:-/tmp}/filedrop_deploy_key"
printf '%s\n' "$VPS_SSH_KEY_VALUE" > "$key_file"
chmod 600 "$key_file"
printf '%s\n' "$PRODUCTION_ENV" > .env
chmod 600 .env
VPS_SSH_KEY="$key_file" ./deploy.sh --no-build
+112
View File
@@ -0,0 +1,112 @@
name: Build & Deploy (Nix)
on:
push:
branches: [main]
workflow_dispatch:
concurrency:
group: deploy
cancel-in-progress: false
permissions:
contents: read
id-token: write
env:
VPS_HOST: ${{ secrets.VPS_HOST }}
VPS_USER: ${{ secrets.VPS_USER }}
jobs:
lint:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v7
with:
fetch-depth: 0
submodules: false
- uses: oven-sh/setup-bun@v2
with:
bun-version: latest
- name: Install deps
run: bun install --frozen-lockfile
- name: Lint (biome)
run: bunx biome check src test
build-and-deploy:
needs: lint
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v7
with:
fetch-depth: 0
submodules: false
- name: Install Nix
uses: DeterminateSystems/nix-installer-action@v22
with:
determinate: false
extra-conf: |
sandbox = false
accept-flake-config = true
- name: Cache Nix
uses: DeterminateSystems/magic-nix-cache-action@v14
with:
use-flakehub: false
- name: Build teleuploader
id: build
run: |
nix build .#teleuploader --impure --option sandbox false --print-build-logs
STORE_PATH=$(readlink result)
echo "store-path=$STORE_PATH" >> "$GITHUB_OUTPUT"
echo "Build OK: $STORE_PATH"
- name: Setup SSH key
env:
SSH_KEY: ${{ secrets.SSH_PRIVATE_KEY }}
run: |
mkdir -p ~/.ssh
echo "$SSH_KEY" > ~/.ssh/id_ed25519
chmod 600 ~/.ssh/id_ed25519
sed -i 's/\r$//' ~/.ssh/id_ed25519
ssh-keygen -y -f ~/.ssh/id_ed25519 >/dev/null 2>&1 || { echo "SSH key invalid"; exit 1; }
ssh-keyscan -H "$VPS_HOST" >> ~/.ssh/known_hosts 2>/dev/null
- name: Deploy teleuploader to VPS
run: |
STORE_PATH="${{ steps.build.outputs.store-path }}"
echo "=== Copying teleuploader: $STORE_PATH ==="
nix copy --to "ssh://$VPS_USER@$VPS_HOST" "$STORE_PATH"
echo "=== Updating profile ==="
ssh "$VPS_USER@$VPS_HOST" "sudo /nix/var/nix/profiles/default/bin/nix-env --profile /nix/var/nix/profiles/teleuploader --set '$STORE_PATH'"
echo "=== Restarting service ==="
ssh "$VPS_USER@$VPS_HOST" "sudo systemctl daemon-reload && sudo systemctl restart teleuploader && sleep 3 && sudo systemctl is-active teleuploader"
echo "✅ teleuploader deployed"
cleanup:
# Bersihkan sampah Nix di VPS SETELAH deploy: hapus generasi profile lama
# + nix store gc. Profil yang sedang dipakai tidak disentuh.
needs: build-and-deploy
if: always()
runs-on: ubuntu-latest
steps:
- name: Nix GC on VPS
env:
VPS_HOST: ${{ secrets.VPS_HOST }}
VPS_USER: ${{ secrets.VPS_USER }}
SSH_KEY: ${{ secrets.SSH_PRIVATE_KEY }}
run: |
mkdir -p ~/.ssh
echo "$SSH_KEY" > ~/.ssh/id_ed25519
chmod 600 ~/.ssh/id_ed25519
ssh-keyscan -H "$VPS_HOST" >> ~/.ssh/known_hosts 2>/dev/null
ssh "$VPS_USER@$VPS_HOST" "sudo /usr/local/bin/nix-gc-vps.sh" || echo "⚠️ Nix GC gagal (non-fatal)"
@@ -0,0 +1,20 @@
name: Publish to FlakeHub
on:
push:
branches: [main, master]
workflow_dispatch:
jobs:
flakehub-publish:
runs-on: ubuntu-latest
permissions:
id-token: write
contents: read
steps:
- uses: actions/checkout@v6
- uses: DeterminateSystems/determinate-nix-action@main
- uses: DeterminateSystems/flakehub-push@main
with:
visibility: public
rolling: true
+26
View File
@@ -0,0 +1,26 @@
name: Mirror to Gitea
on:
push:
branches: [main, master]
workflow_dispatch:
permissions:
contents: write
jobs:
mirror:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v7
with:
fetch-depth: 0
- name: Mirror to Gitea
env:
GITEA_TOKEN: ${{ secrets.GITEA_TOKEN }}
run: |
git remote add gitea "https://oauth2:${GITEA_TOKEN}@git.imrnes.team/MythEclipse/TeleUploader.git"
git push --mirror gitea
echo "✅ Mirrored to Gitea (MythEclipse/TeleUploader)"
+1
View File
@@ -36,3 +36,4 @@ S3_GUIDE.md
# Finder (MacOS) folder config # Finder (MacOS) folder config
.DS_Store .DS_Store
result
+1
View File
@@ -0,0 +1 @@
bun run lint
+7 -1
View File
@@ -20,11 +20,17 @@ Default to using Bun instead of Node.js.
- Bun.$`ls` instead of execa. - Bun.$`ls` instead of execa.
- Rate limiter lokal dinonaktifkan (`checkRateLimit` di `src/utils/rateLimit.ts` selalu mengembalikan `true`). - Rate limiter lokal dinonaktifkan (`checkRateLimit` di `src/utils/rateLimit.ts` selalu mengembalikan `true`).
- Telegram API memiliki auto-retry otomatis jika mengembalikan error 429 (Too Many Requests) menggunakan pool Telegraf multi-bot di `src/utils/telegram.ts`. - Telegram API memiliki auto-retry otomatis jika mengembalikan error 429 (Too Many Requests) menggunakan pool Telegraf multi-bot di `src/utils/telegram.ts`.
- Multi-bot dikonfigurasi melalui `ADDITIONAL_BOT_TOKENS` (koma terpisah) di `.env` yang digabung dengan `BOT_TOKEN` utama (total 4 bot). |- Multi-bot dikonfigurasi melalui `BOT_TOKENS` (koma terpisah) di `.env` — semua token bot digabung dalam satu variabel.
- Menggunakan mekanisme rotasi instan jika ada bot yang terkena rate limit 429 sebelum memutuskan untuk sleep. - Menggunakan mekanisme rotasi instan jika ada bot yang terkena rate limit 429 sebelum memutuskan untuk sleep.
- Pengiriman berkas ke Telegram dieksekusi secara responsif dan paralel penuh tanpa batas konkurensi/antrian. - Pengiriman berkas ke Telegram dieksekusi secara responsif dan paralel penuh tanpa batas konkurensi/antrian.
- Berkas API upload ditulis secara sementara ke disk `/tmp/teleuploader-*` dan di-stream ke Telegram menggunakan `fs.createReadStream` (RAM-optimized) lalu dihapus otomatis setelah 50ms (timeout aman). - Berkas API upload ditulis secara sementara ke disk `/tmp/teleuploader-*` dan di-stream ke Telegram menggunakan `fs.createReadStream` (RAM-optimized) lalu dihapus otomatis setelah 50ms (timeout aman).
## Chunk size (TELEGRAM_CHUNK_SIZE_BYTES)
- Batas keras: Telegram Bot API `getFile` hanya bisa resolve file ≤ 20 MB — di atas itu error `Bad Request: file is too big` dan part tidak bisa di-download.
- Guard fail-fast di `src/env.ts`: service MENOLAK start (exit non-zero) jika `TELEGRAM_CHUNK_SIZE_BYTES` > 19922944 (19 MB, margin aman dari limit 20 MB). Konstanta: `TELEGRAM_CHUNK_SIZE_MAX_BYTES` di `src/shared/utils/validation.ts`, juga dipakai `asSafeChunkSize()` di runtime.
- Default 19 MB; berlaku untuk chunked storage DAN S3 multipart parts (sama-sama disimpan ke Telegram lalu di-resolve via getFile).
## Testing ## Testing
Use `bun test` to run tests. Jalankan tes secara spesifik (misal `bun test test/rateLimit.test.ts`) untuk menghindari polusi mock antar berkas tes ketika dijalankan bersamaan. Use `bun test` to run tests. Jalankan tes secara spesifik (misal `bun test test/rateLimit.test.ts`) untuk menghindari polusi mock antar berkas tes ketika dijalankan bersamaan.
+6 -2
View File
@@ -1,3 +1,7 @@
# ⚠️ LEGACY — pembangunan & deploy sekarang DISARANKAN memakai Nix + systemd
# (lihat flake.nix + .github/workflows/deploy.yml + Caddy reverse proxy di orangevps).
# Dockerfile ini hanya dipertahankan untuk konteks historis / fallback, bukan deploy produksi.
# Stage 1: Builder # Stage 1: Builder
FROM oven/bun:alpine AS builder FROM oven/bun:alpine AS builder
@@ -21,12 +25,12 @@ WORKDIR /usr/src/app
# Copy built files, schema, and package.json # Copy built files, schema, and package.json
COPY --from=builder /usr/src/app/dist/index.js ./dist/index.js COPY --from=builder /usr/src/app/dist/index.js ./dist/index.js
COPY --from=builder /usr/src/app/dist/migrate.js ./dist/migrate.js COPY --from=builder /usr/src/app/dist/migrate.js ./dist/migrate.js
COPY --from=builder /usr/src/app/src/home.html ./home.html COPY --from=builder /usr/src/app/src/home.html ./dist/home.html
COPY schema.sql ./ COPY schema.sql ./
COPY package.json ./ COPY package.json ./
# Expose port # Expose port
EXPOSE 3000 EXPOSE 4000
# Start server # Start server
CMD ["bun", "dist/index.js"] CMD ["bun", "dist/index.js"]
+20 -7
View File
@@ -4,12 +4,11 @@ Backend production-ready untuk upload file ke Telegram yang tersimpan di private
## Setup ## Setup
1. Install PostgreSQL database 1. Siapkan PostgreSQL database (produksi: database `uploader` via PgBouncer pool di `100.121.180.82:6432`)
2. Buat database: `createdb telegram_uploader` 2. Setup environment: `cp .env.example .env`
3. Setup environment: `cp .env.example .env` 3. Edit `.env` dengan nilai yang sesuai (lihat `DATABASE_URL`, `PORT=4000`)
4. Edit `.env` dengan nilai yang sesuai 4. Create table: `bun run db:migrate`
5. Create table: `bun run db:migrate` 5. Install dependencies: `bun install`
6. Install dependencies: `bun install`
## Telegram Private Channel Setup ## Telegram Private Channel Setup
@@ -24,6 +23,20 @@ bun run dev # Development mode
bun run start # Production mode bun run start # Production mode
``` ```
## Deployment (Produksi — Nix + systemd)
> Infra lama berbasis Docker + Traefik sudah dihapus dari orangevps (2026-08-02).
- **Host**: orangevps
- **Service**: systemd unit `teleuploader` (env via `/etc/teleuploader/env` / BWS secrets)
- **Build**: Nix flake (`flake.nix`) — `nix build .#teleuploader``nix copy``systemctl restart teleuploader`
- **CI**: `.github/workflows/deploy.yml` (Gitea Actions / GitHub Actions)
- **Port**: `4000` (`PORT` env)
- **Domain**: `https://upload.asepharyana.my.id`
- **Reverse proxy**: Caddy (bukan Traefik/Docker)
- **Database**: `postgresql://asephs:***@100.121.180.82:6432/uploader` (PgBouncer pool di imrnes, **bukan** 5432/localhost)
- `deploy.sh` & `Dockerfile` & `docker-compose.yml` bersifat **legacy** — jangan dipakai untuk deploy produksi.
## API Endpoints ## API Endpoints
- `POST /api/upload` - Upload file - `POST /api/upload` - Upload file
@@ -34,7 +47,7 @@ bun run start # Production mode
## FAQ ## FAQ
**URL permanen maksudnya apa?** **URL permanen maksudnya apa?**
URL backend tetap permanen: `https://tele.asepharyana.my.id/f/{public_id}` URL backend tetap permanen: `https://upload.asepharyana.my.id/f/{public_id}`
Ini berarti URL service Anda fix, bukan jaminan file Telegram abadi. Ini berarti URL service Anda fix, bukan jaminan file Telegram abadi.
## Testing ## Testing
+3
View File
@@ -19,6 +19,7 @@
"@biomejs/biome": "^2.4.15", "@biomejs/biome": "^2.4.15",
"@types/node": "^25.8.0", "@types/node": "^25.8.0",
"drizzle-kit": "^0.31.10", "drizzle-kit": "^0.31.10",
"husky": "^9.1.7",
"typescript": "^6.0.3", "typescript": "^6.0.3",
}, },
}, },
@@ -216,6 +217,8 @@
"get-tsconfig": ["get-tsconfig@4.14.0", "", { "dependencies": { "resolve-pkg-maps": "^1.0.0" } }, "sha512-yTb+8DXzDREzgvYmh6s9vHsSVCHeC0G3PI5bEXNBHtmshPnO+S5O7qgLEOn0I5QvMy6kpZN8K1NKGyilLb93wA=="], "get-tsconfig": ["get-tsconfig@4.14.0", "", { "dependencies": { "resolve-pkg-maps": "^1.0.0" } }, "sha512-yTb+8DXzDREzgvYmh6s9vHsSVCHeC0G3PI5bEXNBHtmshPnO+S5O7qgLEOn0I5QvMy6kpZN8K1NKGyilLb93wA=="],
"husky": ["husky@9.1.7", "", { "bin": { "husky": "bin.js" } }, "sha512-5gs5ytaNjBrh5Ow3zrvdUUY+0VxIuWVL4i9irt6friV+BqdCfmV11CQTWMiBYWHbXhco+J1kHfTOUkePhCDvMA=="],
"inherits": ["inherits@2.0.4", "", {}, "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ=="], "inherits": ["inherits@2.0.4", "", {}, "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ=="],
"is-stream": ["is-stream@2.0.1", "", {}, "sha512-hFoiJiTl63nn+kstHGBtewWSKnQLpyb155KHheA1l39uvtO9nWIop1p3udqPcUd/xbF1VLMO4n7OI6p7RbngDg=="], "is-stream": ["is-stream@2.0.1", "", {}, "sha512-hFoiJiTl63nn+kstHGBtewWSKnQLpyb155KHheA1l39uvtO9nWIop1p3udqPcUd/xbF1VLMO4n7OI6p7RbngDg=="],
+11 -3
View File
@@ -1,4 +1,9 @@
#!/bin/bash #!/bin/bash
# ⚠️ LEGACY — Script deploy lama berbasis Docker. Sejak 2026-08-02 Docker dihapus
# dari orangevps; produksi kini memakai Nix + systemd (lihat flake.nix, dan CI
# .github/workflows/deploy.yml yang menjalankan `nix build` → `nix copy` →
# `systemctl restart teleuploader`). Berkas ini hanya dipertahankan sebagai
# referensi historis — JANGAN dipakai untuk deploy produksi.
# ─── FileDrop Deploy Script ────────────────────────────────────────────────── # ─── FileDrop Deploy Script ──────────────────────────────────────────────────
# Builds the Bun app locally and deploys to the VPS via Docker. # Builds the Bun app locally and deploys to the VPS via Docker.
# #
@@ -155,12 +160,15 @@ vps "rm -rf '${DEPLOY_DIR}.old' && mv '${DEPLOY_DIR}' '${DEPLOY_DIR}.old' 2>/dev
ok "Files shipped to ${DEPLOY_DIR}" ok "Files shipped to ${DEPLOY_DIR}"
# ── 5. Build Docker image & restart on VPS ─────────────────────────────────── # ── 5. Build Docker image & restart ──────────────────────────────────────
log "Building Docker image on VPS..." log "Building Docker image on VPS..."
vps "cd '${DEPLOY_DIR}' && docker compose build --pull 2>&1" | tail -5 || die "Docker build failed on VPS" vps "cd '${DEPLOY_DIR}' && docker compose build --pull 2>&1" | tail -5 || die "Docker build failed on VPS"
log "Restarting container..." log "Restarting container (zero-downtime via healthcheck)..."
vps "cd '${DEPLOY_DIR}' && docker compose up -d --force-recreate 2>&1" || die "Container restart failed" vps "cd '${DEPLOY_DIR}' && docker compose up -d --force-recreate --wait --wait-timeout 60 2>&1" || {
log "Warn: --wait not supported on this docker-compose version, falling back to basic restart"
vps "cd '${DEPLOY_DIR}' && docker compose up -d --force-recreate 2>&1" || die "Container restart failed"
}
# ── 6. Verify container is running ──────────────────────────────────────────── # ── 6. Verify container is running ────────────────────────────────────────────
log "Waiting for container to be healthy..." log "Waiting for container to be healthy..."
+7 -4
View File
@@ -1,19 +1,22 @@
# ⚠️ LEGACY — Docker/Traefik sudah DIHAPUS dari VPS produksi (orangevps).
# Deploy sekarang Nix + systemd (flake.nix + .github/workflows/deploy.yml) dengan
# Caddy reverse proxy. Berkas ini hanya dipertahankan sebagai referensi historis.
services: services:
app: app:
build: . build: .
container_name: filedrop-app container_name: filedrop-app
restart: always restart: always
environment: environment:
- BOT_TOKENS=${BOT_TOKENS}
- BOT_TOKEN=${BOT_TOKEN} - BOT_TOKEN=${BOT_TOKEN}
- ADDITIONAL_BOT_TOKENS=${ADDITIONAL_BOT_TOKENS:-} - ADDITIONAL_BOT_TOKENS=${ADDITIONAL_BOT_TOKENS:-}
- STORAGE_CHANNEL_ID=${STORAGE_CHANNEL_ID} - STORAGE_CHANNEL_ID=${STORAGE_CHANNEL_ID}
- BASE_URL=${BASE_URL} - BASE_URL=${BASE_URL}
- DATABASE_URL=${DATABASE_URL} - DATABASE_URL=${DATABASE_URL}
- PORT=3000 - PORT=4000
- NODE_ENV=production - NODE_ENV=production
- LOG_LEVEL=info - LOG_LEVEL=info
- TRUST_PROXY=true - TRUST_PROXY=true
- UPLOAD_CONCURRENCY=${UPLOAD_CONCURRENCY:-8}
- BATCH_MAX_ITEMS=${BATCH_MAX_ITEMS:-20} - BATCH_MAX_ITEMS=${BATCH_MAX_ITEMS:-20}
- BATCH_MAX_SIZE_BYTES=${BATCH_MAX_SIZE_BYTES:-524288000} - BATCH_MAX_SIZE_BYTES=${BATCH_MAX_SIZE_BYTES:-524288000}
- MAX_REQUEST_BODY_BYTES=${MAX_REQUEST_BODY_BYTES:-2147483648} - MAX_REQUEST_BODY_BYTES=${MAX_REQUEST_BODY_BYTES:-2147483648}
@@ -46,7 +49,7 @@ services:
healthcheck: healthcheck:
test: test:
- CMD-SHELL - CMD-SHELL
- "bun -e \"fetch('http://localhost:3000/health').then(r => r.status === 200 ? process.exit(0) : process.exit(1))\"" - "bun -e \"fetch('http://localhost:4000/health').then(r => r.status === 200 ? process.exit(0) : process.exit(1))\""
interval: 30s interval: 30s
timeout: 10s timeout: 10s
retries: 3 retries: 3
@@ -59,7 +62,7 @@ services:
- "traefik.http.routers.filedrop.entrypoints=websecure" - "traefik.http.routers.filedrop.entrypoints=websecure"
- "traefik.http.routers.filedrop.tls=true" - "traefik.http.routers.filedrop.tls=true"
- "traefik.http.routers.filedrop.tls.certresolver=cloudflare" - "traefik.http.routers.filedrop.tls.certresolver=cloudflare"
- "traefik.http.services.filedrop.loadbalancer.server.port=3000" - "traefik.http.services.filedrop.loadbalancer.server.port=4000"
- "traefik.http.middlewares.filedrop-rl.ratelimit.average=300" - "traefik.http.middlewares.filedrop-rl.ratelimit.average=300"
- "traefik.http.middlewares.filedrop-rl.ratelimit.burst=100" - "traefik.http.middlewares.filedrop-rl.ratelimit.burst=100"
- "traefik.http.middlewares.filedrop-rl.ratelimit.period=1m" - "traefik.http.middlewares.filedrop-rl.ratelimit.period=1m"
@@ -1,5 +1,10 @@
# Telegram Bot Uploader Backend Implementation Plan # Telegram Bot Uploader Backend Implementation Plan
> ⚠️ **LEGACY** — Dokumen historis (2026-05-17). Port & infrastruktur sudah berubah:
> produksi kini berjalan di port `4000` (Nix + systemd + Caddy, domain `upload.asepharyana.my.id`)
> dan database via PgBouncer pool `100.121.180.82:6432` (bukan port 5432, bukan localhost).
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Production-ready backend for Telegram file uploader with dual upload methods (bot + HTTP API), PostgreSQL storage, and redirect-based downloads. **Goal:** Production-ready backend for Telegram file uploader with dual upload methods (bot + HTTP API), PostgreSQL storage, and redirect-based downloads.
@@ -75,9 +80,9 @@ schema.sql
```bash ```bash
BOT_TOKEN=isi_token_bot_telegram BOT_TOKEN=isi_token_bot_telegram
STORAGE_CHANNEL_ID=-1001234567890 STORAGE_CHANNEL_ID=-1001234567890
BASE_URL=https://tele.asepharyana.my.id BASE_URL=https://upload.asepharyana.my.id
DATABASE_URL=postgresql://user:password@localhost:5432/telegram_uploader DATABASE_URL=postgresql://asephs:***@100.121.180.82:6432/uploader
PORT=3000 PORT=4000
NODE_ENV=production NODE_ENV=production
LOG_LEVEL=info LOG_LEVEL=info
RATE_LIMIT_WINDOW_MS=60000 RATE_LIMIT_WINDOW_MS=60000
@@ -148,7 +153,7 @@ bun run start # Production mode
## FAQ ## FAQ
**URL permanen maksudnya apa?** **URL permanen maksudnya apa?**
URL backend tetap permanen: `https://tele.asepharyana.my.id/f/{public_id}` URL backend tetap permanen: `https://upload.asepharyana.my.id/f/{public_id}`
Ini berarti URL service Anda fix, bukan jaminan file Telegram abadi. Ini berarti URL service Anda fix, bukan jaminan file Telegram abadi.
## Testing ## Testing
@@ -250,7 +255,7 @@ export const config = {
storageChatId: parseInt(process.env.STORAGE_CHANNEL_ID, 10), storageChatId: parseInt(process.env.STORAGE_CHANNEL_ID, 10),
baseUrl: process.env.BASE_URL, baseUrl: process.env.BASE_URL,
databaseUrl: process.env.DATABASE_URL, databaseUrl: process.env.DATABASE_URL,
port: parseInt(process.env.PORT, 10) || 3000, port: parseInt(process.env.PORT, 10) || 4000,
nodeEnv: process.env.NODE_ENV || 'development', nodeEnv: process.env.NODE_ENV || 'development',
logLevel: process.env.LOG_LEVEL || 'info', logLevel: process.env.LOG_LEVEL || 'info',
rateLimitWindowMs: parseInt(process.env.RATE_LIMIT_WINDOW_MS, 10) || 60000, rateLimitWindowMs: parseInt(process.env.RATE_LIMIT_WINDOW_MS, 10) || 60000,
@@ -1009,14 +1014,14 @@ bun run dev
- [ ] **Step 2: Test health endpoint** - [ ] **Step 2: Test health endpoint**
```bash ```bash
curl http://localhost:3000/health curl http://localhost:4000/health
# Expected: {"status":"ok"} # Expected: {"status":"ok"}
``` ```
- [ ] **Step 3: Upload test file via HTTP API (multipart)** - [ ] **Step 3: Upload test file via HTTP API (multipart)**
```bash ```bash
curl -X POST http://localhost:3000/api/upload \ curl -X POST http://localhost:4000/api/upload \
-F "file=@/path/to/testfile.txt" \ -F "file=@/path/to/testfile.txt" \
-F "fileName=test.txt" -F "fileName=test.txt"
``` ```
@@ -1024,14 +1029,14 @@ curl -X POST http://localhost:3000/api/upload \
- [ ] **Step 4: Check file info endpoint** - [ ] **Step 4: Check file info endpoint**
```bash ```bash
curl http://localhost:3000/file/{public_id}/info curl http://localhost:4000/file/{public_id}/info
# Expected: JSON with file metadata # Expected: JSON with file metadata
``` ```
- [ ] **Step 5: Download redirect** - [ ] **Step 5: Download redirect**
```bash ```bash
curl -I http://localhost:3000/f/{public_id} curl -I http://localhost:4000/f/{public_id}
# Expected: HTTP 302 with Location header to Telegram CDN # Expected: HTTP 302 with Location header to Telegram CDN
``` ```
@@ -1047,7 +1052,7 @@ curl -I http://localhost:3000/f/{public_id}
- [ ] **Step 7: Test error handling (file too large)** - [ ] **Step 7: Test error handling (file too large)**
```bash ```bash
curl -X POST http://localhost:3000/api/upload \ curl -X POST http://localhost:4000/api/upload \
-F "file=@/dev/null" \ -F "file=@/dev/null" \
-H "Content-Length: 10000000000" -H "Content-Length: 10000000000"
# Expected: HTTP 400 with error message # Expected: HTTP 400 with error message
@@ -1057,7 +1062,7 @@ curl -X POST http://localhost:3000/api/upload \
```bash ```bash
# Send 31 requests within 1 minute # Send 31 requests within 1 minute
for i in {1..31}; do curl http://localhost:3000/f/{public_id} & done for i in {1..31}; do curl http://localhost:4000/f/{public_id} & done
wait wait
# Expected: First 30 succeed, last one returns 429 # Expected: First 30 succeed, last one returns 429
``` ```
@@ -1066,7 +1071,7 @@ wait
```bash ```bash
# In terminal 1: bun run dev # In terminal 1: bun run dev
# In terminal 2: curl http://localhost:3000/health && sleep 0.1 && curl http://localhost:3000/health # In terminal 2: curl http://localhost:4000/health && sleep 0.1 && curl http://localhost:4000/health
# Send SIGINT to server (Ctrl+C in terminal 1) # Send SIGINT to server (Ctrl+C in terminal 1)
# Check if server stops cleanly, logs show shutdown sequence # Check if server stops cleanly, logs show shutdown sequence
``` ```
@@ -1,5 +1,11 @@
# Docker, Traefik, and GitHub Actions Deployment Plan # Docker, Traefik, and GitHub Actions Deployment Plan
> ⚠️ **LEGACY** — Dokumen historis (2026-05-18). Arsitektur Docker + Traefik +
> GitHub Actions sudah digantikan (2026-08-02) oleh Nix + systemd + Caddy di
> orangevps: port `4000`, domain `upload.asepharyana.my.id`, database via
> PgBouncer pool `100.121.180.82:6432` (bukan 5432/localhost).
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Containerize TeleUploader using Bun, configure Traefik labels for routing `upload.asepharyana.my.id`, and set up full GitHub Actions CI/CD to VPS `45.127.35.244`. **Goal:** Containerize TeleUploader using Bun, configure Traefik labels for routing `upload.asepharyana.my.id`, and set up full GitHub Actions CI/CD to VPS `45.127.35.244`.
@@ -57,7 +63,7 @@ WORKDIR /usr/src/app
# Set production environment variables # Set production environment variables
ENV NODE_ENV=production ENV NODE_ENV=production
ENV PORT=3000 ENV PORT=4000
# Copy necessary files from builder and repo # Copy necessary files from builder and repo
COPY --from=builder /usr/src/app/dist/index.js ./dist/index.js COPY --from=builder /usr/src/app/dist/index.js ./dist/index.js
@@ -65,7 +71,7 @@ COPY --from=builder /usr/src/app/package.json ./package.json
COPY schema.sql ./schema.sql COPY schema.sql ./schema.sql
# Expose server port # Expose server port
EXPOSE 3000 EXPOSE 4000
# Start server # Start server
CMD ["bun", "dist/index.js"] CMD ["bun", "dist/index.js"]
@@ -104,7 +110,7 @@ services:
- STORAGE_CHANNEL_ID=${STORAGE_CHANNEL_ID} - STORAGE_CHANNEL_ID=${STORAGE_CHANNEL_ID}
- BASE_URL=${BASE_URL} - BASE_URL=${BASE_URL}
- DATABASE_URL=${DATABASE_URL} - DATABASE_URL=${DATABASE_URL}
- PORT=3000 - PORT=4000
- NODE_ENV=production - NODE_ENV=production
- LOG_LEVEL=info - LOG_LEVEL=info
networks: networks:
@@ -115,7 +121,7 @@ services:
- "traefik.http.routers.teleuploader.entrypoints=websecure" - "traefik.http.routers.teleuploader.entrypoints=websecure"
- "traefik.http.routers.teleuploader.tls=true" - "traefik.http.routers.teleuploader.tls=true"
- "traefik.http.routers.teleuploader.tls.certresolver=letsencrypt" - "traefik.http.routers.teleuploader.tls.certresolver=letsencrypt"
- "traefik.http.services.teleuploader.loadbalancer.server.port=3000" - "traefik.http.services.teleuploader.loadbalancer.server.port=4000"
networks: networks:
app-shared-net: app-shared-net:
@@ -166,9 +172,9 @@ jobs:
env: env:
BOT_TOKEN: "mock_token" BOT_TOKEN: "mock_token"
STORAGE_CHANNEL_ID: "123456" STORAGE_CHANNEL_ID: "123456"
BASE_URL: "http://localhost:3000" BASE_URL: "http://localhost:4000"
DATABASE_URL: "postgresql://postgres:postgres@localhost:5432/postgres" DATABASE_URL: "postgresql://asephs:***@100.121.180.82:6432/postgres"
PORT: "3000" PORT: "4000"
run: bun run test run: bun run test
build-and-push: build-and-push:
@@ -223,7 +229,7 @@ jobs:
- STORAGE_CHANNEL_ID=${STORAGE_CHANNEL_ID} - STORAGE_CHANNEL_ID=${STORAGE_CHANNEL_ID}
- BASE_URL=${BASE_URL} - BASE_URL=${BASE_URL}
- DATABASE_URL=${DATABASE_URL} - DATABASE_URL=${DATABASE_URL}
- PORT=3000 - PORT=4000
- NODE_ENV=production - NODE_ENV=production
- LOG_LEVEL=info - LOG_LEVEL=info
networks: networks:
@@ -234,7 +240,7 @@ jobs:
- "traefik.http.routers.teleuploader.entrypoints=websecure" - "traefik.http.routers.teleuploader.entrypoints=websecure"
- "traefik.http.routers.teleuploader.tls=true" - "traefik.http.routers.teleuploader.tls=true"
- "traefik.http.routers.teleuploader.tls.certresolver=letsencrypt" - "traefik.http.routers.teleuploader.tls.certresolver=letsencrypt"
- "traefik.http.services.teleuploader.loadbalancer.server.port=3000" - "traefik.http.services.teleuploader.loadbalancer.server.port=4000"
networks: networks:
app-shared-net: app-shared-net:
@@ -248,7 +254,7 @@ jobs:
STORAGE_CHANNEL_ID=${{ secrets.STORAGE_CHANNEL_ID }} STORAGE_CHANNEL_ID=${{ secrets.STORAGE_CHANNEL_ID }}
BASE_URL=${{ secrets.BASE_URL }} BASE_URL=${{ secrets.BASE_URL }}
DATABASE_URL=${{ secrets.DATABASE_URL }} DATABASE_URL=${{ secrets.DATABASE_URL }}
PORT=3000 PORT=4000
EOF EOF
# Pull latest docker image # Pull latest docker image
@@ -1,5 +1,11 @@
# S3-Compatible TeleUploader Implementation Plan # S3-Compatible TeleUploader Implementation Plan
> ⚠️ **LEGACY** — Dokumen historis (2026-07-06). Port & infrastruktur sudah berubah:
> produksi kini berjalan di port `4000` (Nix + systemd + Caddy, domain
> `upload.asepharyana.my.id`), database via PgBouncer pool `100.121.180.82:6432`
> (bukan 5432/localhost). Contoh kode di bawah memakai `localhost:4000` untuk dev.
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Transform TeleUploader into an S3-compatible storage server (Telegram-backed) with a web file manager UI. **Goal:** Transform TeleUploader into an S3-compatible storage server (Telegram-backed) with a web file manager UI.
@@ -2675,7 +2681,7 @@ describe('S3 Bucket Operations', () => {
process.env.S3_DEFAULT_REGION = 'us-east-1'; process.env.S3_DEFAULT_REGION = 'us-east-1';
process.env.BOT_TOKEN = '123456:ABC-DEF'; process.env.BOT_TOKEN = '123456:ABC-DEF';
process.env.STORAGE_CHANNEL_ID = '-1001234567890'; process.env.STORAGE_CHANNEL_ID = '-1001234567890';
process.env.BASE_URL = 'http://localhost:3000'; process.env.BASE_URL = 'http://localhost:4000';
process.env.DATABASE_URL = 'postgresql://localhost/test'; process.env.DATABASE_URL = 'postgresql://localhost/test';
}); });
@@ -2684,7 +2690,7 @@ describe('S3 Bucket Operations', () => {
}); });
it('should return 403 for unauthorized requests', async () => { it('should return 403 for unauthorized requests', async () => {
const req = new Request('http://localhost:3000/', { const req = new Request('http://localhost:4000/', {
method: 'GET', method: 'GET',
headers: { authorization: 'Invalid' }, headers: { authorization: 'Invalid' },
}); });
@@ -2849,7 +2855,7 @@ describe('Web API v1', () => {
mockDbExecute.mockClear(); mockDbExecute.mockClear();
process.env.BOT_TOKEN = '123456:ABC-DEF'; process.env.BOT_TOKEN = '123456:ABC-DEF';
process.env.STORAGE_CHANNEL_ID = '-1001234567890'; process.env.STORAGE_CHANNEL_ID = '-1001234567890';
process.env.BASE_URL = 'http://localhost:3000'; process.env.BASE_URL = 'http://localhost:4000';
process.env.DATABASE_URL = 'postgresql://localhost/test'; process.env.DATABASE_URL = 'postgresql://localhost/test';
}); });
@@ -2858,7 +2864,7 @@ describe('Web API v1', () => {
}); });
it('should list buckets via GET /api/v1/buckets', async () => { it('should list buckets via GET /api/v1/buckets', async () => {
const req = new Request('http://localhost:3000/api/v1/buckets'); const req = new Request('http://localhost:4000/api/v1/buckets');
const res = await handleWebApiV1(req); const res = await handleWebApiV1(req);
expect(res.status).toBe(200); expect(res.status).toBe(200);
const data = await res.json(); const data = await res.json();
@@ -2867,7 +2873,7 @@ describe('Web API v1', () => {
}); });
it('should return 404 for unknown API path', async () => { it('should return 404 for unknown API path', async () => {
const req = new Request('http://localhost:3000/api/v1/unknown'); const req = new Request('http://localhost:4000/api/v1/unknown');
const res = await handleWebApiV1(req); const res = await handleWebApiV1(req);
expect(res.status).toBe(404); expect(res.status).toBe(404);
const data = await res.json(); const data = await res.json();
@@ -2875,7 +2881,7 @@ describe('Web API v1', () => {
}); });
it('should return bucket object listing', async () => { it('should return bucket object listing', async () => {
const req = new Request('http://localhost:3000/api/v1/buckets/test-bucket/objects?prefix='); const req = new Request('http://localhost:4000/api/v1/buckets/test-bucket/objects?prefix=');
const res = await handleWebApiV1(req); const res = await handleWebApiV1(req);
// Should return 200 even with empty results // Should return 200 even with empty results
expect(res.status).toBe(200); expect(res.status).toBe(200);
@@ -2885,7 +2891,7 @@ describe('Web API v1', () => {
}); });
it('should reject invalid bucket name on create', async () => { it('should reject invalid bucket name on create', async () => {
const req = new Request('http://localhost:3000/api/v1/buckets', { const req = new Request('http://localhost:4000/api/v1/buckets', {
method: 'POST', method: 'POST',
headers: { 'content-type': 'application/json' }, headers: { 'content-type': 'application/json' },
body: JSON.stringify({ name: 'INVALID_NAME!' }), body: JSON.stringify({ name: 'INVALID_NAME!' }),
@@ -1,5 +1,7 @@
# Gitea CI/CD Migration Implementation Plan # Gitea CI/CD Migration Implementation Plan
> Catatan (2026-08-02): Produksi sekarang port 4000, deploy Nix+systemd di orangevps, Caddy reverse proxy upload.asepharyana.my.id, DB via pgbouncer pool imrnes 100.121.180.82:6432. Docker/Traefik/Gitea-CI legacy.
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Move `origin` from GitLab to `git.imrnes.team:MythEclipse/TeleUploader` and add Gitea Actions deployment on push to `main` using the existing VPS deploy path. **Goal:** Move `origin` from GitLab to `git.imrnes.team:MythEclipse/TeleUploader` and add Gitea Actions deployment on push to `main` using the existing VPS deploy path.
@@ -1,5 +1,7 @@
# S3 Compatibility Completion Implementation Plan # S3 Compatibility Completion Implementation Plan
> Catatan (2026-08-02): Produksi sekarang port 4000, deploy Nix+systemd di orangevps, Caddy reverse proxy upload.asepharyana.my.id, DB via pgbouncer pool imrnes 100.121.180.82:6432. Docker/Traefik/Gitea-CI legacy.
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Finish TeleUploader S3 compatibility gaps: strict presigned GET, byte ranges, complete multipart GetObject streaming, strict AWS SDK multipart investigation/fix, and warning-free lint. **Goal:** Finish TeleUploader S3 compatibility gaps: strict presigned GET, byte ranges, complete multipart GetObject streaming, strict AWS SDK multipart investigation/fix, and warning-free lint.
@@ -0,0 +1,950 @@
# DDD & Clean Architecture Restructure Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Restructure TeleUploader from flat architecture to DDD/Clean Architecture with repository abstraction, controller/use-case separation, and full JSDoc documentation.
**Architecture:** 5-layer DDD — domain (entities + ports), application (use cases + DTOs), infrastructure (persistence + telegram + cache), interfaces (HTTP controllers + routes + middleware + bot + S3 protocol), and shared/config foundation.
**Tech Stack:** Bun + TypeScript + Drizzle ORM + Telegraf
## Global Constraints
- Zero behavioral changes — only move/restructure code, never alter logic
- Old source files remain until Phase 8 (cleanup), so existing tests keep passing
- JSDoc on every exported function, interface, type, and class (English)
- All new imports use relative paths within new structure
- `config/` and `shared/` have zero dependencies on other new layers
- Domain `ports/` interfaces are implemented by `infrastructure/` repositories
- Use cases constructed via factory functions accepting repository interfaces (no DI framework)
- Controller = HTTP parsing + use case call + response formatting only; no business logic
- Domain entities reuse Drizzle types (no pure domain models)
- `bun test test/<file>` must pass at every phase (old files remain)
---
### Phase 1: Foundation Layer — Config + Shared Utilities
#### Task 1: Create config/index.ts
**Files:**
- Create: `src/config/index.ts`
**Interfaces:**
- Produces: `AppConfig` type, `config: AppConfig` singleton
- [ ] **Create `src/config/index.ts`** — Move content from `src/env.ts`. Same `AppConfig` interface, same `config` export. Add JSDoc to `AppConfig` interface (document every property), the `config` export, `parseNumber`, `parseTokens`, `parseDomains`, `maskSecret`, `maskDatabaseUrl`.
- [ ] **Run tests to verify nothing broken**
```bash
bun test test/env.test.ts
```
Expected: PASS
- [ ] **Create `src/config/__tests__/env.test.ts`** — Mirror of `test/env.test.ts` but imports from `../../config/index.ts`
- [ ] **Run both env tests to verify**
```bash
bun test test/env.test.ts
bun test src/config/__tests__/env.test.ts
```
Expected: Both PASS
- [ ] **Commit**
```bash
git add src/config/
git commit -m "feat: create config layer with JSDoc"
```
#### Task 2: Create shared/ errors and logger
**Files:**
- Create: `src/shared/errors/index.ts`
- Create: `src/shared/logger/index.ts`
**Interfaces:**
- Produces: `DomainError`, `FileNotFoundError`, `BucketNotFoundError`, `FileTooLargeError`, `DuplicateFileError`, `AuthenticationError`, `ValidationError`, `logger` singleton, type `Logger`
- [ ] **Create `src/shared/errors/index.ts`** — Export all error classes from spec Section 11. Add JSDoc for each class.
- [ ] **Create `src/shared/logger/index.ts`** — Move content from `src/utils/logger.ts`. Same implementation. Add JSDoc.
- [ ] **Quick compile check**
```bash
bun build src/config/index.ts --target=bun --outfile=/dev/null 2>&1 | head -5
bun build src/shared/errors/index.ts --target=bun --outfile=/dev/null 2>&1 | head -5
```
Expected: No errors
- [ ] **Commit**
```bash
git add src/shared/errors/ src/shared/logger/
git commit -m "feat: create shared errors and logger layer"
```
#### Task 3: Create shared/ metrics
**Files:**
- Create: `src/shared/metrics/index.ts`
**Interfaces:**
- Produces: `metricsCollector`, `MetricsCollector`, `getSnapshot()`
- [ ] **Create `src/shared/metrics/index.ts`** — Move content from `src/utils/metrics.ts`. Same implementation. Add JSDoc to class, all methods, and `getSnapshot()` return type.
- [ ] **Compile check**
```bash
bun build src/shared/metrics/index.ts --target=bun --outfile=/dev/null 2>&1 | head -5
```
Expected: No errors
- [ ] **Commit**
```bash
git add src/shared/metrics/
git commit -m "feat: create shared metrics layer"
```
#### Task 4: Create shared/utils (file, ip, retry, zip)
**Files:**
- Create: `src/shared/utils/file.ts`
- Create: `src/shared/utils/ip.ts`
- Create: `src/shared/utils/retry.ts`
- Create: `src/shared/utils/zip.ts`
**Interfaces:**
- Produces: `getErrorMessage`, `cleanupTempFile`, `getFileType`, `checkFileSize`, `ensureExtension`, `extractFileName`, `extractMimeType`, `computeHash`, `extractFileFromMessage`, `detectFileType`, `getFileSizeLimit`, `formatCreatedAt`, `buildUploadResponse`, `extractClientIp`, `withRetry`, `withTimeout`, `withFallback`, `createZip`, `extractZipEntry`, `locateZipEntry`
- [ ] **Create `src/shared/utils/file.ts`** — Move all exports from `src/utils/file.ts`. Add JSDoc to every export. Keep all logic identical.
- [ ] **Create `src/shared/utils/ip.ts`** — Move `extractClientIp` from `src/utils/ip.ts`. Add JSDoc.
- [ ] **Create `src/shared/utils/retry.ts`** — Move all exports from `src/utils/retry.ts`. Add JSDoc.
- [ ] **Create `src/shared/utils/zip.ts`** — Move all exports from `src/utils/zip.ts`. Add JSDoc.
- [ ] **Run existing tests to verify**
```bash
bun test test/rateLimit.test.ts
bun test test/file.test.ts
bun test test/zip.test.ts
```
Expected: PASS
- [ ] **Commit**
```bash
git add src/shared/utils/
git commit -m "feat: create shared utilities layer with JSDoc"
```
---
### Phase 2: Domain Layer — Entities + Ports
#### Task 5: Create domain/entities (File, Bucket, FilePart, Multipart)
**Files:**
- Create: `src/domain/entities/file.ts`
- Create: `src/domain/entities/bucket.ts`
- Create: `src/domain/entities/file-part.ts`
- Create: `src/domain/entities/multipart.ts`
**Interfaces:**
- Produces: `File`, `NewFile`, `FilePart`, `NewFilePart`, `Bucket`, `MultipartUpload`, `MultipartPart` types
- [ ] **Create `src/domain/entities/file.ts`** — Re-export `File` and `NewFile` types from Drizzle schema. Add domain JSDoc.
```typescript
// Re-export Drizzle types as domain entities
export type { File, NewFile } from '../../infrastructure/persistence/drizzle/schema';
```
Wait — this creates a circular dependency! Domain shouldn't import from infrastructure. Instead, define standalone interfaces:
```typescript
/**
* Core domain entity representing a file stored in Telegram.
* Contains both Telegram metadata and optional S3-compatible fields.
*/
export interface File {
id: string;
publicId: string;
telegramFileId: string;
telegramFileUniqueId: string;
storageChatId: number;
storageMessageId: number;
fileName: string;
mimeType: string;
sizeBytes: number;
fileType: string;
uploaderId: number;
fileHash: string | null;
archiveTelegramFileId: string | null;
archiveStorageMessageId: number | null;
archiveFileName: string | null;
archiveEntryName: string | null;
archiveMimeType: string | null;
archiveSizeBytes: number | null;
bucketId: string | null;
s3Key: string | null;
storageBackend: string | null;
isDeleted: boolean | null;
multipartUploadId: string | null;
partCount: number | null;
createdAt: Date;
updatedAt: Date;
}
export type NewFile = Omit<File, 'id' | 'createdAt' | 'updatedAt'>;
```
- [ ] **Create `src/domain/entities/bucket.ts`** — Standalone Bucket interface.
- [ ] **Create `src/domain/entities/file-part.ts`** — Standalone FilePart + NewFilePart types with `CompressionAlgorithm`.
- [ ] **Create `src/domain/entities/multipart.ts`** — Standalone MultipartUpload, MultipartPart types.
- [ ] **Compile check**
```bash
bun build src/domain/entities/file.ts --target=bun --outfile=/dev/null 2>&1 | head -5
```
Expected: No errors
- [ ] **Commit**
```bash
git add src/domain/entities/
git commit -m "feat: create domain entities with JSDoc"
```
#### Task 6: Create domain/ports (repository interfaces)
**Files:**
- Create: `src/domain/ports/file-repository.ts`
- Create: `src/domain/ports/bucket-repository.ts`
- Create: `src/domain/ports/file-part-repository.ts`
- Create: `src/domain/ports/multipart-repository.ts`
**Interfaces:**
- Produces: `IFileRepository`, `IBucketRepository`, `IFilePartRepository`, `IMultipartRepository`, `ITelegramService`, `S3FileRecord`, `CompressionAlgorithm`
- [ ] **Create `src/domain/ports/file-repository.ts`** — Define interface for all file operations currently in `db/files.ts` and `db/files-ext.ts`.
```typescript
import type { File, NewFile } from '../entities/file';
export interface S3FileRecord extends File {
bucketId: string;
s3Key: string;
}
export interface IFileRepository {
findByHash(hash: string): Promise<File | null>;
findByPublicId(publicId: string): Promise<File | null>;
findByUniqueId(telegramFileUniqueId: string): Promise<File | null>;
findByBucketAndKey(bucketId: string, s3Key: string): Promise<File | null>;
create(file: NewFile): Promise<File>;
listByPrefix(
bucketId: string,
prefix: string,
delimiter: string | null,
maxKeys: number,
startAfter: string | null,
): Promise<{ objects: S3FileRecord[]; prefixes: string[] }>;
softDelete(bucketId: string, s3Key: string): Promise<boolean>;
softDeleteBatch(bucketId: string, keys: string[]): Promise<number>;
countByBucket(bucketId: string): Promise<number>;
findOrphansByBucket(bucketId: string): Promise<File[]>;
}
```
- [ ] **Create `src/domain/ports/bucket-repository.ts`** — Interface for bucket CRUD.
- [ ] **Create `src/domain/ports/file-part-repository.ts`** — Interface for file parts operations.
- [ ] **Create `src/domain/ports/multipart-repository.ts`** — Interface for multipart upload operations.
- [ ] **Create `src/domain/ports/telegram-service.ts`** — Interface for Telegram operations.
```typescript
/** Result of forwarding a file to Telegram storage */
export interface ForwardResult {
telegramFileId: string;
telegramFileUniqueId: string;
storageMessageId: number;
}
/** File information returned by Telegram's getFile API */
export interface TelegramFileInfo {
file_size: number;
mime_type: string;
file_path: string;
bot_token: string;
}
/** Abstraction over Telegram bot API operations */
export interface ITelegramService {
forwardToStorage(fileChunk: unknown, fileName: string, fileType: string): Promise<ForwardResult>;
getFileInfo(telegramFileId: string): Promise<TelegramFileInfo>;
enqueueUpload<T>(task: () => Promise<T>): Promise<T>;
}
```
- [ ] **Compile check**
```bash
bun build src/domain/ports/file-repository.ts --target=bun --outfile=/dev/null 2>&1
bun build src/domain/ports/telegram-service.ts --target=bun --outfile=/dev/null 2>&1
```
Expected: No errors (domain has no infrastructure dependencies)
- [ ] **Commit**
```bash
git add src/domain/ports/
git commit -m "feat: create domain port interfaces with JSDoc"
```
---
### Phase 3: Infrastructure Layer — Persistence, Telegram, Cache
#### Task 7: Create infrastructure/persistence/drizzle (DB init, schema, migrate)
**Files:**
- Create: `src/infrastructure/persistence/drizzle/index.ts`
- Create: `src/infrastructure/persistence/drizzle/schema.ts`
- Create: `src/infrastructure/persistence/drizzle/migrate.ts`
**Interfaces:**
- Consumes: domain entities (for type alignment)
- Produces: `db` client, `files`, `fileParts` table definitions, `runMigration()`, Drizzle schema types
- [ ] **Create `src/infrastructure/persistence/drizzle/index.ts`** — Move content from `src/db/index.ts`. Same implementation. Add JSDoc to exports.
- [ ] **Create `src/infrastructure/persistence/drizzle/schema.ts`** — Move content from `src/db/schema.ts`. Same implementation. Add JSDoc to table definitions and exported types.
- [ ] **Create `src/infrastructure/persistence/drizzle/migrate.ts`** — Move content from `src/db/migrate.ts`. Same implementation. Add JSDoc.
- [ ] **Compile check**
```bash
bun build src/infrastructure/persistence/drizzle/index.ts --target=bun --outfile=/dev/null 2>&1 | head -10
```
Expected: No errors
- [ ] **Commit**
```bash
git add src/infrastructure/persistence/drizzle/
git commit -m "feat: create persistence drizzle layer"
```
#### Task 8: Create infrastructure/persistence/repositories (implement ports)
**Files:**
- Create: `src/infrastructure/persistence/repositories/file-repository.ts`
- Create: `src/infrastructure/persistence/repositories/bucket-repository.ts`
- Create: `src/infrastructure/persistence/repositories/file-part-repository.ts`
- Create: `src/infrastructure/persistence/repositories/multipart-repository.ts`
**Interfaces:**
- Consumes: `IFileRepository`, `IBucketRepository`, `IFilePartRepository`, `IMultipartRepository` from domain/ports
- Produces: Concrete repository classes implementing each interface
- [ ] **Create `src/infrastructure/persistence/repositories/file-repository.ts`** — Implement `IFileRepository` using Drizzle. Extract logic from `src/db/files.ts` and `src/db/files-ext.ts`. Methods delegate to Drizzle queries (same SQL as original).
```typescript
import { eq, and, sql } from 'drizzle-orm';
import { db, files as fileSchema } from '../drizzle/index';
import type { File, NewFile } from '../../../domain/entities/file';
import type { IFileRepository, S3FileRecord } from '../../../domain/ports/file-repository';
export class DrizzleFileRepository implements IFileRepository {
async findByHash(hash: string): Promise<File | null> {
const result = await db.select().from(fileSchema).where(eq(fileSchema.fileHash, hash)).limit(1);
return result[0] || null;
}
// ... all methods from original files.ts + files-ext.ts
}
```
- [ ] **Create `src/infrastructure/persistence/repositories/bucket-repository.ts`** — Implement `IBucketRepository` from `src/db/buckets.ts`.
- [ ] **Create `src/infrastructure/persistence/repositories/file-part-repository.ts`** — Implement `IFilePartRepository` from `src/db/file-parts.ts`.
- [ ] **Create `src/infrastructure/persistence/repositories/multipart-repository.ts`** — Implement `IMultipartRepository` from `src/db/multipart.ts`.
- [ ] **Compile check**
```bash
bun build src/infrastructure/persistence/repositories/file-repository.ts --target=bun --outfile=/dev/null 2>&1 | head -10
```
Expected: No errors
- [ ] **Commit**
```bash
git add src/infrastructure/persistence/repositories/
git commit -m "feat: create Drizzle repository implementations"
```
#### Task 9: Create infrastructure/telegram (bot-pool, queue, batcher, chunked-storage)
**Files:**
- Create: `src/infrastructure/telegram/types.ts`
- Create: `src/infrastructure/telegram/bot-pool.ts`
- Create: `src/infrastructure/telegram/upload-queue.ts`
- Create: `src/infrastructure/telegram/upload-batcher.ts`
- Create: `src/infrastructure/telegram/chunked-storage.ts`
**Interfaces:**
- Consumes: `ITelegramService`, `IFilePartRepository`, `IFileRepository` from domain/ports
- Produces: `DrizzleFileRepository`, `BotPool`, etc.
- [ ] **Create `src/infrastructure/telegram/types.ts`** — Extract Telegram-specific types from `src/utils/telegram.ts` (no longer inline).
- [ ] **Create `src/infrastructure/telegram/bot-pool.ts`** — Move multi-bot pool + retry logic from `src/utils/telegram.ts`. Implement `ITelegramService` interface. Add JSDoc.
- [ ] **Create `src/infrastructure/telegram/upload-queue.ts`** — Move P-Queue wrapper from `src/utils/telegramQueue.ts`. Add JSDoc.
- [ ] **Create `src/infrastructure/telegram/upload-batcher.ts`** — Move batch upload logic from `src/utils/uploadBatcher.ts`. Add JSDoc. Accept repository interface instead of importing `db` directly.
- [ ] **Create `src/infrastructure/telegram/chunked-storage.ts`** — Move chunked storage logic from `src/utils/chunked-storage.ts`. Accept repository interfaces. Add JSDoc.
- [ ] **Compile check**
```bash
bun build src/infrastructure/telegram/bot-pool.ts --target=bun --outfile=/dev/null 2>&1 | head -10
bun build src/infrastructure/telegram/upload-batcher.ts --target=bun --outfile=/dev/null 2>&1 | head -10
```
Expected: No errors
- [ ] **Commit**
```bash
git add src/infrastructure/telegram/
git commit -m "feat: create infrastructure telegram layer"
```
#### Task 10: Create infrastructure/cache
**Files:**
- Create: `src/infrastructure/cache/index.ts`
**Interfaces:**
- Produces: `Cache<T>`, `fileInfoCache` singleton
- [ ] **Create `src/infrastructure/cache/index.ts`** — Move generic TTL cache from `src/utils/cache.ts`. Add JSDoc. Same implementation.
- [ ] **Compile check**
```bash
bun build src/infrastructure/cache/index.ts --target=bun --outfile=/dev/null 2>&1 | head -5
```
Expected: No errors
- [ ] **Commit**
```bash
git add src/infrastructure/cache/
git commit -m "feat: create infrastructure cache layer"
```
---
### Phase 4: Application Layer — DTOs + Use Cases
#### Task 11: Create application/dto
**Files:**
- Create: `src/application/dto/upload.ts`
- Create: `src/application/dto/file.ts`
- Create: `src/application/dto/bucket.ts`
- Create: `src/application/dto/s3.ts`
- Create: `src/application/dto/auth.ts`
**Interfaces:**
- Produces: `UploadInput`, `UploadOutput`, `FileInfoResponse`, `BucketResponse`, `S3ObjectResponse`, `AuthSession`, etc.
- [ ] **Create `src/application/dto/upload.ts`** — Define upload request/response DTOs.
```typescript
/** Input for the upload file use case */
export interface UploadInput {
tempPath: string;
fileHash: string;
fileName: string;
mimeType: string;
fileType: string;
sizeBytes: number;
uploaderId?: number;
bucketId?: string | null;
s3Key?: string | null;
}
/** Output from the upload file use case */
export interface UploadOutput {
publicId: string;
fileName: string;
mimeType: string;
sizeBytes: number;
fileType: string;
createdAt: Date;
downloadUrl: string;
}
```
- [ ] **Create `src/application/dto/file.ts`** — File info response DTOs.
- [ ] **Create `src/application/dto/bucket.ts`** — Bucket CRUD DTOs.
- [ ] **Create `src/application/dto/s3.ts`** — S3 operation DTOs (list, copy, multipart).
- [ ] **Create `src/application/dto/auth.ts`** — Auth request/response DTOs (login input, session info).
- [ ] **Compile check**
```bash
bun build src/application/dto/upload.ts --target=bun --outfile=/dev/null 2>&1 | head -5
```
Expected: No errors
- [ ] **Commit**
```bash
git add src/application/dto/
git commit -m "feat: create application DTOs"
```
#### Task 12: Create application/use-cases (upload, get-file, auth)
**Files:**
- Create: `src/application/use-cases/upload-file.ts`
- Create: `src/application/use-cases/get-file.ts`
- Create: `src/application/use-cases/authenticate.ts`
**Interfaces:**
- Consumes: Repository interfaces from domain/ports, DTOs from application/dto
- Produces: Factory functions returning use case closures
- [ ] **Create `src/application/use-cases/upload-file.ts`** — Extract upload business logic from `src/routes/upload.ts` + `src/utils/chunked-storage.ts`. Factory function pattern:
```typescript
import { nanoid } from 'nanoid';
import type { IFileRepository } from '../../domain/ports/file-repository';
import type { IFilePartRepository } from '../../domain/ports/file-part-repository';
import type { ITelegramService } from '../../domain/ports/telegram-service';
import type { AppConfig } from '../../config/index';
import type { UploadInput, UploadOutput } from '../dto/upload';
export interface UploadFileUseCaseDeps {
fileRepo: IFileRepository;
filePartRepo: IFilePartRepository;
telegramService: ITelegramService;
config: AppConfig;
}
export function createUploadFileUseCase(deps: UploadFileUseCaseDeps) {
return async (input: UploadInput): Promise<UploadOutput> => {
// 1. Check dedup (fileRepo.findByHash)
// 2. Check file size limits
// 3. Determine storage strategy: chunked vs single vs batch
// 4. Store via telegramService
// 5. Insert DB record via fileRepo
// 6. Build and return UploadOutput
};
}
```
- [ ] **Create `src/application/use-cases/get-file.ts`** — Extract file redirect/stream logic from `src/routes/files.ts`.
- [ ] **Create `src/application/use-cases/authenticate.ts`** — Extract login/logout/session logic from `src/routes/auth.ts` + `src/utils/auth.ts`.
- [ ] **Compile check**
```bash
bun build src/application/use-cases/upload-file.ts --target=bun --outfile=/dev/null 2>&1 | head -15
```
Expected: No errors (may have non-functional code until controllers wired up — skip if not compile-ready)
- [ ] **Commit**
```bash
git add src/application/use-cases/authenticate.ts src/application/use-cases/upload-file.ts src/application/use-cases/get-file.ts
git commit -m "feat: create application use cases (upload, get-file, auth)"
```
#### Task 13: Create application/use-cases (bucket, s3-object, multipart)
**Files:**
- Create: `src/application/use-cases/manage-bucket.ts`
- Create: `src/application/use-cases/s3-object.ts`
- Create: `src/application/use-cases/multipart-upload.ts`
**Interfaces:**
- Consumes: Repository interfaces from domain/ports, DTOs from application/dto
- Produces: Factory functions for bucket CRUD, S3 object ops, multipart upload
- [ ] **Create `src/application/use-cases/manage-bucket.ts`** — Extract bucket CRUD logic from `src/routes/web-api.ts` + `src/routes/s3.ts`.
- [ ] **Create `src/application/use-cases/s3-object.ts`** — Extract S3 get/put/delete/copy logic from `src/routes/s3.ts`. This is the biggest extraction.
- [ ] **Create `src/application/use-cases/multipart-upload.ts`** — Extract S3 multipart upload logic from `src/routes/s3.ts`.
- [ ] **Commit**
```bash
git add src/application/use-cases/manage-bucket.ts src/application/use-cases/s3-object.ts src/application/use-cases/multipart-upload.ts
git commit -m "feat: create application use cases (bucket, s3-object, multipart)"
```
---
### Phase 5: Interfaces Layer — Controllers, Middleware, Routes, S3, Bot
#### Task 14: Create interfaces/s3 protocol files
**Files:**
- Create: `src/interfaces/s3/auth.ts`
- Create: `src/interfaces/s3/headers.ts`
- Create: `src/interfaces/s3/object-stream.ts`
- Create: `src/interfaces/s3/range.ts`
- Create: `src/interfaces/s3/virtual-host.ts`
- Create: `src/interfaces/s3/xml.ts`
**Interfaces:**
- Produces: SigV4 auth, S3 headers, object streaming, range parsing, virtual-host extraction, XML builders — all same as `src/utils/s3/*`
- [ ] **Create each file** — Move content from `src/utils/s3/auth.ts`, `src/utils/s3/headers.ts`, etc. Add JSDoc to each exported function. Same implementations.
- [ ] **Compile check**
```bash
bun build src/interfaces/s3/auth.ts --target=bun --outfile=/dev/null 2>&1 | head -5
```
Expected: No errors
- [ ] **Commit**
```bash
git add src/interfaces/s3/
git commit -m "feat: create interfaces/s3 protocol layer"
```
#### Task 15: Create HTTP middleware (auth, rate-limit)
**Files:**
- Create: `src/interfaces/http/middleware/auth.ts`
- Create: `src/interfaces/http/middleware/rate-limit.ts`
**Interfaces:**
- Produces: `requireAuth`, `getAuthSession`, `withRateLimit`, `checkRateLimit`, `cleanupRateLimitCache`
- [ ] **Create `src/interfaces/http/middleware/auth.ts`** — Move auth middleware from `src/utils/auth.ts`. Update imports to use `config` from `../../config/index`. Add JSDoc.
- [ ] **Create `src/interfaces/http/middleware/rate-limit.ts`** — Move rate limiter from `src/utils/rateLimit.ts`. Add JSDoc.
- [ ] **Compile check**
```bash
bun build src/interfaces/http/middleware/auth.ts --target=bun --outfile=/dev/null 2>&1 | head -10
```
Expected: No errors
- [ ] **Commit**
```bash
git add src/interfaces/http/middleware/
git commit -m "feat: create HTTP middleware layer"
```
#### Task 16: Create HTTP controllers
**Files:**
- Create: `src/interfaces/http/controllers/upload-controller.ts`
- Create: `src/interfaces/http/controllers/file-controller.ts`
- Create: `src/interfaces/http/controllers/auth-controller.ts`
- Create: `src/interfaces/http/controllers/home-controller.ts`
- Create: `src/interfaces/http/controllers/health-controller.ts`
- Create: `src/interfaces/http/controllers/s3-controller.ts`
- Create: `src/interfaces/http/controllers/web-api-controller.ts`
**Interfaces:**
- Consumes: Use case factory functions from application layer
- Produces: HTTP handler functions compatible with `Bun.serve()` routes
- [ ] **Create `src/interfaces/http/controllers/upload-controller.ts`** — Extract upload HTTP handling from `src/routes/upload.ts`. Controller parses request, calls use case, formats response:
```typescript
import { createUploadFileUseCase } from '../../../application/use-cases/upload-file';
// ... setup use case with dependencies
// ... handler functions for multipart and JSON upload paths
```
- [ ] **Create `src/interfaces/http/controllers/file-controller.ts`** — Extract from `src/routes/files.ts`. Controller handles params, calls getFile use case, handles redirect/stream.
- [ ] **Create `src/interfaces/http/controllers/auth-controller.ts`** — Extract from `src/routes/auth.ts`.
- [ ] **Create `src/interfaces/http/controllers/home-controller.ts`** — Move from `src/routes/home.ts`.
- [ ] **Create `src/interfaces/http/controllers/health-controller.ts`** — Move from `src/routes/health.ts`.
- [ ] **Create `src/interfaces/http/controllers/s3-controller.ts`** — Extract S3 dispatching from `src/routes/s3.ts`. This is the largest extraction — split into clear sections (bucket ops, object ops, multipart ops). The 1179-line file becomes a focused controller that delegates to use cases.
- [ ] **Create `src/interfaces/http/controllers/web-api-controller.ts`** — Extract from `src/routes/web-api.ts`.
- [ ] **Move `src/home.html` to `src/interfaces/http/controllers/home.html`** — Update reference in home-controller.
- [ ] **Commit**
```bash
git add src/interfaces/http/controllers/
git commit -m "feat: create HTTP controller layer"
```
#### Task 17: Create HTTP route definitions
**Files:**
- Create: `src/interfaces/http/routes/index.ts`
**Interfaces:**
- Consumes: All controller handler functions
- Produces: Route table for `Bun.serve()`
- [ ] **Create `src/interfaces/http/routes/index.ts`** — Define route table matching current `src/index.ts` routes. Each route maps to its controller handler. Add JSDoc.
```typescript
import type { Server } from 'bun';
import { handleUpload } from '../controllers/upload-controller';
import { handleFileRedirect, handleFileInfo } from '../controllers/file-controller';
// ... other imports
/** Defines all HTTP routes for the application */
export const routes: Server['routes'] = {
'/api/upload': { POST: handleUpload },
'/f/:public_id': { GET: handleFileRedirect },
'/file/:public_id/info': { GET: handleFileInfo },
'/health': { GET: handleHealth },
'/docs': { GET: handleSwaggerHtml },
'/swagger.json': { GET: handleSwaggerJson },
// ... etc
};
```
- [ ] **Commit**
```bash
git add src/interfaces/http/routes/
git commit -m "feat: create HTTP route definitions"
```
#### Task 18: Create interfaces/bot
**Files:**
- Create: `src/interfaces/bot/handler.ts`
**Interfaces:**
- Consumes: `ITelegramService`, use cases
- Produces: `startBot()` function for Telegram bot lifecycle
- [ ] **Create `src/interfaces/bot/handler.ts`** — Move bot handler logic from `src/bot.ts`. Add JSDoc. Update imports to use new structure.
- [ ] **Commit**
```bash
git add src/interfaces/bot/
git commit -m "feat: create bot interface layer"
```
---
### Phase 6: Entry Point Rewire
#### Task 19: Rewrite src/index.ts
**Files:**
- Modify: `src/index.ts`
- Modify (delete): `src/bot.ts`
- [ ] **Rewrite `src/index.ts`** — Update to import from new structure. Server bootstrap, S3 detection, route table from `interfaces/http/routes/index`, lifecycle management.
```typescript
import { serve } from 'bun';
import { config } from './config/index';
import { startBot } from './interfaces/bot/handler';
import { routes } from './interfaces/http/routes/index';
import { isS3Request } from './interfaces/s3/auth';
import { handleS3Request } from './interfaces/http/controllers/s3-controller';
import { extractS3BucketFromHost } from './interfaces/s3/virtual-host';
import { fileInfoCache } from './infrastructure/cache/index';
import { cleanupRateLimitCache } from './interfaces/http/middleware/rate-limit';
import { logger } from './shared/logger/index';
import { metricsCollector } from './shared/metrics/index';
```
- [ ] **Remove `src/bot.ts`** (replaced by `src/interfaces/bot/handler.ts`)
- [ ] **Run full test suite**
```bash
bun test
```
Expected: All tests PASS (old src files still exist for backward compat)
- [ ] **Commit**
```bash
git add src/index.ts src/interfaces/bot/handler.ts
git rm src/bot.ts
git commit -m "feat: rewire entry point to new architecture"
```
---
### Phase 7: Test Restructure
#### Task 20: Create mirror test structure
**Files:**
- Create: `test/unit/domain/entities/` (entity type tests)
- Create: `test/unit/application/use-cases/` (use case tests)
- Create: `test/unit/shared/` (util tests)
- Create: `test/integration/interfaces/http/` (route tests)
- Create: `test/integration/infrastructure/` (repo tests)
- [ ] **Create test folder structure mirroring src/**
```bash
mkdir -p test/unit/domain/entities \
test/unit/application/use-cases \
test/unit/shared \
test/integration/interfaces/http \
test/integration/infrastructure/persistence \
test/integration/infrastructure/telegram
```
- [ ] **Move util tests** — Copy `test/env.test.ts``test/unit/config/env.test.ts`, update import path to `src/config/index`. Copy `test/file.test.ts``test/unit/shared/file.test.ts`, update imports. Copy `test/zip.test.ts``test/unit/shared/zip.test.ts`.
- [ ] **Move route tests** — Copy `test/upload.test.ts``test/integration/interfaces/http/upload.test.ts`, update imports. Copy `test/files.test.ts``test/integration/interfaces/http/files.test.ts`. Copy `test/auth-routes.test.ts`, `test/health.test.ts`, `test/swagger.test.ts`.
- [ ] **Move S3 tests** — Copy `test/s3-auth.test.ts``test/unit/interfaces/s3/auth.test.ts`. Copy `test/s3-range.test.ts`, `test/s3-operations.test.ts`, `test/s3-bucket-config.test.ts`, `test/s3-object-stream.test.ts`.
- [ ] **Move infrastructure tests** — Copy `test/db.test.ts``test/unit/infrastructure/persistence/drizzle/schema.test.ts`. Copy `test/chunked-storage.test.ts``test/unit/infrastructure/telegram/chunked-storage.test.ts`. Copy `test/telegram.test.ts`, `test/telegramQueue.test.ts`.
- [ ] **Keep original test files** — Do NOT delete them yet. Original tests still reference old source paths and need to keep passing for now.
- [ ] **Update test runner scripts in package.json** — Add test patterns for new test locations:
```json
"test:new": "bun test test/unit/ test/integration/",
"test:all": "bun test && bun test test/unit/ test/integration/"
```
- [ ] **Run both old and new tests**
```bash
bun test test/env.test.ts
bun test test/unit/config/env.test.ts
```
Expected: Both PASS
- [ ] **Commit**
```bash
git add test/unit/ test/integration/ package.json
git commit -m "test: restructure tests to mirror new architecture"
```
---
### Phase 8: Cleanup — Delete Old Files
#### Task 21: Remove old source directories
**Files:** Delete old source directories once new structure is verified.
- [ ] **Delete old directories** (only after all tests pass with new imports)
```bash
rm -rf src/db src/routes src/utils src/env.ts src/bot.ts
```
- [ ] **Run full test suite to confirm nothing is broken**
```bash
bun test
bun test test/unit/ test/integration/
```
Expected: All PASS
- [ ] **Commit**
```bash
git rm -r src/db src/routes src/utils src/env.ts src/bot.ts
git commit -m "chore: remove legacy source directories after migration"
```
#### Task 22: Final verification
- [ ] **Full test suite**
```bash
bun run test:all
```
Expected: All PASS
- [ ] **Build verification**
```bash
bun run build
```
Expected: dist/index.js and dist/migrate.js compile without errors
- [ ] **Lint check**
```bash
bunx biome check src test/unit test/integration
```
Expected: No errors
- [ ] **Final commit** if any fixes were needed
```bash
git add -A
git commit -m "chore: final cleanup after architecture migration"
```
---
## Spec Coverage Check
| Spec Requirement | Covered In |
|-----------------|------------|
| New folder structure | Task 5-19 (all phases) |
| DDD layers | Phase 2 (domain), Phase 4 (application), Phase 3 (infrastructure), Phase 5 (interfaces) |
| Repository interfaces | Task 6 |
| Repository implementations | Task 8 |
| Controller/use-case separation | Task 12-13 (use cases), Task 16 (controllers) |
| Dependency injection (factory functions) | Task 12 |
| Telegram service interface | Task 6 (ITelegramService port) |
| Bot moved to interfaces/bot | Task 18 |
| S3 protocol moved to interfaces/s3 | Task 14 |
| Config moved to config/ | Task 1 |
| Shared utilities moved to shared/ | Task 2-4 |
| Error classes | Task 2 |
| JSDoc on all exports | All creation tasks |
| Test restructure | Task 20 |
| Zero behavior changes | No logic altered — only moved/extracted |
| Delete old files | Task 21 |
| Final verification | Task 22 |
@@ -0,0 +1,576 @@
# Per-Bot Queue Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Replace global PQueue with per-bot queues (concurrency=1 per bot) to eliminate 429 collisions and improve rate-limit safety.
**Architecture:** Each bot token gets its own PQueue with concurrency=1. Uploads are assigned to the least-loaded available bot via `selectBot()`. On 429, the bot is marked rate-limited and the upload retries on the next available bot. The global `upload-queue.ts` is removed; `uploadConcurrency` config is replaced by `botCount * perBotConcurrency`.
**Tech Stack:** TypeScript, PQueue, Telegraf
## Global Constraints
- Use `Bun` runtime, not Node.js
- Follow existing code style (Biome lint)
- Each bot queue has concurrency=1 (no two uploads hit same bot simultaneously)
- `selectBot()` picks bot with lowest pending queue count, skipping rate-limited bots
- Remove `uploadConcurrency` from config; derive effective concurrency from bot count
- Remove `upload-queue.ts` entirely
- Remove `enqueueUpload` from `ITelegramService` interface
---
## File Structure
### Files to Modify
| File | Change |
|------|--------|
| `src/infrastructure/telegram/bot-pool.ts` | BotEntry array, selectBot(), per-bot queues, retry logic |
| `src/domain/ports/telegram-service.ts` | Remove `enqueueUpload` from interface |
| `src/env.ts` | Remove `uploadConcurrency` field |
| `src/index.ts` | Remove upload-queue import and usage |
| `src/utils/chunked-storage.ts` | Replace `config.uploadConcurrency` with bot count |
### Files to Delete
| File | Reason |
|------|--------|
| `src/infrastructure/telegram/upload-queue.ts` | Global queue replaced by per-bot queues |
| `test/telegramQueue.test.ts` | Tests for deleted module |
### Files Not Changed
| File | Reason |
|------|--------|
| `test/bot.test.ts` | Only uses ITelegramService interface (via `forwardToStorage`) |
| `src/infrastructure/telegram/chunked-storage.ts` | Uses ITelegramService interface, not BotPool directly |
---
### Task 1: Refresh the spec & plan files after compaction
Due to context compaction, re-read the current spec and plan files to ensure accuracy before implementing.
- [ ] **Step 1: Re-read the spec**
Read: `docs/superpowers/specs/2026-07-29-per-bot-queue-design.md`
- [ ] **Step 2: Re-read key implementation files**
Read: `src/infrastructure/telegram/bot-pool.ts`, `src/env.ts`, `src/utils/chunked-storage.ts`
### Task 2: Refactor ITelegramService interface
Remove `enqueueUpload` from the interface — BotPool handles queueing internally now.
**Files:**
- Modify: `src/domain/ports/telegram-service.ts`
- [ ] **Step 1: Remove `enqueueUpload` from interface**
```typescript
// src/domain/ports/telegram-service.ts — remove entire section:
/**
* Enqueue a task for sequential upload execution.
*
* Ensures only one Telegram upload runs at a time to avoid
* rate limits and resource contention.
*
* @param task - An async function performing the upload.
* @returns The result of the task.
*/
enqueueUpload<T>(task: () => Promise<T>): Promise<T>;
```
- [ ] **Step 2: Run lint to verify**
Run: `bunx biome check src/domain/ports/telegram-service.ts`
Expected: No errors.
- [ ] **Step 3: Commit**
```bash
git add src/domain/ports/telegram-service.ts
git commit -m "refactor: remove enqueueUpload from ITelegramService
Per-bot queue handles queueing internally.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>"
```
### Task 3: Refactor BotPool with per-bot queues
The core of the redesign. Replace `claimBotIndex()` round-robin with per-bot PQueue instances and `selectBot()` for least-loaded assignment.
**Files:**
- Modify: `src/infrastructure/telegram/bot-pool.ts`
**Interfaces:**
- Consumes: `ITelegramService` (no `enqueueUpload` method)
- Produces: `botPool` singleton with per-bot queues, `selectBot()`, per-bot rate-limit tracking
- [ ] **Step 1: Write test file for per-bot queue behavior**
```typescript
// test/bot-pool.test.ts
import { beforeEach, describe, expect, it, mock, spyOn } from 'bun:test';
// We'll test the BotEntry queue behavior and selectBot logic
```
- [ ] **Step 2: Implement BotEntry type and BotPool refactor**
Replace the class body:
```typescript
import PQueue from 'p-queue';
import { Telegraf } from 'telegraf';
import type {
ForwardResult,
ITelegramService,
TelegramFileInfo,
} from '../../domain/ports/telegram-service';
import { config } from '../../env';
import logger from '../../shared/logger/index';
import {
buildSendPayload,
extractUploadedFile,
type SendMethod,
sendMethodMap,
type TelegramMessageResult,
} from './types';
const sleep = (ms: number): Promise<void> =>
new Promise((resolve) => setTimeout(resolve, ms));
const isTransientError = (error: unknown): boolean => {
const str = error instanceof Error ? error.message : String(error);
const transientPatterns = [
'timeout', 'Timed out', 'etimedout', 'econnrefused', 'econnreset',
'ECONNREFUSED', 'ECONNRESET', 'ETIMEDOUT', '5xx', '502', '503', '504',
'Bad Gateway', 'Service Unavailable', 'Gateway Timeout', 'socket hang up',
'socket closed', 'fetch failed', 'network error', 'network timeout',
'API closed', 'read ECONNRESET', 'write EPIPE',
];
return transientPatterns.some((p) => str.toLowerCase().includes(p.toLowerCase()));
};
const MAX_TRANSIENT_RETRIES = 3;
const TELEGRAM_API_TIMEOUT_MS = 120_000;
const PER_BOT_CONCURRENCY = 1;
interface BotEntry {
index: number;
token: string;
instance: Telegraf;
queue: PQueue;
rateLimitedUntil: number; // 0 = not rate-limited
}
export class BotPool implements ITelegramService {
private readonly bots: BotEntry[] = [];
constructor() {
const tokens = Array.from(new Set([config.botToken, ...config.additionalBotTokens]));
this.bots = tokens.map((token, index) => ({
index,
token,
instance: new Telegraf(token),
queue: new PQueue({ concurrency: PER_BOT_CONCURRENCY }),
rateLimitedUntil: 0,
}));
}
/** Number of bots in the pool */
get size(): number {
return this.bots.length;
}
/**
* Select the bot with the fewest pending tasks that isn't rate-limited
* or in the skip set.
*/
private selectBot(skipIndexes?: Set<number>): BotEntry | null {
let best: BotEntry | null = null;
let bestPending = Infinity;
for (const bot of this.bots) {
if (skipIndexes?.has(bot.index)) continue;
if (bot.rateLimitedUntil > Date.now()) continue;
const pending = bot.queue.pending + bot.queue.size;
if (pending < bestPending) {
bestPending = pending;
best = bot;
}
}
return best;
}
/**
* Execute a Telegram API action on a specific bot entry.
* Wraps with timeout.
*/
private async executeBotAction<T>(
bot: BotEntry,
action: (instance: Telegraf, token: string) => Promise<T>,
): Promise<T> {
return Promise.race([
action(bot.instance, bot.token),
new Promise<never>((_, reject) =>
setTimeout(
() => reject(new Error(`Telegram API timeout after ${TELEGRAM_API_TIMEOUT_MS}ms`)),
TELEGRAM_API_TIMEOUT_MS,
),
),
]);
}
/**
* Forward a file chunk to the configured Telegram storage chat.
*
* The upload is submitted to the least-loaded bot's queue. If the bot
* returns 429, it is marked rate-limited and the upload retries on the
* next available bot. If all bots are rate-limited, sleeps before retrying.
*/
async forwardToStorage(
fileChunk: unknown,
fileName: string,
fileType: string,
): Promise<ForwardResult> {
let lastError: unknown;
const attemptedIndexes = new Set<number>();
let transientAttempts = 0;
// Outer retry loop — up to 10 attempts across all bots
for (let attempt = 0; attempt < 10; attempt++) {
const bot = this.selectBot(attemptedIndexes);
if (!bot) {
// No available bots — either all rate-limited or all attempted
if (attemptedIndexes.size > 0) {
// All non-rate-limited bots were tried and failed — wait & reset
logger.warn('All available bots exhausted, sleeping 5s before retry');
await sleep(5000 + Math.random() * 1000);
attemptedIndexes.clear();
continue;
}
// All bots rate-limited — wait for the shortest cooldown
const earliestCooldown = Math.min(
...this.bots.map((b) => b.rateLimitedUntil || Infinity),
);
const waitMs = Math.max(1000, earliestCooldown - Date.now() + 500);
logger.warn('All bots rate-limited, waiting', { waitMs });
await sleep(waitMs);
attemptedIndexes.clear();
continue;
}
attemptedIndexes.add(bot.index);
try {
const result = await bot.queue.add(async () => {
// Inner transient retry loop inside the queue
for (let innerRetry = 0; innerRetry <= MAX_TRANSIENT_RETRIES; innerRetry++) {
try {
const filePayload = { source: fileChunk, filename: fileName };
const sendMethodName = sendMethodMap[fileType] || 'sendDocument';
const payload = buildSendPayload(fileType, fileName);
const tgResult = await this.executeBotAction<TelegramMessageResult>(
bot,
(activeBot) => {
const telegram = activeBot.telegram as unknown as Record<string, SendMethod>;
return telegram[sendMethodName](config.storageChatId, filePayload, payload);
},
);
const uploadedFile = extractUploadedFile(tgResult, fileType);
return {
telegramFileId: uploadedFile?.file_id || '',
telegramFileUniqueId: uploadedFile?.file_unique_id || '',
storageMessageId: tgResult.message_id,
};
} catch (error: unknown) {
const errorStr = error instanceof Error ? error.message : String(error);
const retryAfterMatch = errorStr.match(/retry after (\d+)/i);
if (retryAfterMatch) {
// 429 — mark bot rate-limited, throw to outer loop for retry on different bot
const seconds = parseInt(retryAfterMatch[1], 10);
bot.rateLimitedUntil = Date.now() + seconds * 1000;
logger.info(`Bot #${bot.index} rate-limited for ${seconds}s`, { fileName, attempt });
throw error; // caught by outer retry loop
}
if (innerRetry < MAX_TRANSIENT_RETRIES && isTransientError(error)) {
const backoffMs = Math.min(1000 * 2 ** innerRetry, 10_000);
logger.warn(
`Transient error on bot #${bot.index}, retrying (${innerRetry + 1}/${MAX_TRANSIENT_RETRIES})`,
{ fileName, error: errorStr, backoffMs },
);
await sleep(backoffMs);
continue;
}
throw error; // non-transient — propagate
}
}
throw new Error(`Exhausted transient retries on bot #${bot.index}`);
});
logger.info('File forwarded to storage', { fileName, message: result.storageMessageId });
return result;
} catch (error: unknown) {
lastError = error;
const errorStr = error instanceof Error ? error.message : String(error);
const retryAfterMatch = errorStr.match(/retry after (\d+)/i);
if (retryAfterMatch) {
// Bot was rate-limited — already marked, try next bot
continue;
}
// Transient error at the queue level (timeout, 5xx)
if (transientAttempts < MAX_TRANSIENT_RETRIES && isTransientError(error)) {
transientAttempts++;
continue;
}
// Non-transient — give up
logger.error('Failed to forward file to storage', {
fileName,
error: errorStr,
attempt,
});
throw error;
}
}
throw lastError || new Error('Failed to forward file after all retries');
}
/** Get total effective concurrency across all bots */
getEffectiveConcurrency(): number {
return this.bots.length * PER_BOT_CONCURRENCY;
}
async getFileInfo(telegramFileId: string): Promise<TelegramFileInfo> {
let lastError: unknown;
for (const bot of this.bots) {
for (let retry = 0; retry <= MAX_TRANSIENT_RETRIES; retry++) {
try {
const result = await bot.instance.telegram.getFile(telegramFileId);
const fileData = result as unknown as Omit<TelegramFileInfo, 'bot_token'>;
return {
file_size: fileData.file_size || 0,
mime_type: fileData.mime_type || 'application/octet-stream',
file_path: fileData.file_path || '',
bot_token: bot.token,
};
} catch (error: unknown) {
lastError = error;
const errorStr = error instanceof Error ? error.message : String(error);
if (
errorStr.includes('wrong file_id') ||
errorStr.includes('file is temporarily unavailable')
) {
break;
}
if (retry < MAX_TRANSIENT_RETRIES && isTransientError(error)) {
const backoffMs = Math.min(1000 * 2 ** (retry + 1), 5_000);
await sleep(backoffMs);
continue;
}
break;
}
}
}
logger.error('Failed to get file info from any bot', {
error: lastError instanceof Error ? lastError.message : String(lastError),
});
throw lastError;
}
}
export const botPool = new BotPool();
```
- [ ] **Step 3: Run lint**
Run: `bunx biome check src/infrastructure/telegram/bot-pool.ts`
Expected: No errors.
- [ ] **Step 4: Run existing test suite**
Run: `bun test test/bot.test.ts`
Expected: All tests pass.
- [ ] **Step 5: Commit**
```bash
git add src/infrastructure/telegram/bot-pool.ts
git commit -m "refactor: per-bot queue with selectBot() and rate-limit tracking
Each bot has its own PQueue (concurrency=1). Uploads are assigned to
the least-loaded available bot. On 429, the bot is marked rate-limited
and the upload retries on the next available bot.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>"
```
### Task 4: Remove global upload queue
Delete the global queue, its test, and all references to it from index.ts.
**Files:**
- Delete: `src/infrastructure/telegram/upload-queue.ts`
- Modify: `src/index.ts` (lines 4, 71-81)
- Delete: `test/telegramQueue.test.ts`
- [ ] **Step 1: Delete upload-queue.ts**
Run: `rm src/infrastructure/telegram/upload-queue.ts`
- [ ] **Step 2: Delete the test file**
Run: `rm test/telegramQueue.test.ts`
- [ ] **Step 3: Update index.ts — remove upload-queue import and shutdown drain logic**
Remove line:
```typescript
import { clearQueue, getQueueStats, waitForQueue } from './infrastructure/telegram/upload-queue';
```
Remove the drain block (lines 70-81):
```typescript
// Drain pending upload queue with a timeout
const { pending, size } = getQueueStats();
if (pending > 0 || size > 0) {
logger.info('Draining upload queue', { pending, size });
const drainTimeout = setTimeout(() => {
logger.warn('Upload queue drain timeout — clearing remaining tasks');
clearQueue();
}, 30_000);
await waitForQueue();
clearTimeout(drainTimeout);
logger.info('Upload queue drained');
}
```
- [ ] **Step 4: Run lint**
Run: `bunx biome check src/index.ts`
Expected: No errors.
- [ ] **Step 5: Run tests**
Run: `bun test`
Expected: All tests pass (some may be skipped due to missing queue test).
- [ ] **Step 6: Commit**
```bash
git add src/infrastructure/telegram/upload-queue.ts test/telegramQueue.test.ts src/index.ts
git commit -m "refactor: remove global upload queue
Per-bot queues now handle concontrol internally.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>"
```
### Task 5: Update env.ts and chunked-storage backpressure
Remove `uploadConcurrency` from config and update chunked-storage to derive effective concurrency from bot pool.
**Files:**
- Modify: `src/env.ts`
- Modify: `src/utils/chunked-storage.ts`
- [ ] **Step 1: Remove `uploadConcurrency` from env.ts**
Remove:
```typescript
uploadConcurrency: number;
```
and:
```typescript
uploadConcurrency: parseNumber(process.env.UPLOAD_CONCURRENCY, 8),
```
- [ ] **Step 2: Update chunked-storage.ts backpressure**
Replace:
```typescript
import { config } from '../env';
// ...
if (inFlight.size >= config.uploadConcurrency * 2) {
```
With:
```typescript
import { botPool } from '../infrastructure/telegram/bot-pool';
// ...
if (inFlight.size >= botPool.getEffectiveConcurrency()) {
```
(Use effective concurrency * 2 for backpressure, or just use effective concurrency as the limit.)
Actually let me think about this more carefully. The backpressure in chunked-storage:
```
if (inFlight.size >= config.uploadConcurrency * 2) {
await Promise.race(inFlight);
await new Promise((resolve) => setTimeout(resolve, 0));
}
```
This limits the number of in-flight chunks per file. With `uploadConcurrency: 16`, it was 32. Now with effective concurrency of 6 (6 bots), it would be 12. That's fine as backpressure — it prevents too many chunks from being in memory at once.
Let me use `botPool.getEffectiveConcurrency() * 2` to keep the same multiplier.
- [ ] **Step 3: Run lint and tests**
```bash
bunx biome check src/env.ts src/utils/chunked-storage.ts
bun test test/chunked-storage.test.ts
```
Expected: All checks pass.
- [ ] **Step 4: Commit**
```bash
git add src/env.ts src/utils/chunked-storage.ts
git commit -m "refactor: remove uploadConcurrency from config
Effective concurrency derived from bot pool size. Chunked-storage
backpressure now uses botPool.getEffectiveConcurrency().
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>"
```
### Task 6: Full integration test
- [ ] **Step 1: Run the full test suite**
Run: `bun test`
Expected: All tests pass.
- [ ] **Step 2: Run lint**
Run: `bunx biome check src test`
Expected: No errors.
- [ ] **Step 3: Create summary commit**
```bash
git add -A
git commit -m "refactor: implement per-bot queue architecture
- Each bot has its own PQueue with concurrency=1
- selectBot() assigns uploads to least-loaded available bot
- 429 rate limits are tracked per-bot with cooldown timers
- Failed uploads retry on next available bot
- Removed global upload-queue.ts and uploadConcurrency config
- Updated ITelegramService interface
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>"
```
@@ -1,5 +1,10 @@
# Telegram Bot Uploader Backend Design # Telegram Bot Uploader Backend Design
> ⚠️ **LEGACY** — Dokumen historis (2026-05-17). Port & infrastruktur sudah berubah:
> produksi kini berjalan di port `4000` (Nix + systemd + Caddy, domain `upload.asepharyana.my.id`)
> dan database via PgBouncer pool `100.121.180.82:6432` (bukan port 5432, bukan localhost).
**Date:** 2026-05-17 **Date:** 2026-05-17
**Status:** Approved **Status:** Approved
**Stack:** Bun, Telegraf, PostgreSQL, Drizzle ORM, Winston, nanoid **Stack:** Bun, Telegraf, PostgreSQL, Drizzle ORM, Winston, nanoid
@@ -35,7 +40,7 @@ Production-ready backend untuk Telegram file uploader dengan dual upload methods
4. Bot extracts `telegram_file_id`, `telegram_file_unique_id`, `storage_message_id` 4. Bot extracts `telegram_file_id`, `telegram_file_unique_id`, `storage_message_id`
5. Bot generates `public_id` using nanoid 5. Bot generates `public_id` using nanoid
6. Bot saves metadata to PostgreSQL 6. Bot saves metadata to PostgreSQL
7. Bot replies with download link: `https://tele.asepharyana.my.id/f/{public_id}` 7. Bot replies with download link: `https://upload.asepharyana.my.id/f/{public_id}`
#### Upload via HTTP API #### Upload via HTTP API
1. Client POSTs to `/api/upload` with file (multipart or base64) 1. Client POSTs to `/api/upload` with file (multipart or base64)
@@ -115,7 +120,7 @@ fileName: optional_filename.ext
"size_bytes": 1024000, "size_bytes": 1024000,
"file_type": "document", "file_type": "document",
"created_at": "2026-05-17T23:42:19Z", "created_at": "2026-05-17T23:42:19Z",
"download_url": "https://tele.asepharyana.my.id/f/abc123xyz" "download_url": "https://upload.asepharyana.my.id/f/abc123xyz"
} }
``` ```
@@ -212,9 +217,9 @@ schema.sql
``` ```
BOT_TOKEN=<telegram_bot_token> BOT_TOKEN=<telegram_bot_token>
STORAGE_CHANNEL_ID=<private_channel_id> STORAGE_CHANNEL_ID=<private_channel_id>
BASE_URL=https://tele.asepharyana.my.id BASE_URL=https://upload.asepharyana.my.id
DATABASE_URL=postgresql://user:password@localhost:5432/telegram_uploader DATABASE_URL=postgresql://asephs:***@100.121.180.82:6432/uploader
PORT=3000 PORT=4000
NODE_ENV=production NODE_ENV=production
LOG_LEVEL=info LOG_LEVEL=info
RATE_LIMIT_WINDOW_MS=60000 RATE_LIMIT_WINDOW_MS=60000
@@ -7,6 +7,12 @@ metadata:
# Design: TeleUploader Deployment & CI/CD Setup # Design: TeleUploader Deployment & CI/CD Setup
> ⚠️ **LEGACY** — Dokumen historis (2026-05-18) untuk arsitektur Docker + Traefik +
> GitHub Actions. Docker & Traefik sudah dihapus dari VPS produksi (2026-08-02):
> deploy sekarang Nix + systemd + Caddy di orangevps, port `4000`, domain
> `upload.asepharyana.my.id`, database via PgBouncer pool `100.121.180.82:6432`.
We are setting up production deployment for TeleUploader on VPS `45.127.35.244` behind Traefik utilizing GitHub Actions. We are setting up production deployment for TeleUploader on VPS `45.127.35.244` behind Traefik utilizing GitHub Actions.
## 1. System Architecture ## 1. System Architecture
@@ -22,7 +28,7 @@ TeleUploader is a Bun-based service.
### `Dockerfile` ### `Dockerfile`
- Multi-stage build. - Multi-stage build.
- **Stage 1 (Build)**: Install dependencies, copy source files, run Biome lint/format checks, compile TS build to `dist/index.js` using `bun build`. - **Stage 1 (Build)**: Install dependencies, copy source files, run Biome lint/format checks, compile TS build to `dist/index.js` using `bun build`.
- **Stage 2 (Run)**: Use minimal `oven/bun:1.1-slim` runtime. Copy `dist/index.js`, `schema.sql`, and `package.json`. Expose port `3000`. - **Stage 2 (Run)**: Use minimal `oven/bun:1.1-slim` runtime. Copy `dist/index.js`, `schema.sql`, and `package.json`. Expose port `4000`.
### `docker-compose.yml` ### `docker-compose.yml`
```yaml ```yaml
@@ -38,7 +44,7 @@ services:
- STORAGE_CHANNEL_ID=${STORAGE_CHANNEL_ID} - STORAGE_CHANNEL_ID=${STORAGE_CHANNEL_ID}
- BASE_URL=${BASE_URL} - BASE_URL=${BASE_URL}
- DATABASE_URL=${DATABASE_URL} - DATABASE_URL=${DATABASE_URL}
- PORT=3000 - PORT=4000
- NODE_ENV=production - NODE_ENV=production
- LOG_LEVEL=info - LOG_LEVEL=info
networks: networks:
@@ -49,7 +55,7 @@ services:
- "traefik.http.routers.teleuploader.entrypoints=websecure" - "traefik.http.routers.teleuploader.entrypoints=websecure"
- "traefik.http.routers.teleuploader.tls=true" - "traefik.http.routers.teleuploader.tls=true"
- "traefik.http.routers.teleuploader.tls.certresolver=letsencrypt" - "traefik.http.routers.teleuploader.tls.certresolver=letsencrypt"
- "traefik.http.services.teleuploader.loadbalancer.server.port=3000" - "traefik.http.services.teleuploader.loadbalancer.server.port=4000"
networks: networks:
app-shared-net: app-shared-net:
@@ -1,5 +1,7 @@
# Gitea Remote and CI/CD Migration Design # Gitea Remote and CI/CD Migration Design
> Catatan (2026-08-02): Produksi sekarang port 4000, deploy Nix+systemd di orangevps, Caddy reverse proxy upload.asepharyana.my.id, DB via pgbouncer pool imrnes 100.121.180.82:6432. Docker/Traefik/Gitea-CI legacy.
## Goal ## Goal
Move the repository origin from GitLab to a new Gitea repository and add a Gitea Actions deployment flow that behaves like a GitHub Actions CI/CD pipeline. Move the repository origin from GitLab to a new Gitea repository and add a Gitea Actions deployment flow that behaves like a GitHub Actions CI/CD pipeline.
@@ -1,5 +1,7 @@
# S3 Compatibility Completion Design # S3 Compatibility Completion Design
> Catatan (2026-08-02): Produksi sekarang port 4000, deploy Nix+systemd di orangevps, Caddy reverse proxy upload.asepharyana.my.id, DB via pgbouncer pool imrnes 100.121.180.82:6432. Docker/Traefik/Gitea-CI legacy.
Date: 2026-07-07 Date: 2026-07-07
## Goal ## Goal
@@ -0,0 +1,310 @@
# DDD & Clean Architecture Restructure — TeleUploader (filedrop)
**Date:** 2026-07-28
**Author:** Claude
**Status:** Draft
## 1. Motivation
The current codebase uses a flat structure (`src/routes/`, `src/utils/`, `src/db/`) that mixes concerns:
- Route handlers contain inline DB queries, Telegram API calls, and business logic
- Domain entity types (from Drizzle ORM) leak into all layers
- No clear separation between HTTP concerns, application logic, and infrastructure
- Large files (`routes/s3.ts` at 1179 lines) are hard to maintain, test, and reason about
The goal is to restructure into **Domain-Driven Design (DDD)** with **Clean Architecture** principles, adding **JSDoc documentation** throughout, without changing behaviour or breaking existing tests.
## 2. Approach: Hybrid C
| Aspect | Decision |
|--------|----------|
| **Depth** | Structural DDD + repository abstraction + controller/use-case separation |
| **Domain entities** | Re-use Drizzle-generated types (pragmatic, not creating new pure domain models) |
| **Dependency inversion** | Repository interfaces defined in `domain/ports/`, implementations in `infrastructure/` |
| **Dependency injection** | Simple factory functions (no DI framework) |
| **Testing** | Test files restructured alongside source files |
| **Documentation** | JSDoc on all public functions, classes, interfaces, and types |
| **Language** | English for code and JSDoc |
## 3. Layer Architecture
```
┌─────────────────────────────────────────────┐
│ interfaces/ (Delivery) │
│ HTTP controllers, routes, middleware │
│ Bot handlers │
│ S3 protocol (SigV4, XML, range, stream) │
├─────────────────────────────────────────────┤
│ application/ (Use Cases) │
│ Upload file, get file, auth, bucket ops, │
│ S3 object ops, multipart upload │
├────────────────────┬────────────────────────┤
│ domain/ │ infrastructure/ │
│ entities/ │ persistence/ │
│ ports/ (interfaces)│ telegram/ │
│ │ cache/ │
├────────────────────┴────────────────────────┤
│ config/ + shared/ │
└─────────────────────────────────────────────┘
```
**Dependency Rule:** Outer layers depend on inner layers, never the reverse. Domain knows nothing about HTTP, Drizzle, or Telegram.
## 4. Full Directory Structure
```
src/
├── index.ts # Entry point: server bootstrap, lifecycle
├── domain/
│ ├── entities/
│ │ ├── file.ts # File entity type (re-export from drizzle)
│ │ ├── bucket.ts # Bucket entity type
│ │ ├── file-part.ts # FilePart entity type
│ │ └── multipart.ts # MultipartUpload, MultipartPart types
│ └── ports/
│ ├── file-repository.ts # IFileRepository interface
│ ├── bucket-repository.ts # IBucketRepository interface
│ ├── file-part-repository.ts # IFilePartRepository interface
│ └── multipart-repository.ts # IMultipartRepository interface
├── application/
│ ├── dto/
│ │ ├── upload.ts # Upload request/response DTOs
│ │ ├── file.ts # File response DTOs
│ │ ├── bucket.ts # Bucket DTOs
│ │ ├── s3.ts # S3 operation DTOs
│ │ └── auth.ts # Auth DTOs
│ └── use-cases/
│ ├── upload-file.ts # Upload logic (multipart, JSON base64)
│ ├── get-file.ts # File redirect/stream logic
│ ├── manage-bucket.ts # Bucket CRUD logic
│ ├── s3-object.ts # S3 get/put/delete/copy logic
│ ├── multipart-upload.ts # S3 multipart upload logic
│ └── authenticate.ts # Login/logout/session logic
├── infrastructure/
│ ├── persistence/
│ │ ├── drizzle/
│ │ │ ├── index.ts # DB client + Drizzle init (from db/index.ts)
│ │ │ └── schema.ts # Drizzle table definitions (from db/schema.ts)
│ │ │ └── migrate.ts # Migration runner (from db/migrate.ts)
│ │ └── repositories/
│ │ ├── file-repository.ts # IFileRepository implementation (Drizzle)
│ │ ├── bucket-repository.ts # IBucketRepository implementation
│ │ ├── file-part-repository.ts# IFilePartRepository implementation
│ │ └── multipart-repository.ts# IMultipartRepository implementation
│ ├── telegram/
│ │ ├── bot-pool.ts # Multi-bot pool + retry (from utils/telegram.ts)
│ │ ├── upload-queue.ts # P-Queue wrapper (from utils/telegramQueue.ts)
│ │ ├── upload-batcher.ts # Batch upload (from utils/uploadBatcher.ts)
│ │ ├── chunked-storage.ts # Chunked file storage (from utils/chunked-storage.ts)
│ │ └── types.ts # Telegram-related types
│ └── cache/
│ └── index.ts # Generic TTL cache (from utils/cache.ts)
├── interfaces/
│ ├── http/
│ │ ├── controllers/
│ │ │ ├── upload-controller.ts # Upload HTTP handler
│ │ │ ├── file-controller.ts # File redirect + info handler
│ │ │ ├── auth-controller.ts # Login/logout/me handler
│ │ │ ├── home-controller.ts # Home dashboard handler
│ │ │ ├── health-controller.ts # Health check handler
│ │ │ ├── s3-controller.ts # S3-compatible API handler
│ │ │ └── web-api-controller.ts # JSON REST API handler
│ │ ├── middleware/
│ │ │ ├── auth.ts # requireAuth, getAuthSession
│ │ │ └── rate-limit.ts # Rate limiter
│ │ └── routes/
│ │ └── index.ts # All route definitions
│ ├── bot/
│ │ └── handler.ts # Telegram bot event handlers (from bot.ts)
│ └── s3/
│ ├── auth.ts # SigV4 verification (from utils/s3/auth.ts)
│ ├── headers.ts # CORS/response headers
│ ├── object-stream.ts # Part streaming for GetObject
│ ├── range.ts # HTTP Range header parser
│ ├── virtual-host.ts # Virtual-hosted bucket extraction
│ └── xml.ts # S3 XML response builders/parsers
├── config/
│ └── index.ts # Environment config (from env.ts)
├── shared/
│ ├── errors/
│ │ └── index.ts # Domain + application error classes
│ ├── logger/
│ │ └── index.ts # Winston logger (from utils/logger.ts)
│ ├── metrics/
│ │ └── index.ts # Metrics collector (from utils/metrics.ts)
│ └── utils/
│ ├── file.ts # File type, MIME, hash, extension (from utils/file.ts)
│ ├── ip.ts # Client IP extraction (from utils/ip.ts)
│ ├── retry.ts # Retry/timeout/fallback (from utils/retry.ts)
│ └── zip.ts # ZIP create/extract (from utils/zip.ts)
```
## 5. Dependency Injection Pattern
No DI framework. Use-case factories accept repository interfaces:
```typescript
// domain/ports/file-repository.ts
export interface IFileRepository {
findByHash(hash: string): Promise<File | null>;
findByPublicId(publicId: string): Promise<File | null>;
findByUniqueId(uniqueId: string): Promise<File | null>;
findByBucketAndKey(bucketId: string, key: string): Promise<File | null>;
create(file: NewFile): Promise<File>;
softDelete(bucketId: string, key: string): Promise<boolean>;
listByPrefix(...): Promise<{ objects: File[]; prefixes: string[] }>;
}
// application/use-cases/upload-file.ts
export function createUploadFileUseCase(repos: {
fileRepo: IFileRepository;
filePartRepo: IFilePartRepository;
telegramService: ITelegramService;
config: AppConfig;
}) {
return async (input: UploadInput): Promise<UploadOutput> => {
// business logic here, using repos.xxx() not db directly
};
}
```
This keeps use cases testable — inject mock repositories in tests.
## 6. Controller / Use-Case Boundary
**Controller responsibilities:**
- Parse HTTP request (body, params, headers)
- Call use case
- Format HTTP response
**Use case responsibilities:**
- Business logic (file size checks, dedup, chunking decisions)
- Orchestrate infrastructure calls (Telegram, DB, cache)
- Return DTOs
**Example flow for upload:**
```
UploadController.handleUpload(req)
→ parse multipart form data
→ write to temp file
→ call uploadFileUseCase({ tempPath, fileName, mimeType, ... })
→ check dedup (fileRepo.findByHash)
→ check file size limits
→ if chunked: telegramService.uploadChunks(...), filePartRepo.create(...)
→ if batch: telegramService.enqueueBatch(...)
→ fileRepo.create(...)
→ return UploadOutput
→ format response (buildUploadResponse)
→ return Response
```
## 7. Migration Strategy
The restructure will be done in one pass per layer, with tests updated in lockstep:
1. Create new folder structure
2. Move `shared/` utilities first (no dependencies on other layers)
3. Move `config/`
4. Move `domain/` entities + ports
5. Move `infrastructure/` persistence + telegram + cache
6. Move `application/` use cases + DTOs
7. Move `interfaces/` controllers + middleware + routes
8. Update `src/index.ts` entry point
9. Restructure test files matching new layout
10. Run all tests, fix import paths
11. Delete old folders
## 8. Entity Architecture
```
┌──────────────────────────────────────────────┐
│ Domain Layer │
│ Entities: File, Bucket, FilePart, │
│ MultipartUpload, MultipartPart │
│ Ports: IFileRepository, IBucketRepository, │
│ IFilePartRepository, │
│ IMultipartRepository │
├──────────────────────────────────────────────┤
│ Application Layer │
│ UploadFileUseCase │
│ ├── depends on IFileRepository │
│ ├── depends on IFilePartRepository │
│ └── depends on ITelegramService │
│ │
│ S3ObjectUseCase │
│ ├── depends on IFileRepository │
│ ├── depends on IBucketRepository │
│ ├── depends on IMultipartRepository │
│ └── depends on ITelegramService │
└──────────────────────────────────────────────┘
```
## 9. Telegram Service Interface
A new `ITelegramService` interface in `domain/ports/` to abstract Telegram operations:
```typescript
export interface ITelegramService {
forwardToStorage(fileChunk: unknown, fileName: string, fileType: string): Promise<ForwardResult>;
getFileInfo(telegramFileId: string): Promise<TelegramFileInfo>;
enqueueUpload<T>(task: () => Promise<T>): Promise<T>;
}
```
Implementation stays in `infrastructure/telegram/`.
## 10. JSDoc Standards
Every exported function, interface, type, and class gets JSDoc:
```typescript
/**
* Uploads a file to Telegram storage, handling chunking for large files
* and batch deduplication for small files.
*
* @param input - The prepared upload data and metadata
* @param input.tempPath - Path to the temporary file on disk
* @param input.fileName - Display name for the file
* @param input.mimeType - MIME type of the file
* @param input.fileType - Telegram file category (document, photo, etc.)
* @param input.sizeBytes - File size in bytes
* @returns The stored file entity with Telegram metadata
* @throws {FileTooLargeError} If file exceeds Telegram size limits
*/
```
## 11. Error Handling
Custom error classes replacing generic `Error` throws:
```typescript
// shared/errors/index.ts
export class DomainError extends Error { constructor(msg: string) { super(msg); this.name = 'DomainError'; } }
export class FileNotFoundError extends DomainError {}
export class BucketNotFoundError extends DomainError {}
export class FileTooLargeError extends DomainError {}
export class DuplicateFileError extends DomainError {}
export class AuthenticationError extends DomainError {}
export class ValidationError extends DomainError {}
```
Use cases throw domain errors. Controllers catch and format HTTP responses.
## 12. Existing Files That Remain Unchanged
Files that are purely structural (no business logic restructuring needed):
| Old Path | New Path |
|----------|----------|
| `src/home.html` | `src/interfaces/http/controllers/home.html` (or alongside home controller) |
| `src/utils/s3/*` | `src/interfaces/s3/*` (moved as-is) |
| `src/db/migrate.ts` | `src/infrastructure/persistence/drizzle/migrate.ts` |
## 13. State After Migration
- **Total source files:** ~60 (was ~30) — more focused files, each with clear responsibility
- **JSDoc coverage:** 100% of exported APIs
- **Test files:** ~24, restructured to mirror source layout
- **Behaviour changes:** Zero. No logic is altered, only moved
- **External interfaces:** All API endpoints, S3 XML formats, and response shapes unchanged
@@ -0,0 +1,127 @@
# Per-Bot Queue: Rate-Limit Safe Telegram Upload
**Date:** 2026-07-29
**Status:** Approved Design
## Problem
Telegram Bot API rate-limits each bot to approximately 1-2 concurrent uploads. When multiple upload chunks hit the same bot simultaneously, Telegram returns HTTP 429 (Too Many Requests), causing delays of 30-60 seconds per retry. Under Docker push load, these cumulative delays trigger Gitea client timeouts and `500 Internal Server Error`.
> Catatan (2026-08-02): Produksi sekarang port 4000, deploy Nix+systemd di orangevps, Caddy reverse proxy upload.asepharyana.my.id, DB via pgbouncer pool imrnes 100.121.180.82:6432. Docker/Traefik/Gitea-CI legacy.
The current architecture uses a **global PQueue** with `concurrency=N` where each task picks a bot via round-robin (`claimBotIndex()`). This means two concurrent tasks can both land on the same bot index (after wrap-around), causing 429 collisions.
## Solution: Per-Bot Queue
Each bot has its own PQueue with `concurrency=1`. Uploads are assigned to the bot with the fewest pending tasks. If a bot rate-limits, the upload moves to the next available bot.
### Architecture
```
┌────────────────────────────────────────────────────────┐
│ BotPool │
│ │
│ ┌──────────────────────────────────────────────────┐ │
│ │ BotEntry[0] token=b1 queue=PQueue(conc=1) │ │
│ │ rateLimitedUntil=0 │ │
│ ├──────────────────────────────────────────────────┤ │
│ │ BotEntry[1] token=b2 queue=PQueue(conc=1) │ │
│ │ rateLimitedUntil=0 │ │
│ ├──────────────────────────────────────────────────┤ │
│ │ ... up to N bots │ │
│ └──────────────────────────────────────────────────┘ │
│ │
│ selectBot(skip?): number │
│ └─ bot dengan pending queue paling sedikit │
│ dan tidak sedang rate-limited │
│ │
│ forwardToStorage(file): ForwardResult │
│ └─ retry loop: selectBot → queue.add → handle 429 │
└────────────────────────────────────────────────────────┘
```
### BotEntry Structure
```typescript
interface BotEntry {
index: number;
token: string;
instance: Telegraf;
queue: PQueue; // concurrency: 1
rateLimitedUntil: number; // epoch ms, 0 = not limited
}
```
### Data Flow: Upload
```
forwardToStorage(fileChunk, fileName, fileType)
├─ MAX_RETRIES loop (attempt up to all bots)
│ │
│ ├─ selectBot(attemptedIndexes)
│ │ ├─ Filter out rate-limited bots (rateLimitedUntil > Date.now())
│ │ ├─ Filter out already-attempted bots
│ │ ├─ If none available:
│ │ │ ├─ Wait MIN_SLEEP_MS (5000ms)
│ │ │ ├─ Reset rate-limited timers (clear attemptedIndexes)
│ │ │ └─ Retry selectBot
│ │ └─ Return bot with smallest queue.pending count
│ │
│ ├─ attemptedIndexes.add(selectedBot)
│ │
│ ├─ result = await bots[selectedBot].queue.add(() =>
│ │ executeTelegramCall(bot, fileChunk, fileName)
│ │ )
│ │ │
│ │ ├─ ✅ Success → return ForwardResult
│ │ │
│ │ └─ ❌ Error
│ │ ├─ 429 → markRateLimited(bot, retryAfter)
│ │ │ → continue to next bot in retry loop
│ │ ├─ Transient (timeout, 5xx) → continue
│ │ └─ Non-transient → throw (propagate up)
│ │
│ └─ Attempt counter exhausted → throw lastError
└─ Sorted part tracking (for chunked uploads)
```
### Key Design Decisions
1. **Concurrency=1 per bot**: Guarantees no two Telegram API calls compete for the same bot token. With 6 bots, effective concurrency = 6.
2. **Least-loaded assignment**: `selectBot()` picks the bot with the fewest queued + pending tasks. This naturally load-balances even when some bots are slower.
3. **Rate-limit isolation**: When bot A hits 429, only bot A's queue is paused. Other 5 bots continue serving uploads uninterrupted.
4. **Per-bot rate-limit timer**: `rateLimitedUntil` prevents re-selecting a recently-429'd bot until its cooldown expires.
5. **No global PQueue**: The old `upload-queue.ts` is removed. Each bot owns its queue, eliminating the global backpressure problem.
### Changes by File
| File | Action |
|------|--------|
| `src/infrastructure/telegram/bot-pool.ts` | **Major refactor**: BotEntry array, selectBot(), per-bot queues, retry loop |
| `src/infrastructure/telegram/upload-queue.ts` | **Delete**: No longer needed |
| `src/domain/ports/telegram-service.ts` | **Remove** `enqueueUpload<T>(task: () => Promise<T>): Promise<T>` from interface |
| `src/utils/chunked-storage.ts` | **No changes** — only uses `forwardToStorage()` |
| `src/env.ts` | **Remove** `uploadConcurrency` config (no longer needed) |
### Error Handling
- **429 per bot**: Mark bot rate-limited, move to next. Clear timer after `retryAfter` seconds.
- **All bots 429**: Sleep 5 seconds with jitter, then retry from bot 0.
- **Transient errors** (timeout, 5xx, connection reset): Retry on same bot (inside its queue), then on next bot.
- **Non-transient errors** (4xx other than 429, wrong file_id, auth errors): Propagate immediately.
- **MAX_RETRIES**: 10 attempts across all bots before giving up.
### Testing
- Unit: `selectBot()` returns bot with fewest pending tasks
- Unit: `selectBot()` skips rate-limited bots
- Unit: 429 on bot 0 → retries on bot 1 → succeeds
- Unit: All bots rate-limited → sleeps → retries → succeeds
- Unit: Per-bot queue has concurrency=1 (two tasks to same bot queue sequentially)
- Integration: Forward a real file through the per-bot pool
Generated
+61
View File
@@ -0,0 +1,61 @@
{
"nodes": {
"flake-utils": {
"inputs": {
"systems": "systems"
},
"locked": {
"lastModified": 1731533236,
"narHash": "sha256-l0KFg5HjrsfsO/JpG+r7fRrqm12kzFHyUHqHCVpMMbI=",
"owner": "numtide",
"repo": "flake-utils",
"rev": "11707dc2f618dd54ca8739b309ec4fc024de578b",
"type": "github"
},
"original": {
"owner": "numtide",
"repo": "flake-utils",
"type": "github"
}
},
"nixpkgs": {
"locked": {
"lastModified": 1785301185,
"narHash": "sha256-eoS3KQTO0aPWXZvIaRbRAzSSHW3l5wdMFXtT1ISfoKA=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "9bc02893134c733dd85de46ee4fb2fac696b5529",
"type": "github"
},
"original": {
"owner": "NixOS",
"ref": "nixpkgs-unstable",
"repo": "nixpkgs",
"type": "github"
}
},
"root": {
"inputs": {
"flake-utils": "flake-utils",
"nixpkgs": "nixpkgs"
}
},
"systems": {
"locked": {
"lastModified": 1681028828,
"narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
"owner": "nix-systems",
"repo": "default",
"rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
"type": "github"
},
"original": {
"owner": "nix-systems",
"repo": "default",
"type": "github"
}
}
},
"root": "root",
"version": 7
}
+85
View File
@@ -0,0 +1,85 @@
{
description = "TeleUploader Nix build";
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
flake-utils.url = "github:numtide/flake-utils";
};
outputs = { self, nixpkgs, flake-utils }:
flake-utils.lib.eachSystem [ "x86_64-linux" ] (system:
let
pkgs = import nixpkgs { inherit system; };
# TeleUploader package
teleuploader = pkgs.stdenvNoCC.mkDerivation rec {
pname = "teleuploader";
version = "1.1.0";
src = ./.;
nativeBuildInputs = [
pkgs.bun
pkgs.makeBinaryWrapper
];
# Bun cache di sandbox — prevent online fetch
# Karena bun.lock sudah di repo, bun install --frozen-lockfile
# akan pake cache, tapi di Nix sandbox gak ada internet.
# Solusi: offline flag
preBuild = ''
export HOME=$TMPDIR/home
mkdir -p $HOME
export BUN_INSTALL=$HOME/.bun
'';
buildPhase = ''
echo "=== Installing dependencies ==="
bun install --frozen-lockfile --ignore-scripts 2>&1
echo "=== Building ==="
bun run build 2>&1
'';
installPhase = ''
mkdir -p $out/bin $out/share/teleuploader
# Copy dist files
cp -r dist $out/share/teleuploader/dist
cp src/home.html $out/share/teleuploader/ 2>/dev/null || true
cp schema.sql $out/share/teleuploader/ 2>/dev/null || true
# Wrap with bun from Nix store (dependency sharing!)
# Note: NO --chdir — systemd WorkingDirectory controls this
makeBinaryWrapper ${pkgs.bun}/bin/bun $out/bin/teleuploader \
--add-flags "$out/share/teleuploader/dist/index.js" \
--set-default NODE_ENV production \
--prefix PATH : ${pkgs.lib.makeBinPath [ pkgs.bun ]}
# Also create the migrate wrapper
makeBinaryWrapper ${pkgs.bun}/bin/bun $out/bin/teleuploader-migrate \
--add-flags "$out/share/teleuploader/dist/migrate.js" \
--prefix PATH : ${pkgs.lib.makeBinPath [ pkgs.bun ]}
'';
meta = {
description = "Telegram file uploader backend (S3 Telegram)";
license = pkgs.lib.licenses.mit;
platforms = pkgs.lib.platforms.linux;
};
};
in {
packages = {
inherit teleuploader;
default = teleuploader;
};
# Dev shell with bun for local development
devShells.default = pkgs.mkShell {
buildInputs = [
pkgs.bun
pkgs.nodejs_22
];
};
});
}
+10 -8
View File
@@ -1,20 +1,21 @@
{ {
"name": "filedrop", "name": "filedrop",
"version": "1.0.0", "version": "1.1.0",
"description": "Telegram file uploader backend", "description": "Telegram file uploader backend",
"type": "module", "type": "module",
"scripts": { "scripts": {
"dev": "bun --hot src/index.ts", "dev": "bun --hot src/index.ts",
"build": "bun build src/index.ts --target=bun --outfile=dist/index.js && bun build src/db/migrate.ts --target=bun --outfile=dist/migrate.js", "build": "bun build src/index.ts --target=bun --outfile=dist/index.js && bun build src/infrastructure/persistence/drizzle/migrate.ts --target=bun --outfile=dist/migrate.js",
"start": "NODE_ENV=production bun dist/index.js", "start": "NODE_ENV=production bun dist/index.js",
"db:migrate": "bun dist/migrate.js", "db:migrate": "bun dist/migrate.js",
"test": "bun test test/rateLimit.test.ts && bun test test/file.test.ts && bun test test/telegram.test.ts && bun test test/upload.test.ts && bun test test/files.test.ts && bun test test/health.test.ts && bun test test/db.test.ts && bun test test/bot.test.ts && bun test test/bootstrap.test.ts && bun test test/swagger.test.ts && bun test test/auth.test.ts && bun test test/auth-routes.test.ts && bun test test/s3-auth.test.ts && bun test test/s3-operations.test.ts && bun test test/s3-bucket-config.test.ts && bun test test/web-api.test.ts", "test": "bun test --preload ./test/helpers/setup-env.ts test/rateLimit.test.ts && bun test --preload ./test/helpers/setup-env.ts test/file.test.ts && bun test --preload ./test/helpers/setup-env.ts test/telegram.test.ts && bun test --preload ./test/helpers/setup-env.ts test/upload.test.ts && bun test --preload ./test/helpers/setup-env.ts test/files.test.ts && bun test --preload ./test/helpers/setup-env.ts test/health.test.ts && bun test --preload ./test/helpers/setup-env.ts test/db.test.ts && bun test --preload ./test/helpers/setup-env.ts test/bot.test.ts && bun test --preload ./test/helpers/setup-env.ts test/bootstrap.test.ts && bun test --preload ./test/helpers/setup-env.ts test/swagger.test.ts && bun test --preload ./test/helpers/setup-env.ts test/auth.test.ts && bun test --preload ./test/helpers/setup-env.ts test/auth-routes.test.ts && bun test --preload ./test/helpers/setup-env.ts test/s3-auth.test.ts && bun test --preload ./test/helpers/setup-env.ts test/s3-operations.test.ts && bun test --preload ./test/helpers/setup-env.ts test/s3-bucket-config.test.ts && bun test --preload ./test/helpers/setup-env.ts test/web-api.test.ts && bun test --preload ./test/helpers/setup-env.ts test/env.test.ts && bun test --preload ./test/helpers/setup-env.ts test/bot-pool.test.ts",
"test:s3-auth": "bun test test/s3-auth.test.ts", "test:s3-auth": "bun test --preload ./test/helpers/setup-env.ts test/s3-auth.test.ts",
"test:s3-ops": "bun test test/s3-operations.test.ts", "test:s3-ops": "bun test --preload ./test/helpers/setup-env.ts test/s3-operations.test.ts",
"test:web-api": "bun test test/web-api.test.ts", "test:web-api": "bun test --preload ./test/helpers/setup-env.ts test/web-api.test.ts",
"test:s3": "bun test test/s3-auth.test.ts && bun test test/s3-operations.test.ts && bun test test/web-api.test.ts", "test:s3": "bun test --preload ./test/helpers/setup-env.ts test/s3-auth.test.ts && bun test --preload ./test/helpers/setup-env.ts test/s3-operations.test.ts && bun test --preload ./test/helpers/setup-env.ts test/web-api.test.ts",
"lint": "bunx biome check src test", "lint": "bunx biome check src test",
"format": "bunx biome format --write src test" "format": "bunx biome format --write src test",
"prepare": "husky"
}, },
"dependencies": { "dependencies": {
"@prisma/engines": "^7.8.0", "@prisma/engines": "^7.8.0",
@@ -31,6 +32,7 @@
"@biomejs/biome": "^2.4.15", "@biomejs/biome": "^2.4.15",
"@types/node": "^25.8.0", "@types/node": "^25.8.0",
"drizzle-kit": "^0.31.10", "drizzle-kit": "^0.31.10",
"husky": "^9.1.7",
"typescript": "^6.0.3" "typescript": "^6.0.3"
} }
} }
+2 -1
View File
@@ -56,7 +56,8 @@ CREATE TABLE IF NOT EXISTS multipart_uploads (
s3_key TEXT NOT NULL, s3_key TEXT NOT NULL,
initiated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, initiated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
status VARCHAR DEFAULT 'in_progress', status VARCHAR DEFAULT 'in_progress',
initiated_by TEXT initiated_by TEXT,
content_type TEXT
); );
CREATE TABLE IF NOT EXISTS multipart_parts ( CREATE TABLE IF NOT EXISTS multipart_parts (
+46
View File
@@ -0,0 +1,46 @@
/**
* Input for the login endpoint.
* The caller provides the admin API token to obtain a session cookie.
*/
export interface LoginInput {
/** Admin API token for authentication */
token: string;
}
/**
* Active authentication session information.
*/
export interface AuthSession {
/** Authenticated username (currently always "admin") */
username: string;
/** Session expiry timestamp; null for bearer-token sessions */
expiresAt: Date | null;
/** Authentication method used */
method: 'cookie' | 'bearer';
}
/**
* Response payload for a successful login.
*/
export interface LoginResponse {
/** Authenticated username */
username: string;
}
/**
* Response payload for logout.
*/
export interface LogoutResponse {
/** Whether the logout succeeded */
success: boolean;
}
/**
* Response payload for the current-user (/me) endpoint.
*/
export interface UserInfoResponse {
/** Authenticated username */
username: string;
/** ISO-8601 session expiry timestamp; null when using bearer token */
expiresAt: string | null;
}
+47
View File
@@ -0,0 +1,47 @@
/**
* Input for creating a new bucket.
*/
export interface CreateBucketInput {
/** Bucket name (must match S3 naming rules: 3-63 chars, lowercase, no underscore) */
name: string;
}
/**
* Single bucket representation returned by bucket endpoints.
*/
export interface BucketResponse {
/** Bucket UUID */
id: string;
/** Bucket name */
name: string;
/** ISO-8601 timestamp of when the bucket was created */
createdAt: string;
/** Number of non-deleted objects in the bucket */
objectCount?: number;
}
/**
* Response payload for the list-buckets endpoint.
*/
export interface BucketListResponse {
/** Array of buckets */
buckets: BucketResponse[];
}
/**
* Response payload for bucket creation.
*/
export interface CreateBucketResponse {
/** Bucket UUID */
id: string;
/** Bucket name */
name: string;
}
/**
* Response payload for bucket deletion.
*/
export interface DeleteBucketResponse {
/** Whether the deletion succeeded */
success: boolean;
}
+68
View File
@@ -0,0 +1,68 @@
/**
* Public file information response returned by the file-info endpoint.
* Mirrors the JSON shape of GET /file/:publicId/info.
*/
export interface FileInfoResponse {
/** Public, shareable identifier (nanoid) */
public_id: string;
/** Stored file name */
file_name: string;
/** MIME type of the stored file */
mime_type: string;
/** File size in bytes */
size_bytes: number;
/** High-level file category (e.g. "document", "photo") */
file_type: string;
/** ISO-8601 timestamp of when the file record was created */
created_at: string;
}
/**
* Summary-level file metadata used internally for constructing
* upload responses and object listing entries.
*/
export interface FileMetadata {
/** Public, shareable identifier (nanoid) */
publicId: string;
/** Telegram file identifier used to retrieve the file from Telegram CDN */
telegramFileId: string;
/** Telegram unique file identifier (persists across reuploads) */
telegramFileUniqueId: string;
/** Chat ID where the file or archive was stored */
storageChatId: number;
/** Message ID of the stored file or archive */
storageMessageId: number;
/** Stored file name */
fileName: string;
/** MIME type of the stored file */
mimeType: string;
/** File size in bytes */
sizeBytes: number;
/** High-level file category */
fileType: string;
/** Telegram user ID of the uploader; 0 when unknown or system */
uploaderId: number;
/** Timestamp of file record creation */
createdAt: Date | string | number;
}
/**
* Upload response shape returned to API callers.
* Mirrors the JSON output of the /api/upload endpoint.
*/
export interface UploadResponse {
/** Public, shareable identifier */
public_id: string;
/** Stored file name */
file_name: string;
/** MIME type */
mime_type: string;
/** File size in bytes */
size_bytes: number;
/** High-level file category */
file_type: string;
/** ISO-8601 creation timestamp */
created_at: string;
/** Public download URL */
download_url: string;
}
+87
View File
@@ -0,0 +1,87 @@
/**
* A single S3 object as it appears in listing results.
*/
export interface S3ObjectResponse {
/** The object key (full path within the bucket) */
key: string;
/** Stored file name (basename of the key) */
fileName: string;
/** MIME type of the stored object */
mimeType: string;
/** Object size in bytes */
sizeBytes: number;
/** High-level file category */
fileType: string;
/** SHA-256 hex digest of the object content */
etag: string | null;
/** ISO-8601 timestamp of last modification */
lastModified: string;
/** Public download URL */
downloadUrl: string;
}
/**
* Response payload for S3 ListObjectsV1 / ListObjectsV2.
*/
export interface S3ListObjectsResponse {
/** Array of object summaries */
objects: S3ObjectResponse[];
/** Common prefixes when a delimiter was used (e.g. "folder/" entries) */
prefixes: string[];
/** Whether more results are available */
isTruncated: boolean;
/** Token to pass as continuation-token to retrieve the next page */
nextContinuationToken: string | null;
}
/**
* Input for the copy-object operation (Web API v1).
*/
export interface S3CopyObjectInput {
/** Source object key within the same or source bucket */
sourceKey: string;
/** Destination bucket name; defaults to the source bucket when omitted */
destBucket?: string;
/** Destination object key */
destKey: string;
}
/**
* Response payload for the copy-object operation.
*/
export interface S3CopyObjectResponse {
/** Source object key that was copied */
sourceKey: string;
/** Destination object key */
destKey: string;
/** Destination bucket name */
destBucket: string;
}
/**
* Summary of a multipart upload in listing results.
*/
export interface S3MultipartUploadResponse {
/** The object key being uploaded */
key: string;
/** Upload identifier (nanoid) */
uploadId: string;
/** ISO-8601 timestamp when the upload was initiated */
initiatedAt: Date;
/** Identifier string of the upload initiator */
initiatedBy: string;
}
/**
* Summary of a single part within a multipart upload.
*/
export interface S3MultipartPartResponse {
/** 1-indexed part number */
partNumber: number;
/** ETag of the part content */
etag: string;
/** Part size in bytes */
sizeBytes: number;
/** ISO-8601 timestamp when the part was stored */
createdAt: Date;
}
+46
View File
@@ -0,0 +1,46 @@
/**
* Input for the upload file use case.
* Carries all metadata needed to persist an uploaded file,
* including its temporary location on disk and optional bucket/S3 context.a
*/
export interface UploadInput {
/** Absolute path to the temporary file on disk */
tempPath: string;
/** SHA-256 hex digest of the file content */
fileHash: string;
/** Original file name (may include extension) */
fileName: string;
/** MIME type detected from content inspection or request header */
mimeType: string;
/** High-level file category (e.g. "document", "photo", "video") */
fileType: string;
/** File size in bytes */
sizeBytes: number;
/** Telegram user ID of the uploader; 0 when unknown or system */
uploaderId?: number;
/** Target bucket UUID for S3-compatible storage; null when un-bucketed */
bucketId?: string | null;
/** Object key within the bucket for S3-compatible storage; null when un-bucketed */
s3Key?: string | null;
}
/**
* Output from the upload file use case.
* Contains the public-facing file metadata returned to the caller.
*/
export interface UploadOutput {
/** Public, shareable identifier (nanoid) */
publicId: string;
/** Stored file name (may have been normalized with extension) */
fileName: string;
/** MIME type of the stored file */
mimeType: string;
/** File size in bytes */
sizeBytes: number;
/** High-level file category */
fileType: string;
/** ISO-8601 timestamp of when the file record was created */
createdAt: Date;
/** Public download URL */
downloadUrl: string;
}
+114
View File
@@ -0,0 +1,114 @@
import { timingSafeEqual } from 'node:crypto';
import type {
AuthSession,
LoginInput,
LoginResponse,
LogoutResponse,
UserInfoResponse,
} from '../dto/auth';
/** Subset of application configuration consumed by the authenticate use case. */
export interface AuthUseCaseConfig {
/** Admin API token used to authenticate login requests. */
adminApiToken: string;
/** Name of the session cookie. */
sessionCookieName: string;
/** Session lifetime in milliseconds. */
sessionMaxAgeMs: number;
}
/** Dependencies required by the authenticate use case factory. */
export interface AuthenticateUseCaseDeps {
/** Application configuration subset. */
config: AuthUseCaseConfig;
}
/**
* Performs a constant-time string comparison to prevent timing attacks.
*
* @param left - The first string to compare.
* @param right - The second string to compare.
* @returns `true` if the strings are equal, `false` otherwise.
*/
const timingSafeCompare = (left: string, right: string): boolean => {
const leftBuffer = Buffer.from(left);
const rightBuffer = Buffer.from(right);
if (leftBuffer.length !== rightBuffer.length) {
return false;
}
return timingSafeEqual(leftBuffer, rightBuffer);
};
/**
* Checks whether authentication is enabled based on the configured token.
*
* @param adminApiToken - The admin API token value.
* @returns `true` if the token is non-empty (auth is enabled).
*/
const isAuthEnabled = (adminApiToken: string): boolean => adminApiToken.length > 0;
/**
* Creates a factory function for the login use case.
*
* Validates the provided admin API token and returns session metadata on
* success. The caller (controller/adapter) is responsible for translating
* the result into an HTTP response (e.g. setting a session cookie).
*
* @param deps - The injected dependencies.
* @returns An async function accepting login input and returning a login response.
*/
export function createLoginUseCase(deps: AuthenticateUseCaseDeps) {
return async (input: LoginInput): Promise<LoginResponse> => {
if (!isAuthEnabled(deps.config.adminApiToken)) {
return { username: 'admin' };
}
if (!timingSafeCompare(input.token, deps.config.adminApiToken)) {
throw new Error('Invalid token');
}
return { username: 'admin' };
};
}
/**
* Creates a factory function for the logout use case.
*
* Always succeeds the caller is responsible for clearing the session cookie.
*
* @returns An async function returning a logout response.
*/
export function createLogoutUseCase() {
return async (): Promise<LogoutResponse> => {
return { success: true };
};
}
/**
* Creates a factory function for the current-user (me) use case.
*
* Accepts an already-parsed auth session (from cookie or bearer token) and
* returns the user info response. The caller (controller/adapter) is
* responsible for extracting the session from the raw HTTP request.
*
* @param deps - The injected dependencies.
* @returns An async function accepting an optional session and returning user info.
*/
export function createMeUseCase(deps: AuthenticateUseCaseDeps) {
return async (session: AuthSession | null): Promise<UserInfoResponse | null> => {
if (!isAuthEnabled(deps.config.adminApiToken)) {
return null;
}
if (!session) {
return null;
}
return {
username: session.username,
expiresAt: session.expiresAt?.toISOString() ?? null,
};
};
}
+176
View File
@@ -0,0 +1,176 @@
import type { File } from '../../domain/entities/file';
import type { IFileRepository } from '../../domain/ports/file-repository';
import type { ITelegramService, TelegramFileInfo } from '../../domain/ports/telegram-service';
/**
* Result type for a simple file-info lookup.
*/
export interface FileInfoResult {
/** Whether the file was found. */
found: true;
/** Public unique identifier. */
publicId: string;
/** Original file name. */
fileName: string;
/** MIME type. */
mimeType: string;
/** File size in bytes. */
sizeBytes: number;
/** Telegram file type (document, photo, video, etc.). */
fileType: string;
/** ISO-8601 creation timestamp. */
createdAt: string;
}
/**
* Result type for a file-not-found lookup.
*/
export interface FileNotFoundResult {
/** Always `false` for a not-found result. */
found: false;
}
/**
* Discriminated union of all possible file-info lookup outcomes.
*/
export type GetFileInfoResult = FileInfoResult | FileNotFoundResult;
/**
* Describes a redirect-based file retrieval.
*/
export interface RedirectRetrieval {
/** Discriminant. */
type: 'redirect';
/** The resolved file entity. */
file: File;
/** Full Telegram CDN URL to redirect the client to. */
redirectUrl: string;
/** Cached Telegram file metadata. */
fileInfo: TelegramFileInfo;
}
/**
* Describes a chunked file retrieval that needs a multi-part response.
*/
export interface ChunkedRetrieval {
/** Discriminant. */
type: 'chunked';
/** The resolved file entity. */
file: File;
}
/**
* Describes an archive-entry file retrieval.
*/
export interface ArchiveEntryRetrieval {
/** Discriminant. */
type: 'archive-entry';
/** The resolved file entity. */
file: File;
/** Telegram file metadata for the archive container. */
archiveInfo: TelegramFileInfo;
/** Name of the entry within the archive. */
entryName: string;
}
/**
* Discriminated union of all possible file retrieval outcomes.
*/
export type FileRetrievalResult = RedirectRetrieval | ChunkedRetrieval | ArchiveEntryRetrieval;
/** Subset of application configuration consumed by the get-file use case. */
export interface GetFileConfig {
/** Server base URL (used in constructing archive download URLs). */
baseUrl: string;
}
/** Dependencies required by the get-file use case factory. */
export interface GetFileUseCaseDeps {
/** File repository for looking up file records. */
fileRepo: IFileRepository;
/** Telegram service for resolving file identifiers to download paths. */
telegramService: ITelegramService;
/** Application configuration subset. */
config: GetFileConfig;
}
/**
* Creates a factory function for the get-file info use case.
*
* Looks up a file by its public identifier and returns its metadata.
*
* @param deps - The injected dependencies.
* @returns An async function accepting a public ID and returning file info.
*/
export function createGetFileInfoUseCase(deps: Pick<GetFileUseCaseDeps, 'fileRepo'>) {
return async (publicId: string): Promise<GetFileInfoResult> => {
const file = await deps.fileRepo.findByPublicId(publicId);
if (!file) {
return { found: false };
}
return {
found: true,
publicId: file.publicId,
fileName: file.fileName,
mimeType: file.mimeType,
sizeBytes: file.sizeBytes,
fileType: file.fileType,
createdAt: formatCreatedAtForInfo(file.createdAt),
};
};
}
/**
* Formats a date-like value into an ISO-8601 string.
*
* @param date - A Date instance, date string, or numeric timestamp.
* @returns The ISO-8601 string.
*/
const formatCreatedAtForInfo = (date: Date | string | number): string => {
if (date instanceof Date) return date.toISOString();
return new Date(date).toISOString();
};
/**
* Creates a factory function for the get-file retrieval use case.
*
* Determines how a file should be delivered to the client:
* - **redirect**: For regular (non-chunked, non-archive) files returns a
* Telegram CDN redirect URL.
* - **chunked**: For files stored across multiple Telegram parts returns
* the file entity so the caller can build a multi-part streaming response.
* - **archive-entry**: For files stored inside a Telegram archive (zip)
* returns the archive's Telegram metadata and the entry name so the caller
* can extract and stream the entry.
*
* @param deps - The injected dependencies.
* @returns An async function accepting a public ID and returning a retrieval result.
*/
export function createGetFileUseCase(deps: GetFileUseCaseDeps) {
return async (publicId: string): Promise<FileRetrievalResult | null> => {
const file = await deps.fileRepo.findByPublicId(publicId);
if (!file) {
return null;
}
// Chunked file — return the entity for multi-part response building
if (file.storageBackend === 'chunked') {
return { type: 'chunked', file };
}
// Archive entry — resolve the archive's Telegram location
const archiveEntryName = file.archiveEntryName;
if (archiveEntryName) {
const archiveFileId = file.archiveTelegramFileId || file.telegramFileId;
const archiveInfo = await deps.telegramService.getFileInfo(archiveFileId);
return { type: 'archive-entry', file, archiveInfo, entryName: archiveEntryName };
}
// Regular file — resolve Telegram CDN path for a redirect
const fileInfo = await deps.telegramService.getFileInfo(file.telegramFileId);
const redirectUrl = `https://api.telegram.org/file/bot${fileInfo.bot_token}/${fileInfo.file_path}`;
return { type: 'redirect', file, redirectUrl, fileInfo };
};
}
+156
View File
@@ -0,0 +1,156 @@
import type { Bucket } from '../../domain/entities/bucket';
import type { IBucketRepository } from '../../domain/ports/bucket-repository';
import type { IFileRepository } from '../../domain/ports/file-repository';
/**
* S3 bucket name validation regex.
*
* Bucket names must be 3-63 characters, start/end with a lowercase letter or
* digit, and contain only lowercase letters, digits, dots, and hyphens.
*/
const BUCKET_NAME_REGEX = /^[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$/;
/**
* Error type for bucket-level application errors that carry an S3-compatible
* error code and an HTTP status suggestion.
*/
export class BucketError extends Error {
/** S3-compatible error code (e.g. "NoSuchBucket", "BucketAlreadyExists"). */
readonly code: string;
/** Suggested HTTP status code. */
readonly status: number;
/**
* @param code - The S3 error code.
* @param message - Human-readable error description.
* @param status - Suggested HTTP status.
*/
constructor(code: string, message: string, status: number) {
super(message);
this.name = 'BucketError';
this.code = code;
this.status = status;
}
}
/**
* Describes a bucket entry returned by the list-buckets use case,
* enriched with the current count of non-deleted objects.
*/
export interface BucketWithCount {
/** The bucket domain entity. */
bucket: Bucket;
/** Number of non-deleted objects in the bucket. */
objectCount: number;
}
/** Dependencies required by the manage-bucket use case factories. */
export interface ManageBucketDeps {
/** Bucket repository for CRUD operations. */
bucketRepo: IBucketRepository;
/** File repository for counting and checking objects within buckets. */
fileRepo: IFileRepository;
}
/**
* Creates a use case that lists all buckets together with their object counts.
*
* @param deps - The injected dependencies.
* @returns An async function that returns a list of buckets with counts.
*/
export function createListBucketsUseCase(deps: ManageBucketDeps) {
return async (): Promise<BucketWithCount[]> => {
const buckets = await deps.bucketRepo.list();
const results = await Promise.all(
buckets.map(async (bucket) => ({
bucket,
objectCount: await deps.fileRepo.countByBucket(bucket.id),
})),
);
return results;
};
}
/**
* Creates a use case that retrieves a single bucket by its name.
*
* @param deps - The injected dependencies.
* @returns An async function accepting a bucket name and returning the
* bucket, or `null` when not found.
*/
export function createGetBucketUseCase(deps: ManageBucketDeps) {
return async (name: string): Promise<Bucket | null> => {
return deps.bucketRepo.findByName(name);
};
}
/**
* Creates a use case that creates a new bucket.
*
* Validates the bucket name format and checks for duplicates before
* persisting.
*
* @param deps - The injected dependencies.
* @returns An async function accepting a bucket name and returning the
* newly created bucket.
* @throws {BucketError} When the name is invalid or the bucket already exists.
*/
export function createCreateBucketUseCase(deps: ManageBucketDeps) {
return async (name: string): Promise<Bucket> => {
if (!BUCKET_NAME_REGEX.test(name)) {
throw new BucketError('InvalidBucketName', 'The specified bucket is not valid.', 400);
}
const existing = await deps.bucketRepo.findByName(name);
if (existing) {
throw new BucketError(
'BucketAlreadyExists',
'The requested bucket name is not available.',
409,
);
}
return deps.bucketRepo.create(name);
};
}
/**
* Creates a use case that deletes a bucket.
*
* Ensures the bucket exists and is empty (no non-deleted objects) before
* proceeding with deletion.
*
* @param deps - The injected dependencies.
* @returns An async function accepting a bucket name. Returns `true` when
* the bucket was deleted, throws when the bucket is missing or
* not empty.
* @throws {BucketError} When the bucket does not exist or is not empty.
*/
export function createDeleteBucketUseCase(deps: ManageBucketDeps) {
return async (name: string): Promise<boolean> => {
const bucket = await deps.bucketRepo.findByName(name);
if (!bucket) {
throw new BucketError('NoSuchBucket', 'The specified bucket does not exist.', 404);
}
const objectCount = await deps.fileRepo.countByBucket(bucket.id);
if (objectCount > 0) {
throw new BucketError('BucketNotEmpty', 'The bucket you tried to delete is not empty.', 409);
}
return deps.bucketRepo.delete(name);
};
}
/**
* Creates a use case that checks whether a bucket exists.
*
* @param deps - The injected dependencies.
* @returns An async function accepting a bucket name and returning `true`
* when the bucket exists.
*/
export function createBucketExistsUseCase(deps: ManageBucketDeps) {
return async (name: string): Promise<boolean> => {
return deps.bucketRepo.exists(name);
};
}
@@ -0,0 +1,368 @@
import { nanoid } from 'nanoid';
import { buildNewFile } from '../../domain/entities/file-factory';
import type { MultipartUpload } from '../../domain/entities/multipart';
import type { IBucketRepository } from '../../domain/ports/bucket-repository';
import type { IFileRepository } from '../../domain/ports/file-repository';
import type { IMultipartRepository } from '../../domain/ports/multipart-repository';
import type { ITelegramService } from '../../domain/ports/telegram-service';
import { computeHash, DEFAULT_FILE_TYPE } from '../../shared/utils/file';
// ─── Types ──────────────────────────────────────────────────────────
/**
* A single part reference as submitted in a complete-multipart-upload request.
*/
export interface CompletePartInput {
/** 1-based part number. */
partNumber: number;
/** ETag returned when the part was uploaded. */
etag: string;
}
/**
* Result of initiating a multipart upload.
*/
export interface InitiateMultipartResult {
/** The generated upload identifier (nanoid). */
uploadId: string;
/** The bucket name. */
bucket: string;
/** The S3 object key. */
key: string;
}
/**
* Result of uploading a single part.
*/
export interface UploadPartResult {
/** ETag of the uploaded part (SHA-256 hex digest). */
etag: string;
}
/**
* Result of completing a multipart upload.
*/
export interface CompleteMultipartResult {
/** Public-facing unique identifier of the created file record. */
publicId: string;
/** The S3 location URL of the completed object. */
location: string;
/** Combined ETag (all part etags joined by hyphens). */
etag: string;
/** Total object size in bytes. */
sizeBytes: number;
}
/**
* Summary of a single part within a multipart upload used in listing results.
*/
export interface PartSummary {
/** 1-based part number. */
partNumber: number;
/** ETag of the part content. */
etag: string;
/** Part size in bytes. */
sizeBytes: number;
/** ISO-8601 timestamp when the part was stored. */
createdAt: Date;
}
/**
* Result of listing multipart uploads within a bucket.
*/
export interface ListMultipartUploadsResult {
/** Array of in-progress upload summaries. */
uploads: MultipartUpload[];
/** Whether more results are available. */
isTruncated: boolean;
/** Marker for the next page, or null when not truncated. */
nextKeyMarker: string | null;
}
// ─── Config ─────────────────────────────────────────────────────────
/** Subset of application configuration consumed by the multipart use cases. */
export interface MultipartConfig {
/** Maximum chunk size in bytes for Telegram uploads (part size limit). */
telegramChunkSizeBytes: number;
/** Telegram chat ID where part data is stored. */
storageChatId: number;
/** Server base URL for constructing location URLs. */
baseUrl: string;
}
/** Dependencies required by the multipart upload use case factories. */
export interface MultipartDeps {
/** Bucket repository for bucket lookups. */
bucketRepo: IBucketRepository;
/** File repository for creating the final file record on completion. */
fileRepo: IFileRepository;
/** Multipart repository for managing upload sessions and parts. */
multipartRepo: IMultipartRepository;
/** Telegram service for forwarding part data to storage. */
telegramService: ITelegramService;
/** Application configuration subset. */
config: MultipartConfig;
}
// ─── Use Case Factories ─────────────────────────────────────────────
/**
* Creates a use case that initiates an S3 multipart upload.
*
* Validates the bucket exists and creates a new multipart upload session.
*
* @param deps - The injected dependencies.
* @returns An async function accepting bucket name and object key, returning
* the upload initiation result, or `null` when the bucket is not found.
*/
export function createInitiateMultipartUploadUseCase(deps: MultipartDeps) {
return async (bucketName: string, key: string): Promise<InitiateMultipartResult | null> => {
const bucket = await deps.bucketRepo.findByName(bucketName);
if (!bucket) return null;
const uploadId = await deps.multipartRepo.create(bucket.id, key, 's3');
return { uploadId, bucket: bucketName, key };
};
}
/**
* Creates a use case that uploads a single part of a multipart upload.
*
* Validates the part number range (1-10000), checks the upload session exists
* and matches the expected key, checks part size against the configured limit,
* forwards the part data to Telegram storage, and persists the part record.
*
* @param deps - The injected dependencies.
* @returns An async function accepting upload details and part data, returning
* the part ETag, or `null` when the upload session is not found.
*/
export function createUploadPartUseCase(deps: MultipartDeps) {
return async (input: {
/** Bucket name for the multipart upload. */
bucketName: string;
/** S3 object key for the multipart upload. */
key: string;
/** Upload identifier returned by initiate. */
uploadId: string;
/** 1-based part number (1-10000). */
partNumber: number;
/** Raw part data. */
body: Buffer;
}): Promise<UploadPartResult | null> => {
if (input.partNumber < 1 || input.partNumber > 10000) {
throw new MultipartError(
'InvalidArgument',
'Part number must be an integer between 1 and 10000',
400,
);
}
const multipart = await deps.multipartRepo.findById(input.uploadId);
if (!multipart || multipart.s3Key !== input.key) {
return null;
}
if (input.body.byteLength > deps.config.telegramChunkSizeBytes) {
throw new MultipartError(
'EntityTooLarge',
`Your proposed upload part size (${input.body.byteLength} bytes) exceeds the maximum allowed part size (${deps.config.telegramChunkSizeBytes} bytes) for this storage backend. Use smaller part sizes.`,
400,
);
}
const forwardResult = await deps.telegramService.forwardToStorage(
input.body,
`mp-${input.uploadId}-part-${input.partNumber}`,
'document',
);
const etag = computeHash(input.body);
await deps.multipartRepo.insertPart({
uploadId: input.uploadId,
partNumber: input.partNumber,
telegramFileId: forwardResult.telegramFileId,
telegramFileUniqueId: forwardResult.telegramFileUniqueId,
storageMessageId: forwardResult.storageMessageId,
sizeBytes: input.body.byteLength,
etag,
});
return { etag: `"${etag}"` };
};
}
/**
* Error type for multipart-level application errors.
*/
export class MultipartError extends Error {
/** S3-compatible error code. */
readonly code: string;
/** Suggested HTTP status code. */
readonly status: number;
/**
* @param code - The S3 error code.
* @param message - Human-readable error description.
* @param status - Suggested HTTP status.
*/
constructor(code: string, message: string, status: number) {
super(message);
this.name = 'MultipartError';
this.code = code;
this.status = status;
}
}
/**
* Creates a use case that completes an S3 multipart upload.
*
* Validates the submitted part list (all parts must be present and in ascending
* order), creates the final file record referencing the first part's Telegram
* data, marks the upload session as completed, and returns the combined result.
*
* @param deps - The injected dependencies.
* @returns An async function accepting upload details and submitted parts,
* returning the completion result, or `null` when the upload session
* is not found.
*/
export function createCompleteMultipartUploadUseCase(deps: MultipartDeps) {
return async (input: {
/** Bucket name for the multipart upload. */
bucketName: string;
/** S3 object key for the multipart upload. */
key: string;
/** Upload identifier. */
uploadId: string;
/** Parts submitted by the client (in ascending part number order). */
parts: CompletePartInput[];
}): Promise<CompleteMultipartResult | null> => {
const multipart = await deps.multipartRepo.findById(input.uploadId);
if (!multipart) return null;
const storedParts = await deps.multipartRepo.listParts(input.uploadId);
// Validate ascending part order
const partNumbers = input.parts.map((p) => p.partNumber);
if (partNumbers.length > 1 && partNumbers.some((n, i) => i > 0 && n <= partNumbers[i - 1])) {
throw new MultipartError(
'InvalidPartOrder',
'The list of parts was not in ascending order.',
400,
);
}
// Validate part count matches
if (input.parts.length !== storedParts.length) {
throw new MultipartError(
'InvalidPart',
'One or more specified parts could not be found.',
400,
);
}
const totalSize = storedParts.reduce((sum, p) => sum + p.sizeBytes, 0);
const firstPart = storedParts[0];
if (!firstPart) {
throw new MultipartError('InternalError', 'Multipart object has no parts.', 500);
}
const publicId = nanoid();
await deps.fileRepo.create(
buildNewFile({
publicId,
telegramFileId: firstPart.telegramFileId,
telegramFileUniqueId: firstPart.telegramFileUniqueId,
storageChatId: deps.config.storageChatId,
storageMessageId: firstPart.storageMessageId,
fileName: input.key.split('/').pop() || 'file',
mimeType: 'application/octet-stream',
sizeBytes: totalSize,
fileType: DEFAULT_FILE_TYPE,
storageBackend: 'telegram',
bucketId: multipart.bucketId,
s3Key: input.key,
multipartUploadId: input.uploadId,
}),
);
await deps.multipartRepo.complete(input.uploadId);
const location = `${deps.config.baseUrl}/${input.bucketName}/${input.key}`;
const combinedEtag = storedParts.map((p) => p.etag).join('-');
return {
publicId,
location,
etag: combinedEtag,
sizeBytes: totalSize,
};
};
}
/**
* Creates a use case that aborts an S3 multipart upload.
*
* @param deps - The injected dependencies.
* @returns An async function accepting an upload identifier, returning `true`
* when the upload was aborted, or `null` when the upload session
* is not found.
*/
export function createAbortMultipartUploadUseCase(deps: MultipartDeps) {
return async (uploadId: string): Promise<boolean | null> => {
const multipart = await deps.multipartRepo.findById(uploadId);
if (!multipart) return null;
await deps.multipartRepo.abort(uploadId);
return true;
};
}
/**
* Creates a use case that lists in-progress multipart uploads within a bucket.
*
* @param deps - The injected dependencies.
* @returns An async function accepting query parameters and returning the
* listing result, or `null` when the bucket is not found.
*/
export function createListMultipartUploadsUseCase(deps: MultipartDeps) {
return async (input: {
/** Bucket name to list uploads from. */
bucketName: string;
/** Maximum number of uploads to return (clamped 1-1000). */
maxUploads: number;
/** Return only uploads whose S3 key is strictly greater than this, or null. */
keyMarker: string | null;
}): Promise<ListMultipartUploadsResult | null> => {
const bucket = await deps.bucketRepo.findByName(input.bucketName);
if (!bucket) return null;
return deps.multipartRepo.listByBucket(bucket.id, input.maxUploads, input.keyMarker);
};
}
/**
* Creates a use case that lists parts of a specific multipart upload.
*
* @param deps - The injected dependencies.
* @returns An async function accepting an upload identifier and returning the
* list of parts, or `null` when the upload session is not found.
*/
export function createListPartsUseCase(deps: MultipartDeps) {
return async (uploadId: string): Promise<PartSummary[] | null> => {
const multipart = await deps.multipartRepo.findById(uploadId);
if (!multipart) return null;
const parts = await deps.multipartRepo.listParts(uploadId);
return parts.map((p) => ({
partNumber: p.partNumber,
etag: p.etag,
sizeBytes: p.sizeBytes,
createdAt: p.createdAt,
}));
};
}
+165
View File
@@ -0,0 +1,165 @@
import type { File } from '../../domain/entities/file';
import type { IBucketRepository } from '../../domain/ports/bucket-repository';
import type { IFilePartRepository } from '../../domain/ports/file-part-repository';
import type { IFileRepository } from '../../domain/ports/file-repository';
import type { IMultipartRepository } from '../../domain/ports/multipart-repository';
import type { ITelegramService, TelegramFileInfo } from '../../domain/ports/telegram-service';
import type { CompressionAlgorithm } from '../../shared/utils/compress';
// ─── Types ──────────────────────────────────────────────────────────
/**
* A single part source for building a multi-part streaming response.
* Each part corresponds to a Telegram-stored file chunk.
*/
export interface ObjectPartSource {
/** Telegram file identifier for retrieving this part. */
telegramFileId: string;
/** Telegram CDN URL for downloading this part. */
telegramUrl: string;
/** Original size of this part in bytes. */
sizeBytes: number;
/** 1-based part number within the object. */
partNumber: number;
/** Stored (post-compression) size in bytes, when applicable. */
storedSizeBytes?: number;
/** Compression algorithm applied, or null if uncompressed. */
compressionAlgorithm?: CompressionAlgorithm;
}
/**
* A regular (direct) S3 object resolved to a Telegram CDN URL.
*/
export interface DirectObjectResult {
/** Discriminant. */
type: 'direct';
/** The resolved file entity. */
file: File;
/** Full Telegram CDN URL for downloading the object. */
telegramUrl: string;
/** Telegram file metadata. */
fileInfo: TelegramFileInfo;
}
/**
* A chunked S3 object stored across multiple Telegram file parts.
*/
export interface ChunkedObjectResult {
/** Discriminant. */
type: 'chunked';
/** The resolved file entity. */
file: File;
}
/**
* An S3 object assembled from a completed multipart upload.
*/
export interface MultipartObjectResult {
/** Discriminant. */
type: 'multipart';
/** The resolved file entity. */
file: File;
/** Resolved part sources with Telegram CDN URLs. */
parts: ObjectPartSource[];
}
/**
* Discriminated union of all possible S3 get-object outcomes.
*/
export type GetObjectResult = DirectObjectResult | ChunkedObjectResult | MultipartObjectResult;
/**
* Result of an S3 put-object operation.
*/
export interface PutObjectResult {
/** SHA-256 hex digest of the object content. */
etag: string;
}
/**
* Result of an S3 copy-object operation.
*/
export interface CopyObjectResult {
/** SHA-256 hex digest of the source object content. */
etag: string;
/** ISO-8601 timestamp of the copy operation. */
lastModified: string;
}
/**
* A single S3 object as returned in listing results.
*/
export interface ListObjectEntry {
/** The object key (full path within the bucket). */
key: string;
/** Object size in bytes. */
sizeBytes: number;
/** SHA-256 hex digest or fallback identifier. */
etag: string;
/** ISO-8601 timestamp of last modification. */
lastModified: string;
/** MIME type of the stored object. */
mimeType: string;
}
/**
* Result of an S3 list-objects operation (both V1 and V2).
*/
export interface ListObjectsResult {
/** Array of object summaries. */
objects: ListObjectEntry[];
/** Common prefixes when a delimiter was used. */
prefixes: string[];
/** Whether more results are available. */
isTruncated: boolean;
/** The last key in the returned page, for use as the next marker. */
nextMarker: string | null;
}
/**
* Head-object metadata.
*/
export interface HeadObjectMetadata {
/** MIME type of the object. */
contentType: string;
/** Object size in bytes. */
contentLength: number;
/** Entity tag (SHA-256 hex digest or fallback). */
etag: string;
/** ISO-8601 timestamp of last modification. */
lastModified: string;
}
// ─── Config ─────────────────────────────────────────────────────────
/** Subset of application configuration consumed by the s3-object use cases. */
export interface S3ObjectConfig {
/** Maximum chunk size in bytes for Telegram chunked uploads. */
telegramChunkSizeBytes: number;
/** Whether gzip compression is enabled for chunked uploads. */
compressChunkedUploads: boolean;
/** Minimum chunk size in bytes below which compression is skipped. */
chunkCompressionMinSizeBytes: number;
/** Telegram chat ID where file parts are stored. */
storageChatId: number;
/** Server base URL for constructing download links. */
baseUrl: string;
/** Whether to proxy S3 GET requests through the server. */
proxyS3Get: boolean;
}
/** Dependencies required by the s3-object use case factories. */
export interface S3ObjectDeps {
/** Bucket repository for bucket lookups. */
bucketRepo: IBucketRepository;
/** File repository for object CRUD operations. */
fileRepo: IFileRepository;
/** File-part repository for chunked upload part records. */
filePartRepo: IFilePartRepository;
/** Multipart repository for resolving multipart-upload objects. */
multipartRepo: IMultipartRepository;
/** Telegram service for uploading and resolving file metadata. */
telegramService: ITelegramService;
/** Application configuration subset. */
config: S3ObjectConfig;
}
+165
View File
@@ -0,0 +1,165 @@
import { createReadStream } from 'node:fs';
import { open } from 'node:fs/promises';
import { nanoid } from 'nanoid';
import { buildNewFile } from '../../domain/entities/file-factory';
import type { IFileRepository } from '../../domain/ports/file-repository';
import type { ITelegramService } from '../../domain/ports/telegram-service';
import type { ChunkedStorage } from '../../infrastructure/telegram/chunked-storage';
import { checkFileSize, ensureExtension, getFileType } from '../../shared/utils/file';
import type { UploadInput, UploadOutput } from '../dto/upload';
/** Subset of application configuration consumed by the upload-file use case. */
export interface UploadFileConfig {
/** Server base URL for constructing download links. */
baseUrl: string;
/** Maximum chunk size in bytes for Telegram chunked uploads. */
telegramChunkSizeBytes: number;
/** Telegram chat ID where file parts are stored. */
storageChatId: number;
/** Whether to attempt gzip compression on each chunk. */
compressChunkedUploads: boolean;
/** Minimum chunk size in bytes below which compression is skipped. */
chunkCompressionMinSizeBytes: number;
}
/** Dependencies required by the upload-file use case factory. */
export interface UploadFileUseCaseDeps {
/** File repository for CRUD operations on file records. */
fileRepo: IFileRepository;
/** Telegram service for forwarding file content to storage. */
telegramService: ITelegramService;
/** Chunked storage handler for large file uploads. */
chunkedStorage: ChunkedStorage;
/** Application configuration subset. */
config: UploadFileConfig;
}
/**
* Reads the first 16 bytes from a file on disk for magic-byte detection.
*
* @param tempPath - Absolute path to the temporary file.
* @returns A buffer containing up to 16 bytes.
*/
const readSignatureBuffer = async (tempPath: string): Promise<Buffer> => {
const handle = await open(tempPath, 'r');
try {
const buf = Buffer.alloc(16);
const { bytesRead } = await handle.read(buf, 0, 16, 0);
return buf.subarray(0, bytesRead);
} finally {
await handle.close();
}
};
/**
* Creates a factory function for the upload-file use case.
*
* The returned use case:
* 1. Checks for an existing file with the same SHA-256 hash (deduplication).
* 2. Normalises the file name and MIME type based on magic bytes.
* 3. Validates the file size against Telegram type-specific limits.
* 4. Chooses a storage strategy chunked (delegated to ChunkedStorage) or
* single-message upload.
* 5. Persists the file record.
* 6. Builds and returns the public `UploadOutput` DTO.
*
* @param deps - The injected dependencies.
* @returns An async function accepting `UploadInput` and returning `UploadOutput`.
*/
export function createUploadFileUseCase(deps: UploadFileUseCaseDeps) {
return async (input: UploadInput): Promise<UploadOutput> => {
// 1. Check deduplication by content hash
const existing = await deps.fileRepo.findByHash(input.fileHash);
if (existing) {
return {
publicId: existing.publicId,
fileName: existing.fileName,
mimeType: existing.mimeType,
sizeBytes: existing.sizeBytes,
fileType: existing.fileType,
createdAt:
existing.createdAt instanceof Date ? existing.createdAt : new Date(existing.createdAt),
downloadUrl: `${deps.config.baseUrl}/f/${existing.publicId}`,
};
}
// 2. Read signature bytes for magic-byte-based extension detection
const signatureBuffer = await readSignatureBuffer(input.tempPath);
const { fileName: finalFileName, mimeType } = ensureExtension(
input.fileName,
signatureBuffer,
input.mimeType,
);
// 3. Determine Telegram file type and validate size
const fileType = getFileType(mimeType, finalFileName);
if (!checkFileSize(input.sizeBytes, fileType)) {
throw new Error(`File size exceeds ${fileType} limit`);
}
// 4. Upload — chunked via ChunkedStorage for files above the threshold
if (input.sizeBytes > deps.config.telegramChunkSizeBytes) {
const uploadedFile = await deps.chunkedStorage.storeFileInTelegramChunks({
tempPath: input.tempPath,
partFileNamePrefix: `direct-${input.fileHash.slice(0, 16)}`,
fileName: finalFileName,
mimeType,
sizeBytes: input.sizeBytes,
fileType,
uploaderId: input.uploaderId ?? 0,
bucketId: input.bucketId,
s3Key: input.s3Key,
});
return {
publicId: uploadedFile.publicId,
fileName: uploadedFile.fileName,
mimeType: uploadedFile.mimeType,
sizeBytes: uploadedFile.sizeBytes,
fileType: uploadedFile.fileType,
createdAt: uploadedFile.createdAt,
downloadUrl: `${deps.config.baseUrl}/f/${uploadedFile.publicId}`,
};
}
// 5. Single-message upload path
const forwardResult = await deps.telegramService.forwardToStorage(
createReadStream(input.tempPath),
finalFileName,
fileType,
);
const singlePublicId = nanoid();
const createdFile = await deps.fileRepo.create(
buildNewFile({
publicId: singlePublicId,
telegramFileId: forwardResult.telegramFileId,
telegramFileUniqueId: forwardResult.telegramFileUniqueId,
storageChatId: deps.config.storageChatId,
storageMessageId: forwardResult.storageMessageId,
fileName: finalFileName,
mimeType,
sizeBytes: input.sizeBytes,
fileType,
storageBackend: 'telegram',
uploaderId: input.uploaderId,
fileHash: input.fileHash,
bucketId: input.bucketId,
s3Key: input.s3Key,
}),
);
return {
publicId: createdFile.publicId,
fileName: createdFile.fileName,
mimeType: createdFile.mimeType,
sizeBytes: createdFile.sizeBytes,
fileType: createdFile.fileType,
createdAt: createdFile.createdAt,
downloadUrl: `${deps.config.baseUrl}/f/${createdFile.publicId}`,
};
};
}
-137
View File
@@ -1,137 +0,0 @@
import { nanoid } from 'nanoid';
import { type Context, Telegraf } from 'telegraf';
import { db, files as fileSchema } from './db';
import { findFileByUniqueId } from './db/files';
import { config } from './env';
import {
detectFileType,
extractFileFromMessage,
getErrorMessage,
getFileSizeLimit,
type TelegramMediaMessage,
} from './utils/file';
import logger from './utils/logger';
import { forwardToStorage } from './utils/telegram';
type BotContext = {
message: TelegramMediaMessage;
from: { id: number };
chat?: { id: number };
reply: (text: string, extra?: { reply_parameters: { message_id: number } }) => Promise<unknown>;
};
type MediaEventRegistrar = {
on: (events: string[], handler: (ctx: BotContext) => Promise<unknown>) => void;
};
const replyWithDownloadUrl = async (ctx: BotContext, publicId: string): Promise<void> => {
const url = `${config.baseUrl}/f/${publicId}`;
await ctx.reply(`File berhasil diupload! 📎\n\nDownload: ${url}`, {
reply_parameters: { message_id: ctx.message.message_id },
});
};
export const startBot = async (): Promise<Telegraf<Context>> => {
try {
const bot = new Telegraf(config.botToken);
bot.command('start', async (ctx) => {
await ctx.reply(
`👋 Halo! Kirimkan file (document, photo, video, audio, voice, animation) ke bot ini. ` +
`File akan disimpan di private channel dan kamu dapat download link permanen.`,
);
});
// Logging middleware must be registered BEFORE the media handler so all events are captured
bot.use((ctx, next) => {
logger.info('Telegram event received', {
type: 'type' in ctx.update ? ctx.update.type : undefined,
chat_id: ctx.chat?.id,
});
return next();
});
const mediaBot = bot as unknown as MediaEventRegistrar;
mediaBot.on(
['document', 'photo', 'video', 'audio', 'voice', 'animation', 'sticker', 'video_note'],
async (ctx) => {
try {
const fileType = detectFileType(ctx.message);
const fileObj = extractFileFromMessage(ctx.message, fileType);
const { file_id, mime_type } = fileObj;
const fileSize = fileObj.file_size || 0;
const fileName =
ctx.message.document?.file_name ||
ctx.message.photo?.slice(-1)[0]?.file_name ||
ctx.message.video?.file_name ||
ctx.message.audio?.file_name ||
ctx.message.voice?.file_name ||
'file';
const maxSize = getFileSizeLimit(fileType);
if (fileSize > maxSize) {
return ctx.reply(`File size exceeds ${maxSize / (1024 * 1024)}MB limit`);
}
const existing = await findFileByUniqueId(fileObj.file_unique_id);
if (existing) {
await replyWithDownloadUrl(ctx, existing.publicId);
logger.info('Duplicate file detected in bot, returned existing link', {
publicId: existing.publicId,
fileType,
fileName,
uploader: ctx.from.id,
});
return;
}
const result = await forwardToStorage(file_id, fileName, fileType);
const publicId = nanoid();
const uploaded = {
publicId: publicId,
telegramFileId: result.telegramFileId,
telegramFileUniqueId: result.telegramFileUniqueId,
storageChatId: config.storageChatId,
storageMessageId: result.storageMessageId,
fileName: fileName,
mimeType: mime_type || 'application/octet-stream',
sizeBytes: fileSize,
fileType: fileType,
uploaderId: ctx.from.id,
createdAt: new Date(),
updatedAt: new Date(),
};
await db.insert(fileSchema).values(uploaded);
await replyWithDownloadUrl(ctx, publicId);
logger.info('File uploaded via bot', {
publicId,
fileType,
fileName,
uploader: ctx.from.id,
});
} catch (error: unknown) {
logger.error('Bot file handler error', {
error: getErrorMessage(error),
chat_id: ctx.chat?.id,
});
await ctx.reply('❌ Gagal mengupload file. Coba lagi nanti.');
}
},
);
await bot.launch();
logger.info('Telegram bot started', { botToken: `${config.botToken?.substring(0, 10)}...` });
return bot;
} catch (error: unknown) {
logger.error('Failed to start bot', { error: getErrorMessage(error) });
throw error;
}
};
-81
View File
@@ -1,81 +0,0 @@
import { sql } from 'drizzle-orm';
import { db } from './index';
export interface Bucket {
id: string;
name: string;
createdAt: Date;
updatedAt: Date;
}
type QueryRow = Record<string, unknown>;
type QueryResult = QueryRow[];
export const createBucket = async (name: string): Promise<Bucket> => {
const result = (await db.execute(
sql`INSERT INTO buckets (name) VALUES (${name}) RETURNING id, name, created_at, updated_at`,
)) as unknown as QueryResult;
const row = result[0]!;
return {
id: row.id as string,
name: row.name as string,
createdAt: new Date(row.created_at as string),
updatedAt: new Date(row.updated_at as string),
};
};
export const findBucketByName = async (name: string): Promise<Bucket | null> => {
const result = (await db.execute(
sql`SELECT id, name, created_at, updated_at FROM buckets WHERE name = ${name}`,
)) as unknown as QueryResult;
if (result.length === 0) return null;
const row = result[0]!;
return {
id: row.id as string,
name: row.name as string,
createdAt: new Date(row.created_at as string),
updatedAt: new Date(row.updated_at as string),
};
};
export const listBuckets = async (): Promise<Bucket[]> => {
const result = (await db.execute(
sql`SELECT id, name, created_at, updated_at FROM buckets ORDER BY name`,
)) as unknown as QueryResult;
return result.map((row) => ({
id: row.id as string,
name: row.name as string,
createdAt: new Date(row.created_at as string),
updatedAt: new Date(row.updated_at as string),
}));
};
export const deleteBucket = async (name: string): Promise<boolean> => {
// Cascade-delete rows that hold FK references to the bucket
await db
.execute(
sql`DELETE FROM multipart_parts WHERE upload_id IN (SELECT upload_id FROM multipart_uploads WHERE bucket_id IN (SELECT id FROM buckets WHERE name = ${name}))`,
)
.catch(() => {});
await db
.execute(
sql`DELETE FROM multipart_uploads WHERE bucket_id IN (SELECT id FROM buckets WHERE name = ${name})`,
)
.catch(() => {});
await db
.execute(
sql`DELETE FROM files WHERE bucket_id IN (SELECT id FROM buckets WHERE name = ${name})`,
)
.catch(() => {});
const result = (await db.execute(
sql`DELETE FROM buckets WHERE name = ${name}`,
)) as unknown as QueryResult;
return result.length > 0;
};
export const bucketExists = async (name: string): Promise<boolean> => {
const result = (await db.execute(
sql`SELECT 1 FROM buckets WHERE name = ${name}`,
)) as unknown as QueryResult;
return result.length > 0;
};
-97
View File
@@ -1,97 +0,0 @@
import { sql } from 'drizzle-orm';
import { db } from './index';
export type CompressionAlgorithm = 'gzip' | null;
export interface FilePart {
id: number;
fileId: string;
partNumber: number;
telegramFileId: string;
telegramFileUniqueId: string;
storageChatId: number;
storageMessageId: number;
sizeBytes: number;
storedSizeBytes: number;
compressionAlgorithm: CompressionAlgorithm;
etag: string;
createdAt: Date;
}
export type NewFilePartInput = Omit<FilePart, 'id' | 'createdAt'>;
const toNumber = (value: unknown): number => Number(value ?? 0);
const mapRowToFilePart = (row: Record<string, unknown>): FilePart => ({
id: toNumber(row.id),
fileId: row.file_id as string,
partNumber: toNumber(row.part_number),
telegramFileId: row.telegram_file_id as string,
telegramFileUniqueId: row.telegram_file_unique_id as string,
storageChatId: toNumber(row.storage_chat_id),
storageMessageId: toNumber(row.storage_message_id),
sizeBytes: toNumber(row.size_bytes),
storedSizeBytes: toNumber(row.stored_size_bytes),
compressionAlgorithm: (row.compression_algorithm as CompressionAlgorithm) || null,
etag: row.etag as string,
createdAt: new Date(row.created_at as string),
});
export const insertFileParts = async (parts: NewFilePartInput[]): Promise<void> => {
for (const part of parts) {
await db.execute(
sql`INSERT INTO file_parts (
file_id,
part_number,
telegram_file_id,
telegram_file_unique_id,
storage_chat_id,
storage_message_id,
size_bytes,
stored_size_bytes,
compression_algorithm,
etag
) VALUES (
${part.fileId}::uuid,
${part.partNumber},
${part.telegramFileId},
${part.telegramFileUniqueId},
${part.storageChatId},
${part.storageMessageId},
${part.sizeBytes},
${part.storedSizeBytes},
${part.compressionAlgorithm},
${part.etag}
)`,
);
}
};
export const listFileParts = async (fileId: string): Promise<FilePart[]> => {
const result = (await db.execute(
sql`SELECT id,
file_id,
part_number,
telegram_file_id,
telegram_file_unique_id,
storage_chat_id,
storage_message_id,
size_bytes,
stored_size_bytes,
compression_algorithm,
etag,
created_at
FROM file_parts
WHERE file_id = ${fileId}::uuid
ORDER BY part_number`,
)) as unknown as Record<string, unknown>[];
return result.map(mapRowToFilePart);
};
export const countFileParts = async (fileId: string): Promise<number> => {
const result = (await db.execute(
sql`SELECT COUNT(*) AS count FROM file_parts WHERE file_id = ${fileId}::uuid`,
)) as unknown as Record<string, unknown>[];
return toNumber(result[0]?.count);
};
-143
View File
@@ -1,143 +0,0 @@
import { and, eq, sql } from 'drizzle-orm';
import { db, files as fileSchema } from './index';
import type { File } from './schema';
export interface S3FileRecord extends File {
bucketId: string;
s3Key: string;
}
export const findFileByBucketAndKey = async (
bucketId: string,
s3Key: string,
): Promise<File | null> => {
const result = await db
.select()
.from(fileSchema)
.where(
and(
eq(fileSchema.bucketId, bucketId),
eq(fileSchema.s3Key, s3Key),
eq(fileSchema.isDeleted, false),
),
)
.limit(1);
return result[0] || null;
};
const mapDbRowToS3Record = (row: Record<string, unknown>): S3FileRecord => {
return {
id: row.id as string,
publicId: row.public_id as string,
telegramFileId: row.telegram_file_id as string,
telegramFileUniqueId: row.telegram_file_unique_id as string,
storageChatId: toNumber(row.storage_chat_id),
storageMessageId: toNumber(row.storage_message_id),
fileName: row.file_name as string,
mimeType: row.mime_type as string,
sizeBytes: toNumber(row.size_bytes),
fileType: row.file_type as string,
uploaderId: toNumber(row.uploader_id),
fileHash: row.file_hash as string | null,
archiveTelegramFileId: row.archive_telegram_file_id as string | null,
archiveStorageMessageId:
row.archive_storage_message_id === null ? null : toNumber(row.archive_storage_message_id),
archiveFileName: row.archive_file_name as string | null,
archiveEntryName: row.archive_entry_name as string | null,
archiveMimeType: row.archive_mime_type as string | null,
archiveSizeBytes: row.archive_size_bytes === null ? null : toNumber(row.archive_size_bytes),
bucketId: row.bucket_id as string,
s3Key: row.s3_key as string,
storageBackend: (row.storage_backend as string) || 'telegram',
isDeleted: row.is_deleted as boolean,
multipartUploadId: row.multipart_upload_id as string | null,
partCount:
row.part_count === null || row.part_count === undefined ? null : toNumber(row.part_count),
createdAt: new Date(row.created_at as string),
updatedAt: new Date(row.updated_at as string),
};
};
const escapeLike = (s: string): string => s.replace(/[%_\\]/g, '\\$&');
const toNumber = (value: unknown): number => Number(value ?? 0);
export const listObjectsByPrefix = async (
bucketId: string,
prefix: string,
delimiter: string | null,
maxKeys: number,
startAfter: string | null,
): Promise<{ objects: S3FileRecord[]; prefixes: string[] }> => {
let query = prefix
? sql`SELECT * FROM files WHERE bucket_id = ${bucketId}::uuid AND is_deleted = false AND s3_key LIKE ${`${escapeLike(prefix)}%`}`
: sql`SELECT * FROM files WHERE bucket_id = ${bucketId}::uuid AND is_deleted = false`;
if (startAfter) {
query = sql`${query} AND s3_key > ${startAfter}`;
}
query = sql`${query} ORDER BY s3_key LIMIT ${maxKeys + 1}`;
const rawResult = (await db.execute(query)) as unknown as Record<string, unknown>[];
if (delimiter === '/') {
const prefixSet = new Set<string>();
const objects: S3FileRecord[] = [];
for (const row of rawResult) {
const s3Key = row.s3_key as string;
const relativeKey = s3Key.substring(prefix.length);
const slashIndex = relativeKey.indexOf('/');
if (slashIndex >= 0) {
const folderPrefix = prefix + relativeKey.substring(0, slashIndex + 1);
if (folderPrefix !== prefix) {
prefixSet.add(folderPrefix);
}
} else {
objects.push(mapDbRowToS3Record(row));
}
}
return {
objects: objects.slice(0, maxKeys),
prefixes: Array.from(prefixSet).sort(),
};
}
return {
objects: rawResult.slice(0, maxKeys).map(mapDbRowToS3Record),
prefixes: [],
};
};
export const softDeleteFile = async (bucketId: string, s3Key: string): Promise<boolean> => {
const result = (await db.execute(
sql`UPDATE files SET is_deleted = true WHERE bucket_id = ${bucketId}::uuid AND s3_key = ${s3Key} RETURNING id`,
)) as unknown as Record<string, unknown>[];
return result.length > 0;
};
export const softDeleteFilesBatch = async (bucketId: string, keys: string[]): Promise<number> => {
let deleted = 0;
for (const key of keys) {
const ok = await softDeleteFile(bucketId, key);
if (ok) deleted++;
}
return deleted;
};
export const countBucketObjects = async (bucketId: string): Promise<number> => {
const result = (await db.execute(
sql`SELECT count(*) as count FROM files WHERE bucket_id = ${bucketId}::uuid AND is_deleted = false`,
)) as unknown as Record<string, unknown>[];
return Number(result[0]?.count || 0);
};
export const findOrphanFilesByBucket = async (bucketId: string): Promise<File[]> => {
return await db
.select()
.from(fileSchema)
.where(and(eq(fileSchema.bucketId, bucketId), eq(fileSchema.isDeleted, true)))
.limit(100);
};
-26
View File
@@ -1,26 +0,0 @@
import { eq } from 'drizzle-orm';
import { db, files as fileSchema } from './index';
import type { File } from './schema';
export const findFileByHash = async (hash: string): Promise<File | null> => {
const result = await db.select().from(fileSchema).where(eq(fileSchema.fileHash, hash)).limit(1);
return result[0] || null;
};
export const findFileByPublicId = async (publicId: string): Promise<File | null> => {
const result = await db
.select()
.from(fileSchema)
.where(eq(fileSchema.publicId, publicId))
.limit(1);
return result[0] || null;
};
export const findFileByUniqueId = async (telegramFileUniqueId: string): Promise<File | null> => {
const result = await db
.select()
.from(fileSchema)
.where(eq(fileSchema.telegramFileUniqueId, telegramFileUniqueId))
.limit(1);
return result[0] || null;
};
-129
View File
@@ -1,129 +0,0 @@
import { sql } from 'drizzle-orm';
import { nanoid } from 'nanoid';
import { db } from './index';
export interface MultipartUpload {
uploadId: string;
bucketId: string;
s3Key: string;
initiatedAt: Date;
status: string;
initiatedBy: string;
}
export interface MultipartPart {
id: number;
uploadId: string;
partNumber: number;
telegramFileId: string;
telegramFileUniqueId: string;
storageMessageId: number;
sizeBytes: number;
etag: string;
createdAt: Date;
}
export const createMultipartUpload = async (
bucketId: string,
s3Key: string,
initiatedBy: string,
): Promise<string> => {
const uploadId = nanoid(32);
await db.execute(
sql`INSERT INTO multipart_uploads (upload_id, bucket_id, s3_key, initiated_by) VALUES (${uploadId}, ${bucketId}, ${s3Key}, ${initiatedBy})`,
);
return uploadId;
};
export const findMultipartUpload = async (uploadId: string): Promise<MultipartUpload | null> => {
const result = (await db.execute(
sql`SELECT upload_id, bucket_id, s3_key, initiated_at, status FROM multipart_uploads WHERE upload_id = ${uploadId} AND status = 'in_progress'`,
)) as unknown as Record<string, unknown>[];
if (result.length === 0) return null;
const r = result[0]!;
return {
uploadId: r.upload_id as string,
bucketId: r.bucket_id as string,
s3Key: r.s3_key as string,
initiatedAt: new Date(r.initiated_at as string),
status: r.status as string,
initiatedBy: '',
};
};
export const completeMultipartUpload = async (uploadId: string): Promise<void> => {
await db.execute(
sql`UPDATE multipart_uploads SET status = 'completed' WHERE upload_id = ${uploadId}`,
);
};
export const abortMultipartUpload = async (uploadId: string): Promise<void> => {
await db.execute(
sql`UPDATE multipart_uploads SET status = 'aborted' WHERE upload_id = ${uploadId}`,
);
// Parts are cascade-deleted by FK
};
export const insertMultipartPart = async (
part: Omit<MultipartPart, 'id' | 'createdAt'>,
): Promise<void> => {
await db.execute(
sql`INSERT INTO multipart_parts (upload_id, part_number, telegram_file_id, telegram_file_unique_id, storage_message_id, size_bytes, etag)
VALUES (${part.uploadId}, ${part.partNumber}, ${part.telegramFileId}, ${part.telegramFileUniqueId}, ${part.storageMessageId}, ${part.sizeBytes}, ${part.etag})`,
);
};
export const listMultipartParts = async (uploadId: string): Promise<MultipartPart[]> => {
const result = (await db.execute(
sql`SELECT id, upload_id, part_number, telegram_file_id, telegram_file_unique_id, storage_message_id, size_bytes, etag, created_at
FROM multipart_parts WHERE upload_id = ${uploadId} ORDER BY part_number`,
)) as unknown as Record<string, unknown>[];
return result.map((r) => ({
id: r.id as number,
uploadId: r.upload_id as string,
partNumber: r.part_number as number,
telegramFileId: r.telegram_file_id as string,
telegramFileUniqueId: r.telegram_file_unique_id as string,
storageMessageId: r.storage_message_id as number,
sizeBytes: Number(r.size_bytes),
etag: r.etag as string,
createdAt: new Date(r.created_at as string),
}));
};
const mapRowToMultipartUpload = (r: Record<string, unknown>): MultipartUpload => ({
uploadId: r.upload_id as string,
bucketId: r.bucket_id as string,
s3Key: r.s3_key as string,
initiatedAt: new Date(r.initiated_at as string),
status: r.status as string,
initiatedBy: (r.initiated_by as string | null) || '',
});
export const listMultipartUploadsByBucket = async (
bucketId: string,
maxUploads: number,
keyMarker: string | null,
): Promise<{ uploads: MultipartUpload[]; isTruncated: boolean; nextKeyMarker: string | null }> => {
const limit = Math.min(Math.max(maxUploads || 1000, 1), 1000);
const result = (await db.execute(
keyMarker
? sql`SELECT upload_id, bucket_id, s3_key, initiated_at, status, initiated_by
FROM multipart_uploads
WHERE bucket_id = ${bucketId}::uuid AND status = 'in_progress' AND s3_key > ${keyMarker}
ORDER BY s3_key, initiated_at
LIMIT ${limit + 1}`
: sql`SELECT upload_id, bucket_id, s3_key, initiated_at, status, initiated_by
FROM multipart_uploads
WHERE bucket_id = ${bucketId}::uuid AND status = 'in_progress'
ORDER BY s3_key, initiated_at
LIMIT ${limit + 1}`,
)) as unknown as Record<string, unknown>[];
const uploads = result.slice(0, limit).map(mapRowToMultipartUpload);
return {
uploads,
isTruncated: result.length > limit,
nextKeyMarker: result.length > limit ? uploads.at(-1)?.s3Key || null : null,
};
};
+14
View File
@@ -0,0 +1,14 @@
/**
* Core domain entity representing an S3-compatible storage bucket.
* Buckets group objects for the S3-compatible API layer.
*/
export interface Bucket {
/** Primary key, UUID */
id: string;
/** Bucket name (unique, max 63 chars, S3 naming convention) */
name: string;
/** Record creation timestamp */
createdAt: Date;
/** Record last-updated timestamp */
updatedAt: Date;
}
+85
View File
@@ -0,0 +1,85 @@
/**
* Factory function for building NewFile records with sensible defaults.
*
* Most call sites set the same null defaults for archive/S3/soft-delete fields.
* This factory eliminates ~20 lines of boilerplate per call site (~27 sites).
*/
import type { NewFile } from './file';
/**
* Partial input for creating a file record.
* Only the required unique fields must be provided; optional fields default to null/0/false.
*/
export interface FileInput {
publicId: string;
telegramFileId: string;
telegramFileUniqueId: string;
storageChatId: number;
storageMessageId: number;
fileName: string;
mimeType: string;
sizeBytes: number;
fileType: string;
storageBackend: string | null;
/** Optional overrides */
uploaderId?: number;
fileHash?: string | null;
bucketId?: string | null;
s3Key?: string | null;
partCount?: number | null;
multipartUploadId?: string | null;
/** Archive fields (for batch/zip archives) */
archiveTelegramFileId?: string | null;
archiveStorageMessageId?: number | null;
archiveFileName?: string | null;
archiveEntryName?: string | null;
archiveMimeType?: string | null;
archiveSizeBytes?: number | null;
}
/**
* Build a NewFile record, filling in null/zero defaults for omitted fields.
*
* @example
* ```ts
* await fileRepo.create(buildNewFile({
* publicId,
* telegramFileId: result.telegramFileId,
* telegramFileUniqueId: result.telegramFileUniqueId,
* storageChatId,
* storageMessageId: result.storageMessageId,
* fileName: input.fileName,
* mimeType,
* sizeBytes: input.sizeBytes,
* fileType,
* storageBackend: 'telegram',
* uploaderId: input.uploaderId,
* fileHash: input.fileHash,
* }));
* ```
*/
export const buildNewFile = (input: FileInput): NewFile => ({
publicId: input.publicId,
telegramFileId: input.telegramFileId,
telegramFileUniqueId: input.telegramFileUniqueId,
storageChatId: input.storageChatId,
storageMessageId: input.storageMessageId,
fileName: input.fileName,
mimeType: input.mimeType,
sizeBytes: input.sizeBytes,
fileType: input.fileType,
uploaderId: input.uploaderId ?? 0,
fileHash: input.fileHash ?? null,
archiveTelegramFileId: input.archiveTelegramFileId ?? null,
archiveStorageMessageId: input.archiveStorageMessageId ?? null,
archiveFileName: input.archiveFileName ?? null,
archiveEntryName: input.archiveEntryName ?? null,
archiveMimeType: input.archiveMimeType ?? null,
archiveSizeBytes: input.archiveSizeBytes ?? null,
bucketId: input.bucketId ?? null,
s3Key: input.s3Key ?? null,
storageBackend: input.storageBackend,
isDeleted: false,
multipartUploadId: input.multipartUploadId ?? null,
partCount: input.partCount ?? null,
});
+43
View File
@@ -0,0 +1,43 @@
/**
* Supported compression algorithms for stored file parts.
* - `"gzip"`: Gzip compression was applied
* - `null`: No compression applied
*/
export type CompressionAlgorithm = 'gzip' | null;
/**
* Core domain entity representing a chunk (part) of a file stored in Telegram.
* Large files are split into multiple parts for Telegram-safe storage.
*/
export interface FilePart {
/** Primary key, auto-increment */
id: number;
/** Foreign key to the parent File record (UUID) */
fileId: string;
/** Sequential part number (1-based within the file) */
partNumber: number;
/** Telegram file_id for retrieving this part */
telegramFileId: string;
/** Telegram unique file_id (stable across bot tokens) */
telegramFileUniqueId: string;
/** Chat ID where this part is stored */
storageChatId: number;
/** Message ID within the storage chat */
storageMessageId: number;
/** Original size of this part in bytes */
sizeBytes: number;
/** Stored (post-compression) size in bytes */
storedSizeBytes: number;
/** Compression algorithm applied, or null if uncompressed */
compressionAlgorithm: CompressionAlgorithm;
/** ETag for this part (hash of the stored content) */
etag: string;
/** Record creation timestamp */
createdAt: Date;
}
/**
* Input type for creating a new FilePart record.
* Omits auto-generated fields (id, createdAt).
*/
export type NewFilePart = Omit<FilePart, 'id' | 'createdAt'>;
+64
View File
@@ -0,0 +1,64 @@
/**
* Core domain entity representing a file stored in Telegram.
* Contains both Telegram metadata and optional S3-compatible fields.
*/
export interface File {
/** Primary key, UUID */
id: string;
/** Public-facing unique identifier (short, URL-safe) */
publicId: string;
/** Telegram file_id for retrieving the file */
telegramFileId: string;
/** Telegram unique file_id (stable across bot tokens) */
telegramFileUniqueId: string;
/** Chat ID where the file is stored */
storageChatId: number;
/** Message ID within the storage chat */
storageMessageId: number;
/** Original file name */
fileName: string;
/** MIME type of the file */
mimeType: string;
/** File size in bytes */
sizeBytes: number;
/** File type classification (e.g. "photo", "document", "video") */
fileType: string;
/** Telegram user ID of the uploader */
uploaderId: number;
/** SHA-256 hash of file contents, or null */
fileHash: string | null;
/** Telegram file_id of the archive (zip) containing this file, or null */
archiveTelegramFileId: string | null;
/** Message ID of the archive message, or null */
archiveStorageMessageId: number | null;
/** File name within the archive, or null */
archiveFileName: string | null;
/** Entry name/path within the archive, or null */
archiveEntryName: string | null;
/** MIME type of the archive entry, or null */
archiveMimeType: string | null;
/** Size of the archive entry in bytes, or null */
archiveSizeBytes: number | null;
/** S3 bucket ID if stored via S3-compatible API, or null */
bucketId: string | null;
/** S3 object key if stored via S3-compatible API, or null */
s3Key: string | null;
/** Storage backend identifier, defaults to "telegram" */
storageBackend: string | null;
/** Soft-delete flag */
isDeleted: boolean | null;
/** S3 multipart upload ID if uploaded in parts, or null */
multipartUploadId: string | null;
/** Number of file_parts for chunked storage, or null */
partCount: number | null;
/** Record creation timestamp */
createdAt: Date;
/** Record last-updated timestamp */
updatedAt: Date;
}
/**
* Input type for creating a new File record.
* Omits auto-generated fields (id, createdAt, updatedAt).
*/
export type NewFile = Omit<File, 'id' | 'createdAt' | 'updatedAt'>;
+43
View File
@@ -0,0 +1,43 @@
/**
* Core domain entity representing an S3 multipart upload session.
* Tracks in-progress multipart uploads within a bucket.
*/
export interface MultipartUpload {
/** Unique upload identifier (nanoid) */
uploadId: string;
/** Foreign key to the parent Bucket (UUID) */
bucketId: string;
/** S3 object key being uploaded */
s3Key: string;
/** Timestamp when the upload was initiated */
initiatedAt: Date;
/** Upload status: "in_progress", "completed", or "aborted" */
status: string;
/** Identifier of the entity that initiated the upload */
initiatedBy: string;
}
/**
* Core domain entity representing an individual part of an S3 multipart upload.
* Each part is stored as a separate Telegram message.
*/
export interface MultipartPart {
/** Primary key, auto-increment */
id: number;
/** Foreign key to the parent MultipartUpload */
uploadId: string;
/** Sequential part number (1-based within the upload) */
partNumber: number;
/** Telegram file_id for retrieving this part */
telegramFileId: string;
/** Telegram unique file_id (stable across bot tokens) */
telegramFileUniqueId: string;
/** Message ID within the storage chat */
storageMessageId: number;
/** Part size in bytes */
sizeBytes: number;
/** ETag for this part */
etag: string;
/** Record creation timestamp */
createdAt: Date;
}
+42
View File
@@ -0,0 +1,42 @@
import type { Bucket } from '../entities/bucket';
/**
* Repository interface for Bucket entity persistence.
*
* Abstracts the bucket CRUD operations currently in `src/db/buckets.ts`.
*/
export interface IBucketRepository {
/**
* Create a new bucket with the given name.
* @param name - The unique bucket name (S3 naming convention).
* @returns The newly created bucket record.
*/
create(name: string): Promise<Bucket>;
/**
* Find a bucket by its unique name.
* @param name - The bucket name to look up.
* @returns The matching bucket, or `null` when not found.
*/
findByName(name: string): Promise<Bucket | null>;
/**
* List all buckets, ordered alphabetically by name.
* @returns An array of all bucket records.
*/
list(): Promise<Bucket[]>;
/**
* Delete a bucket and cascade-delete all associated files and multipart data.
* @param name - The name of the bucket to delete.
* @returns `true` if the bucket was deleted, `false` if it did not exist.
*/
delete(name: string): Promise<boolean>;
/**
* Check whether a bucket with the given name exists.
* @param name - The bucket name to check.
* @returns `true` if the bucket exists, `false` otherwise.
*/
exists(name: string): Promise<boolean>;
}
+30
View File
@@ -0,0 +1,30 @@
import type { FilePart, NewFilePart } from '../entities/file-part';
/**
* Repository interface for FilePart entity persistence.
*
* Abstracts the file-part operations currently in `src/db/file-parts.ts`.
* File parts represent the chunks of a large file stored across multiple
* Telegram messages for Telegram-safe storage.
*/
export interface IFilePartRepository {
/**
* Insert multiple file parts in a single operation.
* @param parts - An array of new file part records (auto-generated fields omitted).
*/
insert(parts: NewFilePart[]): Promise<void>;
/**
* List all file parts for a given file, ordered by part number.
* @param fileId - The UUID of the parent file record.
* @returns An array of file parts.
*/
listByFileId(fileId: string): Promise<FilePart[]>;
/**
* Count the number of file parts associated with a file.
* @param fileId - The UUID of the parent file record.
* @returns The part count.
*/
countByFileId(fileId: string): Promise<number>;
}
+108
View File
@@ -0,0 +1,108 @@
import type { File, NewFile } from '../entities/file';
/**
* An S3-synced file record: a File entity guaranteed to have non-null
* bucketId and s3Key values.
*/
export interface S3FileRecord extends File {
/** S3 bucket UUID (non-null refinement) */
bucketId: string;
/** S3 object key (non-null refinement) */
s3Key: string;
}
/**
* Repository interface for File entity persistence.
*
* Abstracts all file CRUD operations currently spread across
* `src/db/files.ts` and `src/db/files-ext.ts`.
*/
export interface IFileRepository {
/**
* Find a single file by its SHA-256 content hash.
* @param hash - The SHA-256 hash to search for.
* @returns The matching file, or `null` when not found.
*/
findByHash(hash: string): Promise<File | null>;
/**
* Find a single file by its public-facing short identifier.
* @param publicId - The public ID to look up.
* @returns The matching file, or `null` when not found.
*/
findByPublicId(publicId: string): Promise<File | null>;
/**
* Find a single file by its Telegram file unique ID (stable across bot tokens).
* @param telegramFileUniqueId - The Telegram unique file ID.
* @returns The matching file, or `null` when not found.
*/
findByUniqueId(telegramFileUniqueId: string): Promise<File | null>;
/**
* Find a single file by its S3 bucket and object key.
* @param bucketId - The bucket UUID.
* @param s3Key - The S3 object key.
* @returns The matching file, or `null` when not found.
*/
findByBucketAndKey(bucketId: string, s3Key: string): Promise<File | null>;
/**
* Create a new file record.
* @param file - The file data (auto-generated fields omitted).
* @returns The newly created file record with all fields populated.
*/
create(file: NewFile): Promise<File>;
/**
* List objects within a bucket, optionally filtered by prefix and delimiter.
*
* When `delimiter` is `"/"`, common prefixes (pseudo-directories) are
* returned separately and objects whose key continues past the delimiter
* are omitted from the `objects` array.
*
* @param bucketId - The bucket UUID to list from.
* @param prefix - Key prefix to filter by.
* @param delimiter - Delimiter character (e.g. `"/"`) or `null` for flat listing.
* @param maxKeys - Maximum number of object records to return.
* @param startAfter - Return only keys strictly greater than this value, or `null`.
* @returns A list of matching S3 file records and discovered common prefixes.
*/
listByPrefix(
bucketId: string,
prefix: string,
delimiter: string | null,
maxKeys: number,
startAfter: string | null,
): Promise<{ objects: S3FileRecord[]; prefixes: string[] }>;
/**
* Soft-delete a single file by bucket and key.
* @param bucketId - The bucket UUID.
* @param s3Key - The S3 object key.
* @returns `true` if a row was soft-deleted, `false` otherwise.
*/
softDelete(bucketId: string, s3Key: string): Promise<boolean>;
/**
* Soft-delete multiple files within a bucket in batch.
* @param bucketId - The bucket UUID.
* @param keys - Array of S3 object keys to delete.
* @returns The number of rows actually soft-deleted.
*/
softDeleteBatch(bucketId: string, keys: string[]): Promise<number>;
/**
* Count non-deleted objects in a bucket.
* @param bucketId - The bucket UUID.
* @returns The object count.
*/
countByBucket(bucketId: string): Promise<number>;
/**
* Find soft-deleted (orphaned) file records in a bucket.
* @param bucketId - The bucket UUID.
* @returns An array of orphaned file records.
*/
findOrphansByBucket(bucketId: string): Promise<File[]>;
}
+70
View File
@@ -0,0 +1,70 @@
import type { MultipartPart, MultipartUpload } from '../entities/multipart';
/**
* Repository interface for S3 multipart upload persistence.
*
* Abstracts the multipart upload operations currently in `src/db/multipart.ts`.
* Manages both multipart upload sessions and their individual parts.
*/
export interface IMultipartRepository {
/**
* Initiate a new multipart upload session.
* @param bucketId - The UUID of the target bucket.
* @param s3Key - The S3 object key being uploaded.
* @param initiatedBy - Identifier of the entity that initiated the upload.
* @returns The newly generated upload ID (nanoid).
*/
create(bucketId: string, s3Key: string, initiatedBy: string): Promise<string>;
/**
* Find an in-progress multipart upload by its upload ID.
* @param uploadId - The upload identifier.
* @returns The matching upload, or `null` if not found or not in progress.
*/
findById(uploadId: string): Promise<MultipartUpload | null>;
/**
* Mark a multipart upload as completed.
* @param uploadId - The upload identifier to complete.
*/
complete(uploadId: string): Promise<void>;
/**
* Mark a multipart upload as aborted.
* @param uploadId - The upload identifier to abort.
*/
abort(uploadId: string): Promise<void>;
/**
* Insert a single part record for a multipart upload.
* @param part - The part data (auto-generated fields omitted).
*/
insertPart(part: Omit<MultipartPart, 'id' | 'createdAt'>): Promise<void>;
/**
* List all parts for a multipart upload, ordered by part number.
* @param uploadId - The upload identifier.
* @returns An array of multipart parts.
*/
listParts(uploadId: string): Promise<MultipartPart[]>;
/**
* List in-progress multipart uploads within a bucket, with pagination.
*
* Results are ordered by S3 key and initiation timestamp.
*
* @param bucketId - The UUID of the bucket.
* @param maxUploads - Maximum number of uploads to return (clamped 1-1000).
* @param keyMarker - Return only uploads whose S3 key is strictly greater than this, or `null`.
* @returns A list of uploads and pagination metadata.
*/
listByBucket(
bucketId: string,
maxUploads: number,
keyMarker: string | null,
): Promise<{
uploads: MultipartUpload[];
isTruncated: boolean;
nextKeyMarker: string | null;
}>;
}
+53
View File
@@ -0,0 +1,53 @@
/**
* Result of forwarding a file to Telegram storage.
*/
export interface ForwardResult {
/** The Telegram file_id for retrieving the file */
telegramFileId: string;
/** The Telegram unique file_id (stable across bot tokens) */
telegramFileUniqueId: string;
/** The message ID within the storage chat */
storageMessageId: number;
}
/**
* File information returned by Telegram's getFile API.
*/
export interface TelegramFileInfo {
/** File size in bytes */
file_size: number;
/** MIME type of the file */
mime_type: string;
/** Path on Telegram's file server for downloading */
file_path: string;
/** Bot token that owns the retrieved file */
bot_token: string;
}
/**
* Abstraction over Telegram bot API operations.
*
* Defines the contract for forwarding files to Telegram storage,
* retrieving file metadata, and managing concurrent uploads.
*/
export interface ITelegramService {
/**
* Forward a file chunk to the configured Telegram storage chat.
*
* @param fileChunk - The file data (ReadStream, Buffer, or file path).
* @param fileName - The original file name.
* @param fileType - The file type classification (e.g. "photo", "document").
* @returns The Telegram identifiers of the stored file.
*/
forwardToStorage(fileChunk: unknown, fileName: string, fileType: string): Promise<ForwardResult>;
/**
* Retrieve file metadata from Telegram by file ID.
*
* Tries all configured bots; returns info from the first that owns the file.
*
* @param telegramFileId - The Telegram file_id to look up.
* @returns Metadata including size, MIME type, download path, and bot token.
*/
getFileInfo(telegramFileId: string): Promise<TelegramFileInfo>;
}
+67 -13
View File
@@ -1,8 +1,11 @@
import logger from './utils/logger'; import logger from './shared/logger/index';
import { TELEGRAM_CHUNK_SIZE_MAX_BYTES } from './shared/utils/validation';
interface AppConfig { interface AppConfig {
botToken: string; /** All bot tokens merged from BOT_TOKENS (or BOT_TOKEN + ADDITIONAL_BOT_TOKENS fallback) */
additionalBotTokens: string[]; botTokens: string[];
/** Per-bot concurrency for Telegram API calls (default 1). */
telegramBotConcurrency: number;
storageChatId: number; storageChatId: number;
baseUrl: string; baseUrl: string;
databaseUrl: string; databaseUrl: string;
@@ -12,7 +15,6 @@ interface AppConfig {
rateLimitWindowMs: number; rateLimitWindowMs: number;
rateLimitMaxRequests: number; rateLimitMaxRequests: number;
trustProxy: boolean; trustProxy: boolean;
uploadConcurrency: number;
batchMaxItems: number; batchMaxItems: number;
batchMaxSizeBytes: number; batchMaxSizeBytes: number;
maxRequestBodyBytes: number; maxRequestBodyBytes: number;
@@ -29,8 +31,21 @@ interface AppConfig {
s3VhostDomains: string[]; s3VhostDomains: string[];
} }
// Validate bot tokens: BOT_TOKENS (new) or fallback to BOT_TOKEN + ADDITIONAL_BOT_TOKENS
const botTokensRaw =
process.env.BOT_TOKENS ||
[process.env.BOT_TOKEN, process.env.ADDITIONAL_BOT_TOKENS].filter(Boolean).join(',');
if (!botTokensRaw) {
logger.error(
'Missing required environment variables: BOT_TOKENS (or BOT_TOKEN + ADDITIONAL_BOT_TOKENS)',
);
throw new Error(
'Missing environment variables: BOT_TOKENS (or BOT_TOKEN + ADDITIONAL_BOT_TOKENS)',
);
}
const requiredEnv = { const requiredEnv = {
BOT_TOKEN: process.env.BOT_TOKEN,
STORAGE_CHANNEL_ID: process.env.STORAGE_CHANNEL_ID, STORAGE_CHANNEL_ID: process.env.STORAGE_CHANNEL_ID,
BASE_URL: process.env.BASE_URL, BASE_URL: process.env.BASE_URL,
DATABASE_URL: process.env.DATABASE_URL, DATABASE_URL: process.env.DATABASE_URL,
@@ -46,6 +61,25 @@ if (missing.length > 0) {
throw new Error(`Missing environment variables: ${missing.join(', ')}`); throw new Error(`Missing environment variables: ${missing.join(', ')}`);
} }
// Validate S3 credentials: if S3_ACCESS_KEY is explicitly set (env var present,
// not relying on default), S3_SECRET_KEY must also be set. An empty secret key
// would cause HMAC-SHA256 to "succeed" silently — a security hole.
const s3AccessKeyExplicit = 'S3_ACCESS_KEY' in process.env;
const s3SecretKeyExplicit = 'S3_SECRET_KEY' in process.env;
if (s3AccessKeyExplicit || s3SecretKeyExplicit) {
const s3Key = (process.env.S3_ACCESS_KEY || '').trim();
const s3Secret = (process.env.S3_SECRET_KEY || '').trim();
if (s3Key && !s3Secret) {
logger.error('S3_ACCESS_KEY is set but S3_SECRET_KEY is empty — this is a security risk');
throw new Error(
'S3_ACCESS_KEY requires S3_SECRET_KEY to be set. Set S3_SECRET_KEY or unset S3_ACCESS_KEY.',
);
}
if (s3Secret && !s3Key) {
logger.warn('S3_SECRET_KEY is set but S3_ACCESS_KEY is not — S3 auth will use the default key');
}
}
const parseNumber = (value: string | undefined, fallback: number): number => { const parseNumber = (value: string | undefined, fallback: number): number => {
const parsed = Number.parseInt(value || '', 10); const parsed = Number.parseInt(value || '', 10);
return Number.isFinite(parsed) && parsed > 0 ? parsed : fallback; return Number.isFinite(parsed) && parsed > 0 ? parsed : fallback;
@@ -74,24 +108,45 @@ const maskSecret = (value: string): string => {
const maskDatabaseUrl = (value: string): string => const maskDatabaseUrl = (value: string): string =>
value.replace(/:\/\/([^:]+):([^@]+)@/, '://$1:***@'); value.replace(/:\/\/([^:]+):([^@]+)@/, '://$1:***@');
// Fail-fast guard for TELEGRAM_CHUNK_SIZE_BYTES: chunked uploads store every
// part as a Telegram document and later resolve it via getFile, which only
// supports files up to 20 MB ("Bad Request: file is too big" above that).
// A chunk above the limit makes every part undownloadable — refuse to start
// instead of failing on the first large-file download.
const telegramChunkSizeBytes = parseNumber(
process.env.TELEGRAM_CHUNK_SIZE_BYTES,
TELEGRAM_CHUNK_SIZE_MAX_BYTES,
);
if (telegramChunkSizeBytes > TELEGRAM_CHUNK_SIZE_MAX_BYTES) {
logger.error(
`TELEGRAM_CHUNK_SIZE_BYTES=${telegramChunkSizeBytes} exceeds the maximum allowed chunk size ` +
`${TELEGRAM_CHUNK_SIZE_MAX_BYTES} bytes (${TELEGRAM_CHUNK_SIZE_MAX_BYTES / (1024 * 1024)} MB). ` +
'Telegram Bot API getFile cannot download files larger than 20 MB, so every stored part would ' +
'be undownloadable ("Bad Request: file is too big"). ' +
`Set TELEGRAM_CHUNK_SIZE_BYTES to ${TELEGRAM_CHUNK_SIZE_MAX_BYTES} or lower.`,
);
throw new Error(
`TELEGRAM_CHUNK_SIZE_BYTES=${telegramChunkSizeBytes} exceeds the maximum allowed chunk size ` +
`${TELEGRAM_CHUNK_SIZE_MAX_BYTES} bytes (${TELEGRAM_CHUNK_SIZE_MAX_BYTES / (1024 * 1024)} MB)`,
);
}
export const config: AppConfig = { export const config: AppConfig = {
botToken: process.env.BOT_TOKEN!, botTokens: parseTokens(botTokensRaw),
additionalBotTokens: telegramBotConcurrency: parseNumber(process.env.TELEGRAM_BOT_CONCURRENCY, 1),
process.env.NODE_ENV === 'test' ? [] : parseTokens(process.env.ADDITIONAL_BOT_TOKENS),
storageChatId: parseInt(process.env.STORAGE_CHANNEL_ID!, 10), storageChatId: parseInt(process.env.STORAGE_CHANNEL_ID!, 10),
baseUrl: process.env.BASE_URL!, baseUrl: process.env.BASE_URL!,
databaseUrl: process.env.DATABASE_URL!, databaseUrl: process.env.DATABASE_URL!,
port: parseInt(process.env.PORT!, 10) || 3000, port: parseInt(process.env.PORT!, 10) || 4000,
nodeEnv: process.env.NODE_ENV || 'development', nodeEnv: process.env.NODE_ENV || 'development',
logLevel: process.env.LOG_LEVEL || 'info', logLevel: process.env.LOG_LEVEL || 'info',
rateLimitWindowMs: parseNumber(process.env.RATE_LIMIT_WINDOW_MS, 60000), rateLimitWindowMs: parseNumber(process.env.RATE_LIMIT_WINDOW_MS, 60000),
rateLimitMaxRequests: parseNumber(process.env.RATE_LIMIT_MAX_REQUESTS, 150), rateLimitMaxRequests: parseNumber(process.env.RATE_LIMIT_MAX_REQUESTS, 150),
trustProxy: process.env.TRUST_PROXY === 'true', trustProxy: process.env.TRUST_PROXY === 'true',
uploadConcurrency: parseNumber(process.env.UPLOAD_CONCURRENCY, 8),
batchMaxItems: parseNumber(process.env.BATCH_MAX_ITEMS, 20), batchMaxItems: parseNumber(process.env.BATCH_MAX_ITEMS, 20),
batchMaxSizeBytes: parseNumber(process.env.BATCH_MAX_SIZE_BYTES, 500 * 1024 * 1024), batchMaxSizeBytes: parseNumber(process.env.BATCH_MAX_SIZE_BYTES, 500 * 1024 * 1024),
maxRequestBodyBytes: parseNumber(process.env.MAX_REQUEST_BODY_BYTES, 2 * 1024 * 1024 * 1024), maxRequestBodyBytes: parseNumber(process.env.MAX_REQUEST_BODY_BYTES, 2 * 1024 * 1024 * 1024),
telegramChunkSizeBytes: parseNumber(process.env.TELEGRAM_CHUNK_SIZE_BYTES, 20 * 1024 * 1024), telegramChunkSizeBytes,
compressChunkedUploads: process.env.COMPRESS_CHUNKED_UPLOADS !== 'false', compressChunkedUploads: process.env.COMPRESS_CHUNKED_UPLOADS !== 'false',
chunkCompressionMinSizeBytes: parseNumber(process.env.CHUNK_COMPRESSION_MIN_SIZE_BYTES, 4096), chunkCompressionMinSizeBytes: parseNumber(process.env.CHUNK_COMPRESSION_MIN_SIZE_BYTES, 4096),
adminApiToken: process.env.ADMIN_API_TOKEN || '', adminApiToken: process.env.ADMIN_API_TOKEN || '',
@@ -110,8 +165,7 @@ export const config: AppConfig = {
logger.info('Environment variables loaded', { logger.info('Environment variables loaded', {
config: { config: {
...config, ...config,
botToken: maskSecret(config.botToken), botTokens: config.botTokens.map(maskSecret),
additionalBotTokens: config.additionalBotTokens.map(maskSecret),
databaseUrl: maskDatabaseUrl(config.databaseUrl), databaseUrl: maskDatabaseUrl(config.databaseUrl),
adminApiToken: maskSecret(config.adminApiToken), adminApiToken: maskSecret(config.adminApiToken),
adminApiTokenEnabled: config.adminApiToken.length > 0, adminApiTokenEnabled: config.adminApiToken.length > 0,
+59 -27
View File
@@ -136,6 +136,10 @@
} }
.auth-card button:disabled { opacity: 0.7; cursor: wait; } .auth-card button:disabled { opacity: 0.7; cursor: wait; }
.auth-error { color: var(--danger); font-size: 0.85rem; margin-bottom: 12px; } .auth-error { color: var(--danger); font-size: 0.85rem; margin-bottom: 12px; }
.readonly-badge {
font-size: 0.75rem; color: var(--text2); background: var(--bg2);
border: 1px solid var(--border); border-radius: 999px; padding: 2px 10px;
}
</style> </style>
</head> </head>
<body> <body>
@@ -153,9 +157,11 @@
<select id="bucketSelect" onchange="window.switchBucket(this.value)"> <select id="bucketSelect" onchange="window.switchBucket(this.value)">
<option value="">— Select bucket —</option> <option value="">— Select bucket —</option>
</select> </select>
<button type="button" onclick="window.showCreateBucketModal()">+ New</button> <button id="newBucketBtn" type="button" onclick="window.showCreateBucketModal()">+ New</button>
<button type="button" onclick="window.showCredentialsModal()" title="S3 Credentials">🔑</button> <button id="credsBtn" type="button" onclick="window.showCredentialsModal()" title="S3 Credentials">🔑</button>
<button id="loginBtn" type="button" onclick="window.showAuthScreen()" style="display:none">Login</button>
<button id="logoutBtn" type="button" onclick="window.logout()" style="display:none">Logout</button> <button id="logoutBtn" type="button" onclick="window.logout()" style="display:none">Logout</button>
<span id="readonlyBadge" class="readonly-badge" style="display:none">👀 read-only</span>
<span class="spacer"></span> <span class="spacer"></span>
<div class="search"> <div class="search">
<input id="searchInput" type="text" placeholder="Filter prefix..." oninput="window.debouncedSearch()"> <input id="searchInput" type="text" placeholder="Filter prefix..." oninput="window.debouncedSearch()">
@@ -179,6 +185,7 @@
</div> </div>
<script> <script>
let currentBucket = null, currentPrefix = '', currentObjects = [], currentPrefixes = [], allBuckets = [], searchTimer = null; let currentBucket = null, currentPrefix = '', currentObjects = [], currentPrefixes = [], allBuckets = [], searchTimer = null;
let isAdmin = false;
const setAuthError = (message) => { const setAuthError = (message) => {
const errorEl = document.getElementById('authError'); const errorEl = document.getElementById('authError');
errorEl.textContent = message; errorEl.textContent = message;
@@ -186,25 +193,36 @@
}; };
const showAuthScreen = () => { const showAuthScreen = () => {
document.getElementById('authScreen').style.display = 'flex'; document.getElementById('authScreen').style.display = 'flex';
document.getElementById('logoutBtn').style.display = 'none';
setTimeout(() => document.getElementById('authTokenInput')?.focus(), 50); setTimeout(() => document.getElementById('authTokenInput')?.focus(), 50);
}; };
const hideAuthScreen = (showLogout) => { const hideAuthScreen = () => {
document.getElementById('authScreen').style.display = 'none'; document.getElementById('authScreen').style.display = 'none';
document.getElementById('logoutBtn').style.display = showLogout ? 'inline-block' : 'none';
}; };
// Applies the admin/read-only UI state based on isAdmin.
const applyAdminUI = () => {
document.getElementById('newBucketBtn').style.display = isAdmin ? 'inline-block' : 'none';
document.getElementById('credsBtn').style.display = isAdmin ? 'inline-block' : 'none';
document.getElementById('loginBtn').style.display = isAdmin ? 'none' : 'inline-block';
document.getElementById('logoutBtn').style.display = isAdmin ? 'inline-block' : 'none';
document.getElementById('readonlyBadge').style.display = isAdmin ? 'none' : 'inline-block';
// Dropzone (upload) is admin-only.
document.getElementById('dropzone').style.display = isAdmin && currentBucket ? 'block' : 'none';
if (currentObjects.length || currentPrefixes.length) renderFileList();
};
// Non-blocking auth check: read-only visitors still get the file browser.
const checkAuth = async () => { const checkAuth = async () => {
try { try {
const res = await fetch('/api/v1/auth/me'); const res = await fetch('/api/v1/auth/me');
if (res.ok) { hideAuthScreen(true); return true; } if (res.ok) { isAdmin = true; }
if (res.status === 401) { showAuthScreen(); return false; } else if (res.status === 401) { isAdmin = false; }
if (res.status === 404) { hideAuthScreen(false); return true; } else if (res.status === 404) { isAdmin = true; } // auth disabled — full access
setAuthError('Unable to verify login status. Please try again.'); else { isAdmin = false; }
showAuthScreen(); return false;
} catch { } catch {
setAuthError('Network error while checking login status.'); isAdmin = false;
showAuthScreen(); return false;
} }
hideAuthScreen();
applyAdminUI();
return isAdmin;
}; };
const handleLogin = async () => { const handleLogin = async () => {
const input = document.getElementById('authTokenInput'); const input = document.getElementById('authTokenInput');
@@ -217,7 +235,7 @@
method: 'POST', headers: { 'content-type': 'application/json' }, method: 'POST', headers: { 'content-type': 'application/json' },
body: JSON.stringify({ token }), body: JSON.stringify({ token }),
}); });
if (res.ok) { hideAuthScreen(true); input.value = ''; await loadBuckets(); return; } if (res.ok) { isAdmin = true; hideAuthScreen(); input.value = ''; applyAdminUI(); await loadBuckets(); return; }
const body = await res.json().catch(() => ({ error: 'Login failed' })); const body = await res.json().catch(() => ({ error: 'Login failed' }));
setAuthError(body.error || 'Login failed'); setAuthError(body.error || 'Login failed');
} catch { } catch {
@@ -228,11 +246,8 @@
}; };
const logout = async () => { const logout = async () => {
await fetch('/api/v1/auth/logout', { method: 'POST' }).catch(() => {}); await fetch('/api/v1/auth/logout', { method: 'POST' }).catch(() => {});
currentBucket = null; currentPrefix = ''; currentObjects = []; currentPrefixes = []; isAdmin = false;
document.getElementById('bucketSelect').innerHTML = '<option value="">— Select bucket —</option>'; applyAdminUI();
document.getElementById('fileList').innerHTML = '<div class="empty"><h2>Logged out</h2><p>Enter the admin token to continue.</p></div>';
document.getElementById('dropzone').style.display = 'none';
showAuthScreen();
}; };
const api = async (path, opts = {}) => { const api = async (path, opts = {}) => {
const res = await fetch(path, opts); const res = await fetch(path, opts);
@@ -249,11 +264,13 @@
}; };
const switchBucket = async (name) => { const switchBucket = async (name) => {
currentBucket = name || null; currentPrefix = ''; currentBucket = name || null; currentPrefix = '';
if (name) { await loadObjects(); document.getElementById('dropzone').style.display = 'block'; } if (name) { await loadObjects(); }
else { else {
document.getElementById('fileList').innerHTML = '<div class="empty"><h2>Select a bucket</h2><p>Choose a bucket from the dropdown above.</p></div>'; document.getElementById('fileList').innerHTML = '<div class="empty"><h2>Select a bucket</h2><p>Choose a bucket from the dropdown above.</p></div>';
document.getElementById('breadcrumb').style.display = 'none'; document.getElementById('dropzone').style.display = 'none'; document.getElementById('breadcrumb').style.display = 'none';
} }
// Dropzone (upload) is admin-only.
document.getElementById('dropzone').style.display = isAdmin && currentBucket ? 'block' : 'none';
}; };
const renderBreadcrumb = () => { const renderBreadcrumb = () => {
const bc = document.getElementById('breadcrumb'); const bc = document.getElementById('breadcrumb');
@@ -278,7 +295,12 @@
}; };
const renderFileList = () => { const renderFileList = () => {
const container = document.getElementById('fileList'); const container = document.getElementById('fileList');
if (currentPrefixes.length === 0 && currentObjects.length === 0) { container.innerHTML = '<div class="empty"><h2>This bucket is empty</h2><p>Drop files here to upload.</p></div>'; return; } if (currentPrefixes.length === 0 && currentObjects.length === 0) {
container.innerHTML = isAdmin
? '<div class="empty"><h2>This bucket is empty</h2><p>Drop files here to upload.</p></div>'
: '<div class="empty"><h2>This bucket is empty</h2></div>';
return;
}
let html = ''; let html = '';
for (const prefix of currentPrefixes) { for (const prefix of currentPrefixes) {
const displayName = prefix.replace(currentPrefix, ''); const displayName = prefix.replace(currentPrefix, '');
@@ -286,7 +308,9 @@
} }
for (const obj of currentObjects) { for (const obj of currentObjects) {
const displayName = obj.key.replace(currentPrefix, ''); const displayName = obj.key.replace(currentPrefix, '');
html += `<div class="file-row"><span class="icon">📄</span><span class="name">${escapeHtml(displayName)}</span><span class="size">${formatSize(obj.sizeBytes)}</span><span class="date">${formatDate(obj.lastModified)}</span><span class="actions"><button onclick="event.stopPropagation();downloadObject('${obj.key}')" title="Download"></button><button onclick="event.stopPropagation();copyLink('${obj.key}')" title="Copy link">🔗</button><button onclick="event.stopPropagation();deleteObject('${obj.key}')" title="Delete">🗑</button></span></div>`; // Delete is admin-only; download + copy link are always available.
const deleteBtn = isAdmin ? `<button onclick="event.stopPropagation();deleteObject('${obj.key}')" title="Delete">🗑</button>` : '';
html += `<div class="file-row"><span class="icon">📄</span><span class="name">${escapeHtml(displayName)}</span><span class="size">${formatSize(obj.sizeBytes)}</span><span class="date">${formatDate(obj.lastModified)}</span><span class="actions"><button onclick="event.stopPropagation();downloadObject('${obj.key}')" title="Download"></button><button onclick="event.stopPropagation();copyLink('${obj.key}')" title="Copy link">🔗</button>${deleteBtn}</span></div>`;
} }
container.innerHTML = html; container.innerHTML = html;
}; };
@@ -297,11 +321,13 @@
const downloadObject = async (key) => { window.open(`/api/v1/buckets/${encodeURIComponent(currentBucket)}/download/${encodeURIComponent(key)}`,'_blank'); }; const downloadObject = async (key) => { window.open(`/api/v1/buckets/${encodeURIComponent(currentBucket)}/download/${encodeURIComponent(key)}`,'_blank'); };
const copyLink = (key) => { navigator.clipboard.writeText(`${window.location.origin}/api/v1/buckets/${encodeURIComponent(currentBucket)}/download/${encodeURIComponent(key)}`).catch(()=>{}); }; const copyLink = (key) => { navigator.clipboard.writeText(`${window.location.origin}/api/v1/buckets/${encodeURIComponent(currentBucket)}/download/${encodeURIComponent(key)}`).catch(()=>{}); };
const deleteObject = async (key) => { const deleteObject = async (key) => {
if (!isAdmin) { alert('Read-only mode — login as admin to delete.'); return; }
if(!confirm(`Delete "${key}"?`))return; if(!confirm(`Delete "${key}"?`))return;
try{await api(`/api/v1/buckets/${encodeURIComponent(currentBucket)}/${encodeURIComponent(key)}`,{method:'DELETE'});await loadObjects();} try{await api(`/api/v1/buckets/${encodeURIComponent(currentBucket)}/${encodeURIComponent(key)}`,{method:'DELETE'});await loadObjects();}
catch(e){alert(`Delete failed: ${e.message}`);} catch(e){alert(`Delete failed: ${e.message}`);}
}; };
const uploadFiles = async (files) => { const uploadFiles = async (files) => {
if (!isAdmin) { alert('Read-only mode — login as admin to upload.'); return; }
if(!currentBucket||files.length===0)return; if(!currentBucket||files.length===0)return;
const overlay=document.getElementById('progressOverlay'), fill=document.getElementById('progressFill'), pn=document.getElementById('progressFileName'), pp=document.getElementById('progressPercent'); const overlay=document.getElementById('progressOverlay'), fill=document.getElementById('progressFill'), pn=document.getElementById('progressFileName'), pp=document.getElementById('progressPercent');
overlay.style.display='flex'; overlay.style.display='flex';
@@ -325,13 +351,19 @@
dropzone.addEventListener('click',()=>{const i=document.createElement('input');i.type='file';i.multiple=true;i.onchange=()=>{if(i.files.length>0)uploadFiles(i.files);};i.click();}); dropzone.addEventListener('click',()=>{const i=document.createElement('input');i.type='file';i.multiple=true;i.onchange=()=>{if(i.files.length>0)uploadFiles(i.files);};i.click();});
const showModal=(html)=>{document.getElementById('modalContent').innerHTML=html;document.getElementById('modalOverlay').style.display='flex';}; const showModal=(html)=>{document.getElementById('modalContent').innerHTML=html;document.getElementById('modalOverlay').style.display='flex';};
const closeModal=(e)=>{if(e&&e.target!==e.currentTarget)return;document.getElementById('modalOverlay').style.display='none';}; const closeModal=(e)=>{if(e&&e.target!==e.currentTarget)return;document.getElementById('modalOverlay').style.display='none';};
const showCreateBucketModal=()=>{showModal(`<h3>Create Bucket</h3><input id="bucketNameInput" type="text" placeholder="my-bucket-name" pattern="[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]"><p style="font-size:0.8rem;color:var(--text2);margin-bottom:12px">Lowercase, 3-63 chars, no underscores</p><div class="buttons"><button onclick="closeModal()">Cancel</button><button class="primary" onclick="createBucket()">Create</button></div>`);setTimeout(()=>document.getElementById('bucketNameInput')?.focus(),100);}; const showCreateBucketModal=()=>{
const createBucket=async()=>{const n=document.getElementById('bucketNameInput').value.trim();if(!n)return;try{await apiJson('/api/v1/buckets',{method:'POST',body:JSON.stringify({name:n})});closeModal();await loadBuckets();document.getElementById('bucketSelect').value=n;await switchBucket(n);}catch(e){alert(`Failed: ${e.message}`);}}; if (!isAdmin) { alert('Read-only mode — login as admin to create buckets.'); return; }
const showCredentialsModal=()=>{showModal(`<h3>S3 Credentials</h3><p style="margin-bottom:12px;font-size:0.85rem;color:var(--text2)">Use these in any S3 client (aws-cli, rclone, s3cmd, etc.)</p><label style="font-size:0.85rem;font-weight:600">Endpoint URL</label><input type="text" value="${window.location.origin}" readonly onclick="this.select()"><label style="font-size:0.85rem;font-weight:600">Region</label><input type="text" value="us-east-1" readonly onclick="this.select()"><label style="font-size:0.85rem;font-weight:600">Access Key</label><input id="s3AccessKey" type="text" readonly onclick="this.select()"><label style="font-size:0.85rem;font-weight:600">Secret Key</label><input id="s3SecretKey" type="password" readonly onclick="this.select()"><div class="buttons"><button type="button" onclick="window.closeModal()">Close</button></div>`);}; showModal(`<h3>Create Bucket</h3><input id="bucketNameInput" type="text" placeholder="my-bucket-name" pattern="[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]"><p style="font-size:0.8rem;color:var(--text2);margin-bottom:12px">Lowercase, 3-63 chars, no underscores</p><div class="buttons"><button onclick="closeModal()">Cancel</button><button class="primary" onclick="createBucket()">Create</button></div>`);setTimeout(()=>document.getElementById('bucketNameInput')?.focus(),100);};
const init=async()=>{if(await checkAuth())await loadBuckets();}; const createBucket=async()=>{
if (!isAdmin) { alert('Read-only mode — login as admin to create buckets.'); return; }
const n=document.getElementById('bucketNameInput').value.trim();if(!n)return;try{await apiJson('/api/v1/buckets',{method:'POST',body:JSON.stringify({name:n})});closeModal();await loadBuckets();document.getElementById('bucketSelect').value=n;await switchBucket(n);}catch(e){alert(`Failed: ${e.message}`);}};
const showCredentialsModal=()=>{
if (!isAdmin) { alert('Read-only mode — login as admin to view S3 credentials.'); return; }
showModal(`<h3>S3 Credentials</h3><p style="margin-bottom:12px;font-size:0.85rem;color:var(--text2)">Use these in any S3 client (aws-cli, rclone, s3cmd, etc.)</p><label style="font-size:0.85rem;font-weight:600">Endpoint URL</label><input type="text" value="${window.location.origin}" readonly onclick="this.select()"><label style="font-size:0.85rem;font-weight:600">Region</label><input type="text" value="us-east-1" readonly onclick="this.select()"><label style="font-size:0.85rem;font-weight:600">Access Key</label><input id="s3AccessKey" type="text" readonly onclick="this.select()"><label style="font-size:0.85rem;font-weight:600">Secret Key</label><input id="s3SecretKey" type="password" readonly onclick="this.select()"><div class="buttons"><button type="button" onclick="window.closeModal()">Close</button></div>`);};
const init=async()=>{await checkAuth();await loadBuckets();};
document.getElementById('authLoginBtn').addEventListener('click',handleLogin); document.getElementById('authLoginBtn').addEventListener('click',handleLogin);
document.getElementById('authTokenInput').addEventListener('keydown',e=>{if(e.key==='Enter')handleLogin();}); document.getElementById('authTokenInput').addEventListener('keydown',e=>{if(e.key==='Enter')handleLogin();});
Object.assign(window, { switchBucket, navigateTo, debouncedSearch, downloadObject, copyLink, deleteObject, closeModal, showCreateBucketModal, createBucket, showCredentialsModal, logout }); Object.assign(window, { switchBucket, navigateTo, debouncedSearch, downloadObject, copyLink, deleteObject, closeModal, showCreateBucketModal, createBucket, showCredentialsModal, showAuthScreen, logout });
init(); init();
</script> </script>
</body> </body>
+11 -101
View File
@@ -1,116 +1,26 @@
import { serve } from 'bun'; import { serve } from 'bun';
import { startBot } from './bot';
import { config } from './env'; import { config } from './env';
import { handleLogin, handleLogout, handleMe } from './routes/auth'; import { fileInfoCache } from './infrastructure/cache/index';
import { handleFileInfo, handleFileRedirect } from './routes/files'; import { startBot } from './interfaces/bot/handler';
import { handleHealth } from './routes/health'; import { handleS3Request } from './interfaces/http/controllers/s3-controller';
import { handleHome } from './routes/home'; import { cleanupRateLimitCache } from './interfaces/http/middleware/rate-limit';
import { handleS3Request } from './routes/s3'; import { routes } from './interfaces/http/routes/index';
import { handleSwaggerHtml, handleSwaggerJson } from './routes/swagger'; import { logger } from './shared/logger/index';
import { handleUpload } from './routes/upload'; import { metricsCollector } from './shared/metrics/index';
import { handleWebApiV1 } from './routes/web-api'; import { getS3RouteBucket, shouldHandleS3 } from './shared/utils/s3-detection';
import { requireAuth } from './utils/auth';
import { fileInfoCache } from './utils/cache';
import logger from './utils/logger';
import { metricsCollector } from './utils/metrics';
import { cleanupRateLimitCache, withRateLimit } from './utils/rateLimit';
import { isS3Request } from './utils/s3/auth';
import { extractS3BucketFromHost } from './utils/s3/virtual-host';
// ─── Auto-run migration at startup ────────────────────────────────────────── // ─── Auto-run migration at startup ──────────────────────────────────────────
try { try {
const { runMigration } = await import('./db/migrate'); const { runMigration } = await import('./infrastructure/persistence/drizzle/migrate');
await runMigration(); await runMigration();
} catch { } catch {
logger.warn('Auto-migration skipped (non-fatal)'); logger.warn('Auto-migration skipped (non-fatal)');
} }
// ─── Auto-run migration at startup ───
try {
await import('./db/migrate');
} catch {
// migrate.ts calls process.exit(1) on failure — if it throws, log and continue
logger.warn('Auto-migration warning (non-fatal)');
}
const getS3RouteBucket = (req: Request): string | null => {
const host = req.headers.get('host') || '';
return extractS3BucketFromHost(host, config.s3VhostDomains);
};
const shouldHandleS3 = (req: Request, headers: Record<string, string>): boolean => {
const url = new URL(req.url);
return Boolean(
getS3RouteBucket(req) || isS3Request(headers) || url.searchParams.has('X-Amz-Signature'),
);
};
const handleMaybeS3Root = (req: Request): Response | Promise<Response> => {
if (req.method === 'OPTIONS') {
return handleS3Request(req, getS3RouteBucket(req));
}
const headers = Object.fromEntries(req.headers);
if (shouldHandleS3(req, headers)) {
return handleS3Request(req, getS3RouteBucket(req));
}
return new Response('Not Allowed', { status: 405 });
};
const server = serve({ const server = serve({
port: config.port, port: config.port,
routes: { routes,
'/api/upload': {
POST: withRateLimit(handleUpload),
},
'/f/:public_id': {
GET: withRateLimit(handleFileRedirect),
},
'/file/:public_id/info': {
GET: withRateLimit(handleFileInfo),
},
'/health': {
GET: handleHealth,
},
'/docs': {
GET: handleSwaggerHtml,
},
'/swagger.json': {
GET: handleSwaggerJson,
},
'/': {
GET: (req: Request) => {
const headers = Object.fromEntries(req.headers);
if (shouldHandleS3(req, headers)) {
return handleS3Request(req, getS3RouteBucket(req));
}
return handleHome();
},
PUT: handleMaybeS3Root,
HEAD: handleMaybeS3Root,
DELETE: handleMaybeS3Root,
POST: handleMaybeS3Root,
OPTIONS: handleMaybeS3Root,
},
'/api/v1/auth/login': {
POST: withRateLimit(handleLogin),
},
'/api/v1/auth/logout': {
POST: handleLogout,
},
'/api/v1/auth/me': {
GET: handleMe,
},
'/api/v1/*': {
GET: requireAuth(handleWebApiV1),
POST: requireAuth(handleWebApiV1),
DELETE: requireAuth(handleWebApiV1),
PUT: requireAuth(handleWebApiV1),
},
},
fetch: async (req: Request) => { fetch: async (req: Request) => {
if (req.method === 'OPTIONS') {
return handleS3Request(req, getS3RouteBucket(req));
}
const headers = Object.fromEntries(req.headers); const headers = Object.fromEntries(req.headers);
if (shouldHandleS3(req, headers)) { if (shouldHandleS3(req, headers)) {
return handleS3Request(req, getS3RouteBucket(req)); return handleS3Request(req, getS3RouteBucket(req));
@@ -126,7 +36,7 @@ logger.info('Server started', { port: config.port, url: config.baseUrl });
const gracefulShutdown = async (signal: string): Promise<void> => { const gracefulShutdown = async (signal: string): Promise<void> => {
logger.info('Graceful shutdown signal received', { signal }); logger.info('Graceful shutdown signal received', { signal });
logger.info('Closing HTTP server'); logger.info('Closing HTTP server — no new requests accepted');
server.stop(); server.stop();
logger.info('Stopping Telegram bot'); logger.info('Stopping Telegram bot');
+14 -23
View File
@@ -1,74 +1,65 @@
// Simple in-memory cache with TTL support /**
interface CacheEntry<T> { * Generic in-memory cache with TTL (time-to-live) support.
value: T; * Entries expire after a configurable duration and are lazily evicted on access.
expiresAt: number; *
} * @typeParam T - The type of values stored in the cache
*/
class Cache<T> { export class Cache<T> {
private store = new Map<string, CacheEntry<T>>(); private store = new Map<string, CacheEntry<T>>();
private ttlMs: number; private ttlMs: number;
constructor(ttlSeconds: number = 3600) { constructor(ttlSeconds = 3600) {
this.ttlMs = ttlSeconds * 1000; this.ttlMs = ttlSeconds * 1000;
} }
set(key: string, value: T): void { set(key: string, value: T): void {
this.store.set(key, { this.store.set(key, { value, expiresAt: Date.now() + this.ttlMs });
value,
expiresAt: Date.now() + this.ttlMs,
});
} }
get(key: string): T | null { get(key: string): T | null {
const entry = this.store.get(key); const entry = this.store.get(key);
if (!entry) return null; if (!entry) return null;
if (Date.now() > entry.expiresAt) { if (Date.now() > entry.expiresAt) {
this.store.delete(key); this.store.delete(key);
return null; return null;
} }
return entry.value; return entry.value;
} }
has(key: string): boolean { has(key: string): boolean {
return this.get(key) !== null; return this.get(key) !== null;
} }
delete(key: string): void { delete(key: string): void {
this.store.delete(key); this.store.delete(key);
} }
clear(): void { clear(): void {
this.store.clear(); this.store.clear();
} }
size(): number { size(): number {
return this.store.size; return this.store.size;
} }
// Cleanup expired entries
cleanup(): number { cleanup(): number {
let removed = 0; let removed = 0;
const now = Date.now(); const now = Date.now();
for (const [key, entry] of this.store.entries()) { for (const [key, entry] of this.store.entries()) {
if (now > entry.expiresAt) { if (now > entry.expiresAt) {
this.store.delete(key); this.store.delete(key);
removed++; removed++;
} }
} }
return removed; return removed;
} }
} }
// File info cache (1 hour TTL) interface CacheEntry<T> {
value: T;
expiresAt: number;
}
export const fileInfoCache = new Cache<{ export const fileInfoCache = new Cache<{
file_size: number; file_size: number;
mime_type: string; mime_type: string;
file_path: string; file_path: string;
bot_token: string; bot_token: string;
}>(3600); }>(3600);
export { Cache };
+47
View File
@@ -0,0 +1,47 @@
/**
* Dependency Injection container.
*
* Wires up singleton instances of all repositories and application services,
* making them available to controllers and other adapters without requiring
* a full DI framework.
*
* @module infrastructure/di
*/
import type { IBucketRepository } from '../domain/ports/bucket-repository';
import type { IFilePartRepository } from '../domain/ports/file-part-repository';
import type { IFileRepository } from '../domain/ports/file-repository';
import type { IMultipartRepository } from '../domain/ports/multipart-repository';
import type { ITelegramService } from '../domain/ports/telegram-service';
import { DrizzleBucketRepository } from './persistence/repositories/bucket-repository';
import { DrizzleFilePartRepository } from './persistence/repositories/file-part-repository';
import { DrizzleFileRepository } from './persistence/repositories/file-repository';
import { DrizzleMultipartRepository } from './persistence/repositories/multipart-repository';
import { botPool } from './telegram/bot-pool';
import { ChunkedStorage } from './telegram/chunked-storage';
// ─── Repository Singletons ──────────────────────────────────────────
/** Singleton IFileRepository instance backed by Drizzle ORM. */
export const fileRepository: IFileRepository = new DrizzleFileRepository();
/** Singleton IBucketRepository instance backed by Drizzle ORM. */
export const bucketRepository: IBucketRepository = new DrizzleBucketRepository();
/** Singleton IFilePartRepository instance backed by Drizzle ORM. */
export const filePartRepository: IFilePartRepository = new DrizzleFilePartRepository();
/** Singleton IMultipartRepository instance backed by Drizzle ORM. */
export const multipartRepository: IMultipartRepository = new DrizzleMultipartRepository();
/** Singleton ITelegramService instance backed by the bot pool. */
export const telegramService: ITelegramService = botPool;
// ─── Service Singletons ─────────────────────────────────────────────
/** Singleton ChunkedStorage for large file chunked uploads. */
export const chunkedStorage = new ChunkedStorage(
fileRepository,
filePartRepository,
telegramService,
);
@@ -8,6 +8,9 @@ const client = postgres(process.env.DATABASE_URL!, {
connect_timeout: 10, connect_timeout: 10,
}); });
/** Drizzle ORM database client initialized with the files and fileParts schema. */
export const db = drizzle(client, { schema: { fileParts, files } }); export const db = drizzle(client, { schema: { fileParts, files } });
export { fileParts, files }; export { fileParts, files };
export default db; export default db;
@@ -1,21 +1,24 @@
import postgres from 'postgres'; import postgres from 'postgres';
import { config } from '../env'; import { config } from '../../../env';
import { getErrorMessage } from '../utils/file'; import logger from '../../../shared/logger/index';
import logger from '../utils/logger'; import { getErrorMessage } from '../../../shared/utils/file';
/** /**
* Run raw SQL migration from schema.sql. * Run raw SQL migration from schema.sql.
* Safe to call multiple times all statements use IF NOT EXISTS. * Safe to call multiple times all statements use IF NOT EXISTS.
* Searches multiple relative paths to support execution from compiled dist,
* bun --hot, or direct script invocation.
*/ */
export const runMigration = async (): Promise<void> => { export const runMigration = async (): Promise<void> => {
// In compiled dist: import.meta.dir = .../dist/ // In compiled dist: import.meta.dir = .../dist/infrastructure/persistence/drizzle/
// In source via bun --hot: import.meta.dir = .../src/db/ // In source via bun --hot: import.meta.dir = .../src/infrastructure/persistence/drizzle/
const dir = import.meta.dir || ''; const dir = import.meta.dir || '';
const candidates = [ const candidates = [
`${dir}/../../schema.sql`, // from dist/ `${dir}/../../../../schema.sql`, // from dist/
`${dir}/../schema.sql`, // from src/ (bun --hot src/index.ts) `${dir}/../../../schema.sql`, // from src/infrastructure/persistence/
`${dir}/../schema.sql`, // from src/db/ (bun --hot src/db/migrate.ts) `${dir}/../../schema.sql`, // from src/infrastructure/
`${dir}/schema.sql`, // from src/ (bun run db:migrate) `${dir}/../schema.sql`, // from src/infrastructure/persistence/drizzle/
`${dir}/schema.sql`, // from next to file (bun run directly)
]; ];
let schemaSql: string | null = null; let schemaSql: string | null = null;
@@ -47,7 +50,7 @@ export const runMigration = async (): Promise<void> => {
} }
}; };
// When run directly: `bun src/db/migrate.ts` or `bun dist/migrate.js` // When run directly: `bun src/infrastructure/persistence/drizzle/migrate.ts`
if (import.meta.path === Bun.main) { if (import.meta.path === Bun.main) {
await runMigration(); await runMigration();
} }
@@ -1,4 +1,3 @@
import type { InferInsertModel, InferSelectModel } from 'drizzle-orm';
import { import {
bigint, bigint,
boolean, boolean,
@@ -10,6 +9,11 @@ import {
uuid, uuid,
} from 'drizzle-orm/pg-core'; } from 'drizzle-orm/pg-core';
/**
* Files table definition.
* Stores metadata about uploaded files including Telegram storage references,
* S3 bucket information, multipart upload tracking, and archive entries.
*/
export const files = pgTable('files', { export const files = pgTable('files', {
id: uuid('id').primaryKey().defaultRandom(), id: uuid('id').primaryKey().defaultRandom(),
publicId: text('public_id').unique().notNull(), publicId: text('public_id').unique().notNull(),
@@ -39,6 +43,11 @@ export const files = pgTable('files', {
updatedAt: timestamp('updated_at').defaultNow().notNull(), updatedAt: timestamp('updated_at').defaultNow().notNull(),
}); });
/**
* File parts table definition.
* Stores chunks of multipart uploads with per-part Telegram storage references
* and compression metadata.
*/
export const fileParts = pgTable('file_parts', { export const fileParts = pgTable('file_parts', {
id: serial('id').primaryKey(), id: serial('id').primaryKey(),
fileId: uuid('file_id').notNull(), fileId: uuid('file_id').notNull(),
@@ -53,8 +62,3 @@ export const fileParts = pgTable('file_parts', {
etag: text('etag').notNull(), etag: text('etag').notNull(),
createdAt: timestamp('created_at').defaultNow().notNull(), createdAt: timestamp('created_at').defaultNow().notNull(),
}); });
export type File = InferSelectModel<typeof files>;
export type NewFile = InferInsertModel<typeof files>;
export type FilePart = InferSelectModel<typeof fileParts>;
export type NewFilePart = InferInsertModel<typeof fileParts>;
@@ -0,0 +1,100 @@
import { sql } from 'drizzle-orm';
import type { Bucket } from '../../../domain/entities/bucket';
import type { IBucketRepository } from '../../../domain/ports/bucket-repository';
import { db } from '../drizzle/index';
/** Raw result row from `db.execute()`. */
type QueryRow = Record<string, unknown>;
/** Array of raw result rows. */
type QueryResult = QueryRow[];
/**
* Maps a raw database row to a {@link Bucket} domain entity.
*/
const mapRowToBucket = (row: Record<string, unknown>): Bucket => ({
id: row.id as string,
name: row.name as string,
createdAt: new Date(row.created_at as string),
updatedAt: new Date(row.updated_at as string),
});
/**
* Drizzle-backed implementation of {@link IBucketRepository}.
*
* Delegates to the same SQL queries as the original `src/db/buckets.ts`
* module, using raw SQL for drizzle tables that are not part of the
* typed schema.
*/
export class DrizzleBucketRepository implements IBucketRepository {
/**
* {@inheritDoc IBucketRepository.create}
*/
async create(name: string): Promise<Bucket> {
const result = (await db.execute(
sql`INSERT INTO buckets (name) VALUES (${name}) RETURNING id, name, created_at, updated_at`,
)) as unknown as QueryResult;
return mapRowToBucket(result[0]!);
}
/**
* {@inheritDoc IBucketRepository.findByName}
*/
async findByName(name: string): Promise<Bucket | null> {
const result = (await db.execute(
sql`SELECT id, name, created_at, updated_at FROM buckets WHERE name = ${name}`,
)) as unknown as QueryResult;
if (result.length === 0) return null;
return mapRowToBucket(result[0]!);
}
/**
* {@inheritDoc IBucketRepository.list}
*/
async list(): Promise<Bucket[]> {
const result = (await db.execute(
sql`SELECT id, name, created_at, updated_at FROM buckets ORDER BY name`,
)) as unknown as QueryResult;
return result.map(mapRowToBucket);
}
/**
* {@inheritDoc IBucketRepository.delete}
*
* Cascade-deletes multipart and file rows that hold foreign-key
* references to the bucket before deleting the bucket itself.
* Failures during cascade are silently caught to match the original
* defensive-cleanup behaviour.
*/
async delete(name: string): Promise<boolean> {
// Cascade-delete rows that hold FK references to the bucket
await db
.execute(
sql`DELETE FROM multipart_parts WHERE upload_id IN (SELECT upload_id FROM multipart_uploads WHERE bucket_id IN (SELECT id FROM buckets WHERE name = ${name}))`,
)
.catch(() => {});
await db
.execute(
sql`DELETE FROM multipart_uploads WHERE bucket_id IN (SELECT id FROM buckets WHERE name = ${name})`,
)
.catch(() => {});
await db
.execute(
sql`DELETE FROM files WHERE bucket_id IN (SELECT id FROM buckets WHERE name = ${name})`,
)
.catch(() => {});
const result = (await db.execute(
sql`DELETE FROM buckets WHERE name = ${name}`,
)) as unknown as QueryResult;
return result.length > 0;
}
/**
* {@inheritDoc IBucketRepository.exists}
*/
async exists(name: string): Promise<boolean> {
const result = (await db.execute(
sql`SELECT 1 FROM buckets WHERE name = ${name}`,
)) as unknown as QueryResult;
return result.length > 0;
}
}
@@ -0,0 +1,105 @@
import { sql } from 'drizzle-orm';
import type { FilePart, NewFilePart } from '../../../domain/entities/file-part';
import type { IFilePartRepository } from '../../../domain/ports/file-part-repository';
import { db } from '../drizzle/index';
/** Compression algorithm type matching the domain entity. */
type CompressionAlgorithm = 'gzip' | null;
/** Safely converts a raw value to a number, defaulting to 0. */
const toNumber = (value: unknown): number => Number(value ?? 0);
/**
* Maps a raw database row (snake_case keys) to a {@link FilePart}
* domain entity.
*/
const mapRowToFilePart = (row: Record<string, unknown>): FilePart => ({
id: toNumber(row.id),
fileId: row.file_id as string,
partNumber: toNumber(row.part_number),
telegramFileId: row.telegram_file_id as string,
telegramFileUniqueId: row.telegram_file_unique_id as string,
storageChatId: toNumber(row.storage_chat_id),
storageMessageId: toNumber(row.storage_message_id),
sizeBytes: toNumber(row.size_bytes),
storedSizeBytes: toNumber(row.stored_size_bytes),
compressionAlgorithm: (row.compression_algorithm as CompressionAlgorithm) || null,
etag: row.etag as string,
createdAt: new Date(row.created_at as string),
});
/**
* Drizzle-backed implementation of {@link IFilePartRepository}.
*
* Delegates to the same SQL queries as the original `src/db/file-parts.ts`
* module, using raw SQL for all operations.
*/
export class DrizzleFilePartRepository implements IFilePartRepository {
/**
* {@inheritDoc IFilePartRepository.insert}
*/
async insert(parts: NewFilePart[]): Promise<void> {
for (const part of parts) {
await db.execute(
sql`INSERT INTO file_parts (
file_id,
part_number,
telegram_file_id,
telegram_file_unique_id,
storage_chat_id,
storage_message_id,
size_bytes,
stored_size_bytes,
compression_algorithm,
etag
) VALUES (
${part.fileId}::uuid,
${part.partNumber},
${part.telegramFileId},
${part.telegramFileUniqueId},
${part.storageChatId},
${part.storageMessageId},
${part.sizeBytes},
${part.storedSizeBytes},
${part.compressionAlgorithm},
${part.etag}
)`,
);
}
}
/**
* {@inheritDoc IFilePartRepository.listByFileId}
*/
async listByFileId(fileId: string): Promise<FilePart[]> {
const result = (await db.execute(
sql`SELECT id,
file_id,
part_number,
telegram_file_id,
telegram_file_unique_id,
storage_chat_id,
storage_message_id,
size_bytes,
stored_size_bytes,
compression_algorithm,
etag,
created_at
FROM file_parts
WHERE file_id = ${fileId}::uuid
ORDER BY part_number`,
)) as unknown as Record<string, unknown>[];
return result.map(mapRowToFilePart);
}
/**
* {@inheritDoc IFilePartRepository.countByFileId}
*/
async countByFileId(fileId: string): Promise<number> {
const result = (await db.execute(
sql`SELECT COUNT(*) AS count FROM file_parts WHERE file_id = ${fileId}::uuid`,
)) as unknown as Record<string, unknown>[];
return toNumber(result[0]?.count);
}
}
@@ -0,0 +1,210 @@
import { and, eq, sql } from 'drizzle-orm';
import type { File, NewFile } from '../../../domain/entities/file';
import type { IFileRepository, S3FileRecord } from '../../../domain/ports/file-repository';
import { db, files as fileSchema } from '../drizzle/index';
/** Safely converts a raw value to a number, defaulting to 0. */
const toNumber = (value: unknown): number => Number(value ?? 0);
/**
* Escape special LIKE wildcard characters (`%`, `_`, `\`) so that
* a user-supplied prefix can be safely used in a LIKE expression.
*/
const escapeLike = (s: string): string => s.replace(/[%_\\]/g, '\\$&');
/**
* Maps a raw database row (snake_case keys) to an {@link S3FileRecord}
* domain entity. Used only when raw SQL via `db.execute()` returns
* un-typed result sets.
*/
const mapDbRowToS3Record = (row: Record<string, unknown>): S3FileRecord => ({
id: row.id as string,
publicId: row.public_id as string,
telegramFileId: row.telegram_file_id as string,
telegramFileUniqueId: row.telegram_file_unique_id as string,
storageChatId: toNumber(row.storage_chat_id),
storageMessageId: toNumber(row.storage_message_id),
fileName: row.file_name as string,
mimeType: row.mime_type as string,
sizeBytes: toNumber(row.size_bytes),
fileType: row.file_type as string,
uploaderId: toNumber(row.uploader_id),
fileHash: row.file_hash as string | null,
archiveTelegramFileId: row.archive_telegram_file_id as string | null,
archiveStorageMessageId:
row.archive_storage_message_id === null ? null : toNumber(row.archive_storage_message_id),
archiveFileName: row.archive_file_name as string | null,
archiveEntryName: row.archive_entry_name as string | null,
archiveMimeType: row.archive_mime_type as string | null,
archiveSizeBytes: row.archive_size_bytes === null ? null : toNumber(row.archive_size_bytes),
bucketId: row.bucket_id as string,
s3Key: row.s3_key as string,
storageBackend: (row.storage_backend as string) || 'telegram',
isDeleted: row.is_deleted as boolean,
multipartUploadId: row.multipart_upload_id as string | null,
partCount:
row.part_count === null || row.part_count === undefined ? null : toNumber(row.part_count),
createdAt: new Date(row.created_at as string),
updatedAt: new Date(row.updated_at as string),
});
/**
* Drizzle-backed implementation of {@link IFileRepository}.
*
* Delegates to the same SQL queries as the original `src/db/files.ts` and
* `src/db/files-ext.ts` modules while presenting a clean domain interface.
*/
export class DrizzleFileRepository implements IFileRepository {
/**
* {@inheritDoc IFileRepository.findByHash}
*/
async findByHash(hash: string): Promise<File | null> {
const result = await db.select().from(fileSchema).where(eq(fileSchema.fileHash, hash)).limit(1);
return result[0] || null;
}
/**
* {@inheritDoc IFileRepository.findByPublicId}
*/
async findByPublicId(publicId: string): Promise<File | null> {
const result = await db
.select()
.from(fileSchema)
.where(eq(fileSchema.publicId, publicId))
.limit(1);
return result[0] || null;
}
/**
* {@inheritDoc IFileRepository.findByUniqueId}
*/
async findByUniqueId(telegramFileUniqueId: string): Promise<File | null> {
const result = await db
.select()
.from(fileSchema)
.where(eq(fileSchema.telegramFileUniqueId, telegramFileUniqueId))
.limit(1);
return result[0] || null;
}
/**
* {@inheritDoc IFileRepository.findByBucketAndKey}
*/
async findByBucketAndKey(bucketId: string, s3Key: string): Promise<File | null> {
const result = await db
.select()
.from(fileSchema)
.where(
and(
eq(fileSchema.bucketId, bucketId),
eq(fileSchema.s3Key, s3Key),
eq(fileSchema.isDeleted, false),
),
)
.limit(1);
return result[0] || null;
}
/**
* {@inheritDoc IFileRepository.create}
*/
async create(file: NewFile): Promise<File> {
const result = await db.insert(fileSchema).values(file).returning();
return result[0]!;
}
/**
* {@inheritDoc IFileRepository.listByPrefix}
*/
async listByPrefix(
bucketId: string,
prefix: string,
delimiter: string | null,
maxKeys: number,
startAfter: string | null,
): Promise<{ objects: S3FileRecord[]; prefixes: string[] }> {
let query = prefix
? sql`SELECT * FROM files WHERE bucket_id = ${bucketId}::uuid AND is_deleted = false AND s3_key LIKE ${`${escapeLike(prefix)}%`}`
: sql`SELECT * FROM files WHERE bucket_id = ${bucketId}::uuid AND is_deleted = false`;
if (startAfter) {
query = sql`${query} AND s3_key > ${startAfter}`;
}
query = sql`${query} ORDER BY s3_key LIMIT ${maxKeys + 1}`;
const rawResult = (await db.execute(query)) as unknown as Record<string, unknown>[];
if (delimiter === '/') {
const prefixSet = new Set<string>();
const objects: S3FileRecord[] = [];
for (const row of rawResult) {
const s3Key = row.s3_key as string;
const relativeKey = s3Key.substring(prefix.length);
const slashIndex = relativeKey.indexOf('/');
if (slashIndex >= 0) {
const folderPrefix = prefix + relativeKey.substring(0, slashIndex + 1);
if (folderPrefix !== prefix) {
prefixSet.add(folderPrefix);
}
} else {
objects.push(mapDbRowToS3Record(row));
}
}
return {
objects: objects.slice(0, maxKeys),
prefixes: Array.from(prefixSet).sort(),
};
}
return {
objects: rawResult.slice(0, maxKeys).map(mapDbRowToS3Record),
prefixes: [],
};
}
/**
* {@inheritDoc IFileRepository.softDelete}
*/
async softDelete(bucketId: string, s3Key: string): Promise<boolean> {
const result = (await db.execute(
sql`UPDATE files SET is_deleted = true WHERE bucket_id = ${bucketId}::uuid AND s3_key = ${s3Key} RETURNING id`,
)) as unknown as Record<string, unknown>[];
return result.length > 0;
}
/**
* {@inheritDoc IFileRepository.softDeleteBatch}
*/
async softDeleteBatch(bucketId: string, keys: string[]): Promise<number> {
let deleted = 0;
for (const key of keys) {
const ok = await this.softDelete(bucketId, key);
if (ok) deleted++;
}
return deleted;
}
/**
* {@inheritDoc IFileRepository.countByBucket}
*/
async countByBucket(bucketId: string): Promise<number> {
const result = (await db.execute(
sql`SELECT count(*) as count FROM files WHERE bucket_id = ${bucketId}::uuid AND is_deleted = false`,
)) as unknown as Record<string, unknown>[];
return Number(result[0]?.count || 0);
}
/**
* {@inheritDoc IFileRepository.findOrphansByBucket}
*/
async findOrphansByBucket(bucketId: string): Promise<File[]> {
return await db
.select()
.from(fileSchema)
.where(and(eq(fileSchema.bucketId, bucketId), eq(fileSchema.isDeleted, true)))
.limit(100);
}
}
@@ -0,0 +1,140 @@
import { sql } from 'drizzle-orm';
import { nanoid } from 'nanoid';
import type { MultipartPart, MultipartUpload } from '../../../domain/entities/multipart';
import type { IMultipartRepository } from '../../../domain/ports/multipart-repository';
import { db } from '../drizzle/index';
/**
* Maps a raw database row to a {@link MultipartUpload} domain entity.
*/
const mapRowToMultipartUpload = (r: Record<string, unknown>): MultipartUpload => ({
uploadId: r.upload_id as string,
bucketId: r.bucket_id as string,
s3Key: r.s3_key as string,
initiatedAt: new Date(r.initiated_at as string),
status: r.status as string,
initiatedBy: (r.initiated_by as string | null) || '',
});
/**
* Drizzle-backed implementation of {@link IMultipartRepository}.
*
* Delegates to the same SQL queries as the original `src/db/multipart.ts`
* module, using raw SQL for all operations on the un-typed
* `multipart_uploads` and `multipart_parts` tables.
*/
export class DrizzleMultipartRepository implements IMultipartRepository {
/**
* {@inheritDoc IMultipartRepository.create}
*/
async create(bucketId: string, s3Key: string, initiatedBy: string): Promise<string> {
const uploadId = nanoid(32);
await db.execute(
sql`INSERT INTO multipart_uploads (upload_id, bucket_id, s3_key, initiated_by) VALUES (${uploadId}, ${bucketId}, ${s3Key}, ${initiatedBy})`,
);
return uploadId;
}
/**
* {@inheritDoc IMultipartRepository.findById}
*/
async findById(uploadId: string): Promise<MultipartUpload | null> {
const result = (await db.execute(
sql`SELECT upload_id, bucket_id, s3_key, initiated_at, status FROM multipart_uploads WHERE upload_id = ${uploadId} AND status = 'in_progress'`,
)) as unknown as Record<string, unknown>[];
if (result.length === 0) return null;
const r = result[0]!;
return {
uploadId: r.upload_id as string,
bucketId: r.bucket_id as string,
s3Key: r.s3_key as string,
initiatedAt: new Date(r.initiated_at as string),
status: r.status as string,
initiatedBy: '',
};
}
/**
* {@inheritDoc IMultipartRepository.complete}
*/
async complete(uploadId: string): Promise<void> {
await db.execute(
sql`UPDATE multipart_uploads SET status = 'completed' WHERE upload_id = ${uploadId}`,
);
}
/**
* {@inheritDoc IMultipartRepository.abort}
*/
async abort(uploadId: string): Promise<void> {
await db.execute(
sql`UPDATE multipart_uploads SET status = 'aborted' WHERE upload_id = ${uploadId}`,
);
}
/**
* {@inheritDoc IMultipartRepository.insertPart}
*/
async insertPart(part: Omit<MultipartPart, 'id' | 'createdAt'>): Promise<void> {
await db.execute(
sql`INSERT INTO multipart_parts (upload_id, part_number, telegram_file_id, telegram_file_unique_id, storage_message_id, size_bytes, etag)
VALUES (${part.uploadId}, ${part.partNumber}, ${part.telegramFileId}, ${part.telegramFileUniqueId}, ${part.storageMessageId}, ${part.sizeBytes}, ${part.etag})`,
);
}
/**
* {@inheritDoc IMultipartRepository.listParts}
*/
async listParts(uploadId: string): Promise<MultipartPart[]> {
const result = (await db.execute(
sql`SELECT id, upload_id, part_number, telegram_file_id, telegram_file_unique_id, storage_message_id, size_bytes, etag, created_at
FROM multipart_parts WHERE upload_id = ${uploadId} ORDER BY part_number`,
)) as unknown as Record<string, unknown>[];
return result.map((r) => ({
id: r.id as number,
uploadId: r.upload_id as string,
partNumber: r.part_number as number,
telegramFileId: r.telegram_file_id as string,
telegramFileUniqueId: r.telegram_file_unique_id as string,
storageMessageId: r.storage_message_id as number,
sizeBytes: Number(r.size_bytes),
etag: r.etag as string,
createdAt: new Date(r.created_at as string),
}));
}
/**
* {@inheritDoc IMultipartRepository.listByBucket}
*/
async listByBucket(
bucketId: string,
maxUploads: number,
keyMarker: string | null,
): Promise<{
uploads: MultipartUpload[];
isTruncated: boolean;
nextKeyMarker: string | null;
}> {
const limit = Math.min(Math.max(maxUploads || 1000, 1), 1000);
const result = (await db.execute(
keyMarker
? sql`SELECT upload_id, bucket_id, s3_key, initiated_at, status, initiated_by
FROM multipart_uploads
WHERE bucket_id = ${bucketId}::uuid AND status = 'in_progress' AND s3_key > ${keyMarker}
ORDER BY s3_key, initiated_at
LIMIT ${limit + 1}`
: sql`SELECT upload_id, bucket_id, s3_key, initiated_at, status, initiated_by
FROM multipart_uploads
WHERE bucket_id = ${bucketId}::uuid AND status = 'in_progress'
ORDER BY s3_key, initiated_at
LIMIT ${limit + 1}`,
)) as unknown as Record<string, unknown>[];
const uploads = result.slice(0, limit).map(mapRowToMultipartUpload);
return {
uploads,
isTruncated: result.length > limit,
nextKeyMarker: result.length > limit ? uploads.at(-1)?.s3Key || null : null,
};
}
}
+303
View File
@@ -0,0 +1,303 @@
import PQueue from 'p-queue';
import { Telegraf } from 'telegraf';
import type {
ForwardResult,
ITelegramService,
TelegramFileInfo,
} from '../../domain/ports/telegram-service';
import { config } from '../../env';
import logger from '../../shared/logger/index';
import {
buildSendPayload,
extractUploadedFile,
type SendMethod,
sendMethodMap,
type TelegramMessageResult,
} from './types';
const sleep = (ms: number): Promise<void> => new Promise((resolve) => setTimeout(resolve, ms));
const isTransientError = (error: unknown): boolean => {
const str = error instanceof Error ? error.message : String(error);
const transientPatterns = [
'timeout',
'Timed out',
'etimedout',
'econnrefused',
'econnreset',
'ECONNREFUSED',
'ECONNRESET',
'ETIMEDOUT',
'5xx',
'502',
'503',
'504',
'Bad Gateway',
'Service Unavailable',
'Gateway Timeout',
'socket hang up',
'socket closed',
'fetch failed',
'network error',
'network timeout',
'API closed',
'read ECONNRESET',
'write EPIPE',
];
return transientPatterns.some((p) => str.toLowerCase().includes(p.toLowerCase()));
};
const MAX_TRANSIENT_RETRIES = 3;
const MAX_OUTER_RETRIES = 10;
const TELEGRAM_API_TIMEOUT_MS = 120_000;
interface BotEntry {
index: number;
token: string;
instance: Telegraf;
queue: PQueue;
rateLimitedUntil: number; // 0 = not rate-limited
}
export class BotPool implements ITelegramService {
private readonly bots: BotEntry[] = [];
constructor() {
const tokens = Array.from(new Set(config.botTokens));
this.bots = tokens.map((token, index) => ({
index,
token,
instance: new Telegraf(token),
queue: new PQueue({ concurrency: config.telegramBotConcurrency }),
rateLimitedUntil: 0,
}));
}
/** Number of bots in the pool */
get size(): number {
return this.bots.length;
}
/**
* Select the bot with the fewest pending tasks that isn't rate-limited
* or in the skip set.
*/
private selectBot(skipIndexes?: Set<number>): BotEntry | null {
if (this.bots.length === 0) return null;
let best: BotEntry | null = null;
let bestPending = Infinity;
for (const bot of this.bots) {
if (skipIndexes?.has(bot.index)) continue;
if (bot.rateLimitedUntil > Date.now()) continue;
const pending = bot.queue.pending + bot.queue.size;
if (pending < bestPending) {
bestPending = pending;
best = bot;
}
}
return best;
}
/**
* Execute a Telegram API action on a specific bot entry.
* Wraps with timeout.
*/
private async executeBotAction<T>(
bot: BotEntry,
action: (instance: Telegraf, token: string) => Promise<T>,
): Promise<T> {
return Promise.race([
action(bot.instance, bot.token),
new Promise<never>((_, reject) =>
setTimeout(
() => reject(new Error(`Telegram API timeout after ${TELEGRAM_API_TIMEOUT_MS}ms`)),
TELEGRAM_API_TIMEOUT_MS,
),
),
]);
}
/**
* Forward a file chunk to the configured Telegram storage chat.
*
* The upload is submitted to the least-loaded bot's queue. If the bot
* returns 429, it is marked rate-limited and the upload retries on the
* next available bot. If all bots are rate-limited, sleeps before retrying.
*/
async forwardToStorage(
fileChunk: unknown,
fileName: string,
fileType: string,
): Promise<ForwardResult> {
let lastError: unknown;
const attemptedIndexes = new Set<number>();
let transientAttempts = 0;
// Outer retry loop — up to MAX_OUTER_RETRIES attempts across all bots
for (let attempt = 0; attempt < MAX_OUTER_RETRIES; attempt++) {
const bot = this.selectBot(attemptedIndexes);
if (!bot) {
// No available bots — either all rate-limited or all attempted
if (attemptedIndexes.size > 0) {
// All non-rate-limited bots were tried and failed — wait & reset
logger.warn('All available bots exhausted, sleeping 5s before retry');
await sleep(5000 + Math.random() * 1000);
attemptedIndexes.clear();
continue;
}
// All bots rate-limited — wait for the shortest cooldown
const earliestCooldown = Math.min(...this.bots.map((b) => b.rateLimitedUntil || Infinity));
const waitMs = Math.max(1000, earliestCooldown - Date.now() + 500);
logger.warn('All bots rate-limited, waiting', { waitMs });
await sleep(waitMs);
attemptedIndexes.clear();
continue;
}
attemptedIndexes.add(bot.index);
try {
const result = await bot.queue.add(async () => {
// Inner transient retry loop inside the queue
for (let innerRetry = 0; innerRetry <= MAX_TRANSIENT_RETRIES; innerRetry++) {
try {
const filePayload = { source: fileChunk, filename: fileName };
const sendMethodName = sendMethodMap[fileType] || 'sendDocument';
const payload = buildSendPayload(fileType, fileName);
const tgResult = await this.executeBotAction<TelegramMessageResult>(
bot,
(activeBot) => {
const telegram = activeBot.telegram as unknown as Record<string, SendMethod>;
return telegram[sendMethodName](config.storageChatId, filePayload, payload);
},
);
const uploadedFile = extractUploadedFile(tgResult, fileType);
return {
telegramFileId: uploadedFile?.file_id || '',
telegramFileUniqueId: uploadedFile?.file_unique_id || '',
storageMessageId: tgResult.message_id,
};
} catch (error: unknown) {
const errorStr = error instanceof Error ? error.message : String(error);
const retryAfterMatch = errorStr.match(/retry after (\d+)/i);
if (retryAfterMatch) {
// 429 — mark bot rate-limited, throw to outer loop for retry on different bot
const seconds = parseInt(retryAfterMatch[1], 10);
bot.rateLimitedUntil = Date.now() + seconds * 1000;
logger.info(`Bot #${bot.index} rate-limited for ${seconds}s`, {
fileName,
attempt,
});
throw error; // caught by outer retry loop
}
if (innerRetry < MAX_TRANSIENT_RETRIES && isTransientError(error)) {
const backoffMs = Math.min(1000 * 2 ** innerRetry, 10_000);
logger.warn(
`Transient error on bot #${bot.index}, retrying (${innerRetry + 1}/${MAX_TRANSIENT_RETRIES})`,
{ fileName, error: errorStr, backoffMs },
);
await sleep(backoffMs);
continue;
}
throw error; // non-transient — propagate
}
}
throw new Error(`Exhausted transient retries on bot #${bot.index}`);
});
logger.info('File forwarded to storage', { fileName, message: result.storageMessageId });
return result;
} catch (error: unknown) {
lastError = error;
const errorStr = error instanceof Error ? error.message : String(error);
const retryAfterMatch = errorStr.match(/retry after (\d+)/i);
if (retryAfterMatch) {
// 429 catch in outer block: serves as a safety net for errors that
// contain "retry after N" wording but were rethrown from the inner
// queue task's fallback path (e.g., non-429 errors with similar text).
logger.warn('Retry-after pattern caught in outer loop (safety net)', {
fileName,
error: errorStr,
});
continue;
}
// Transient error at the queue level — retry on next bot
if (transientAttempts < MAX_TRANSIENT_RETRIES && isTransientError(error)) {
transientAttempts++;
continue;
}
// Non-transient — give up
logger.error('Failed to forward file to storage', {
fileName,
error: errorStr,
attempt,
});
throw error;
}
}
throw lastError || new Error('Failed to forward file after all retries');
}
/** Get total effective concurrency across all bots */
getEffectiveConcurrency(): number {
return this.bots.length * config.telegramBotConcurrency;
}
async getFileInfo(telegramFileId: string): Promise<TelegramFileInfo> {
let lastError: unknown;
for (const bot of this.bots) {
for (let retry = 0; retry <= MAX_TRANSIENT_RETRIES; retry++) {
try {
const result = await bot.instance.telegram.getFile(telegramFileId);
const fileData = result as unknown as Omit<TelegramFileInfo, 'bot_token'>;
return {
file_size: fileData.file_size || 0,
mime_type: fileData.mime_type || 'application/octet-stream',
file_path: fileData.file_path || '',
bot_token: bot.token,
};
} catch (error: unknown) {
lastError = error;
const errorStr = error instanceof Error ? error.message : String(error);
if (
errorStr.includes('wrong file_id') ||
errorStr.includes('file is temporarily unavailable')
) {
break;
}
if (retry < MAX_TRANSIENT_RETRIES && isTransientError(error)) {
const backoffMs = Math.min(1000 * 2 ** (retry + 1), 5_000);
logger.warn(
`Transient error getting file info, retrying bot ${bot.token.slice(0, 8)}... (${retry + 1}/${MAX_TRANSIENT_RETRIES})`,
{ telegramFileId, error: errorStr, backoffMs },
);
await sleep(backoffMs);
continue;
}
break;
}
}
}
logger.error('Failed to get file info from any bot', {
error: lastError instanceof Error ? lastError.message : String(lastError),
});
throw lastError;
}
}
export const botPool = new BotPool();
@@ -0,0 +1,282 @@
import { createReadStream } from 'node:fs';
import { nanoid } from 'nanoid';
import type { File as FileEntity } from '../../domain/entities/file';
import { buildNewFile } from '../../domain/entities/file-factory';
import type { NewFilePart } from '../../domain/entities/file-part';
import type { IFilePartRepository } from '../../domain/ports/file-part-repository';
import type { IFileRepository } from '../../domain/ports/file-repository';
import type { ITelegramService } from '../../domain/ports/telegram-service';
import { config } from '../../env';
import { createGetObjectResponse, type ObjectPartSource } from '../../interfaces/s3/object-stream';
import type { RangeParseResult } from '../../interfaces/s3/range';
import { type CompressionAlgorithm, maybeCompressChunk } from '../../shared/utils/compress';
import { computeHash } from '../../shared/utils/file';
import { asSafeChunkSize } from '../../shared/utils/validation';
/**
* Metadata about a single uploaded chunk (part) stored in Telegram.
*/
export interface ChunkedUploadPart {
/** 1-based part number within the file */
partNumber: number;
/** Telegram file_id for retrieving this part */
telegramFileId: string;
/** Telegram unique file_id (stable across bot tokens) */
telegramFileUniqueId: string;
/** Message ID within the storage chat */
storageMessageId: number;
/** Original (pre-compression) size in bytes */
sizeBytes: number;
/** Stored (post-compression) size in bytes */
storedSizeBytes: number;
/** Compression algorithm applied, or null */
compressionAlgorithm: CompressionAlgorithm;
/** ETag (SHA-256 hash) of the original chunk */
etag: string;
}
/**
* Result of uploading a file in Telegram chunks.
*/
export interface ChunkedUploadResult {
/** Ordered list of uploaded parts */
parts: ChunkedUploadPart[];
/** SHA-256 hash of the complete file content */
fileHash: string;
/** Total file size in bytes */
totalSizeBytes: number;
}
/**
* Input parameters for storing a file via chunked Telegram uploads.
*/
export interface ChunkedFileInput {
/** Path to the temporary file on disk */
tempPath: string;
/** Prefix for generated part file names */
partFileNamePrefix: string;
/** Original file name */
fileName: string;
/** MIME type of the file */
mimeType: string;
/** File size in bytes */
sizeBytes: number;
/** File type classification (e.g. "document", "video") */
fileType: string;
/** Telegram user ID of the uploader */
uploaderId: number;
/** S3 bucket ID if the file is also tracked in S3, or null */
bucketId?: string | null;
/** S3 object key if the file is also tracked in S3, or null */
s3Key?: string | null;
}
/**
* Manages chunked storage of large files in Telegram.
*
* Large files are split into smaller chunks, each uploaded as a separate
* Telegram document. File and part metadata is persisted through the
* provided repository interfaces.
*
* Injects dependencies via constructor can be used with any
* {@link IFileRepository}, {@link IFilePartRepository}, and
* {@link ITelegramService} implementation.
*/
export class ChunkedStorage {
/**
* @param fileRepository - Repository for File entity persistence.
* @param filePartRepository - Repository for FilePart entity persistence.
* @param telegramService - Service for Telegram API interactions.
*/
constructor(
private readonly fileRepository: IFileRepository,
private readonly filePartRepository: IFilePartRepository,
private readonly telegramService: ITelegramService,
) {}
/**
* Upload a file to Telegram in chunks and return chunk metadata.
*
* Reads the file from disk in fixed-size chunks, compresses each chunk
* if beneficial, and forwards each chunk to Telegram storage.
*
* @param input - Upload parameters including temp path, chunk size, and compression settings.
* @returns Metadata about all uploaded chunks and the file hash.
*/
async uploadFileInTelegramChunks(input: {
tempPath: string;
partFileNamePrefix: string;
chunkSizeBytes: number;
compress: boolean;
compressionMinSizeBytes: number;
}): Promise<ChunkedUploadResult> {
const chunkSizeBytes = asSafeChunkSize(input.chunkSizeBytes);
const hasher = new Bun.CryptoHasher('sha256');
const parts: ChunkedUploadPart[] = [];
let totalSizeBytes = 0;
let partNumber = 0;
const stream = createReadStream(input.tempPath, { highWaterMark: chunkSizeBytes });
for await (const data of stream) {
const chunk = Buffer.isBuffer(data) ? data : Buffer.from(data as Uint8Array);
if (chunk.byteLength === 0) continue;
partNumber += 1;
totalSizeBytes += chunk.byteLength;
hasher.update(chunk);
const { bytes, compressionAlgorithm } = maybeCompressChunk(
chunk,
input.compress,
input.compressionMinSizeBytes,
);
const forwardResult = await this.telegramService.forwardToStorage(
bytes,
`${input.partFileNamePrefix}.part-${partNumber}`,
'document',
);
parts.push({
partNumber,
telegramFileId: forwardResult.telegramFileId,
telegramFileUniqueId: forwardResult.telegramFileUniqueId,
storageMessageId: forwardResult.storageMessageId,
sizeBytes: chunk.byteLength,
storedSizeBytes: bytes.byteLength,
compressionAlgorithm,
etag: computeHash(chunk),
});
}
return {
parts,
fileHash: hasher.digest('hex'),
totalSizeBytes,
};
}
/**
* Upload a file to Telegram in chunks and persist file + part records.
*
* Combines chunk upload ({@link uploadFileInTelegramChunks}) with
* repository persistence for both the File and FilePart entities.
*
* @param input - The file metadata and upload parameters.
* @returns The persisted File entity.
*/
async storeFileInTelegramChunks(input: ChunkedFileInput): Promise<FileEntity> {
const upload = await this.uploadFileInTelegramChunks({
tempPath: input.tempPath,
partFileNamePrefix: input.partFileNamePrefix,
chunkSizeBytes: config.telegramChunkSizeBytes,
compress: config.compressChunkedUploads,
compressionMinSizeBytes: config.chunkCompressionMinSizeBytes,
});
const firstPart = upload.parts[0];
if (!firstPart) {
throw new Error('Chunked upload produced no parts');
}
const publicId = nanoid();
const file = await this.fileRepository.create(
buildNewFile({
publicId,
telegramFileId: firstPart.telegramFileId,
telegramFileUniqueId: firstPart.telegramFileUniqueId,
storageChatId: config.storageChatId,
storageMessageId: firstPart.storageMessageId,
fileName: input.fileName,
mimeType: input.mimeType,
sizeBytes: upload.totalSizeBytes,
fileType: input.fileType,
storageBackend: 'chunked',
uploaderId: input.uploaderId,
fileHash: upload.fileHash,
bucketId: input.bucketId,
s3Key: input.s3Key,
partCount: upload.parts.length,
}),
);
const fileParts: NewFilePart[] = upload.parts.map((part) => ({
fileId: file.id,
partNumber: part.partNumber,
telegramFileId: part.telegramFileId,
telegramFileUniqueId: part.telegramFileUniqueId,
storageChatId: config.storageChatId,
storageMessageId: part.storageMessageId,
sizeBytes: part.sizeBytes,
storedSizeBytes: part.storedSizeBytes,
compressionAlgorithm: part.compressionAlgorithm,
etag: part.etag,
}));
await this.filePartRepository.insert(fileParts);
return file;
}
/**
* Build a list of object-part sources for reconstructing a chunked file.
*
* Queries the file-part repository and enriches each part with
* the Telegram download URL by calling {@link ITelegramService.getFileInfo}.
*
* @param file - The File entity whose parts should be resolved.
* @returns An ordered list of object part sources ready for streaming.
*/
async buildChunkedObjectSources(file: FileEntity): Promise<ObjectPartSource[]> {
const parts = await this.filePartRepository.listByFileId(file.id);
const sources: ObjectPartSource[] = [];
for (const part of parts) {
const fileInfo = await this.telegramService.getFileInfo(part.telegramFileId);
sources.push({
telegramFileId: part.telegramFileId,
telegramUrl: `https://api.telegram.org/file/bot${fileInfo.bot_token}/${fileInfo.file_path}`,
sizeBytes: part.sizeBytes,
storedSizeBytes: part.storedSizeBytes,
compressionAlgorithm: part.compressionAlgorithm,
partNumber: part.partNumber,
});
}
return sources;
}
/**
* Create an HTTP Response that streams a chunked file's content.
*
* Supports HTTP range requests for partial content delivery.
* The response is constructed by reassembling parts in order and
* optionally decompressing gzip-compressed parts.
*
* @param input - Parameters including the file entity, range, and request ID.
* @returns A Response object streaming the requested byte range.
*/
async createChunkedObjectResponse(input: {
file: FileEntity;
range: RangeParseResult;
reqId: string;
}): Promise<Response> {
const parts = await this.buildChunkedObjectSources(input.file);
if (parts.length === 0) {
throw new Error('Chunked object has no parts');
}
return createGetObjectResponse({
reqId: input.reqId,
contentType: input.file.mimeType,
etag: input.file.fileHash || parts.map((p) => p.telegramFileId).join('-'),
lastModified:
input.file.createdAt instanceof Date
? input.file.createdAt
: new Date(input.file.createdAt),
totalSize: Number(input.file.sizeBytes),
parts,
range: input.range,
});
}
}
+130
View File
@@ -0,0 +1,130 @@
/**
* Telegram-specific types used internally by the infrastructure layer.
*
* These types represent the raw Telegram Bot API response shapes and
* the internal abstractions built on top of them. The higher-level domain
* types (ForwardResult, TelegramFileInfo) are defined in
* src/domain/ports/telegram-service.ts.
*/
/**
* File reference within a Telegram message result.
* Contains identifiers returned by the Telegram API for uploaded media.
*/
export interface UploadedTelegramFile {
/** Telegram file_id for retrieving the file */
file_id?: string;
/** Telegram unique file_id (stable across bot tokens) */
file_unique_id?: string;
}
/**
* Result structure returned by Telegram send* API methods.
* Covers all media types a Telegram message can carry.
*/
export interface TelegramMessageResult {
/** Unique message identifier inside the chat */
message_id: number;
/** Sent document, if applicable */
document?: UploadedTelegramFile;
/** Sent photo (array of sizes, last element is largest), if applicable */
photo?: UploadedTelegramFile[];
/** Sent video, if applicable */
video?: UploadedTelegramFile;
/** Sent audio, if applicable */
audio?: UploadedTelegramFile;
/** Sent voice message, if applicable */
voice?: UploadedTelegramFile;
/** Sent animation (GIF), if applicable */
animation?: UploadedTelegramFile;
/** Sent sticker, if applicable */
sticker?: UploadedTelegramFile;
/** Sent video note, if applicable */
video_note?: UploadedTelegramFile;
/** Catch-all for any additional Telegram response fields */
[key: string]: unknown;
}
/**
* Payload structure for sending a file via the Telegram Bot API.
*
* @internal
*/
export type FilePayload = { source: unknown; filename: string };
/**
* Additional optional payload for Telegram send method calls.
*
* @internal
*/
export type SendPayload = { caption?: string };
/**
* Function signature for Telegram send* method calls on a bot instance.
*
* @internal
*/
export type SendMethod = (
chatId: number,
filePayload: FilePayload,
payload?: SendPayload,
) => Promise<TelegramMessageResult>;
/**
* Mapping from file type identifier to Telegram Bot API method name.
*
* Each key corresponds to a Telegram media type; the value is the
* method name to call on `bot.telegram`.
*/
export const sendMethodMap: Record<string, string> = {
photo: 'sendPhoto',
audio: 'sendAudio',
video: 'sendVideo',
voice: 'sendVoice',
animation: 'sendAnimation',
sticker: 'sendSticker',
document: 'sendDocument',
video_note: 'sendDocument',
};
/**
* Extract the uploaded file reference from a Telegram message result
* based on the media type present in the result.
*
* Falls back to looking up the file type key directly on the result object.
*
* @param result - The message result from a Telegram send* call.
* @param fileType - The file type classification (e.g. "document", "photo").
* @returns The uploaded file reference, or `undefined` if none was found.
*/
export const extractUploadedFile = (
result: TelegramMessageResult,
fileType: string,
): UploadedTelegramFile | undefined => {
if (result.document) return result.document;
if (result.photo) return result.photo?.slice(-1)[0];
if (result.video) return result.video;
if (result.audio) return result.audio;
if (result.voice) return result.voice;
if (result.animation) return result.animation;
if (result.sticker) return result.sticker;
if (result.video_note) return result.video_note;
return result[fileType] as UploadedTelegramFile | undefined;
};
/**
* Build the send payload (caption, etc.) for a Telegram send* method call.
*
* Stickers do not support captions. Documents get a labelled caption
* with the file name. All other types use the plain file name as caption.
*
* @param fileType - The file type (e.g. "document", "photo", "sticker").
* @param fileName - The file name to use in the caption.
* @returns The payload object with caption (or empty for sticker).
*/
export const buildSendPayload = (fileType: string, fileName: string): SendPayload => {
const basePayload: SendPayload = { caption: fileName };
if (fileType === 'sticker') return {};
if (fileType === 'document') return { caption: `📁 ${fileName}` };
return basePayload;
};
+198
View File
@@ -0,0 +1,198 @@
import { nanoid } from 'nanoid';
import { type Context, Telegraf } from 'telegraf';
import { buildNewFile } from '../../domain/entities/file-factory';
import type { IFileRepository } from '../../domain/ports/file-repository';
import type { ITelegramService } from '../../domain/ports/telegram-service';
import { config } from '../../env';
import { DrizzleFileRepository } from '../../infrastructure/persistence/repositories/file-repository';
import { botPool } from '../../infrastructure/telegram/bot-pool';
import logger from '../../shared/logger/index';
import {
checkFileSize,
detectFileType,
extractFileFromMessage,
getErrorMessage,
getFileSizeLimit,
type TelegramMediaMessage,
} from '../../shared/utils/file';
/**
* Minimal bot context shape used by the media event handler.
*
* Represents the subset of Telegraf's Context that the handler requires
* for processing incoming media messages.
*/
type BotContext = {
/** The incoming media message with file attachments. */
message: TelegramMediaMessage;
/** The sender of the message. */
from: { id: number };
/** The chat where the message was sent, if available. */
chat?: { id: number };
/**
* Reply to the message with text.
*
* @param text - The reply text.
* @param extra - Optional reply parameters (e.g. reply_parameters for threading).
*/
reply: (text: string, extra?: { reply_parameters: { message_id: number } }) => Promise<unknown>;
};
/**
* Duck-typed object that exposes a Telegraf-style `on()` method
* for registering event handlers on multiple event types.
*/
type MediaEventRegistrar = {
/**
* Register a handler for the given event types.
*
* @param events - Array of event type strings (e.g. "document", "photo").
* @param handler - Async handler receiving the bot context.
*/
on: (events: string[], handler: (ctx: BotContext) => Promise<unknown>) => void;
};
/**
* Replies to a Telegram message with a download URL for the uploaded file.
*
* @param ctx - The bot context for the incoming message.
* @param publicId - The public identifier of the uploaded file.
*/
const replyWithDownloadUrl = async (ctx: BotContext, publicId: string): Promise<void> => {
const url = `${config.baseUrl}/f/${publicId}`;
await ctx.reply(`File berhasil diupload! 📎\n\nDownload: ${url}`, {
reply_parameters: { message_id: ctx.message.message_id },
});
};
/**
* Start the Telegram bot and register message handlers.
*
* Creates a Telegraf instance, registers a `/start` command handler,
* logging middleware, and media event handlers for all supported file types.
* Incoming media files are deduplicated by their Telegram unique ID,
* forwarded to the storage channel, and persisted with a public download URL.
*
* @param deps - Optional external dependencies for testing or DI override.
* @param deps.telegramService - The Telegram service used to forward files to
* the storage channel. Defaults to the singleton BotPool instance.
* @param deps.fileRepo - The file repository used for deduplication queries
* and persisting new file records. Defaults to a new DrizzleFileRepository.
* @returns The launched Telegraf bot instance, suitable for graceful shutdown
* via `bot.stop(signal)`.
*/
export async function startBot(
deps: {
/** The Telegram service to forward files to storage. */
telegramService?: ITelegramService;
/** The file repository for deduplication and persistence. */
fileRepo?: IFileRepository;
} = {},
): Promise<Telegraf<Context>> {
const telegramService = deps.telegramService ?? botPool;
const fileRepo = deps.fileRepo ?? new DrizzleFileRepository();
try {
const bot = new Telegraf(config.botTokens[0]);
bot.command('start', async (ctx) => {
await ctx.reply(
`👋 Halo! Kirimkan file (document, photo, video, audio, voice, animation) ke bot ini. ` +
`File akan disimpan di private channel dan kamu dapat download link permanen.`,
);
});
// Logging middleware must be registered BEFORE the media handler so all events are captured
bot.use((ctx, next) => {
logger.info('Telegram event received', {
type: 'type' in ctx.update ? ctx.update.type : undefined,
chat_id: ctx.chat?.id,
});
return next();
});
const mediaBot = bot as unknown as MediaEventRegistrar;
mediaBot.on(
['document', 'photo', 'video', 'audio', 'voice', 'animation', 'sticker', 'video_note'],
async (ctx) => {
try {
const fileType = detectFileType(ctx.message);
const fileObj = extractFileFromMessage(ctx.message, fileType);
const { file_id, mime_type } = fileObj;
const fileSize = fileObj.file_size || 0;
const fileName =
ctx.message.document?.file_name ||
ctx.message.photo?.slice(-1)[0]?.file_name ||
ctx.message.video?.file_name ||
ctx.message.audio?.file_name ||
ctx.message.voice?.file_name ||
'file';
if (!checkFileSize(fileSize, fileType)) {
return ctx.reply(
`File size exceeds ${getFileSizeLimit(fileType) / (1024 * 1024)}MB limit`,
);
}
const existing = await fileRepo.findByUniqueId(fileObj.file_unique_id);
if (existing) {
await replyWithDownloadUrl(ctx, existing.publicId);
logger.info('Duplicate file detected in bot, returned existing link', {
publicId: existing.publicId,
fileType,
fileName,
uploader: ctx.from.id,
});
return;
}
const result = await telegramService.forwardToStorage(file_id, fileName, fileType);
const publicId = nanoid();
await fileRepo.create(
buildNewFile({
publicId,
telegramFileId: result.telegramFileId,
telegramFileUniqueId: result.telegramFileUniqueId,
storageChatId: config.storageChatId,
storageMessageId: result.storageMessageId,
fileName,
mimeType: mime_type || 'application/octet-stream',
sizeBytes: fileSize,
fileType,
uploaderId: ctx.from.id,
storageBackend: 'telegram',
}),
);
await replyWithDownloadUrl(ctx, publicId);
logger.info('File uploaded via bot', {
publicId,
fileType,
fileName,
uploader: ctx.from.id,
});
} catch (error: unknown) {
logger.error('Bot file handler error', {
error: getErrorMessage(error),
chat_id: ctx.chat?.id,
});
await ctx.reply('❌ Gagal mengupload file. Coba lagi nanti.');
}
},
);
await bot.launch();
logger.info('Telegram bot started', {
botToken: `${config.botTokens[0]?.substring(0, 10)}...`,
});
return bot;
} catch (error: unknown) {
logger.error('Failed to start bot', { error: getErrorMessage(error) });
throw error;
}
}
@@ -0,0 +1,151 @@
import {
type AuthSession,
createLoginUseCase,
createLogoutUseCase,
createMeUseCase,
} from '../../../application/use-cases/authenticate';
import { config } from '../../../env';
import {
checkBearerToken,
clearSessionCookie,
createSessionCookie,
getAuthSession,
isAuthEnabled,
} from '../middleware/auth';
/**
* Helper that builds a JSON Response with optional extra headers.
*
* @param data - The JSON-serialisable body.
* @param status - HTTP status code (default 200).
* @param headers - Optional extra response headers.
* @returns A JSON Response.
*/
const json = (data: unknown, status = 200, headers: Record<string, string> = {}): Response =>
Response.json(data, { status, headers });
/**
* Returns a standard 404 Not Found JSON response.
*
* Used to hide auth endpoints when auth is disabled.
*
* @returns A 404 JSON response.
*/
const notFound = (): Response => json({ error: 'Not found' }, 404);
/**
* Parses the login request body, extracting the `token` field.
*
* @param req - The incoming HTTP request with a JSON body.
* @returns The login token payload, or `null` when the body is invalid.
*/
const readLoginBody = async (req: Request): Promise<{ token: string } | null> => {
try {
const body = (await req.json()) as { token?: unknown };
if (typeof body.token !== 'string' || body.token.length === 0) return null;
return { token: body.token };
} catch {
return null;
}
};
/**
* Handles the login endpoint.
*
* Reads the admin API token from the request body, validates it via the
* login use case, and sets a session cookie on success.
*
* When auth is disabled the endpoint returns 404.
*
* @param req - The incoming HTTP request.
* @returns A JSON response with login status and a Set-Cookie header.
*/
export const handleLogin = async (req: Request): Promise<Response> => {
if (!isAuthEnabled()) return notFound();
const body = await readLoginBody(req);
if (!body) return json({ error: 'Token is required' }, 400);
try {
const loginUseCase = createLoginUseCase({
config: {
adminApiToken: config.adminApiToken,
sessionCookieName: config.sessionCookieName,
sessionMaxAgeMs: config.sessionMaxAgeMs,
},
});
const result = await loginUseCase({ token: body.token });
return json({ username: result.username }, 200, {
'set-cookie': createSessionCookie('admin'),
});
} catch (error: unknown) {
const message = error instanceof Error ? error.message : 'Invalid token';
if (message === 'Invalid token') {
return json({ error: 'Invalid token' }, 401);
}
return json({ error: message }, 500);
}
};
/**
* Handles the logout endpoint.
*
* Clears the session cookie and returns a success response.
*
* @returns A JSON response with a cleared Set-Cookie header.
*/
export const handleLogout = async (): Promise<Response> => {
const logoutUseCase = createLogoutUseCase();
await logoutUseCase();
return json({ success: true }, 200, {
'set-cookie': clearSessionCookie(),
});
};
/**
* Handles the current-user (me) endpoint.
*
* Extracts the authentication session from the request (cookie or bearer
* token) and returns the user info via the me use case.
*
* When auth is disabled the endpoint returns 404.
*
* @param req - The incoming HTTP request.
* @returns A JSON response with user info, or 401 when unauthenticated.
*/
export const handleMe = async (req: Request): Promise<Response> => {
if (!isAuthEnabled()) return notFound();
const session: AuthSession | null = getAuthSession(req);
if (!session && !checkBearerToken(req.headers.get('authorization'))) {
return json({ error: 'Unauthorized' }, 401);
}
const meUseCase = createMeUseCase({
config: {
adminApiToken: config.adminApiToken,
sessionCookieName: config.sessionCookieName,
sessionMaxAgeMs: config.sessionMaxAgeMs,
},
});
const activeSession = session ?? {
username: 'admin',
expiresAt: null,
method: 'bearer' as const,
};
const result = await meUseCase(activeSession);
if (!result) {
return json({ error: 'Unauthorized' }, 401);
}
return json({
username: result.username,
expiresAt: result.expiresAt,
});
};
@@ -0,0 +1,208 @@
import { createReadStream } from 'node:fs';
import { nanoid } from 'nanoid';
import type { TelegramFileInfo } from '../../../domain/ports/telegram-service';
import { fileInfoCache } from '../../../infrastructure/cache/index';
import { chunkedStorage, fileRepository } from '../../../infrastructure/di';
import { botPool } from '../../../infrastructure/telegram/bot-pool';
import logger from '../../../shared/logger/index';
import { cleanupTempFile, formatCreatedAt, getErrorMessage } from '../../../shared/utils/file';
import { locateZipEntry } from '../../../shared/utils/zip';
/**
* Extended Request type that includes route parameter access.
*/
type RequestWithParams = Request & {
/** Route parameters extracted by the router. */
params?: {
/** Public file identifier. */
public_id?: string;
};
};
/**
* Resolves Telegram file metadata for a given file ID, using the in-memory
* cache to avoid repeated API calls to Telegram.
*
* @param telegramFileId - The Telegram file identifier to resolve.
* @param publicId - The public file ID (used for logging).
* @returns The resolved Telegram file info.
*/
const getTelegramFileInfo = async (
telegramFileId: string,
publicId: string,
): Promise<TelegramFileInfo> => {
const cacheKey = `file_info_${telegramFileId}`;
const cached = fileInfoCache.get(cacheKey) as TelegramFileInfo | null;
if (cached) {
logger.debug('File info from cache', { publicId, cacheKey });
return cached;
}
const fileInfo = await botPool.getFileInfo(telegramFileId);
fileInfoCache.set(cacheKey, fileInfo);
logger.debug('File info cached', { publicId, cacheKey });
return fileInfo;
};
/**
* Builds a Telegram CDN download URL from a file path and bot token.
*
* @param filePath - The Telegram file path returned by getFile.
* @param botToken - The bot token used to authenticate the download.
* @returns The full Telegram CDN URL.
*/
const buildTelegramFileUrl = (filePath: string, botToken: string): string =>
`https://api.telegram.org/file/bot${botToken}/${filePath}`;
/**
* Sanitises a file name for use in a Content-Disposition header, removing
* characters that could enable header injection.
*
* @param fileName - The raw file name.
* @returns The sanitised file name.
*/
const sanitizeFilenameHeader = (fileName: string): string =>
fileName.replace(/[\\"]/g, '').replace(/[\n\r]/g, '');
/**
* Returns a JSON error response with the given status code and message.
*
* @param status - HTTP status code.
* @param error - Error message.
* @returns A JSON Response.
*/
const fail = (status: number, error: string): Response => Response.json({ error }, { status });
/**
* Handles file redirect requests.
*
* Looks up a file by its public identifier and determines the best delivery
* method:
* - **chunked** files are streamed via the chunked-object response builder.
* - **archive-entry** files are extracted from a Telegram-stored zip archive
* and streamed as a single file.
* - **regular** files are redirected to the Telegram CDN URL (302).
*
* @param req - The incoming HTTP request with a `public_id` route parameter.
* @returns A redirect or streaming response, or a JSON error.
*/
export const handleFileRedirect = async (req: RequestWithParams): Promise<Response> => {
const publicId = req.params?.public_id;
try {
if (!publicId) {
return fail(400, 'Missing file id');
}
const file = await fileRepository.findByPublicId(publicId);
if (!file) {
logger.warn('File not found', { publicId });
return fail(404, 'File not found');
}
if (file.storageBackend === 'chunked') {
if (file.archiveEntryName) {
return fail(501, 'Archive entry extraction is not supported for chunked files');
}
const range = { type: 'none' as const };
return chunkedStorage.createChunkedObjectResponse({ file, range, reqId: '' });
}
const archiveEntryName = file.archiveEntryName;
if (archiveEntryName) {
const archiveFileId = file.archiveTelegramFileId || file.telegramFileId;
const archiveInfo = await getTelegramFileInfo(archiveFileId, publicId);
const archiveResponse = await fetch(
buildTelegramFileUrl(archiveInfo.file_path, archiveInfo.bot_token),
);
if (!archiveResponse.ok) {
logger.error('Archive download failed', { publicId, status: archiveResponse.status });
return fail(500, 'Server error');
}
const tempZipPath = `/tmp/filedrop-dl-${nanoid()}.zip`;
await Bun.write(tempZipPath, archiveResponse);
const loc = await locateZipEntry(tempZipPath, archiveEntryName);
if (!loc) {
await cleanupTempFile(tempZipPath);
logger.error('Archive entry not found', { publicId, archiveEntryName });
return fail(404, 'File not found');
}
const fileStream = createReadStream(tempZipPath, {
start: loc.start,
end: loc.start + loc.length - 1,
});
fileStream.on('close', () => {
void cleanupTempFile(tempZipPath);
});
fileStream.on('error', () => {
void cleanupTempFile(tempZipPath);
});
return new Response(fileStream as unknown as ReadableStream, {
status: 200,
headers: {
'Content-Type': file.mimeType || 'application/octet-stream',
'Content-Disposition': `attachment; filename="${sanitizeFilenameHeader(file.fileName)}"`,
'Content-Length': String(loc.length),
},
});
}
const fileInfo = await getTelegramFileInfo(file.telegramFileId, publicId);
const redirectUrl = buildTelegramFileUrl(fileInfo.file_path, fileInfo.bot_token);
return new Response(null, {
status: 302,
headers: {
Location: redirectUrl,
},
});
} catch (error: unknown) {
logger.error('File redirect error', { publicId, error: getErrorMessage(error) });
return fail(500, 'Server error');
}
};
/**
* Handles file info requests.
*
* Looks up a file by its public identifier and returns its metadata as JSON.
*
* @param req - The incoming HTTP request with a `public_id` route parameter.
* @returns A JSON response with file metadata, or 404 when not found.
*/
export const handleFileInfo = async (req: RequestWithParams): Promise<Response> => {
const publicId = req.params?.public_id;
try {
if (!publicId) {
return fail(400, 'Missing file id');
}
const file = await fileRepository.findByPublicId(publicId);
if (!file) {
logger.warn('File not found', { publicId });
return fail(404, 'File not found');
}
return Response.json(
{
public_id: file.publicId,
file_name: file.fileName,
mime_type: file.mimeType,
size_bytes: file.sizeBytes,
file_type: file.fileType,
created_at: formatCreatedAt(file.createdAt),
},
{ status: 200 },
);
} catch (error: unknown) {
logger.error('File info error', { publicId, error: getErrorMessage(error) });
return fail(500, 'Server error');
}
};
@@ -0,0 +1,25 @@
import { sql } from 'drizzle-orm';
import { db } from '../../../infrastructure/persistence/drizzle/index';
import logger from '../../../shared/logger/index';
import { getErrorMessage } from '../../../shared/utils/file';
/**
* Handles the health-check endpoint.
*
* Verifies database connectivity by executing a simple `SELECT 1` query.
* Returns a 200 response with `{ status: 'ok' }` when the database is
* reachable, or a 500 response with the error details when it is not.
*
* @param _req - The incoming HTTP request (unused).
* @returns A JSON response indicating the database health status.
*/
export const handleHealth = async (_req: Request): Promise<Response> => {
try {
await db.execute(sql`SELECT 1`);
return Response.json({ status: 'ok' }, { status: 200 });
} catch (error: unknown) {
const message = getErrorMessage(error);
logger.error('Health check failed', { error: message });
return Response.json({ status: 'error', error: message }, { status: 500 });
}
};
@@ -0,0 +1,64 @@
import { existsSync } from 'node:fs';
import { dirname, join } from 'node:path';
import type { BunFile } from 'bun';
/**
* Maximum number of parent directories to walk up when locating home.html.
* Deep enough for the dev layout (controllers/ -> src/ = 4 levels) with margin.
*/
const MAX_PARENT_WALK = 6;
/**
* Resolves the absolute path to `home.html` by walking up from `startDir`.
*
* The file lives at different depths depending on how the app is run:
* - Dev (`bun --hot src/index.ts`): `import.meta.dir` is
* `src/interfaces/http/controllers/`, home.html lives at `src/home.html`
* (4 levels up).
* - Prod (bundled `dist/index.js`): `import.meta.dir` is
* `$out/share/teleuploader/dist/`, home.html lives next to dist/
* (1 level up, per flake.nix installPhase).
*
* Returns the first existing candidate, or `null` if none is found within
* the walk bound.
*
* @param startDir - Directory to start the search from (typically `import.meta.dir`).
* @param maxDepth - Maximum number of parent directories to walk (default: 6).
* @returns Absolute path to home.html, or `null` if not found.
*/
export const resolveHomeHtml = (startDir: string, maxDepth = MAX_PARENT_WALK): string | null => {
let dir = startDir;
for (let depth = 0; depth <= maxDepth; depth++) {
const candidate = join(dir, 'home.html');
if (existsSync(candidate)) return candidate;
const parent = dirname(dir);
if (parent === dir) break;
dir = parent;
}
return null;
};
/**
* Handles the home/dashboard page request.
*
* Reads the `home.html` file and serves it as an HTML response with UTF-8
* charset. Fails fast with a clear error when the file cannot be located
* instead of letting Bun.serve swallow the ENOENT into a bare 500.
*
* @returns An HTML response containing the home page content.
*/
export const handleHome = async (): Promise<Response> => {
const homeHtml = resolveHomeHtml(import.meta.dir);
if (!homeHtml) {
throw new Error(
`home.html not found — looked up from ${import.meta.dir} and ${MAX_PARENT_WALK} parent dirs`,
);
}
const html = await (Bun.file(homeHtml) as BunFile).text();
return new Response(html, {
status: 200,
headers: {
'content-type': 'text/html; charset=utf-8',
},
});
};
+370
View File
@@ -0,0 +1,370 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>FileDrop · S3 File Manager</title>
<style>
:root {
--bg: #ffffff; --bg2: #f5f5f5; --text: #1a1a1a;
--text2: #666; --border: #e0e0e0; --accent: #2563eb;
--danger: #dc2626; --radius: 8px;
}
@media (prefers-color-scheme: dark) {
:root {
--bg: #0d1117; --bg2: #161b22; --text: #c9d1d9;
--text2: #8b949e; --border: #30363d; --accent: #58a6ff;
--danger: #f85149;
}
}
* { box-sizing: border-box; margin: 0; padding: 0; }
body {
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
background: var(--bg); color: var(--text); line-height: 1.5;
min-height: 100vh;
}
.topbar {
display: flex; align-items: center; gap: 12px;
padding: 12px 24px; background: var(--bg2);
border-bottom: 1px solid var(--border);
position: sticky; top: 0; z-index: 50;
}
.topbar .logo { font-weight: 700; font-size: 1.1rem; }
.topbar select, .topbar button {
padding: 6px 12px; border: 1px solid var(--border);
border-radius: var(--radius); background: var(--bg);
color: var(--text); font-size: 0.875rem; cursor: pointer;
}
.modal input {
width: 100%; padding: 8px 12px; border: 1px solid var(--border);
border-radius: var(--radius); background: var(--bg);
color: var(--text); margin-bottom: 12px;
}
.topbar button.primary { background: var(--accent); color: #fff; border-color: var(--accent); }
.topbar .spacer { flex: 1; }
.topbar .search input {
padding: 6px 12px; border: 1px solid var(--border);
border-radius: var(--radius); background: var(--bg);
color: var(--text); font-size: 0.875rem; width: 200px;
}
.file-list { padding: 16px 24px; }
.breadcrumb {
padding: 8px 0; margin-bottom: 8px; font-size: 0.9rem;
color: var(--accent); cursor: pointer;
}
.breadcrumb span:hover { text-decoration: underline; }
.breadcrumb .sep { color: var(--text2); margin: 0 4px; }
.file-row {
display: flex; align-items: center; gap: 12px;
padding: 10px 12px; border-radius: var(--radius);
cursor: pointer; transition: background 0.1s;
}
.file-row:hover { background: var(--bg2); }
.file-row .icon { font-size: 1.2rem; width: 28px; text-align: center; flex-shrink: 0; }
.file-row .name { flex: 1; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
.file-row .size { width: 80px; text-align: right; color: var(--text2); font-size: 0.85rem; }
.file-row .date { width: 140px; color: var(--text2); font-size: 0.85rem; }
.file-row .actions { display: flex; gap: 4px; }
.file-row .actions button {
padding: 4px 8px; border: none; border-radius: 4px;
background: transparent; color: var(--text2); cursor: pointer; font-size: 0.8rem;
}
.file-row .actions button:hover { color: var(--text); background: var(--border); }
.dropzone {
position: fixed; bottom: 0; left: 0; right: 0;
padding: 12px 24px; background: var(--bg2);
border-top: 1px solid var(--border);
text-align: center; color: var(--text2); font-size: 0.85rem; cursor: pointer;
}
.dropzone.dragover { background: var(--accent); color: #fff; }
.progress-overlay {
position: fixed; top: 0; left: 0; right: 0; bottom: 0;
background: rgba(0,0,0,0.5); display: flex;
align-items: center; justify-content: center; z-index: 100;
}
.progress-card {
background: var(--bg); padding: 24px; border-radius: var(--radius);
min-width: 300px; max-width: 500px;
}
.progress-bar {
height: 8px; background: var(--border); border-radius: 4px;
margin: 12px 0; overflow: hidden;
}
.progress-bar .fill {
height: 100%; background: var(--accent);
transition: width 0.2s; width: 0%;
}
.modal-overlay {
position: fixed; top: 0; left: 0; right: 0; bottom: 0;
background: rgba(0,0,0,0.5); display: flex;
align-items: center; justify-content: center; z-index: 100;
}
.modal {
background: var(--bg); padding: 24px; border-radius: var(--radius);
min-width: 360px; max-width: 500px;
}
.modal h3 { margin-bottom: 16px; }
.modal .buttons { display: flex; gap: 8px; justify-content: flex-end; }
.modal .buttons button {
padding: 8px 16px; border: 1px solid var(--border);
border-radius: var(--radius); background: var(--bg); color: var(--text); cursor: pointer;
}
.modal .buttons .primary { background: var(--accent); color: #fff; border-color: var(--accent); }
.modal .buttons .danger { background: var(--danger); color: #fff; border-color: var(--danger); }
.empty { text-align: center; padding: 48px 24px; color: var(--text2); }
.empty h2 { font-size: 1.2rem; margin-bottom: 8px; }
.auth-screen {
position: fixed; inset: 0; z-index: 200; display: none;
align-items: center; justify-content: center; padding: 24px;
background: linear-gradient(135deg, var(--bg), var(--bg2));
}
.auth-card {
width: min(100%, 380px); padding: 28px; border: 1px solid var(--border);
border-radius: 16px; background: var(--bg); box-shadow: 0 20px 60px rgba(0,0,0,0.18);
}
.auth-card h1 { font-size: 1.45rem; margin-bottom: 8px; }
.auth-card p { color: var(--text2); margin-bottom: 18px; }
.auth-card input {
width: 100%; padding: 10px 12px; border: 1px solid var(--border);
border-radius: var(--radius); background: var(--bg2); color: var(--text);
margin-bottom: 12px;
}
.auth-card button {
width: 100%; padding: 10px 14px; border: 1px solid var(--accent);
border-radius: var(--radius); background: var(--accent); color: #fff;
cursor: pointer; font-weight: 600;
}
.auth-card button:disabled { opacity: 0.7; cursor: wait; }
.auth-error { color: var(--danger); font-size: 0.85rem; margin-bottom: 12px; }
.readonly-badge {
font-size: 0.75rem; color: var(--text2); background: var(--bg2);
border: 1px solid var(--border); border-radius: 999px; padding: 2px 10px;
}
</style>
</head>
<body>
<div id="authScreen" class="auth-screen">
<div class="auth-card">
<h1>📦 FileDrop</h1>
<p>Enter admin token to continue.</p>
<input id="authTokenInput" type="password" placeholder="Admin token" autocomplete="current-password">
<div id="authError" class="auth-error" style="display:none"></div>
<button id="authLoginBtn" type="button">Login</button>
</div>
</div>
<div class="topbar">
<span class="logo">📦 FileDrop</span>
<select id="bucketSelect" onchange="window.switchBucket(this.value)">
<option value="">— Select bucket —</option>
</select>
<button id="newBucketBtn" type="button" onclick="window.showCreateBucketModal()">+ New</button>
<button id="credsBtn" type="button" onclick="window.showCredentialsModal()" title="S3 Credentials">🔑</button>
<button id="loginBtn" type="button" onclick="window.showAuthScreen()" style="display:none">Login</button>
<button id="logoutBtn" type="button" onclick="window.logout()" style="display:none">Logout</button>
<span id="readonlyBadge" class="readonly-badge" style="display:none">👀 read-only</span>
<span class="spacer"></span>
<div class="search">
<input id="searchInput" type="text" placeholder="Filter prefix..." oninput="window.debouncedSearch()">
</div>
</div>
<div id="breadcrumb" class="breadcrumb" style="display:none;padding:8px 24px"></div>
<div id="fileList" class="file-list">
<div class="empty"><h2>Select a bucket to get started</h2><p>Choose a bucket from the dropdown above, or create a new one.</p></div>
</div>
<div id="dropzone" class="dropzone" style="display:none">📁 Drop files here or click to upload</div>
<div id="progressOverlay" class="progress-overlay" style="display:none">
<div class="progress-card">
<h3>Uploading...</h3>
<div id="progressFileName"></div>
<div class="progress-bar"><div id="progressFill" class="fill"></div></div>
<div id="progressPercent" style="font-size:0.85rem;color:var(--text2)">0%</div>
</div>
</div>
<div id="modalOverlay" class="modal-overlay" style="display:none" onclick="closeModal(event)">
<div id="modalContent" class="modal" onclick="event.stopPropagation()"></div>
</div>
<script>
let currentBucket = null, currentPrefix = '', currentObjects = [], currentPrefixes = [], allBuckets = [], searchTimer = null;
let isAdmin = false;
const setAuthError = (message) => {
const errorEl = document.getElementById('authError');
errorEl.textContent = message;
errorEl.style.display = message ? 'block' : 'none';
};
const showAuthScreen = () => {
document.getElementById('authScreen').style.display = 'flex';
setTimeout(() => document.getElementById('authTokenInput')?.focus(), 50);
};
const hideAuthScreen = () => {
document.getElementById('authScreen').style.display = 'none';
};
// Applies the admin/read-only UI state based on isAdmin.
const applyAdminUI = () => {
document.getElementById('newBucketBtn').style.display = isAdmin ? 'inline-block' : 'none';
document.getElementById('credsBtn').style.display = isAdmin ? 'inline-block' : 'none';
document.getElementById('loginBtn').style.display = isAdmin ? 'none' : 'inline-block';
document.getElementById('logoutBtn').style.display = isAdmin ? 'inline-block' : 'none';
document.getElementById('readonlyBadge').style.display = isAdmin ? 'none' : 'inline-block';
// Dropzone (upload) is admin-only.
document.getElementById('dropzone').style.display = isAdmin && currentBucket ? 'block' : 'none';
if (currentObjects.length || currentPrefixes.length) renderFileList();
};
// Non-blocking auth check: read-only visitors still get the file browser.
const checkAuth = async () => {
try {
const res = await fetch('/api/v1/auth/me');
if (res.ok) { isAdmin = true; }
else if (res.status === 401) { isAdmin = false; }
else if (res.status === 404) { isAdmin = true; } // auth disabled — full access
else { isAdmin = false; }
} catch {
isAdmin = false;
}
hideAuthScreen();
applyAdminUI();
return isAdmin;
};
const handleLogin = async () => {
const input = document.getElementById('authTokenInput');
const btn = document.getElementById('authLoginBtn');
const token = input.value.trim();
if (!token) { setAuthError('Admin token is required.'); input.focus(); return; }
btn.disabled = true; btn.textContent = 'Logging in...'; setAuthError('');
try {
const res = await fetch('/api/v1/auth/login', {
method: 'POST', headers: { 'content-type': 'application/json' },
body: JSON.stringify({ token }),
});
if (res.ok) { isAdmin = true; hideAuthScreen(); input.value = ''; applyAdminUI(); await loadBuckets(); return; }
const body = await res.json().catch(() => ({ error: 'Login failed' }));
setAuthError(body.error || 'Login failed');
} catch {
setAuthError('Network error while logging in.');
} finally {
btn.disabled = false; btn.textContent = 'Login';
}
};
const logout = async () => {
await fetch('/api/v1/auth/logout', { method: 'POST' }).catch(() => {});
isAdmin = false;
applyAdminUI();
};
const api = async (path, opts = {}) => {
const res = await fetch(path, opts);
if (!res.ok) { const body = await res.json().catch(() => ({ error: res.statusText })); throw new Error(body.error || res.statusText); }
return res;
};
const apiJson = async (path, opts = {}) => { const res = await api(path, { headers: { 'content-type': 'application/json' }, ...opts }); return res.json(); };
const loadBuckets = async () => {
const data = await apiJson('/api/v1/buckets');
allBuckets = data.buckets || [];
const sel = document.getElementById('bucketSelect');
sel.innerHTML = `<option value="">— Select bucket —</option>${allBuckets.map(b => `<option value="${b.name}">${b.name} (${b.objectCount})</option>`).join('')}`;
if (currentBucket) sel.value = currentBucket;
};
const switchBucket = async (name) => {
currentBucket = name || null; currentPrefix = '';
if (name) { await loadObjects(); }
else {
document.getElementById('fileList').innerHTML = '<div class="empty"><h2>Select a bucket</h2><p>Choose a bucket from the dropdown above.</p></div>';
document.getElementById('breadcrumb').style.display = 'none';
}
// Dropzone (upload) is admin-only.
document.getElementById('dropzone').style.display = isAdmin && currentBucket ? 'block' : 'none';
};
const renderBreadcrumb = () => {
const bc = document.getElementById('breadcrumb');
if (!currentPrefix) { bc.style.display = 'none'; return; }
bc.style.display = 'block';
const parts = currentPrefix.split('/').filter(Boolean);
bc.innerHTML = `<span onclick="window.navigateTo('')">${currentBucket}</span>`;
let accumulated = '';
for (const part of parts) { accumulated += `${part}/`; bc.innerHTML += `<span class="sep">/</span><span onclick="window.navigateTo('${accumulated}')">${part}</span>`; }
};
const navigateTo = (prefix) => { currentPrefix = prefix; loadObjects(); };
const loadObjects = async () => {
if (!currentBucket) return;
const searchVal = document.getElementById('searchInput').value;
const prefix = searchVal || currentPrefix;
const url = `/api/v1/buckets/${encodeURIComponent(currentBucket)}/objects?prefix=${encodeURIComponent(prefix)}&delimiter=/&max-keys=200`;
try {
const data = await apiJson(url);
currentObjects = data.objects || []; currentPrefixes = data.prefixes || [];
renderFileList(); renderBreadcrumb();
} catch (e) { document.getElementById('fileList').innerHTML = `<div class="empty"><h2>Error</h2><p>${e.message}</p></div>`; }
};
const renderFileList = () => {
const container = document.getElementById('fileList');
if (currentPrefixes.length === 0 && currentObjects.length === 0) {
container.innerHTML = isAdmin
? '<div class="empty"><h2>This bucket is empty</h2><p>Drop files here to upload.</p></div>'
: '<div class="empty"><h2>This bucket is empty</h2></div>';
return;
}
let html = '';
for (const prefix of currentPrefixes) {
const displayName = prefix.replace(currentPrefix, '');
html += `<div class="file-row" onclick="window.navigateTo('${prefix}')"><span class="icon">🗂</span><span class="name">${displayName.endsWith('/') ? displayName : `${displayName}/`}</span><span class="size"></span><span class="date"></span><span class="actions"></span></div>`;
}
for (const obj of currentObjects) {
const displayName = obj.key.replace(currentPrefix, '');
// Delete is admin-only; download + copy link are always available.
const deleteBtn = isAdmin ? `<button onclick="event.stopPropagation();deleteObject('${obj.key}')" title="Delete">🗑</button>` : '';
html += `<div class="file-row"><span class="icon">📄</span><span class="name">${escapeHtml(displayName)}</span><span class="size">${formatSize(obj.sizeBytes)}</span><span class="date">${formatDate(obj.lastModified)}</span><span class="actions"><button onclick="event.stopPropagation();downloadObject('${obj.key}')" title="Download"></button><button onclick="event.stopPropagation();copyLink('${obj.key}')" title="Copy link">🔗</button>${deleteBtn}</span></div>`;
}
container.innerHTML = html;
};
const formatSize = (bytes) => { const size = Number(bytes); if (!Number.isFinite(size) || size <= 0) return '0 B'; const u = ['B','KB','MB','GB','TB']; let i=0,s=size; while(s>=1024&&i<u.length-1){s/=1024;i++} return `${s.toFixed(i>0?1:0)} ${u[i]}`; };
const formatDate = (iso) => { if(!iso)return ''; return new Date(iso).toLocaleDateString(undefined,{month:'short',day:'numeric',year:'numeric'}); };
const escapeHtml = (s) => { const d=document.createElement('div');d.textContent=s;return d.innerHTML; };
const debouncedSearch = () => { clearTimeout(searchTimer); searchTimer = setTimeout(loadObjects, 300); };
const downloadObject = async (key) => { window.open(`/api/v1/buckets/${encodeURIComponent(currentBucket)}/download/${encodeURIComponent(key)}`,'_blank'); };
const copyLink = (key) => { navigator.clipboard.writeText(`${window.location.origin}/api/v1/buckets/${encodeURIComponent(currentBucket)}/download/${encodeURIComponent(key)}`).catch(()=>{}); };
const deleteObject = async (key) => {
if (!isAdmin) { alert('Read-only mode — login as admin to delete.'); return; }
if(!confirm(`Delete "${key}"?`))return;
try{await api(`/api/v1/buckets/${encodeURIComponent(currentBucket)}/${encodeURIComponent(key)}`,{method:'DELETE'});await loadObjects();}
catch(e){alert(`Delete failed: ${e.message}`);}
};
const uploadFiles = async (files) => {
if (!isAdmin) { alert('Read-only mode — login as admin to upload.'); return; }
if(!currentBucket||files.length===0)return;
const overlay=document.getElementById('progressOverlay'), fill=document.getElementById('progressFill'), pn=document.getElementById('progressFileName'), pp=document.getElementById('progressPercent');
overlay.style.display='flex';
for(let i=0;i<files.length;i++){
const file=files[i]; pn.textContent=`${i+1}/${files.length}: ${file.name}`; fill.style.width='0%'; pp.textContent='0%';
await new Promise((resolve,reject)=>{
const fd=new FormData(); fd.append('file',file); fd.append('key',currentPrefix+file.name);
const xhr=new XMLHttpRequest();
xhr.upload.onprogress=(e)=>{if(e.lengthComputable){const p=Math.round((e.loaded/e.total)*100);fill.style.width=`${p}%`;pp.textContent=`${p}%`;}};
xhr.onload=()=>{if(xhr.status>=200&&xhr.status<300)resolve();else reject(new Error(xhr.statusText));};
xhr.onerror=()=>reject(new Error('Upload failed'));
xhr.open('POST',`/api/v1/buckets/${encodeURIComponent(currentBucket)}/upload`); xhr.send(fd);
});
}
overlay.style.display='none'; await loadObjects();
};
const dropzone=document.getElementById('dropzone');
dropzone.addEventListener('dragover',e=>{e.preventDefault();dropzone.classList.add('dragover');});
dropzone.addEventListener('dragleave',()=>dropzone.classList.remove('dragover'));
dropzone.addEventListener('drop',e=>{e.preventDefault();dropzone.classList.remove('dragover');if(e.dataTransfer.files.length>0)uploadFiles(e.dataTransfer.files);});
dropzone.addEventListener('click',()=>{const i=document.createElement('input');i.type='file';i.multiple=true;i.onchange=()=>{if(i.files.length>0)uploadFiles(i.files);};i.click();});
const showModal=(html)=>{document.getElementById('modalContent').innerHTML=html;document.getElementById('modalOverlay').style.display='flex';};
const closeModal=(e)=>{if(e&&e.target!==e.currentTarget)return;document.getElementById('modalOverlay').style.display='none';};
const showCreateBucketModal=()=>{
if (!isAdmin) { alert('Read-only mode — login as admin to create buckets.'); return; }
showModal(`<h3>Create Bucket</h3><input id="bucketNameInput" type="text" placeholder="my-bucket-name" pattern="[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]"><p style="font-size:0.8rem;color:var(--text2);margin-bottom:12px">Lowercase, 3-63 chars, no underscores</p><div class="buttons"><button onclick="closeModal()">Cancel</button><button class="primary" onclick="createBucket()">Create</button></div>`);setTimeout(()=>document.getElementById('bucketNameInput')?.focus(),100);};
const createBucket=async()=>{
if (!isAdmin) { alert('Read-only mode — login as admin to create buckets.'); return; }
const n=document.getElementById('bucketNameInput').value.trim();if(!n)return;try{await apiJson('/api/v1/buckets',{method:'POST',body:JSON.stringify({name:n})});closeModal();await loadBuckets();document.getElementById('bucketSelect').value=n;await switchBucket(n);}catch(e){alert(`Failed: ${e.message}`);}};
const showCredentialsModal=()=>{
if (!isAdmin) { alert('Read-only mode — login as admin to view S3 credentials.'); return; }
showModal(`<h3>S3 Credentials</h3><p style="margin-bottom:12px;font-size:0.85rem;color:var(--text2)">Use these in any S3 client (aws-cli, rclone, s3cmd, etc.)</p><label style="font-size:0.85rem;font-weight:600">Endpoint URL</label><input type="text" value="${window.location.origin}" readonly onclick="this.select()"><label style="font-size:0.85rem;font-weight:600">Region</label><input type="text" value="us-east-1" readonly onclick="this.select()"><label style="font-size:0.85rem;font-weight:600">Access Key</label><input id="s3AccessKey" type="text" readonly onclick="this.select()"><label style="font-size:0.85rem;font-weight:600">Secret Key</label><input id="s3SecretKey" type="password" readonly onclick="this.select()"><div class="buttons"><button type="button" onclick="window.closeModal()">Close</button></div>`);};
const init=async()=>{await checkAuth();await loadBuckets();};
document.getElementById('authLoginBtn').addEventListener('click',handleLogin);
document.getElementById('authTokenInput').addEventListener('keydown',e=>{if(e.key==='Enter')handleLogin();});
Object.assign(window, { switchBucket, navigateTo, debouncedSearch, downloadObject, copyLink, deleteObject, closeModal, showCreateBucketModal, createBucket, showCredentialsModal, showAuthScreen, logout });
init();
</script>
</body>
</html>
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,393 @@
import { Readable } from 'node:stream';
import { nanoid } from 'nanoid';
import { buildNewFile } from '../../../domain/entities/file-factory';
import { config } from '../../../env';
import { chunkedStorage, fileRepository, telegramService } from '../../../infrastructure/di';
import logger from '../../../shared/logger/index';
import { metricsCollector } from '../../../shared/metrics/index';
import {
buildUploadResponse,
checkFileSize,
cleanupTempFile,
computeHash,
ensureExtension,
extractMimeType,
getErrorMessage,
getFileType,
} from '../../../shared/utils/file';
import { streamToTemp } from '../../../shared/utils/temp-stream';
/** Prepared upload metadata before submission to storage. */
interface PreparedUpload {
tempPath: string;
fileHash: string;
sizeBytes: number;
signatureBuffer: Buffer;
}
/**
* Maximum allowed size (in bytes) for a base64 JSON upload.
* JSON uploads are limited to 50 MB because base64 encoding adds ~33%
* overhead and large payloads strain the JSON parser.
*/
const JSON_UPLOAD_LIMIT_BYTES = 50 * 1024 * 1024;
/** Number of leading bytes read for magic-byte / signature detection. */
const SIGNATURE_BYTES = 16;
/**
* Payload structure accepted by the JSON upload endpoint.
*/
interface JsonUploadPayload {
/** Base64-encoded file data (optionally with a data URI prefix). */
file?: unknown;
/** Optional file name. */
fileName?: string;
}
/**
* Parses a base64-encoded file string, optionally stripping the data URI
* prefix.
*
* Accepts both bare base64 strings and RFC 2397 data URIs (e.g.
* `data:image/png;base64,...`).
*
* @param file - The base64 string, with or without a data URI prefix.
* @returns The raw base64 payload and the detected MIME type.
*/
const parseBase64File = (file: string): { base64Data: string; mimeType: string } => {
if (!file.startsWith('data:')) {
return { base64Data: file, mimeType: 'application/octet-stream' };
}
const match = file.match(/^data:([^;]+);base64,(.+)$/);
return match
? { base64Data: match[2], mimeType: match[1] }
: { base64Data: file, mimeType: 'application/octet-stream' };
};
/**
* Extracts the Content-Length header value as a number.
*
* @param req - The incoming HTTP request.
* @returns The content length in bytes, or `null` when the header is missing
* or invalid.
*/
const getContentLength = (req: Request): number | null => {
const value = req.headers.get('content-length');
if (!value) return null;
const parsed = Number.parseInt(value, 10);
return Number.isFinite(parsed) && parsed >= 0 ? parsed : null;
};
/**
* Checks whether the request body exceeds the configured maximum size and
* returns an error response if it does.
*
* @param req - The incoming HTTP request.
* @returns A 413 Response when the request is too large, or `null` when
* the size is within bounds (or unknown).
*/
const rejectOversizedRequest = (req: Request): Response | null => {
const contentLength = getContentLength(req);
if (contentLength !== null && contentLength > config.maxRequestBodyBytes) {
return Response.json({ error: 'Request body too large' }, { status: 413 });
}
return null;
};
/**
* Streams a multipart `File` to a temporary file on disk while computing
* its SHA-256 hash and extracting the signature (first 16 bytes).
*
* Delegates to the shared {@link streamToTemp} utility.
*
* @param file - The multipart `File` object.
* @param maxSizeBytes - Maximum allowed file size; an error is thrown if
* the stream exceeds this limit.
* @returns A fully prepared upload descriptor with hash, size, and temp path.
* @throws {Error} When the file size exceeds `maxSizeBytes`.
*/
const streamFileToTemp = async (file: File, maxSizeBytes: number): Promise<PreparedUpload> => {
const result = await streamToTemp(file.stream().getReader(), { maxSizeBytes });
return result;
};
/**
* Writes an in-memory buffer to a temporary file on disk.
*
* Used for base64 JSON uploads where the decoded data is already in a Buffer.
*
* @param fileBuffer - The decoded file content.
* @param fileHash - Pre-computed SHA-256 hex digest.
* @returns A prepared upload descriptor.
*/
const writeBufferToTemp = async (fileBuffer: Buffer, fileHash: string): Promise<PreparedUpload> => {
const tempPath = `/tmp/filedrop-${nanoid()}`;
try {
await Bun.write(tempPath, fileBuffer);
return {
tempPath,
fileHash,
sizeBytes: fileBuffer.byteLength,
signatureBuffer: fileBuffer.subarray(0, SIGNATURE_BYTES),
};
} catch (error) {
await cleanupTempFile(tempPath);
throw error;
}
};
/**
* Handles a multipart/form-data file upload.
*
* Steps:
* 1. Parse the multipart form and extract the file.
* 2. Stream the file to a temp location, computing its hash.
* 3. Check for deduplication by content hash.
* 4. Determine the MIME type, file name, and Telegram file type.
* 5. Validate file size limits.
* 6. Upload to Telegram (chunked or single-message).
* 7. Return the upload response JSON.
*
* @param req - The incoming HTTP request with a multipart body.
* @returns A JSON response with the uploaded file metadata.
*/
const handleMultipartUpload = async (req: Request): Promise<Response> => {
try {
const formData = await req.formData();
const file = formData.get('file');
const fileName =
(formData.get('fileName') as string) || (file instanceof File ? file.name : null) || 'file';
if (!file || !(file instanceof File)) {
return Response.json({ error: 'No file provided' }, { status: 400 });
}
if (file.size > config.maxRequestBodyBytes) {
return Response.json({ error: 'File size exceeds upload limit' }, { status: 413 });
}
const prepared = await streamFileToTemp(file, config.maxRequestBodyBytes);
const existingFile = await fileRepository.findByHash(prepared.fileHash);
if (existingFile) {
await cleanupTempFile(prepared.tempPath);
return Response.json(buildUploadResponse(existingFile, config.baseUrl), { status: 200 });
}
const rawMimeType = file.type || extractMimeType({}, req) || 'application/octet-stream';
const { fileName: finalFileName, mimeType } = ensureExtension(
fileName,
prepared.signatureBuffer,
rawMimeType,
);
const fileType = getFileType(mimeType, finalFileName);
if (!checkFileSize(prepared.sizeBytes, fileType)) {
await cleanupTempFile(prepared.tempPath);
return Response.json({ error: `File size exceeds ${fileType} limit` }, { status: 400 });
}
if (prepared.sizeBytes > config.telegramChunkSizeBytes) {
const uploadedFile = await chunkedStorage.storeFileInTelegramChunks({
tempPath: prepared.tempPath,
partFileNamePrefix: `direct-${prepared.fileHash?.slice(0, 16) || 'upload'}`,
fileName: finalFileName,
mimeType,
sizeBytes: prepared.sizeBytes,
fileType,
uploaderId: 0,
});
await cleanupTempFile(prepared.tempPath);
return Response.json(buildUploadResponse(uploadedFile, config.baseUrl), { status: 200 });
}
// Single-message — direct to Telegram storage
const forwardResult = await telegramService.forwardToStorage(
Readable.from(Bun.file(prepared.tempPath).stream()),
finalFileName,
fileType,
);
const publicId = nanoid();
const createdFile = await fileRepository.create(
buildNewFile({
publicId,
telegramFileId: forwardResult.telegramFileId,
telegramFileUniqueId: forwardResult.telegramFileUniqueId,
storageChatId: config.storageChatId,
storageMessageId: forwardResult.storageMessageId,
fileName: finalFileName,
mimeType,
sizeBytes: prepared.sizeBytes,
fileType,
storageBackend: 'telegram',
uploaderId: 0,
fileHash: prepared.fileHash,
}),
);
await cleanupTempFile(prepared.tempPath);
return Response.json(buildUploadResponse(createdFile, config.baseUrl), { status: 200 });
} catch (error: unknown) {
const message = getErrorMessage(error);
logger.error('Multipart upload error', { error: message });
return Response.json({ error: message }, { status: 500 });
}
};
/**
* Handles an application/json file upload where the file is sent as a
* base64-encoded string.
*
* Steps:
* 1. Parse the JSON body and extract the base64 file data.
* 2. Decode and estimate the file size; reject if too large for JSON.
* 3. Write the decoded buffer to a temp file.
* 4. Check deduplication by content hash.
* 5. Determine MIME type, file name, and Telegram file type.
* 6. Validate file size limits.
* 7. Upload to Telegram (chunked or single-message).
* 8. Return the upload response JSON.
*
* @param req - The incoming HTTP request with a JSON body.
* @returns A JSON response with the uploaded file metadata.
*/
const handleJSONUpload = async (req: Request): Promise<Response> => {
try {
const { file, fileName = 'file' } = (await req.json()) as JsonUploadPayload;
if (!file || typeof file !== 'string') {
return Response.json(
{ error: 'Invalid JSON. Must include "file" (base64) and optional "fileName"' },
{ status: 400 },
);
}
const { base64Data, mimeType: rawMimeType } = parseBase64File(file);
const estimatedSizeBytes = Math.floor((base64Data.length * 3) / 4);
if (
estimatedSizeBytes > JSON_UPLOAD_LIMIT_BYTES ||
estimatedSizeBytes > config.maxRequestBodyBytes
) {
return Response.json(
{
error:
'JSON base64 uploads are limited to 50MB. Use multipart/form-data for larger files',
},
{ status: 400 },
);
}
const fileBytes = Buffer.from(base64Data, 'base64');
const hash = computeHash(fileBytes);
const existingFile = await fileRepository.findByHash(hash);
if (existingFile) {
return Response.json(buildUploadResponse(existingFile, config.baseUrl), { status: 200 });
}
const fileTypeRaw = getFileType(rawMimeType, fileName);
const fileType = fileTypeRaw === 'application' ? 'document' : fileTypeRaw;
const { fileName: finalFileName, mimeType } = ensureExtension(fileName, fileBytes, rawMimeType);
if (!checkFileSize(fileBytes.byteLength, fileType)) {
return Response.json({ error: `File size exceeds ${fileType} limit` }, { status: 400 });
}
const prepared = await writeBufferToTemp(fileBytes, hash);
if (prepared.sizeBytes > config.telegramChunkSizeBytes) {
const uploadedFile = await chunkedStorage.storeFileInTelegramChunks({
tempPath: prepared.tempPath,
partFileNamePrefix: `direct-${prepared.fileHash?.slice(0, 16) || 'json'}`,
fileName: finalFileName,
mimeType,
sizeBytes: prepared.sizeBytes,
fileType,
uploaderId: 0,
});
await cleanupTempFile(prepared.tempPath);
return Response.json(buildUploadResponse(uploadedFile, config.baseUrl), { status: 200 });
}
// Single-message — direct to Telegram storage
const forwardResult = await telegramService.forwardToStorage(
Readable.from(Bun.file(prepared.tempPath).stream()),
finalFileName,
fileType,
);
const publicId = nanoid();
const createdFile = await fileRepository.create(
buildNewFile({
publicId,
telegramFileId: forwardResult.telegramFileId,
telegramFileUniqueId: forwardResult.telegramFileUniqueId,
storageChatId: config.storageChatId,
storageMessageId: forwardResult.storageMessageId,
fileName: finalFileName,
mimeType,
sizeBytes: prepared.sizeBytes,
fileType,
storageBackend: 'telegram',
uploaderId: 0,
fileHash: prepared.fileHash,
}),
);
await cleanupTempFile(prepared.tempPath);
return Response.json(buildUploadResponse(createdFile, config.baseUrl), { status: 200 });
} catch (error: unknown) {
const message = getErrorMessage(error);
logger.error('JSON upload error', { error: message });
return Response.json({ error: message }, { status: 500 });
}
};
/**
* Main upload request handler.
*
* Dispatches to either the multipart or JSON handler based on the request
* Content-Type header, returning an appropriate error for unsupported
* content types.
*
* Recording of upload metrics is handled centrally in this function.
*
* @param req - The incoming HTTP request.
* @returns A JSON response with the uploaded file metadata or an error.
*/
export const handleUpload = async (req: Request): Promise<Response> => {
const startTime = performance.now();
try {
const contentType = req.headers.get('content-type') || '';
const oversizedResponse = rejectOversizedRequest(req);
if (oversizedResponse) return oversizedResponse;
if (contentType.includes('multipart/form-data')) {
return handleMultipartUpload(req);
} else if (contentType.includes('application/json')) {
return handleJSONUpload(req);
}
return Response.json(
{ error: 'Unsupported content type. Use multipart/form-data or application/json' },
{ status: 400 },
);
} catch (error: unknown) {
metricsCollector.recordError();
const message = getErrorMessage(error);
logger.error('Upload error', { error: message });
return Response.json({ error: message }, { status: 500 });
} finally {
metricsCollector.recordUploadTime(performance.now() - startTime);
}
};
@@ -1,75 +1,121 @@
import { createReadStream } from 'node:fs'; import { createReadStream } from 'node:fs';
import { nanoid } from 'nanoid'; import { nanoid } from 'nanoid';
import { createBucket, deleteBucket, findBucketByName, listBuckets } from '../db/buckets'; import { buildNewFile } from '../../../domain/entities/file-factory';
import { config } from '../../../env';
import { bucketRepository, chunkedStorage, fileRepository } from '../../../infrastructure/di';
import { botPool } from '../../../infrastructure/telegram/bot-pool';
import logger from '../../../shared/logger/index';
import { import {
countBucketObjects, cleanupTempFile,
findFileByBucketAndKey, DEFAULT_FILE_TYPE,
listObjectsByPrefix, ensureExtension,
softDeleteFile, getErrorMessage,
} from '../db/files-ext'; } from '../../../shared/utils/file';
import { config } from '../env'; import { streamToTemp } from '../../../shared/utils/temp-stream';
import { createChunkedObjectResponse, storeFileInTelegramChunks } from '../utils/chunked-storage';
import { cleanupTempFile, computeHash, ensureExtension, getErrorMessage } from '../utils/file';
import logger from '../utils/logger';
import { forwardToStorage, getFileInfo } from '../utils/telegram';
/**
* Route parameters extracted from the URL path.
*/
type RouteParams = { bucket?: string; key?: string }; type RouteParams = { bucket?: string; key?: string };
const json = (data: unknown, status = 200) => Response.json(data, { status }); /**
* Returns a successful JSON Response.
*
* @param data - The JSON-serialisable body.
* @param status - HTTP status code (default 200).
* @returns A JSON Response.
*/
const json = (data: unknown, status = 200): Response => Response.json(data, { status });
const jsonError = (error: string, status: number) => Response.json({ error }, { status }); /**
* Returns a JSON error Response.
*
* @param error - The error message.
* @param status - HTTP status code.
* @returns A JSON Response.
*/
const jsonError = (error: string, status: number): Response => Response.json({ error }, { status });
// ─────── Bucket endpoints ─────── // ─────── Bucket endpoints ───────
/**
* Lists all buckets together with their object counts.
*
* @returns A JSON response with the bucket list.
*/
export const handleListBucketsV1 = async (): Promise<Response> => { export const handleListBucketsV1 = async (): Promise<Response> => {
const buckets = await listBuckets(); const buckets = await bucketRepository.list();
const result = await Promise.all( const result = await Promise.all(
buckets.map(async (b) => ({ buckets.map(async (b) => ({
id: b.id, id: b.id,
name: b.name, name: b.name,
createdAt: b.createdAt.toISOString(), createdAt: b.createdAt.toISOString(),
objectCount: await countBucketObjects(b.id), objectCount: await fileRepository.countByBucket(b.id),
})), })),
); );
return json({ buckets: result }); return json({ buckets: result });
}; };
/**
* Creates a new bucket.
*
* Validates the bucket name format and checks for duplicates before creating.
*
* @param req - The incoming HTTP request with a JSON body containing `name`.
* @returns A JSON response with the created bucket or an error.
*/
export const handleCreateBucketV1 = async (req: Request): Promise<Response> => { export const handleCreateBucketV1 = async (req: Request): Promise<Response> => {
const body = (await req.json()) as { name?: string }; const body = (await req.json()) as { name?: string };
if (!body.name || !/^[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$/.test(body.name)) { if (!body.name || !/^[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$/.test(body.name)) {
return jsonError('Invalid bucket name. Use lowercase, 3-63 chars, no underscore', 400); return jsonError('Invalid bucket name. Use lowercase, 3-63 chars, no underscore', 400);
} }
const existing = await findBucketByName(body.name); const existing = await bucketRepository.findByName(body.name);
if (existing) return jsonError('Bucket already exists', 409); if (existing) return jsonError('Bucket already exists', 409);
const bucket = await createBucket(body.name); const bucket = await bucketRepository.create(body.name);
return json({ id: bucket.id, name: bucket.name }, 201); return json({ id: bucket.id, name: bucket.name }, 201);
}; };
/**
* Deletes a bucket by name.
*
* Ensures the bucket exists and is empty before deletion.
*
* @param _req - The incoming HTTP request (unused).
* @param params - Route parameters containing the bucket name.
* @returns A JSON response indicating success or an error.
*/
export const handleDeleteBucketV1 = async ( export const handleDeleteBucketV1 = async (
_req: Request, _req: Request,
params: RouteParams, params: RouteParams,
): Promise<Response> => { ): Promise<Response> => {
const bucket = await findBucketByName(params.bucket!); const bucket = await bucketRepository.findByName(params.bucket!);
if (!bucket) return jsonError('Bucket not found', 404); if (!bucket) return jsonError('Bucket not found', 404);
const count = await countBucketObjects(bucket.id); const count = await fileRepository.countByBucket(bucket.id);
if (count > 0) return jsonError('Bucket is not empty', 409); if (count > 0) return jsonError('Bucket is not empty', 409);
await deleteBucket(params.bucket!); await bucketRepository.delete(params.bucket!);
return json({ success: true }); return json({ success: true });
}; };
// ─────── Object endpoints ─────── // ─────── Object endpoints ───────
/**
* Lists objects within a bucket (with prefix filtering and pagination).
*
* @param req - The incoming HTTP request with query parameters.
* @param params - Route parameters containing the bucket name.
* @returns A JSON response with the object list.
*/
export const handleListObjectsV1 = async (req: Request, params: RouteParams): Promise<Response> => { export const handleListObjectsV1 = async (req: Request, params: RouteParams): Promise<Response> => {
const bucket = await findBucketByName(params.bucket!); const bucket = await bucketRepository.findByName(params.bucket!);
if (!bucket) return jsonError('Bucket not found', 404); if (!bucket) return jsonError('Bucket not found', 404);
const url = new URL(req.url); const url = new URL(req.url);
const prefix = url.searchParams.get('prefix') || ''; const prefix = url.searchParams.get('prefix') || '';
const delimiter = url.searchParams.get('delimiter') || '/'; const delimiter = url.searchParams.get('delimiter') || '/';
const maxKeys = parseInt(url.searchParams.get('max-keys') || '1000', 10); const maxKeys = Number.parseInt(url.searchParams.get('max-keys') || '1000', 10);
const continuationToken = url.searchParams.get('continuation-token') || null; const continuationToken = url.searchParams.get('continuation-token') || null;
const { objects, prefixes } = await listObjectsByPrefix( const { objects, prefixes } = await fileRepository.listByPrefix(
bucket.id, bucket.id,
prefix, prefix,
delimiter, delimiter,
@@ -99,11 +145,20 @@ export const handleListObjectsV1 = async (req: Request, params: RouteParams): Pr
}); });
}; };
/**
* Uploads an object to a bucket (Web API V1).
*
* Accepts multipart/form-data with a `file` field and optional `key` field.
*
* @param req - The incoming HTTP request with a multipart body.
* @param params - Route parameters containing the bucket name.
* @returns A JSON response with the object metadata.
*/
export const handleUploadObjectV1 = async ( export const handleUploadObjectV1 = async (
req: Request, req: Request,
params: RouteParams, params: RouteParams,
): Promise<Response> => { ): Promise<Response> => {
const bucket = await findBucketByName(params.bucket!); const bucket = await bucketRepository.findByName(params.bucket!);
if (!bucket) return jsonError('Bucket not found', 404); if (!bucket) return jsonError('Bucket not found', 404);
const formData = await req.formData(); const formData = await req.formData();
@@ -114,55 +169,49 @@ export const handleUploadObjectV1 = async (
} }
const key = (formData.get('key') as string) || file.name; const key = (formData.get('key') as string) || file.name;
const buffer = Buffer.from(await file.arrayBuffer()); const streamed = await streamToTemp(file.stream().getReader(), { prefix: '/tmp/filedrop-web-' });
const hash = computeHash(buffer);
const tempPath = `/tmp/filedrop-web-${nanoid()}`;
await Bun.write(tempPath, buffer);
const signatureBuffer = buffer.subarray(0, 16);
const { fileName: finalFileName, mimeType } = ensureExtension( const { fileName: finalFileName, mimeType } = ensureExtension(
key.split('/').pop() || 'file', key.split('/').pop() || 'file',
signatureBuffer, streamed.signatureBuffer,
file.type || 'application/octet-stream', file.type || 'application/octet-stream',
); );
const partFileNamePrefix = `s3-${bucket.name}-${key.replace(/\//g, '_')}`; const partFileNamePrefix = `s3-${bucket.name}-${key.replace(/\//g, '_')}`;
if (buffer.byteLength > config.telegramChunkSizeBytes) { if (streamed.sizeBytes > config.telegramChunkSizeBytes) {
const file = await storeFileInTelegramChunks({ const uploadedFile = await chunkedStorage.storeFileInTelegramChunks({
tempPath, tempPath: streamed.tempPath,
partFileNamePrefix, partFileNamePrefix,
fileName: finalFileName, fileName: finalFileName,
mimeType, mimeType,
sizeBytes: buffer.byteLength, sizeBytes: streamed.sizeBytes,
fileType: 'document', fileType: DEFAULT_FILE_TYPE,
uploaderId: 0, uploaderId: 0,
bucketId: bucket.id, bucketId: bucket.id,
s3Key: key, s3Key: key,
}); });
await cleanupTempFile(tempPath); await cleanupTempFile(streamed.tempPath);
return json( return json(
{ {
key, key,
size: buffer.byteLength, size: streamed.sizeBytes,
etag: hash, etag: streamed.fileHash,
downloadUrl: `${config.baseUrl}/f/${file.publicId}`, downloadUrl: `${config.baseUrl}/f/${uploadedFile.publicId}`,
}, },
201, 201,
); );
} }
const forwardResult = await forwardToStorage( const forwardResult = await botPool.forwardToStorage(
createReadStream(tempPath), createReadStream(streamed.tempPath),
partFileNamePrefix, partFileNamePrefix,
'document', 'document',
); );
const publicId = nanoid(); const publicId = nanoid();
const { db, files: fileSchema } = await import('../db/index');
await db.insert(fileSchema).values({ await fileRepository.create(
buildNewFile({
publicId, publicId,
telegramFileId: forwardResult.telegramFileId, telegramFileId: forwardResult.telegramFileId,
telegramFileUniqueId: forwardResult.telegramFileUniqueId, telegramFileUniqueId: forwardResult.telegramFileUniqueId,
@@ -170,57 +219,89 @@ export const handleUploadObjectV1 = async (
storageMessageId: forwardResult.storageMessageId, storageMessageId: forwardResult.storageMessageId,
fileName: finalFileName, fileName: finalFileName,
mimeType, mimeType,
sizeBytes: buffer.byteLength, sizeBytes: streamed.sizeBytes,
fileType: 'document', fileType: DEFAULT_FILE_TYPE,
uploaderId: 0, uploaderId: 0,
fileHash: hash, fileHash: streamed.fileHash,
bucketId: bucket.id, bucketId: bucket.id,
s3Key: key, s3Key: key,
storageBackend: 'telegram', storageBackend: 'telegram',
isDeleted: false, }),
createdAt: new Date(), );
updatedAt: new Date(),
});
await cleanupTempFile(tempPath); await cleanupTempFile(streamed.tempPath);
return json( return json(
{ key, size: buffer.byteLength, etag: hash, downloadUrl: `${config.baseUrl}/f/${publicId}` }, {
key,
size: streamed.sizeBytes,
etag: streamed.fileHash,
downloadUrl: `${config.baseUrl}/f/${publicId}`,
},
201, 201,
); );
}; };
/**
* Deletes an object from a bucket (soft delete).
*
* @param _req - The incoming HTTP request (unused).
* @param params - Route parameters containing the bucket name and object key.
* @returns A JSON response indicating success.
*/
export const handleDeleteObjectV1 = async ( export const handleDeleteObjectV1 = async (
_req: Request, _req: Request,
params: RouteParams, params: RouteParams,
): Promise<Response> => { ): Promise<Response> => {
const bucket = await findBucketByName(params.bucket!); const bucket = await bucketRepository.findByName(params.bucket!);
if (!bucket) return jsonError('Bucket not found', 404); if (!bucket) return jsonError('Bucket not found', 404);
await softDeleteFile(bucket.id, params.key!); await fileRepository.softDelete(bucket.id, params.key!);
return json({ success: true }); return json({ success: true });
}; };
/**
* Downloads (or redirects to) an object from a bucket.
*
* For chunked objects, builds a streaming response. For regular Telegram
* objects, issues a 302 redirect to the Telegram CDN URL.
*
* @param _req - The incoming HTTP request (unused).
* @param params - Route parameters containing the bucket name and object key.
* @returns A redirect or streaming response, or a JSON error.
*/
export const handleDownloadObjectV1 = async ( export const handleDownloadObjectV1 = async (
_req: Request, _req: Request,
params: RouteParams, params: RouteParams,
): Promise<Response> => { ): Promise<Response> => {
const bucket = await findBucketByName(params.bucket!); const bucket = await bucketRepository.findByName(params.bucket!);
if (!bucket) return jsonError('Bucket not found', 404); if (!bucket) return jsonError('Bucket not found', 404);
const file = await findFileByBucketAndKey(bucket.id, params.key!); const file = await fileRepository.findByBucketAndKey(bucket.id, params.key!);
if (!file) return jsonError('Object not found', 404); if (!file) return jsonError('Object not found', 404);
if (file.storageBackend === 'chunked') { if (file.storageBackend === 'chunked') {
const range = { type: 'none' as const }; const range = { type: 'none' as const };
return createChunkedObjectResponse({ file, range, reqId: '' }); return chunkedStorage.createChunkedObjectResponse({ file, range, reqId: '' });
} }
const fileInfo = await getFileInfo(file.telegramFileId); const fileInfo = await botPool.getFileInfo(file.telegramFileId);
const redirectUrl = `https://api.telegram.org/file/bot${fileInfo.bot_token}/${fileInfo.file_path}`; const redirectUrl = `https://api.telegram.org/file/bot${fileInfo.bot_token}/${fileInfo.file_path}`;
return new Response(null, { status: 302, headers: { Location: redirectUrl } }); return new Response(null, { status: 302, headers: { Location: redirectUrl } });
}; };
/**
* Copies an object from one location to another within the same or a
* different bucket.
*
* Creates a new file record referencing the same Telegram-stored data as
* the source object.
*
* @param req - The incoming HTTP request with a JSON body specifying source
* and destination keys and the destination bucket.
* @param params - Route parameters containing the source bucket name.
* @returns A JSON response with the copy result, or an error.
*/
export const handleCopyObjectV1 = async (req: Request, params: RouteParams): Promise<Response> => { export const handleCopyObjectV1 = async (req: Request, params: RouteParams): Promise<Response> => {
const body = (await req.json()) as { const body = (await req.json()) as {
sourceKey?: string; sourceKey?: string;
@@ -233,12 +314,12 @@ export const handleCopyObjectV1 = async (req: Request, params: RouteParams): Pro
} }
const destBucketName = body.destBucket || params.bucket!; const destBucketName = body.destBucket || params.bucket!;
const sourceBucket = await findBucketByName(params.bucket!); const sourceBucket = await bucketRepository.findByName(params.bucket!);
const destBucket = await findBucketByName(destBucketName); const destBucket = await bucketRepository.findByName(destBucketName);
if (!sourceBucket || !destBucket) return jsonError('Bucket not found', 404); if (!sourceBucket || !destBucket) return jsonError('Bucket not found', 404);
const sourceFile = await findFileByBucketAndKey(sourceBucket.id, body.sourceKey); const sourceFile = await fileRepository.findByBucketAndKey(sourceBucket.id, body.sourceKey);
if (!sourceFile) return jsonError('Source object not found', 404); if (!sourceFile) return jsonError('Source object not found', 404);
if (sourceFile.storageBackend === 'chunked') { if (sourceFile.storageBackend === 'chunked') {
@@ -246,9 +327,9 @@ export const handleCopyObjectV1 = async (req: Request, params: RouteParams): Pro
} }
const publicId = nanoid(); const publicId = nanoid();
const { db, files: fileSchema } = await import('../db/index');
await db.insert(fileSchema).values({ await fileRepository.create(
buildNewFile({
publicId, publicId,
telegramFileId: sourceFile.telegramFileId, telegramFileId: sourceFile.telegramFileId,
telegramFileUniqueId: sourceFile.telegramFileUniqueId, telegramFileUniqueId: sourceFile.telegramFileUniqueId,
@@ -256,23 +337,28 @@ export const handleCopyObjectV1 = async (req: Request, params: RouteParams): Pro
storageMessageId: sourceFile.storageMessageId, storageMessageId: sourceFile.storageMessageId,
fileName: sourceFile.fileName, fileName: sourceFile.fileName,
mimeType: sourceFile.mimeType, mimeType: sourceFile.mimeType,
sizeBytes: sourceFile.sizeBytes, sizeBytes: Number(sourceFile.sizeBytes),
fileType: sourceFile.fileType, fileType: sourceFile.fileType,
uploaderId: 0, uploaderId: 0,
fileHash: sourceFile.fileHash, fileHash: sourceFile.fileHash,
bucketId: destBucket.id, bucketId: destBucket.id,
s3Key: body.destKey, s3Key: body.destKey,
storageBackend: 'telegram', storageBackend: 'telegram',
isDeleted: false, }),
createdAt: new Date(), );
updatedAt: new Date(),
});
return json({ sourceKey: body.sourceKey, destKey: body.destKey, destBucket: destBucketName }); return json({ sourceKey: body.sourceKey, destKey: body.destKey, destBucket: destBucketName });
}; };
// ─────── Router ─────── /**
* Main Web API V1 request router.
*
* Parses the request path and method, then dispatches to the appropriate
* handler function for bucket and object operations.
*
* @param req - The incoming HTTP request.
* @returns A JSON response from the matched handler, or 404.
*/
export const handleWebApiV1 = async (req: Request): Promise<Response> => { export const handleWebApiV1 = async (req: Request): Promise<Response> => {
const url = new URL(req.url); const url = new URL(req.url);
const pathname = url.pathname.replace(/^\/api\/v1/, ''); const pathname = url.pathname.replace(/^\/api\/v1/, '');
@@ -1,23 +1,39 @@
import { createHmac, timingSafeEqual } from 'node:crypto'; import { createHmac, timingSafeEqual } from 'node:crypto';
import { config } from '../env'; import { config } from '../../../env';
const ADMIN_USERNAME = 'admin'; const ADMIN_USERNAME = 'admin';
const SIGNATURE_SEPARATOR = '.'; const SIGNATURE_SEPARATOR = '.';
/** A request handler function that returns a Response. */
type Handler = (req: Request) => Response | Promise<Response>; type Handler = (req: Request) => Response | Promise<Response>;
/**
* Represents an authenticated user session after successful
* authentication via cookie or bearer token.
*/
export interface AuthSession { export interface AuthSession {
/** The authenticated username (always "admin" in this implementation). */
username: string; username: string;
/**
* Expiration date of the session, or `null` for bearer-token
* sessions which do not expire at the session level.
*/
expiresAt: Date | null; expiresAt: Date | null;
/** The authentication method used to establish this session. */
method: 'cookie' | 'bearer'; method: 'cookie' | 'bearer';
} }
/** Options for configuring cookie-based session behaviour. */
interface CookieOptions { interface CookieOptions {
/** HMAC signing secret (defaults to {@link config.adminApiToken}). */
secret?: string; secret?: string;
/** Name of the session cookie (defaults to {@link config.sessionCookieName}). */
cookieName?: string; cookieName?: string;
/** Session lifetime in milliseconds (defaults to {@link config.sessionMaxAgeMs}). */
maxAgeMs?: number; maxAgeMs?: number;
} }
/** Shape of the serialised cookie payload. */
interface SessionPayload { interface SessionPayload {
u: string; u: string;
e: number; e: number;
@@ -37,8 +53,25 @@ const decodePayload = (value: string): string | null => {
} }
}; };
/**
* Checks whether authentication is enabled.
*
* Authentication is considered enabled when the admin API token is
* non-empty.
*
* @param secret - Secret to check (defaults to `config.adminApiToken`).
* @returns `true` when auth is enabled, `false` otherwise.
*/
export const isAuthEnabled = (secret = config.adminApiToken): boolean => secret.length > 0; export const isAuthEnabled = (secret = config.adminApiToken): boolean => secret.length > 0;
/**
* Compares two strings using a timing-safe algorithm to prevent
* timing side-channel attacks.
*
* @param left - First string to compare.
* @param right - Second string to compare.
* @returns `true` when the strings are equal, `false` otherwise.
*/
export const timingSafeCompare = (left: string, right: string): boolean => { export const timingSafeCompare = (left: string, right: string): boolean => {
const leftBuffer = Buffer.from(left); const leftBuffer = Buffer.from(left);
const rightBuffer = Buffer.from(right); const rightBuffer = Buffer.from(right);
@@ -50,9 +83,29 @@ export const timingSafeCompare = (left: string, right: string): boolean => {
return timingSafeEqual(leftBuffer, rightBuffer); return timingSafeEqual(leftBuffer, rightBuffer);
}; };
/**
* Signs an arbitrary payload string with HMAC-SHA256 using the given
* secret, producing a base64url-encoded signature.
*
* @param payload - The value to sign.
* @param secret - HMAC signing key.
* @returns The base64url-encoded signature.
*/
export const signCookiePayload = (payload: string, secret: string): string => export const signCookiePayload = (payload: string, secret: string): string =>
createHmac('sha256', secret).update(payload).digest('base64url'); createHmac('sha256', secret).update(payload).digest('base64url');
/**
* Verifies the HMAC signature on a cookie value and returns the
* original signed payload.
*
* The cookie value is expected to be in the format
* `<payload>.<signature>`. Returns `null` when the format is
* invalid or the signature does not match.
*
* @param cookieValue - The full cookie value including signature.
* @param secret - HMAC signing key.
* @returns The unsigned payload string, or `null` on failure.
*/
export const verifyCookieSignature = (cookieValue: string, secret: string): string | null => { export const verifyCookieSignature = (cookieValue: string, secret: string): string | null => {
const separatorIndex = cookieValue.lastIndexOf(SIGNATURE_SEPARATOR); const separatorIndex = cookieValue.lastIndexOf(SIGNATURE_SEPARATOR);
if (separatorIndex <= 0 || separatorIndex === cookieValue.length - 1) { if (separatorIndex <= 0 || separatorIndex === cookieValue.length - 1) {
@@ -70,9 +123,28 @@ export const verifyCookieSignature = (cookieValue: string, secret: string): stri
return payload; return payload;
}; };
/**
* Builds the `Set-Cookie` attribute string for a given max-age in
* seconds. The cookie is HttpOnly, SameSite=Lax, Secure, and
* scoped to the root path.
*
* @param maxAgeSeconds - Max-Age in seconds.
* @returns The cookie attribute string (excluding name=value).
*/
const cookieAttributes = (maxAgeSeconds: number): string => const cookieAttributes = (maxAgeSeconds: number): string =>
[`Max-Age=${maxAgeSeconds}`, 'Path=/', 'HttpOnly', 'SameSite=Lax', 'Secure'].join('; '); [`Max-Age=${maxAgeSeconds}`, 'Path=/', 'HttpOnly', 'SameSite=Lax', 'Secure'].join('; ');
/**
* Creates a signed session cookie string suitable for use as a
* `Set-Cookie` header value.
*
* The cookie embeds a base64url-encoded JSON payload containing the
* username and expiration timestamp, signed with HMAC-SHA256.
*
* @param username - Session username (default `"admin"`).
* @param options - Optional cookie settings.
* @returns A fully-formed `Set-Cookie` header value.
*/
export const createSessionCookie = ( export const createSessionCookie = (
username = ADMIN_USERNAME, username = ADMIN_USERNAME,
options: CookieOptions = {}, options: CookieOptions = {},
@@ -90,9 +162,25 @@ export const createSessionCookie = (
return `${cookieName}=${payload}${SIGNATURE_SEPARATOR}${signature}; ${cookieAttributes(maxAgeSeconds)}`; return `${cookieName}=${payload}${SIGNATURE_SEPARATOR}${signature}; ${cookieAttributes(maxAgeSeconds)}`;
}; };
/**
* Creates a `Set-Cookie` header value that immediately expires the
* session cookie, effectively logging the user out.
*
* @param cookieName - Name of the cookie to clear (defaults to
* `config.sessionCookieName`).
* @returns A `Set-Cookie` header value with Max-Age=0.
*/
export const clearSessionCookie = (cookieName = config.sessionCookieName): string => export const clearSessionCookie = (cookieName = config.sessionCookieName): string =>
`${cookieName}=; ${cookieAttributes(0)}`; `${cookieName}=; ${cookieAttributes(0)}`;
/**
* Finds the value of a named cookie from a raw `Cookie` header
* string.
*
* @param cookieHeader - The raw `Cookie` header value, or `null`.
* @param cookieName - Name of the cookie to look for.
* @returns The cookie value, or `null` if not found.
*/
const findCookieValue = (cookieHeader: string | null, cookieName: string): string | null => { const findCookieValue = (cookieHeader: string | null, cookieName: string): string | null => {
if (!cookieHeader) return null; if (!cookieHeader) return null;
@@ -110,6 +198,19 @@ const findCookieValue = (cookieHeader: string | null, cookieName: string): strin
return null; return null;
}; };
/**
* Parses an {@link AuthSession} from a signed session cookie.
*
* The function verifies the HMAC signature, decodes the payload,
* and validates the expiration timestamp. Returns `null` when the
* cookie is missing, malformed, expired, or the signature is
* invalid. Also returns `null` when auth is disabled (empty
* admin API token).
*
* @param cookieHeader - The `Cookie` header value, or `null`.
* @param options - Optional overrides for secret / cookie name.
* @returns The parsed session, or `null`.
*/
export const parseSessionFromCookie = ( export const parseSessionFromCookie = (
cookieHeader: string | null, cookieHeader: string | null,
options: Pick<CookieOptions, 'secret' | 'cookieName'> = {}, options: Pick<CookieOptions, 'secret' | 'cookieName'> = {},
@@ -142,6 +243,15 @@ export const parseSessionFromCookie = (
} }
}; };
/**
* Validates a `Bearer` token from the `Authorization` header using
* timing-safe comparison.
*
* @param authorizationHeader - The raw `Authorization` header, or `null`.
* @param secret - Expected bearer token (defaults to
* `config.adminApiToken`).
* @returns `true` when the token is valid, `false` otherwise.
*/
export const checkBearerToken = ( export const checkBearerToken = (
authorizationHeader: string | null, authorizationHeader: string | null,
secret = config.adminApiToken, secret = config.adminApiToken,
@@ -155,6 +265,19 @@ export const checkBearerToken = (
return token.length > 0 && timingSafeCompare(token, secret); return token.length > 0 && timingSafeCompare(token, secret);
}; };
/**
* Extracts the authenticated session from a request.
*
* Tries cookie-based authentication first, then falls back to a
* Bearer token in the `Authorization` header. When auth is
* disabled (empty API token) the function returns a synthetic
* session with method `"bearer"` and no expiry, effectively
* granting access to all requests.
*
* @param req - The incoming HTTP request.
* @param options - Optional overrides for secret / cookie name.
* @returns The authenticated session, or `null` when unauthenticated.
*/
export const getAuthSession = ( export const getAuthSession = (
req: Request, req: Request,
options: Pick<CookieOptions, 'secret' | 'cookieName'> = {}, options: Pick<CookieOptions, 'secret' | 'cookieName'> = {},
@@ -182,9 +305,28 @@ export const getAuthSession = (
return null; return null;
}; };
/**
* Creates a 401 Unauthorized JSON response with a standard error
* body.
*
* @returns A `Response` with status 401 and JSON body
* `{ error: "Unauthorized" }`.
*/
export const unauthorizedResponse = (): Response => export const unauthorizedResponse = (): Response =>
Response.json({ error: 'Unauthorized' }, { status: 401 }); Response.json({ error: 'Unauthorized' }, { status: 401 });
/**
* Middleware that wraps a request handler with authentication.
*
* When auth is enabled the wrapper checks for a valid session
* (cookie or Bearer token) before delegating to the handler.
* Unauthenticated requests receive a 401 response. When auth is
* disabled the handler is always invoked.
*
* @param handler - The request handler to protect.
* @param options - Optional overrides for secret / cookie name.
* @returns A wrapped handler that performs the auth check.
*/
export const requireAuth = ( export const requireAuth = (
handler: Handler, handler: Handler,
options: Pick<CookieOptions, 'secret' | 'cookieName'> = {}, options: Pick<CookieOptions, 'secret' | 'cookieName'> = {},
@@ -1,6 +1,6 @@
import { config } from '../env'; import { config } from '../../../env';
import { extractClientIp } from './ip'; import logger from '../../../shared/logger/index';
import logger from './logger'; import { extractClientIp } from '../../../shared/utils/ip';
interface RateLimitEntry { interface RateLimitEntry {
count: number; count: number;
+121
View File
@@ -0,0 +1,121 @@
import { handleSwaggerHtml, handleSwaggerJson } from '../../../routes/swagger';
import { getS3RouteBucket, shouldHandleS3 } from '../../../shared/utils/s3-detection';
import { handleLogin, handleLogout, handleMe } from '../controllers/auth-controller';
import { handleFileInfo, handleFileRedirect } from '../controllers/file-controller';
import { handleHealth } from '../controllers/health-controller';
import { handleHome } from '../controllers/home-controller';
import { handleS3Request } from '../controllers/s3-controller';
import { handleUpload } from '../controllers/upload-controller';
import { handleWebApiV1 } from '../controllers/web-api-controller';
import { requireAuth } from '../middleware/auth';
import { withRateLimit } from '../middleware/rate-limit';
/**
* Dispatches an S3 request directly, bypassing rate limiting.
*
* S3 API calls (used by Docker registry for blob pushes) must not be
* rate-limited large concurrent layer uploads would hit the limit and
* fail. The Docker registry client retries on 5xx, not 4xx, so a 429
* would abort the entire push.
*
* @param req - The incoming S3 request.
* @returns The S3 response.
*/
const handleS3Direct = (req: Request): Promise<Response> => {
return handleS3Request(req, getS3RouteBucket(req));
};
/**
* Defines all HTTP routes for the application.
*
* Each route maps a URL pattern to its corresponding handler function(s),
* with middleware such as rate limiting and authentication applied where needed.
* This table is designed to be passed as the `routes` option to `Bun.serve()`.
*
* Route patterns follow Bun's routing syntax:
* - Static paths: `/health`
* - Parameterized paths: `/f/:public_id`
* - Wildcard paths: `/api/v1/*`
*/
export const routes = {
'/api/upload': {
POST: withRateLimit(handleUpload),
},
'/f/:public_id': {
GET: withRateLimit(handleFileRedirect),
},
'/file/:public_id/info': {
GET: withRateLimit(handleFileInfo),
},
'/health': {
GET: handleHealth,
},
'/docs': {
GET: handleSwaggerHtml,
},
'/swagger.json': {
GET: handleSwaggerJson,
},
'/': {
GET: (req: Request): Promise<Response> => {
if (shouldHandleS3(req)) return handleS3Direct(req);
return handleHome();
},
PUT: (req: Request): Promise<Response> => {
const headers = Object.fromEntries(req.headers);
if (shouldHandleS3(req, headers)) return handleS3Direct(req);
return Promise.resolve(new Response('Not Allowed', { status: 405 }));
},
HEAD: handleS3Direct,
DELETE: handleS3Direct,
POST: handleS3Direct,
OPTIONS: handleS3Direct,
},
// Catch-all for S3 path-style requests (/{bucket}/{key} ...)
// Only intercepts requests with S3 auth headers; others get 404.
'/*': {
GET: (req: Request): Promise<Response> => {
if (shouldHandleS3(req, Object.fromEntries(req.headers))) return handleS3Direct(req);
return Promise.resolve(new Response('Not Found', { status: 404 }));
},
PUT: (req: Request): Promise<Response> => {
if (shouldHandleS3(req, Object.fromEntries(req.headers))) return handleS3Direct(req);
return Promise.resolve(new Response('Not Found', { status: 404 }));
},
HEAD: (req: Request): Promise<Response> => {
if (shouldHandleS3(req, Object.fromEntries(req.headers))) return handleS3Direct(req);
return Promise.resolve(new Response('Not Found', { status: 404 }));
},
DELETE: (req: Request): Promise<Response> => {
if (shouldHandleS3(req, Object.fromEntries(req.headers))) return handleS3Direct(req);
return Promise.resolve(new Response('Not Found', { status: 404 }));
},
POST: (req: Request): Promise<Response> => {
if (shouldHandleS3(req, Object.fromEntries(req.headers))) return handleS3Direct(req);
return Promise.resolve(new Response('Not Found', { status: 404 }));
},
PATCH: (req: Request): Promise<Response> => {
if (shouldHandleS3(req, Object.fromEntries(req.headers))) return handleS3Direct(req);
return Promise.resolve(new Response('Not Found', { status: 404 }));
},
OPTIONS: handleS3Direct,
},
'/api/v1/auth/login': {
POST: withRateLimit(handleLogin),
},
'/api/v1/auth/logout': {
POST: handleLogout,
},
'/api/v1/auth/me': {
GET: handleMe,
},
// Read endpoints (GET) are public — anyone can list buckets/objects and
// download files. Write endpoints (POST/DELETE/PUT) require admin auth so
// visitors cannot upload, edit, copy, or delete.
'/api/v1/*': {
GET: handleWebApiV1,
POST: requireAuth(handleWebApiV1),
DELETE: requireAuth(handleWebApiV1),
PUT: requireAuth(handleWebApiV1),
},
};
@@ -1,3 +1,27 @@
import { timingSafeEqual } from 'node:crypto';
/**
* Timing-safe string comparison that prevents timing attacks.
*
* Uses `crypto.timingSafeEqual` which runs in constant time regardless of
* where the strings differ. Returns false for mismatched-length inputs
* to avoid leaking length information via early return.
*
* @param left - The first string to compare.
* @param right - The second string to compare.
* @returns True if both strings are equal.
*/
const timingSafeCompare = (left: string, right: string): boolean => {
const leftBuffer = Buffer.from(left);
const rightBuffer = Buffer.from(right);
if (leftBuffer.length !== rightBuffer.length) {
return false;
}
return timingSafeEqual(leftBuffer, rightBuffer);
};
export interface SigV4Result { export interface SigV4Result {
isValid: boolean; isValid: boolean;
credential: { credential: {
@@ -22,6 +46,12 @@ export interface VerifyPresignedUrlInput {
const SERVICE = 's3'; const SERVICE = 's3';
const TERMINATION = 'aws4_request'; const TERMINATION = 'aws4_request';
/**
* Maximum acceptable clock skew between client and server for header-based
* SigV4 authentication. AWS allows 15 minutes.
*/
const MAX_CLOCK_SKEW_MS = 15 * 60 * 1000;
// eslint-disable-next-line @typescript-eslint/no-explicit-any // eslint-disable-next-line @typescript-eslint/no-explicit-any
const buf = (data: string | ArrayBuffer | Uint8Array): Uint8Array => { const buf = (data: string | ArrayBuffer | Uint8Array): Uint8Array => {
if (data instanceof Uint8Array) return data; if (data instanceof Uint8Array) return data;
@@ -105,10 +135,37 @@ const buildCanonicalRequest = (
return `${method}\n${canonicalUri}\n${canonicalQueryString}\n${canonicalHeaders}\n${signedHeaders}\n${hashedPayload}`; return `${method}\n${canonicalUri}\n${canonicalQueryString}\n${canonicalHeaders}\n${signedHeaders}\n${hashedPayload}`;
}; };
/**
* Normalizes a URI per AWS SigV4 requirements plus RFC 3986:
*
* 1. Decode percent-encoded characters
* 2. Remove dot-segments (`.` and `..`) per RFC 3986 section 5.2.4
*
* @param uri - The raw URI path to normalize.
* @returns The normalized URI path.
*/
const normalizeUri = (uri: string): string => { const normalizeUri = (uri: string): string => {
if (!uri || uri === '') return '/'; if (!uri || uri === '') return '/';
// AWS SigV4 requires URI-decoded paths in the canonical request // AWS SigV4 requires URI-decoded paths in the canonical request
return decodeURIComponent(uri); // Only `.` and `..` segments are removed per RFC 3986 section 5.2.4
// Empty segments (from `//` or trailing `/`) are preserved — they are
// part of the URI and the SDK signs them.
const decoded = decodeURIComponent(uri);
const segments = decoded.split('/');
const result: string[] = [];
for (const segment of segments) {
if (segment === '.') continue;
if (segment === '..') {
result.pop();
continue;
}
result.push(segment);
}
// Join preserves empty first segment (from leading /) automatically
return result.join('/') || '/';
}; };
const awsEncode = (value: string): string => const awsEncode = (value: string): string =>
@@ -125,23 +182,57 @@ export const buildCanonicalQueryString = (
for (const [key, value] of searchParams.entries()) { for (const [key, value] of searchParams.entries()) {
if (!excludeKeys.has(key)) pairs.push([key, value]); if (!excludeKeys.has(key)) pairs.push([key, value]);
} }
// AWS SigV4 requires UTF-8 byte-order (code point) comparison, NOT localeCompare
pairs.sort(([ak, av], [bk, bv]) => { pairs.sort(([ak, av], [bk, bv]) => {
const a = `${awsEncode(ak)}=${awsEncode(av)}`; const a = `${awsEncode(ak)}=${awsEncode(av)}`;
const b = `${awsEncode(bk)}=${awsEncode(bv)}`; const b = `${awsEncode(bk)}=${awsEncode(bv)}`;
return a.localeCompare(b); if (a < b) return -1;
if (a > b) return 1;
return 0;
}); });
return pairs.map(([key, value]) => `${awsEncode(key)}=${awsEncode(value)}`).join('&'); return pairs.map(([key, value]) => `${awsEncode(key)}=${awsEncode(value)}`).join('&');
}; };
const getHashedPayload = async ( const getHashedPayload = async (body: string | null): Promise<string> => {
body: string | null,
contentSha256: string | null,
): Promise<string> => {
if (contentSha256) return contentSha256;
if (!body || body.length === 0) return await sha256Hex(''); if (!body || body.length === 0) return await sha256Hex('');
return await sha256Hex(body); return await sha256Hex(body);
}; };
/**
* Parses an AWS SigV4 `x-amz-date` value (e.g. `20260707T120000Z`) into a Date.
*
* @param amzDate - The date string in `YYYYMMDDTHHmmssZ` format.
* @returns The parsed Date, or null if the format is invalid.
*/
const parseAmzDateUtc = (amzDate: string): Date | null => {
const match = amzDate.match(/^(\d{4})(\d{2})(\d{2})T(\d{2})(\d{2})(\d{2})Z$/);
if (!match) return null;
const [, year, month, day, hour, minute, second] = match;
return new Date(
Date.UTC(
Number.parseInt(year, 10),
Number.parseInt(month, 10) - 1,
Number.parseInt(day, 10),
Number.parseInt(hour, 10),
Number.parseInt(minute, 10),
Number.parseInt(second, 10),
),
);
};
/**
* Validates that `host` is included in the signed headers list.
*
* AWS SigV4 mandates that `host` is always signed. Reject requests that
* omit it to prevent header injection / replay variants.
*
* @param signedHeaders - The semicolon-separated signed headers string.
* @returns True if `host` is present.
*/
const validateSignedHeaders = (signedHeaders: string): boolean => {
return signedHeaders.split(';').some((h) => h.toLowerCase() === 'host');
};
export const verifySignature = async ( export const verifySignature = async (
method: string, method: string,
url: string, url: string,
@@ -161,14 +252,24 @@ export const verifySignature = async (
return { isValid: false, credential: null, errorCode: 'AccessDenied' }; return { isValid: false, credential: null, errorCode: 'AccessDenied' };
} }
if (parsed.accessKey !== s3AccessKey) { if (!timingSafeCompare(parsed.accessKey, s3AccessKey)) {
return { isValid: false, credential: null, errorCode: 'SignatureDoesNotMatch' }; return { isValid: false, credential: null, errorCode: 'SignatureDoesNotMatch' };
} }
if (parsed.region !== region) { if (!timingSafeCompare(parsed.region, region)) {
return { isValid: false, credential: null, errorCode: 'SignatureDoesNotMatch' }; return { isValid: false, credential: null, errorCode: 'SignatureDoesNotMatch' };
} }
// Validate service and termination in credential scope (M2)
if (parsed.service !== SERVICE || parsed.termination !== TERMINATION) {
return { isValid: false, credential: null, errorCode: 'SignatureDoesNotMatch' };
}
// Validate host is in signed headers (LOW/host)
if (!validateSignedHeaders(parsed.signedHeaders)) {
return { isValid: false, credential: null, errorCode: 'AccessDenied' };
}
const parsedUrl = new URL(url, 'http://localhost'); const parsedUrl = new URL(url, 'http://localhost');
const canonicalUri = normalizeUri(parsedUrl.pathname); const canonicalUri = normalizeUri(parsedUrl.pathname);
const canonicalQueryString = buildCanonicalQueryString(parsedUrl.searchParams); const canonicalQueryString = buildCanonicalQueryString(parsedUrl.searchParams);
@@ -177,7 +278,12 @@ export const verifySignature = async (
if (contentSha256?.startsWith('STREAMING-')) { if (contentSha256?.startsWith('STREAMING-')) {
return { isValid: false, credential: null, errorCode: 'NotImplemented' }; return { isValid: false, credential: null, errorCode: 'NotImplemented' };
} }
const hashedPayload = await getHashedPayload(body, contentSha256);
// CRITICAL: Use the x-amz-content-sha256 header value in the canonical
// request because that's what the client signed. The actual body hash is
// verified by verifyBodyHash() after streaming, ensuring integrity without
// breaking SigV4.
const hashedPayload = contentSha256 || (await getHashedPayload(body));
const canonicalRequest = buildCanonicalRequest( const canonicalRequest = buildCanonicalRequest(
method, method,
@@ -190,8 +296,31 @@ export const verifySignature = async (
const hashedCanonicalRequest = await sha256Hex(canonicalRequest); const hashedCanonicalRequest = await sha256Hex(canonicalRequest);
const amzDate = headers['x-amz-date'] || ''; // M1: Fall back to Date header if x-amz-date is missing
const amzDate = headers['x-amz-date'] || headers.date || '';
// H5: Validate request freshness (clock skew / replay protection)
if (amzDate) {
const requestDate = parseAmzDateUtc(amzDate);
if (requestDate) {
const now = Date.now();
const skew = Math.abs(now - requestDate.getTime());
if (skew > MAX_CLOCK_SKEW_MS) {
return { isValid: false, credential: null, errorCode: 'RequestExpired' };
}
}
}
const dateStamp = parsed.date; const dateStamp = parsed.date;
// M3: Ensure date in credential scope matches x-amz-date
if (amzDate) {
const amzDateStamp = amzDate.slice(0, 8); // "YYYYMMDD"
if (amzDateStamp !== dateStamp) {
return { isValid: false, credential: null, errorCode: 'SignatureDoesNotMatch' };
}
}
const credentialScope = `${dateStamp}/${region}/${parsed.service}/${parsed.termination}`; const credentialScope = `${dateStamp}/${region}/${parsed.service}/${parsed.termination}`;
const stringToSign = `AWS4-HMAC-SHA256\n${amzDate}\n${credentialScope}\n${hashedCanonicalRequest}`; const stringToSign = `AWS4-HMAC-SHA256\n${amzDate}\n${credentialScope}\n${hashedCanonicalRequest}`;
@@ -199,7 +328,7 @@ export const verifySignature = async (
const signingKey = await getSigningKey(s3SecretKey, dateStamp, region); const signingKey = await getSigningKey(s3SecretKey, dateStamp, region);
const expectedSignature = await hmacHex(signingKey, stringToSign); const expectedSignature = await hmacHex(signingKey, stringToSign);
if (expectedSignature !== parsed.signature) { if (!timingSafeCompare(expectedSignature, parsed.signature)) {
return { isValid: false, credential: null, errorCode: 'SignatureDoesNotMatch' }; return { isValid: false, credential: null, errorCode: 'SignatureDoesNotMatch' };
} }
@@ -214,22 +343,6 @@ export const verifySignature = async (
}; };
}; };
const parseAmzDateUtc = (amzDate: string): Date | null => {
const match = amzDate.match(/^(\d{4})(\d{2})(\d{2})T(\d{2})(\d{2})(\d{2})Z$/);
if (!match) return null;
const [, year, month, day, hour, minute, second] = match;
return new Date(
Date.UTC(
Number.parseInt(year, 10),
Number.parseInt(month, 10) - 1,
Number.parseInt(day, 10),
Number.parseInt(hour, 10),
Number.parseInt(minute, 10),
Number.parseInt(second, 10),
),
);
};
export const verifyPresignedUrl = async ({ export const verifyPresignedUrl = async ({
url, url,
method, method,
@@ -265,7 +378,12 @@ export const verifyPresignedUrl = async ({
if (!Number.isFinite(expires) || expires <= 0 || !signedAt) { if (!Number.isFinite(expires) || expires <= 0 || !signedAt) {
return { isValid: false, credential: null, errorCode: 'AccessDenied' }; return { isValid: false, credential: null, errorCode: 'AccessDenied' };
} }
if (now.getTime() > signedAt.getTime() + expires * 1000) { // AWS S3 spec limits presigned URLs to 7 days (604800 seconds)
const MAX_PRESIGNED_EXPIRY_SECONDS = 604800;
if (
now.getTime() > signedAt.getTime() + expires * 1000 ||
expires > MAX_PRESIGNED_EXPIRY_SECONDS
) {
return { isValid: false, credential: null, errorCode: 'AccessDenied' }; return { isValid: false, credential: null, errorCode: 'AccessDenied' };
} }
@@ -275,14 +393,19 @@ export const verifyPresignedUrl = async ({
} }
const [accessKey, dateStamp, credentialRegion, service, termination] = credParts; const [accessKey, dateStamp, credentialRegion, service, termination] = credParts;
if ( if (
accessKey !== s3AccessKey || !timingSafeCompare(accessKey, s3AccessKey) ||
credentialRegion !== region || !timingSafeCompare(credentialRegion, region) ||
service !== SERVICE || service !== SERVICE ||
termination !== TERMINATION termination !== TERMINATION
) { ) {
return { isValid: false, credential: null, errorCode: 'SignatureDoesNotMatch' }; return { isValid: false, credential: null, errorCode: 'SignatureDoesNotMatch' };
} }
// Validate host is in signed headers for presigned URLs too
if (!validateSignedHeaders(signedHeaders)) {
return { isValid: false, credential: null, errorCode: 'AccessDenied' };
}
const signedHeaderList = signedHeaders.split(';').filter(Boolean); const signedHeaderList = signedHeaders.split(';').filter(Boolean);
const canonicalHeaders = signedHeaderList const canonicalHeaders = signedHeaderList
.map((headerName) => { .map((headerName) => {
@@ -301,7 +424,7 @@ export const verifyPresignedUrl = async ({
stringToSign, stringToSign,
); );
if (expectedSignature !== signature) { if (!timingSafeCompare(expectedSignature, signature)) {
return { isValid: false, credential: null, errorCode: 'SignatureDoesNotMatch' }; return { isValid: false, credential: null, errorCode: 'SignatureDoesNotMatch' };
} }
return { isValid: true, credential: { accessKey, date: dateStamp, region, service } }; return { isValid: true, credential: { accessKey, date: dateStamp, region, service } };
@@ -311,3 +434,30 @@ export const isS3Request = (headers: Record<string, string>): boolean => {
const auth = headers.authorization || ''; const auth = headers.authorization || '';
return auth.startsWith('AWS4-HMAC-SHA256'); return auth.startsWith('AWS4-HMAC-SHA256');
}; };
/**
* Verifies that the actual body SHA-256 matches the `x-amz-content-sha256`
* header from the original request.
*
* This MUST be called AFTER the body has been fully streamed and hashed,
* as a second pass after `verifySignature` (which cannot hash a streaming
* body without consuming it).
*
* @param bodySha256 - The SHA-256 hex digest of the actual body content.
* @param headers - The original request headers.
* @returns An error result on mismatch, or null if the check passes.
*/
export const verifyBodyHash = (
bodySha256: string,
headers: Record<string, string>,
): SigV4Result | null => {
const claimedHash = headers['x-amz-content-sha256'];
// If the client sent UNSIGNED-PAYLOAD, skip verification
if (!claimedHash || claimedHash === 'UNSIGNED-PAYLOAD' || claimedHash.startsWith('STREAMING-')) {
return null;
}
if (claimedHash !== bodySha256) {
return { isValid: false, credential: null, errorCode: 'BadDigest' };
}
return null;
};
@@ -1,3 +1,5 @@
import { nanoid } from 'nanoid';
export const S3_CORS_HEADERS: Record<string, string> = { export const S3_CORS_HEADERS: Record<string, string> = {
'access-control-allow-origin': '*', 'access-control-allow-origin': '*',
'access-control-allow-methods': 'GET, PUT, HEAD, DELETE, POST, OPTIONS', 'access-control-allow-methods': 'GET, PUT, HEAD, DELETE, POST, OPTIONS',
@@ -31,7 +33,13 @@ export const s3Headers = (
extraHeaders: Record<string, string> = {}, extraHeaders: Record<string, string> = {},
): Record<string, string> => ({ ): Record<string, string> => ({
...S3_CORS_HEADERS, ...S3_CORS_HEADERS,
...(requestId ? { 'x-amz-request-id': requestId, 'x-amz-id-2': requestId } : {}), server: 'AmazonS3',
...(requestId
? {
'x-amz-request-id': requestId,
'x-amz-id-2': `${requestId}+${nanoid(16)}`,
}
: {}),
...extraHeaders, ...extraHeaders,
}); });
@@ -2,6 +2,9 @@ import { gunzipSync } from 'node:zlib';
import { applyS3Headers } from './headers'; import { applyS3Headers } from './headers';
import { contentRange, type RangeParseResult } from './range'; import { contentRange, type RangeParseResult } from './range';
/** Timeout (ms) for individual Telegram CDN chunk fetches. */
const TELEGRAM_FETCH_TIMEOUT_MS = 30_000;
export interface ObjectPartSource { export interface ObjectPartSource {
telegramFileId: string; telegramFileId: string;
telegramUrl: string; telegramUrl: string;
@@ -61,9 +64,15 @@ const streamFromBytes = (bytes: Uint8Array): ReadableStream<Uint8Array> =>
new Response(bytes).body!; new Response(bytes).body!;
const fetchWholePartBytes = async (telegramUrl: string): Promise<Uint8Array> => { const fetchWholePartBytes = async (telegramUrl: string): Promise<Uint8Array> => {
const res = await fetch(telegramUrl); const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), TELEGRAM_FETCH_TIMEOUT_MS);
try {
const res = await fetch(telegramUrl, { signal: controller.signal });
if (!res.ok) throw new Error(`Telegram fetch failed: ${res.status}`); if (!res.ok) throw new Error(`Telegram fetch failed: ${res.status}`);
return new Uint8Array(await res.arrayBuffer()); return new Uint8Array(await res.arrayBuffer());
} finally {
clearTimeout(timer);
}
}; };
const fetchPartBody = async (planned: PlannedPart): Promise<ReadableStream<Uint8Array>> => { const fetchPartBody = async (planned: PlannedPart): Promise<ReadableStream<Uint8Array>> => {
@@ -77,10 +86,10 @@ const fetchPartBody = async (planned: PlannedPart): Promise<ReadableStream<Uint8
} }
const rangeHeader = `bytes=${planned.relativeStart}-${planned.relativeEnd}`; const rangeHeader = `bytes=${planned.relativeStart}-${planned.relativeEnd}`;
const res = await fetch( const fetchOpts = wantsWholePart
planned.part.telegramUrl, ? { signal: AbortSignal.timeout(TELEGRAM_FETCH_TIMEOUT_MS) }
wantsWholePart ? undefined : { headers: { range: rangeHeader } }, : { headers: { range: rangeHeader }, signal: AbortSignal.timeout(TELEGRAM_FETCH_TIMEOUT_MS) };
); const res = await fetch(planned.part.telegramUrl, fetchOpts);
if (!res.ok) throw new Error(`Telegram fetch failed: ${res.status}`); if (!res.ok) throw new Error(`Telegram fetch failed: ${res.status}`);
if (wantsWholePart || res.status === 206) return res.body!; if (wantsWholePart || res.status === 206) return res.body!;
@@ -1,5 +1,9 @@
const stripPort = (host: string): string => { const stripPort = (host: string): string => {
if (host.startsWith('[')) return host; // Handle IPv6: [::1]:4321 -> [::1]
if (host.startsWith('[')) {
const closeBracket = host.indexOf(']');
return host.slice(0, closeBracket + 1).toLowerCase();
}
return host.split(':')[0].toLowerCase().replace(/\.$/, ''); return host.split(':')[0].toLowerCase().replace(/\.$/, '');
}; };
@@ -277,8 +277,10 @@ export const s3ErrorResponse = (
// ─────── DeleteObjects XML parser ─────── // ─────── DeleteObjects XML parser ───────
export const parseDeleteObjectsBody = (body: string): { keys: string[]; quiet: boolean } => { export const parseDeleteObjectsBody = (body: string): { keys: string[]; quiet: boolean } => {
const keys = Array.from(body.matchAll(/<Key>([^<]+)<\/Key>/g), (match) => match[1]); // H9: Use non-greedy match to handle keys containing < character
const quiet = body.includes('<Quiet>true</Quiet>') || body.includes('<Quiet>true '); const keys = Array.from(body.matchAll(/<Key>([\s\S]*?)<\/Key>/g), (match) => match[1]);
// Handle whitespace inside <Quiet> element + namespace prefix support
const quiet = /<\w*:?Quiet\w*>\s*true\s*<\/\w*:?Quiet\w*>/i.test(body);
return { keys, quiet }; return { keys, quiet };
}; };
@@ -299,7 +301,7 @@ export const parseCompleteMultipartBody = (body: string): CompletePart[] => {
const etagMatch = partXml.match(/<ETag>"?([^"<\s]+)"?<\/ETag>/); const etagMatch = partXml.match(/<ETag>"?([^"<\s]+)"?<\/ETag>/);
if (numMatch && etagMatch) { if (numMatch && etagMatch) {
parts.push({ parts.push({
partNumber: parseInt(numMatch[1], 10), partNumber: Number.parseInt(numMatch[1], 10),
etag: etagMatch[1].replace(/^"/, '').replace(/"$/, ''), etag: etagMatch[1].replace(/^"/, '').replace(/"$/, ''),
}); });
} }
-64
View File
@@ -1,64 +0,0 @@
import { config } from '../env';
import {
checkBearerToken,
clearSessionCookie,
createSessionCookie,
getAuthSession,
isAuthEnabled,
timingSafeCompare,
} from '../utils/auth';
const json = (data: unknown, status = 200, headers: Record<string, string> = {}): Response =>
Response.json(data, { status, headers });
const notFound = (): Response => json({ error: 'Not found' }, 404);
const readLoginBody = async (req: Request): Promise<{ token: string } | null> => {
try {
const body = (await req.json()) as { token?: unknown };
if (typeof body.token !== 'string' || body.token.length === 0) return null;
return { token: body.token };
} catch {
return null;
}
};
export const handleLogin = async (req: Request): Promise<Response> => {
if (!isAuthEnabled()) return notFound();
const body = await readLoginBody(req);
if (!body) return json({ error: 'Token is required' }, 400);
if (!timingSafeCompare(body.token, config.adminApiToken)) {
return json({ error: 'Invalid token' }, 401);
}
return json({ username: 'admin' }, 200, {
'set-cookie': createSessionCookie('admin'),
});
};
export const handleLogout = async (): Promise<Response> =>
json({ success: true }, 200, {
'set-cookie': clearSessionCookie(),
});
export const handleMe = async (req: Request): Promise<Response> => {
if (!isAuthEnabled()) return notFound();
const session = getAuthSession(req);
if (!session && !checkBearerToken(req.headers.get('authorization'))) {
return json({ error: 'Unauthorized' }, 401);
}
const activeSession = session ?? {
username: 'admin',
expiresAt: null,
method: 'bearer' as const,
};
return json({
username: activeSession.username,
expiresAt: activeSession.expiresAt?.toISOString() ?? null,
});
};
-152
View File
@@ -1,152 +0,0 @@
import { createReadStream } from 'node:fs';
import { nanoid } from 'nanoid';
import { findFileByPublicId } from '../db/files';
import { fileInfoCache } from '../utils/cache';
import { createChunkedObjectResponse } from '../utils/chunked-storage';
import { cleanupTempFile, formatCreatedAt, getErrorMessage } from '../utils/file';
import logger from '../utils/logger';
import { metricsCollector } from '../utils/metrics';
import { getFileInfo, type TelegramFileInfo } from '../utils/telegram';
import { locateZipEntry } from '../utils/zip';
type RequestWithParams = Request & {
params?: {
public_id?: string;
};
};
const getTelegramFileInfo = async (telegramFileId: string, public_id: string) => {
const cacheKey = `file_info_${telegramFileId}`;
let fileInfo = fileInfoCache.get(cacheKey) as TelegramFileInfo | null;
if (!fileInfo) {
metricsCollector.recordCacheMiss();
fileInfo = await getFileInfo(telegramFileId);
fileInfoCache.set(cacheKey, fileInfo);
logger.debug('File info cached', { public_id, cacheKey });
} else {
metricsCollector.recordCacheHit();
logger.debug('File info from cache', { public_id, cacheKey });
}
return fileInfo;
};
const buildTelegramFileUrl = (filePath: string, botToken: string): string =>
`https://api.telegram.org/file/bot${botToken}/${filePath}`;
const sanitizeFilenameHeader = (fileName: string): string =>
fileName.replace(/[\\"]/g, '').replace(/[\n\r]/g, '');
const fail = (status: number, error: string): Response => Response.json({ error }, { status });
export const handleFileRedirect = async (req: RequestWithParams): Promise<Response> => {
const public_id = req.params?.public_id;
try {
if (!public_id) {
return fail(400, 'Missing file id');
}
const file = await findFileByPublicId(public_id);
if (!file) {
logger.warn('File not found', { public_id });
return fail(404, 'File not found');
}
if (file.storageBackend === 'chunked') {
if (file.archiveEntryName) {
return fail(501, 'Archive entry extraction is not supported for chunked files');
}
const range = { type: 'none' as const };
return createChunkedObjectResponse({ file, range, reqId: '' });
}
const archiveEntryName = file.archiveEntryName;
if (archiveEntryName) {
const archiveFileId = file.archiveTelegramFileId || file.telegramFileId;
const archiveInfo = await getTelegramFileInfo(archiveFileId, public_id);
const archiveResponse = await fetch(
buildTelegramFileUrl(archiveInfo.file_path, archiveInfo.bot_token),
);
if (!archiveResponse.ok) {
logger.error('Archive download failed', { public_id, status: archiveResponse.status });
return fail(500, 'Server error');
}
const tempZipPath = `/tmp/filedrop-dl-${nanoid()}.zip`;
await Bun.write(tempZipPath, archiveResponse);
const loc = await locateZipEntry(tempZipPath, archiveEntryName);
if (!loc) {
await cleanupTempFile(tempZipPath);
logger.error('Archive entry not found', { public_id, archiveEntryName });
return fail(404, 'File not found');
}
const fileStream = createReadStream(tempZipPath, {
start: loc.start,
end: loc.start + loc.length - 1,
});
fileStream.on('close', () => {
void cleanupTempFile(tempZipPath);
});
fileStream.on('error', () => {
void cleanupTempFile(tempZipPath);
});
return new Response(fileStream as unknown as ReadableStream, {
status: 200,
headers: {
'Content-Type': file.mimeType || 'application/octet-stream',
'Content-Disposition': `attachment; filename="${sanitizeFilenameHeader(file.fileName)}"`,
'Content-Length': String(loc.length),
},
});
}
const fileInfo = await getTelegramFileInfo(file.telegramFileId, public_id);
const redirectUrl = buildTelegramFileUrl(fileInfo.file_path, fileInfo.bot_token);
return new Response(null, {
status: 302,
headers: {
Location: redirectUrl,
},
});
} catch (error: unknown) {
logger.error('File redirect error', { public_id, error: getErrorMessage(error) });
return fail(500, 'Server error');
}
};
export const handleFileInfo = async (req: RequestWithParams): Promise<Response> => {
const public_id = req.params?.public_id;
try {
if (!public_id) {
return fail(400, 'Missing file id');
}
const file = await findFileByPublicId(public_id);
if (!file) {
logger.warn('File not found', { public_id });
return fail(404, 'File not found');
}
return Response.json(
{
public_id: file.publicId,
file_name: file.fileName,
mime_type: file.mimeType,
size_bytes: file.sizeBytes,
file_type: file.fileType,
created_at: formatCreatedAt(file.createdAt),
},
{ status: 200 },
);
} catch (error: unknown) {
logger.error('File info error', { public_id, error: getErrorMessage(error) });
return fail(500, 'Server error');
}
};
-15
View File
@@ -1,15 +0,0 @@
import { sql } from 'drizzle-orm';
import { db } from '../db';
import { getErrorMessage } from '../utils/file';
import logger from '../utils/logger';
export const handleHealth = async (_req: Request): Promise<Response> => {
try {
await db.execute(sql`SELECT 1`);
return Response.json({ status: 'ok' }, { status: 200 });
} catch (error: unknown) {
const message = getErrorMessage(error);
logger.error('Health check failed', { error: message });
return Response.json({ status: 'error', error: message }, { status: 500 });
}
};
-9
View File
@@ -1,9 +0,0 @@
export const handleHome = async (): Promise<Response> => {
const html = await Bun.file(`${import.meta.dir}/../home.html`).text();
return new Response(html, {
status: 200,
headers: {
'content-type': 'text/html; charset=utf-8',
},
});
};
-1178
View File
File diff suppressed because it is too large Load Diff
-303
View File
@@ -1,303 +0,0 @@
import { createWriteStream } from 'node:fs';
import { nanoid } from 'nanoid';
import { findFileByHash } from '../db/files';
import { config } from '../env';
import { storeFileInTelegramChunks } from '../utils/chunked-storage';
import {
buildUploadResponse,
checkFileSize,
cleanupTempFile,
computeHash,
ensureExtension,
extractMimeType,
getErrorMessage,
getFileType,
} from '../utils/file';
import logger from '../utils/logger';
import { metricsCollector } from '../utils/metrics';
import { enqueuePreparedUpload, type PreparedUpload } from '../utils/uploadBatcher';
interface JsonUploadPayload {
file?: unknown;
fileName?: string;
}
const parseBase64File = (file: string): { base64Data: string; mimeType: string } => {
if (!file.startsWith('data:')) {
return { base64Data: file, mimeType: 'application/octet-stream' };
}
const match = file.match(/^data:([^;]+);base64,(.+)$/);
return match
? { base64Data: match[2], mimeType: match[1] }
: { base64Data: file, mimeType: 'application/octet-stream' };
};
const normalizeFileType = (mimeType: string, fileName: string): string => {
const fileType = getFileType(mimeType, fileName);
return fileType === 'application' ? 'document' : fileType;
};
const JSON_UPLOAD_LIMIT_BYTES = 50 * 1024 * 1024;
const SIGNATURE_BYTES = 16;
const getContentLength = (req: Request): number | null => {
const value = req.headers.get('content-length');
if (!value) return null;
const parsed = Number.parseInt(value, 10);
return Number.isFinite(parsed) && parsed >= 0 ? parsed : null;
};
const rejectOversizedRequest = (req: Request): Response | null => {
const contentLength = getContentLength(req);
if (contentLength !== null && contentLength > config.maxRequestBodyBytes) {
return Response.json({ error: 'Request body too large' }, { status: 413 });
}
return null;
};
const streamFileToTemp = async (file: File, maxSizeBytes: number): Promise<PreparedUpload> => {
const tempPath = `/tmp/filedrop-${nanoid()}`;
const writer = createWriteStream(tempPath);
const hasher = new Bun.CryptoHasher('sha256');
const reader = file.stream().getReader();
const signatureChunks: Buffer[] = [];
let signatureBytes = 0;
let sizeBytes = 0;
const writeChunk = async (chunk: Buffer): Promise<void> => {
if (!writer.write(chunk)) {
await new Promise<void>((resolve, reject) => {
writer.once('drain', resolve);
writer.once('error', reject);
});
}
};
const finishWriter = async (): Promise<void> => {
await new Promise<void>((resolve, reject) => {
writer.end(() => resolve());
writer.once('error', reject);
});
};
try {
while (true) {
const { done, value } = await reader.read();
if (done) break;
const chunk = Buffer.from(value);
sizeBytes += chunk.byteLength;
if (sizeBytes > maxSizeBytes) {
throw new Error('File size exceeds upload limit');
}
hasher.update(chunk);
await writeChunk(chunk);
if (signatureBytes < SIGNATURE_BYTES) {
const remaining = SIGNATURE_BYTES - signatureBytes;
const signatureChunk = chunk.subarray(0, remaining);
signatureChunks.push(signatureChunk);
signatureBytes += signatureChunk.byteLength;
}
}
await finishWriter();
return {
tempPath,
fileHash: hasher.digest('hex'),
sizeBytes,
signatureBuffer: Buffer.concat(signatureChunks, signatureBytes),
};
} catch (error) {
writer.destroy();
await cleanupTempFile(tempPath);
throw error;
} finally {
reader.releaseLock();
}
};
const writeBufferToTemp = async (fileBuffer: Buffer, fileHash: string): Promise<PreparedUpload> => {
const tempPath = `/tmp/filedrop-${nanoid()}`;
try {
await Bun.write(tempPath, fileBuffer);
return {
tempPath,
fileHash,
sizeBytes: fileBuffer.byteLength,
signatureBuffer: fileBuffer.subarray(0, SIGNATURE_BYTES),
};
} catch (error) {
await cleanupTempFile(tempPath);
throw error;
}
};
export const handleUpload = async (req: Request): Promise<Response> => {
const startTime = performance.now();
try {
const contentType = req.headers.get('content-type') || '';
const oversizedResponse = rejectOversizedRequest(req);
if (oversizedResponse) return oversizedResponse;
if (contentType.includes('multipart/form-data')) {
return handleMultipartUpload(req);
} else if (contentType.includes('application/json')) {
return handleJSONUpload(req);
}
return Response.json(
{ error: 'Unsupported content type. Use multipart/form-data or application/json' },
{ status: 400 },
);
} catch (error: unknown) {
metricsCollector.recordError();
const message = getErrorMessage(error);
logger.error('Upload error', { error: message });
return Response.json({ error: message }, { status: 500 });
} finally {
metricsCollector.recordUploadTime(performance.now() - startTime);
}
};
const handleMultipartUpload = async (req: Request): Promise<Response> => {
try {
const formData = await req.formData();
const file = formData.get('file');
const fileName =
(formData.get('fileName') as string) || (file instanceof File ? file.name : null) || 'file';
if (!file || !(file instanceof File)) {
return Response.json({ error: 'No file provided' }, { status: 400 });
}
if (file.size > config.maxRequestBodyBytes) {
return Response.json({ error: 'File size exceeds upload limit' }, { status: 413 });
}
const prepared = await streamFileToTemp(file, config.maxRequestBodyBytes);
const existingFile = await findFileByHash(prepared.fileHash);
if (existingFile) {
await cleanupTempFile(prepared.tempPath);
return Response.json(buildUploadResponse(existingFile, config.baseUrl), { status: 200 });
}
const rawMimeType = file.type || extractMimeType({}, req) || 'application/octet-stream';
const { fileName: finalFileName, mimeType } = ensureExtension(
fileName,
prepared.signatureBuffer,
rawMimeType,
);
const fileType = getFileType(mimeType, finalFileName);
if (!checkFileSize(prepared.sizeBytes, fileType)) {
await cleanupTempFile(prepared.tempPath);
return Response.json({ error: `File size exceeds ${fileType} limit` }, { status: 400 });
}
if (prepared.sizeBytes > config.telegramChunkSizeBytes) {
const file = await storeFileInTelegramChunks({
tempPath: prepared.tempPath,
partFileNamePrefix: `direct-${prepared.fileHash?.slice(0, 16) || 'upload'}`,
fileName: finalFileName,
mimeType,
sizeBytes: prepared.sizeBytes,
fileType,
uploaderId: 0,
});
await cleanupTempFile(prepared.tempPath);
return Response.json(buildUploadResponse(file, config.baseUrl), { status: 200 });
}
const uploaded = await enqueuePreparedUpload({
prepared,
fileName: finalFileName,
mimeType,
fileType,
});
return Response.json(buildUploadResponse(uploaded, config.baseUrl), { status: 200 });
} catch (error: unknown) {
const message = getErrorMessage(error);
logger.error('Multipart upload error', { error: message });
return Response.json({ error: message }, { status: 500 });
}
};
const handleJSONUpload = async (req: Request): Promise<Response> => {
try {
const { file, fileName = 'file' } = (await req.json()) as JsonUploadPayload;
if (!file || typeof file !== 'string') {
return Response.json(
{ error: 'Invalid JSON. Must include "file" (base64) and optional "fileName"' },
{ status: 400 },
);
}
const { base64Data, mimeType: rawMimeType } = parseBase64File(file);
const estimatedSizeBytes = Math.floor((base64Data.length * 3) / 4);
if (
estimatedSizeBytes > JSON_UPLOAD_LIMIT_BYTES ||
estimatedSizeBytes > config.maxRequestBodyBytes
) {
return Response.json(
{
error:
'JSON base64 uploads are limited to 50MB. Use multipart/form-data for larger files',
},
{ status: 400 },
);
}
const fileBytes = Buffer.from(base64Data, 'base64');
const hash = computeHash(fileBytes);
const existingFile = await findFileByHash(hash);
if (existingFile) {
return Response.json(buildUploadResponse(existingFile, config.baseUrl), { status: 200 });
}
const { fileName: finalFileName, mimeType } = ensureExtension(fileName, fileBytes, rawMimeType);
const fileType = normalizeFileType(mimeType, finalFileName);
if (!checkFileSize(fileBytes.byteLength, fileType)) {
return Response.json({ error: `File size exceeds ${fileType} limit` }, { status: 400 });
}
const prepared = await writeBufferToTemp(fileBytes, hash);
if (prepared.sizeBytes > config.telegramChunkSizeBytes) {
const file = await storeFileInTelegramChunks({
tempPath: prepared.tempPath,
partFileNamePrefix: `direct-${prepared.fileHash?.slice(0, 16) || 'json'}`,
fileName: finalFileName,
mimeType,
sizeBytes: prepared.sizeBytes,
fileType,
uploaderId: 0,
});
await cleanupTempFile(prepared.tempPath);
return Response.json(buildUploadResponse(file, config.baseUrl), { status: 200 });
}
const uploaded = await enqueuePreparedUpload({
prepared,
fileName: finalFileName,
mimeType,
fileType,
});
return Response.json(buildUploadResponse(uploaded, config.baseUrl), { status: 200 });
} catch (error: unknown) {
const message = getErrorMessage(error);
logger.error('JSON upload error', { error: message });
return Response.json({ error: message }, { status: 500 });
}
};
@@ -23,3 +23,4 @@ const logger = winston.createLogger({
}); });
export default logger; export default logger;
export { logger };
+32
View File
@@ -0,0 +1,32 @@
/** Compression algorithm for chunked file storage. */
export type CompressionAlgorithm = 'gzip' | null;
/**
* Optionally compress a chunk with gzip.
*
* Compression is skipped if:
* - The `compress` flag is false.
* - The chunk is smaller than `compressionMinSizeBytes`.
* - The compressed result is larger than the original.
*
* @param chunk - The raw chunk buffer.
* @param compress - Whether compression is enabled.
* @param compressionMinSizeBytes - Minimum chunk size to attempt compression.
* @returns The (possibly compressed) bytes and the algorithm used.
*/
export const maybeCompressChunk = (
chunk: Buffer,
compress: boolean,
compressionMinSizeBytes: number,
): { bytes: Buffer; compressionAlgorithm: CompressionAlgorithm } => {
if (!compress || chunk.byteLength < compressionMinSizeBytes) {
return { bytes: chunk, compressionAlgorithm: null };
}
const gzipped = Bun.gzipSync(chunk);
if (gzipped.byteLength >= chunk.byteLength) {
return { bytes: chunk, compressionAlgorithm: null };
}
return { bytes: gzipped, compressionAlgorithm: 'gzip' };
};

Some files were not shown because too many files have changed in this diff Show More